# iFixAi

> Audit whether an agent is actually doing the job it was given

- Category: Evaluation & Production Ops / Evaluation & Red-Teaming
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/ifixai-ai/iFixAi
- Website: https://www.ifixai.ai
- Canonical page: https://ai-tldr.dev/tools/ifixai/
- Updated: 2026-10-15

## In short

iFixAi audits deployed AI agents rather than benchmarking models. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0. To get started: install and run the wizard.

## Overview

iFixAi audits deployed AI agents rather than benchmarking models. Its argument is that existing eval, red-teaming and observability tools measure technical capability — token efficiency, latency, prompt-injection resistance — but cannot answer whether the agent is doing the job it was given, within the permissions and process its organisation expects. iFixAi runs a fixed diagnostic and returns that answer as a letter grade.

A run executes inspections grouped into 25 categories. Five of them are core pillars and are the only ones that feed the grade: fabrication (ungranted tool use, no audit trail, unsourced claims), manipulation (privilege escalation, policy breaking, prompt injection, poisoned retrieval), deception (sandbagging, hidden side-goals, silent failure), unpredictability (context distortion, instruction drift) and opacity (weak risk scoring, broken human escalation). The grade is their weighted average, with manipulation weighted heaviest, and three mandatory minimums that cap the overall score at 60% if missed. A further 20 premium categories are scored and reported separately so grades stay comparable between agents that expose different capabilities — all of it Apache-2.0, with no paywall.

Credibility rests on who grades the run. Every run has two roles — the system under test and the judge — and iFixAi calls a result citable only when a judge from a *different* vendor graded it, auto-paired from a provider key in your environment with the subject's own vendor excluded. Self-judged runs still print a grade but are flagged as smoke tests. The system under test is usually your real deployed agent: point the HTTP provider at its endpoint with `--grounding sut` to measure the governance it already enforces, or implement one `send_message` method for anything else. Capabilities your adapter does not expose are reported as `insufficient_evidence` and count neither for nor against the grade.

## What it does

- 60 inspections across 25 categories; five core pillars produce a weighted A–F grade, the other 20 are reported alongside it
- Cross-vendor judging — the judge is auto-paired from a different provider, and the subject's own vendor is excluded so it never grades itself
- Points at your real deployed agent over its own HTTP endpoint, measuring the guardrails it ships with, not a bare model
- Three ways to run: a guided `ifixai setup` wizard, fully scriptable CLI flags for CI, or a plugin/skill that lets the agent operate the audit itself
- Suites sized for the job — `smoke` (3), `strategic` (8), the default `essential` (10), `core` (32), `extended` (28) or `all` (60, via `--full-run`) — plus security, reliability, compliance and frontier themes
- JSON and Markdown reports plus a terminal scorecard; config lives in `ifixai.yaml` and stores the key's env-var name, never the secret
- Telemetry is pseudonymous, disclosed on first run, off in CI, and disabled with `--no-telemetry`, `IFIXAI_TELEMETRY=0` or `DO_NOT_TRACK=1`

## Getting started

Install the CLI with the extra for the provider you intend to test, then either run the wizard or pass flags. A citable grade needs two keys from different vendors — one for the agent being graded, one for the judge.

1. Install and run the wizard

   `ifixai setup` detects API keys already in your environment, then writes `ifixai.yaml` so later runs need no flags. Reports land in ./ifixai-results/.

```bash
pip install "ifixai[openai]"   # or anthropic, gemini, …
ifixai setup
ifixai run
```

2. Prove the pipeline first

   The built-in mock needs no keys and no network. Expect a failing scorecard — the bundled default fixture ships seeded defects on purpose so you can see what failures look like.

```bash
ifixai run --provider mock --api-key not-used --eval-mode self
```

3. Get a citable grade

   Two keys from different vendors: the subject's key is always passed explicitly, the judge is auto-paired from the environment. Pass your own `--fixture`, or the seeded-defect default's failures land on your scorecard.

```bash
pip install "ifixai[anthropic,openai]"
export ANTHROPIC_API_KEY=sk-ant-...   # the agent under test
export OPENAI_API_KEY=sk-...          # the judge
ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fixture.yaml
```

4. Audit your real agent

   This is the recommended target: your deployed agent over its own OpenAI-compatible endpoint, observed as shipped.

```bash
ifixai run --provider http --endpoint <agent-url> --grounding sut
ifixai run --provider openai --suite strategic   # quick bare-model read
ifixai list suites
```

5. Run it from inside your coding agent

   The plugin self-provisions in Claude Code and Codex; the skill scaffolds a /ifixai-skill command into any of a dozen agents.

```bash
# Claude Code
/plugin marketplace add ifixai-ai/iFixAi
/plugin install ifixai@ifixai-community

# any agent
uvx ifixai install --agents cursor
uvx ifixai install --list
```

## When to use it

- Get a defensible, cross-vendor-judged grade for an agent before it goes in front of customers
- Catch an agent using tools it was never granted, escalating its own privileges or drifting off-task on long runs
- Add an audit gate to CI with scripted flags and JSON output, failing a build on a dropped mandatory minimum
- Compare two candidate agents on the same scale, since only the five core pillars feed the grade

## Version history

- 2026-10-15 v4.2.0: `ifixai run` now defaults to a 10-inspection essential suite (`--full-run` runs all 60), adds Vercel and Cloudflare AI Gateway providers, and ships two offline commands: `ifixai trajectory` checks recorded agent actions against a policy, and `ifixai policy-lab` finds risky combinations of permitted tools.