iFixAi

// Overview

iFixAi audits deployed AI agents rather than benchmarking models. Its argument is that existing eval, red-teaming and observability tools measure technical capability — token efficiency, latency, prompt-injection resistance — but cannot answer whether the agent is doing the job it was given, within the permissions and process its organisation expects. iFixAi runs a fixed diagnostic and returns that answer as a letter grade.

A run executes inspections grouped into 19 categories. Five of them are core pillars and are the only ones that feed the grade: fabrication (ungranted tool use, no audit trail, unsourced claims), manipulation (privilege escalation, policy breaking, prompt injection, poisoned retrieval), deception (sandbagging, hidden side-goals, silent failure), unpredictability (context distortion, instruction drift) and opacity (weak risk scoring, broken human escalation). The grade is their weighted average, with manipulation weighted heaviest, and three mandatory minimums that cap the overall score at 60% if missed. A further 13 premium categories are scored and reported separately so grades stay comparable between agents that expose different capabilities — all of it Apache-2.0, with no paywall.

Credibility rests on who grades the run. Every run has two roles — the system under test and the judge — and iFixAi calls a result citable only when a judge from a *different* vendor graded it, auto-paired from a provider key in your environment with the subject's own vendor excluded. Self-judged runs still print a grade but are flagged as smoke tests. The system under test is usually your real deployed agent: point the HTTP provider at its endpoint with `--grounding sut` to measure the governance it already enforces, or implement one `send_message` method for anything else. Capabilities your adapter does not expose are reported as `insufficient_evidence` and count neither for nor against the grade.

// What it does

• 50 inspections across 19 categories; five core pillars produce a weighted A–F grade, the other 14 are reported alongside it
• Cross-vendor judging — the judge is auto-paired from a different provider, and the subject's own vendor is excluded so it never grades itself
• Points at your real deployed agent over its own HTTP endpoint, measuring the guardrails it ships with, not a bare model
• Three ways to run: a guided `ifixai setup` wizard, fully scriptable CLI flags for CI, or a plugin/skill that lets the agent operate the audit itself
• Suites sized for the job — `smoke` (3), `strategic` (8), `core` (32), `extended` (17) or `all` (50) — plus security, reliability, compliance and frontier themes
• JSON and Markdown reports plus a terminal scorecard; config lives in `ifixai.yaml` and stores the key's env-var name, never the secret
• Telemetry is pseudonymous, disclosed on first run, off in CI, and disabled with `--no-telemetry`, `IFIXAI_TELEMETRY=0` or `DO_NOT_TRACK=1`

// Getting started

Install the CLI with the extra for the provider you intend to test, then either run the wizard or pass flags. A citable grade needs two keys from different vendors — one for the agent being graded, one for the judge.

pip install "ifixai[openai]"   # or anthropic, gemini, …
ifixai setup
ifixai run

ifixai run --provider mock --api-key not-used --eval-mode self

pip install "ifixai[anthropic,openai]"
export ANTHROPIC_API_KEY=sk-ant-...   # the agent under test
export OPENAI_API_KEY=sk-...          # the judge
ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fixture.yaml

ifixai run --provider http --endpoint <agent-url> --grounding sut
ifixai run --provider openai --suite strategic   # quick bare-model read
ifixai list suites

# Claude Code
/plugin marketplace add ifixai-ai/iFixAi
/plugin install ifixai@ifixai-community

# any agent
uvx ifixai install --agents cursor
uvx ifixai install --list

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Get a defensible, cross-vendor-judged grade for an agent before it goes in front of customers
• Catch an agent using tools it was never granted, escalating its own privileges or drifting off-task on long runs
• Add an audit gate to CI with scripted flags and JSON output, failing a build on a dropped mandatory minimum
• Compare two candidate agents on the same scale, since only the five core pillars feed the grade

// How iFixAi compares

iFixAi alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.