What Is Garak? LLM Vulnerability Scanner

// In plain English

A large language model is supposed to refuse harmful requests, keep its secret system instructions private, and stay polite. But does your deployed model actually do that? You can ask it a few tricky questions by hand, but a determined attacker will try hundreds of phrasings you never thought of. Testing one prompt at a time does not scale.

Garak is an open-source tool from NVIDIA that does this testing for you, automatically. You point it at a model, and it fires a large battery of known attacks at it — jailbreak attempts, prompt injection, tricks to leak training data or secrets, prompts that fish for toxic output — and then it reads the model's replies and reports which attacks got through. The name stands for Generative AI Red-teaming and Assessment Kit.

The easiest analogy is a security scanner like nmap or a vulnerability scanner for a website. You do not manually try every known exploit against your server; you run a scanner that already knows the exploit catalog and checks them all for you. Garak is that scanner, but the target is a language model and the "exploits" are adversarial prompts. It turns slow, manual red-teaming into a repeatable command you can run on every model you ship.

// Why it matters

Once you put an LLM in front of users, its weaknesses become your incident. A support bot that can be talked into insulting customers, an assistant that reveals its hidden system prompt, a model that regurgitates a secret API key it saw in training — these are real, reputational, and sometimes legal problems. Garak exists to find these holes before an attacker does.

Manual testing has two problems Garak solves. First, coverage: a person remembers maybe a dozen attack tricks; Garak ships with a large, curated library of attack families drawn from public research, so it probes corners you would never think to try. Second, repeatability: a human red-team session is hard to reproduce. Garak is a command, so you can run the exact same suite against every new model version and compare the results — a vulnerability that was fixed last month and quietly came back will show up.

• Before launch — scan a candidate model to see what it is vulnerable to, so you know what guardrails you actually need.
• Model selection — run the same suite against several models and compare how each one holds up before you commit.
• Regression testing — re-run on every upgrade or fine-tune to make sure a fix stayed fixed and a new model did not get worse.
• Validating guardrails — point Garak at your full system (model plus filters) to check the defenses you added actually block the attacks.

It is part of a broader move to treat LLM safety like software security: a known catalog of weaknesses, an automated tool to check for them, and a report you can act on. Garak is one of the most established open tools in that space, alongside others like Microsoft's PyRIT for automated red-teaming.

// How it works

Garak is built from four kinds of moving parts. Understanding them is most of understanding the tool, because everything you configure maps onto one of them.

• Generator — the thing being tested. This is your model, wrapped so Garak can send it prompts and read replies. Generators exist for many APIs and for locally hosted models.
• Probe — one family of attack. A probe knows how to generate many adversarial prompts of a certain type (for example, a jailbreak style, or a prompt-injection pattern) and sends them to the generator.
• Detector — the grader. After the probe gets replies, a detector inspects each reply and decides whether the attack succeeded. Detectors range from simple keyword/pattern checks to small classifier models.
• Report / scorer — the output. Garak aggregates pass/fail across every probe and prompt into a report showing which attack families got through and how often.

The flow is a loop: a probe produces attack prompts, the generator answers them, a detector scores each answer, and the result is logged. Repeat across every probe in the suite, then summarize.

The key insight is the probe/detector split. A probe knows how to attack; a detector knows how to recognize a successful attack. Keeping them separate means the same toxicity detector can grade the output of many different probes, and a new probe can reuse existing detectors. It is the same separation that makes a security scanner extensible — add a new exploit check without rewriting the engine.

In practice you run one command. You name the model (the generator) and, optionally, which probes to run. Leave the probes unset and Garak runs its default suite across all the attack families it knows.

# Scan a hosted model across the full default probe suite
python -m garak --model_type openai --model_name <model-id>

# Or target one attack family — here, prompt-injection probes
python -m garak --model_type openai --model_name <model-id> \
    --probes promptinject

# List everything available
python -m garak --list_probes
python -m garak --list_detectors

// What Garak probes for

Garak's probes are grouped by the kind of weakness they hunt. You do not need to memorize the catalog, but knowing the broad families tells you what a scan actually covers — and what it does not.

Each family is implemented as one or more probes, and each probe ships many concrete prompts. When you read a Garak report, you are seeing, per family, how often the model fell for the attack. A model might shrug off jailbreaks but leak its system prompt — the breakdown is the whole point.

// Reading the results (and their limits)

A Garak run prints a per-probe summary as it goes and writes a detailed log plus an HTML report. For each probe you get a pass rate — what fraction of that family's attacks the model resisted. Lower means more vulnerable. The report links back to the actual prompts and replies, so you can read exactly how a model failed instead of trusting a number.

How to think about a score

• Compare, don't worship. The most useful thing is relative: model A vs model B, or this version vs last version. An absolute pass rate depends on which probes ran and how strict the detectors are.
• Detectors are imperfect. A keyword-based detector can miss a cleverly worded harmful reply (a false negative) or flag a harmless one (a false positive). Treat the report as a prioritized list of things to read, not a verdict.
• Coverage is not completeness. Garak tests the attacks it knows. A clean scan means it found nothing in its catalog — not that the model is unbreakable. New attacks appear constantly.

Used well, the report does the triage for you: it surfaces the handful of prompts where your model misbehaved, and you decide what to do about each one — add a filter, harden the system prompt, or accept the risk. That human review step is essential. Garak narrows thousands of prompts down to the ones worth a person's attention.

// Garak vs guardrails vs manual red-teaming

Garak is easy to confuse with neighbouring tools. They are complements, not substitutes — each does a different job in the safety lifecycle.

The natural pipeline uses all three. Manual red-teamers discover a new attack. That attack becomes a Garak probe so it can be checked automatically forever after. And whatever still gets through, you block at runtime with guardrails. Garak is the automated middle layer — see automated vs manual red-teaming for where the human stays essential.

One more distinction: Garak is a security scanner, not a quality evaluator. It asks "can this model be made to misbehave?", not "is this answer accurate or helpful?". For the latter you want an evals framework. Many teams run both.

// Going deeper

Once the basic scan makes sense, a few areas reward a closer look.

Scanning the whole system, not just the model. The most useful target is often not a bare model but your application — the model plus its system prompt, retrieval context, and any output filters. Garak can point at an API endpoint that represents your full stack, so the scan measures the defenses you actually shipped. A bare model and the same model behind guardrails can score very differently, and the gap is exactly what your guardrails bought you.

Writing your own probes and detectors. Because of the probe/detector split, Garak is extensible. If your manual red-team finds an attack specific to your domain, you can encode it as a custom probe so it gets checked on every future run. You can likewise add a detector — for instance, one that flags whenever a reply contains your real system-prompt text or a known secret. This is how a one-off finding becomes permanent regression coverage.

Cost, time, and non-determinism. A full default scan fans out into many requests, so it can be slow and, on paid APIs, expensive — scope your probes deliberately. And because models are stochastic, two runs can give slightly different results; an attack that fails once may succeed on a retry. For high-stakes checks, run a probe multiple times rather than trusting a single pass.

Where it fits the bigger picture. Garak focuses on text-prompt attacks against the model's behavior. It is one tool in a layered defense, not the whole thing. To understand what it is throwing at your model, the adversarial prompts and jailbreak techniques articles cover the attacks themselves; pair its findings with runtime guardrails and ongoing human red-teaming. The honest takeaway: an automated scanner is a fast, repeatable floor for LLM safety, never the ceiling. It proves a model fails on known attacks; it can never prove a model is safe.