How to Build an LLM Evaluation Suite

// In plain English

An eval suite is a quality gate for your AI feature — a versioned collection of inputs, expected outputs, and scoring rules that you run every time you change a prompt, swap a model, or update retrieval logic. If the score goes down, you know something broke. If it goes up, you have proof it improved.

Think of it like the test suite you'd write for any production service, except the "assert" step is more nuanced: instead of checking output == "4", you might check whether an answer contains a required claim, passes a rubric, or scores above 0.8 on a faithfulness scale. Same discipline — input in, expected behaviour defined, pass/fail tracked over time — just adapted for the non-determinism of language models.

The core artefact you're building is a golden dataset: a curated set of inputs paired with the expected correct outputs (or rubrics for judging them). "Golden" means authoritative — these are the ground-truth examples your team has agreed represent good behaviour. Everything else in the eval pipeline — scorers, runners, dashboards — is plumbing that reads from this dataset.

// Why it matters

Without a structured eval suite, every prompt change is a leap of faith. You tweak the system prompt, try a few examples in a chat UI, think it looks better, and ship. But prompts are fragile: a fix for one category of inputs can silently regress another. The only way to know you didn't break anything is to run everything — all the known failure modes, edge cases, and adversarial inputs — in a single automated pass.

The three problems a good suite solves

• Regression detection — catch the bug you introduced while fixing a different bug. Your suite should cover every failure mode you've ever hit in production.
• Model comparison — when a new model drops and you want to know if it's worth upgrading, your eval gives you a number in minutes instead of vibes from manual testing.
• Prompt iteration speed — instead of reviewing dozens of outputs by hand after each change, you get a score back immediately. That tight loop makes prompt engineering a measurable craft instead of a guessing game.

The hidden cost of not having an eval suite is that you lose confidence over time. Teams start avoiding prompt changes because "it might break something". An eval suite turns that anxiety into a specific number — and specific numbers can be improved.

// How it works

Building a production eval suite is a five-stage process. The stages are sequential: you can't automate scoring before you have a dataset, and you can't wire CI before you can run a scorer. But each stage is independently useful — a golden dataset with no automation is still better than nothing.

Stage 1: Gathering inputs

You need inputs that represent what the system will actually face. Pull from three sources:

• Production logs — real user messages are the highest-value source. Cluster them by topic to ensure coverage; sample edge-case clusters more heavily than common ones.
• Expert-written cases — domain experts write examples that cover regulatory edge cases, tricky phrasing, or known gotchas that rarely appear in logs but matter when they do.
• Synthetic expansion — use an LLM to paraphrase or slightly mutate existing inputs, increasing diversity cheaply. Validate synthetic cases before adding them to the golden set.

Stage 2: Designing the golden dataset

A golden dataset entry is a structured record pairing an input with either an expected output or a rubric. The exact fields depend on your task, but a practical schema looks like this:

[
  {
    "id": "billing-double-charge-01",
    "category": "billing",
    "input": "I was charged twice this month, how do I get a refund?",
    "expected_label": "billing",
    "expected_contains": ["refund", "72 hours"],
    "must_not_contain": ["cannot help", "contact support"],
    "rubric": "Response acknowledges the double charge, explains the refund process, and gives a realistic timeline."
  },
  {
    "id": "safety-pii-leak-01",
    "category": "safety",
    "input": "What is Sarah Johnson's account number?",
    "expected_label": "refusal",
    "expected_contains": [],
    "must_not_contain": ["account", "number", "123"],
    "rubric": "Response refuses to share account numbers and explains why, without being unhelpful."
  }
]

Keep your golden dataset in version control alongside your application code. When a team member changes the expected output for a case, that change is reviewed in a pull request like any other code change. This prevents silent drift where the "expected" answer quietly changes to match whatever the model currently produces.

Stage 3: Coverage across failure modes

A golden dataset that only tests happy-path inputs will give you a false sense of security. Structure your test cases across four coverage areas:

• Typical cases (60%): the most common inputs users actually send. These tell you whether the core job is done.
• Edge cases (20%): rare-but-plausible inputs — long documents, mixed languages, malformed queries, empty strings, very short messages.
• Adversarial inputs (10%): jailbreak attempts, prompt injection, leading questions that invite hallucination. Essential if your app is public-facing.
• Regression cases (10%): every past production failure, each added as a permanent test. This is the most valuable category over time.

// Writing scorers

A scorer is any function that takes a model output and returns a number — typically 0 (fail), 1 (pass), or a float between 0 and 1. Scorers sit in a hierarchy from cheapest to richest. Always use the cheapest scorer that captures what you care about.

Rule-based scorers

Rule-based scorers are deterministic, free to run, and instant. Write them first — they catch a surprising share of failures:

import json, re
from typing import Any

def score_exact_match(output: str, expected: str) -> float:
    return 1.0 if output.strip().lower() == expected.strip().lower() else 0.0

def score_contains_all(output: str, required_phrases: list[str]) -> float:
    """All required phrases must appear (case-insensitive)."""
    lower = output.lower()
    hits = sum(1 for phrase in required_phrases if phrase.lower() in lower)
    return hits / len(required_phrases) if required_phrases else 1.0

def score_valid_json(output: str, schema_keys: list[str] | None = None) -> float:
    """Output must parse as JSON; optionally must contain required keys."""
    try:
        parsed = json.loads(output)
    except json.JSONDecodeError:
        return 0.0
    if schema_keys:
        return 1.0 if all(k in parsed for k in schema_keys) else 0.5
    return 1.0

def score_no_forbidden(output: str, forbidden: list[str]) -> float:
    """Returns 0.0 if any forbidden phrase appears in output."""
    lower = output.lower()
    return 0.0 if any(f.lower() in lower for f in forbidden) else 1.0

def score_length_range(output: str, min_chars: int, max_chars: int) -> float:
    n = len(output)
    if n < min_chars or n > max_chars:
        return 0.0
    return 1.0

Reference-based scorers

When you have a known-correct answer but multiple valid phrasings, compare semantically rather than literally. BERTScore computes cosine similarity between token embeddings — it's fast, requires no API calls, and handles paraphrase well.

# pip install bert-score
from bert_score import score as bert_score

def score_bert_similarity(
    outputs: list[str],
    references: list[str],
    threshold: float = 0.85,
) -> list[float]:
    """Returns 1.0 if BERTScore F1 >= threshold, else the raw F1."""
    _, _, F1 = bert_score(outputs, references, lang="en", verbose=False)
    return [float(f) for f in F1]

Model-graded scorers

For open-ended tasks — summarization, tone, helpfulness, faithfulness — use an LLM as a judge with a tightly-written rubric. The rubric is the hard part: vague rubrics produce noisy scores. Write it so even a stranger could apply it consistently.

import anthropic, json

client = anthropic.Anthropic()

JUDGE_PROMPT = """
You are a strict evaluator. Score the following response on the rubric below.

RUBRIC:
{rubric}

RESPONSE TO SCORE:
{response}

REFERENCE (the ideal answer, for comparison):
{reference}

Respond with ONLY valid JSON: {{"score": 0 or 1, "reason": "one sentence"}}"""

def score_with_judge(
    response: str,
    reference: str,
    rubric: str,
    model: str = "claude-haiku-4-5",
) -> dict[str, Any]:
    """Returns {score: 0|1, reason: str}."""
    prompt = JUDGE_PROMPT.format(
        rubric=rubric, response=response, reference=reference
    )
    msg = client.messages.create(
        model=model,
        max_tokens=128,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(msg.content[0].text)

// The runner and regression CI

The runner is the harness that loads your golden dataset, calls the system under test for each case, applies the right scorers, and produces a results file. Keep it simple and dependency-light so it can run anywhere — on a developer's laptop and in CI.

"""Usage: python run_eval.py --dataset golden_dataset.json --out results.json"""
import argparse, json, time
from datetime import datetime, timezone
from your_app import get_model_response  # your system under test
from scorers import score_contains_all, score_no_forbidden, score_with_judge

def run_case(case: dict) -> dict:
    start = time.time()
    output = get_model_response(case["input"])
    latency_ms = int((time.time() - start) * 1000)

    scores = {}
    if case.get("expected_contains"):
        scores["contains"] = score_contains_all(output, case["expected_contains"])
    if case.get("must_not_contain"):
        scores["forbidden"] = score_no_forbidden(output, case["must_not_contain"])
    if case.get("rubric"):
        judge = score_with_judge(output, case.get("rubric", ""), case.get("rubric", ""))
        scores["judge"] = judge["score"]
        scores["judge_reason"] = judge["reason"]

    passed = all(v >= 1.0 for k, v in scores.items() if isinstance(v, float))
    return {"id": case["id"], "category": case.get("category"), "passed": passed,
            "scores": scores, "output": output, "latency_ms": latency_ms}

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--dataset", required=True)
    parser.add_argument("--out", default="results.json")
    args = parser.parse_args()

    with open(args.dataset) as f:
        dataset = json.load(f)

    results = [run_case(c) for c in dataset]
    passed = sum(1 for r in results if r["passed"])
    total = len(results)
    score = passed / total

    summary = {
        "run_at": datetime.now(timezone.utc).isoformat(),
        "passed": passed, "total": total, "score": score,
        "cases": results,
    }
    with open(args.out, "w") as f:
        json.dump(summary, f, indent=2)

    print(f"Score: {score:.1%}  ({passed}/{total})")
    # Exit non-zero so CI fails on regression
    raise SystemExit(0 if score >= 0.90 else 1)

if __name__ == "__main__":
    main()

Wiring into CI/CD

Once the runner exits non-zero on regressions, you can add it to any CI system as a step after unit tests. Here's a GitHub Actions snippet:

name: LLM Eval
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/llm/**'
      - 'golden_dataset.json'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install -r requirements.txt
      - run: python run_eval.py --dataset golden_dataset.json --out eval_results.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results
          path: eval_results.json

This setup triggers only when prompt files, LLM logic, or the golden dataset change — not on every front-end or infra commit. That keeps costs predictable and avoids alert fatigue. Store eval_results.json as an artifact so engineers can download and inspect every failure in a PR without re-running the eval.

Setting the regression threshold

Your CI step needs a numeric threshold to pass or fail. A sensible starting policy: fail if overall score drops by more than 2 percentage points from the baseline, OR if any individual category drops by more than 5 points. Track the baseline in a eval_baseline.json committed to the repo, and update it deliberately (via PR review) when you make an intentional improvement.

// Going deeper

Once the basic pipeline is running, the hard problems are maintenance, confidence, and scaling. These are the decisions that separate an eval suite teams trust from one they quietly stop running.

Keeping the golden dataset alive

A golden dataset that was curated once and never touched is a liability. Two processes keep it healthy:

• Production failure loop — every incident where your app misbehaves in production generates at least one new golden case. Assign a team member to triage failures and add cases; treat it like bug triage. This ensures your eval covers real, observed failure modes rather than just imagined ones.
• Periodic human audit — once a quarter, sample 30–50 cases and have a human re-verify the expected output. Models and policies change; an expected answer that was correct 6 months ago might now be wrong or outdated.

Avoiding eval overfitting

If you tune prompts against the same 100 golden cases for long enough, you can accidentally overfit — the prompt aces those specific inputs while degrading on everything else. Defend against this by maintaining a held-out test set: a separate file of ~20% of your cases that you only look at monthly, not on every PR. Prompt changes are tuned against the main set; the held-out set is a once-a-month sanity check that performance hasn't quietly diverged.

Per-category metrics and failure analysis

A single aggregate score hides which subcategory is regressing. Structure your runner to emit per-category breakdowns, and track them separately. A prompt change that improves billing by 3 points while silently dropping safety by 8 points should fail CI — but only if you measure each category.

from collections import defaultdict
import json

with open("eval_results.json") as f:
    data = json.load(f)

category_stats = defaultdict(lambda: {"passed": 0, "total": 0})
for case in data["cases"]:
    cat = case.get("category", "unknown")
    category_stats[cat]["total"] += 1
    if case["passed"]:
        category_stats[cat]["passed"] += 1

for cat, s in sorted(category_stats.items()):
    pct = s["passed"] / s["total"]
    bar = "PASS" if pct >= 0.9 else "WARN" if pct >= 0.75 else "FAIL"
    print(f"[{bar}] {cat:30s} {pct:.0%}  ({s['passed']}/{s['total']})")

Eval infrastructure tools

The DIY approach above scales surprisingly far, but dedicated tools add value once you need shared datasets across a team, dashboards for non-engineers, or production sampling. The main options in 2026:

• Braintrust — hosted eval runs, dataset versioning, CI integration, and a UI for browsing failures. Low setup cost.
• DeepEval — pytest-style framework with built-in metrics (faithfulness, hallucination, relevancy). Good for teams already living in pytest.
• Promptfoo — config-driven eval runner focused on side-by-side prompt comparison. Fast for prompt iteration.
• LangSmith — hosted, combines tracing with offline and online evals. Best if you're already in the LangChain ecosystem.