Evaluation & Safety · TRACK 01/05
Evaluation Basics
Because "it looks good to me" is not a test suite.
// THE TRACK
01 · START HERELLM EvalsUnderstand what an LLM eval is, why eyeballing outputs stops working past ten test cases, and what a real eval pipeline looks like.BEGINNERBuilding an Eval SuiteLearn to build a complete LLM evaluation suite from scratch: curate a golden dataset, write test cases for every failure mode, automate scoring, and catch regressions before users do.INTERMEDIATEGolden DatasetsKnow what a golden dataset is, how big it needs to be, and how to keep it honest as your product and models change.BEGINNERYour First EvalWalk away knowing exactly how to turn a handful of real user examples into a small, runnable eval suite for your app.BEGINNERCode vs Model vs Human GradingUnderstand the three grading methods for LLM outputs and be able to pick the right mix for any eval.INTERMEDIATEEval MetricsBe able to read accuracy, pass@k, F1, and rubric scores — and know which metric actually answers your question.INTERMEDIATERegression Testing PromptsYou'll understand how to set up regression tests that flag when a prompt edit quietly makes outputs worse.INTERMEDIATEEval in CI/CDYou'll understand how to make evals a pipeline gate so quality regressions block a merge instead of reaching production.INTERMEDIATEOffline vs Online EvalsYou'll understand the difference between testing on a frozen dataset and measuring quality on live traffic, and why you need both.INTERMEDIATEEval Sample SizeYou'll understand how to tell whether a higher eval score is a real improvement or just statistical noise from too few examples.INTERMEDIATERagas LibraryYou will understand what Ragas measures, the core RAG metrics it computes, and where it fits in an evaluation workflow.INTERMEDIATEDeepEvalYou will understand how DeepEval frames LLM evaluation as unit tests and which metrics it provides out of the box.INTERMEDIATEPromptfooYou will understand how Promptfoo compares prompts and models side by side and runs evals plus red-team probes in CI.INTERMEDIATElm-evaluation-harnessYou will understand why lm-evaluation-harness is the de facto standard for running academic LLM benchmarks reproducibly.INTERMEDIATEInspect (AISI)You will understand how Inspect structures rigorous frontier and safety evaluations using datasets, solvers, and scorers.INTERMEDIATEOpenAI EvalsYou will understand what the OpenAI Evals framework does and how the open-source repo relates to the hosted Evals API.INTERMEDIATE