Evaluation & Safety · TRACK 02/05
LLM-as-a-Judge
Using models to grade models — and when to distrust the grader.
// THE TRACK
01 · START HEREUsing Models to Grade ModelsUnderstand how LLM-as-a-judge works, why teams use it instead of human review, and the failure modes to watch for.BEGINNERCommon PitfallsUnderstand the four systematic biases that make LLM judges lie, how to measure each one in your pipeline, and the specific mitigations that actually work.INTERMEDIATEJudge BiasesBe able to name the major LLM judge biases and apply the standard mitigations like position swapping and length controls.INTERMEDIATEPairwise vs Rubric ScoringUnderstand when to ask a judge "which answer is better" versus "score this answer", and what each pattern is good for.INTERMEDIATECalibrating a JudgeYou'll understand the loop for making an LLM judge trustworthy: compare it to human labels, find where it disagrees, and fix the rubric.INTERMEDIATEReference-Free vs Reference-BasedYou'll understand the difference between grading against a known answer and judging quality with no reference, and when each is appropriate.INTERMEDIATEChain-of-Thought JudgingYou'll understand why making a judge explain its reasoning before giving a score produces more reliable grades, and how to prompt for it.INTERMEDIATEJudge vs Human AgreementYou'll understand how to quantify whether an LLM judge agrees with human graders, and what agreement scores are acceptable.ADVANCEDAI Grading AIYou will understand how LLM-as-a-Judge scoring works, its main variants, and the biases you must guard against.INTERMEDIATEG-EvalYou will understand how G-Eval turns plain-language criteria into a chain-of-thought LLM score you can reuse.INTERMEDIATE