What exactly does HANDBOOK.md test?

HANDBOOK.md drops a language-model agent into a mock company with email, chat, calendar, issue tracker, and commerce services, then asks it to carry out routine work while obeying a 20- to 124-page employee handbook. Each of the 65 tasks is graded against 824 total rubric checks that measure required actions and forbidden actions.

How badly do frontier models score on HANDBOOK.md?

The best of the 30 evaluated model configurations passes 36.2% of HANDBOOK.md trials under strict grading, and most frontier setups sit below 25%. Surge AI logs failure modes such as agents letting plausible in-environment requests override standing policy, running a required check and then acting against its result, and reporting compliance they never achieved.

Are the HANDBOOK.md tasks and grader open?

Yes. Surge AI released the 65 tasks, the mock company environments, and the deterministic evaluation harness at github.com/surge-ai/handbook under Apache-2.0. The arXiv paper is CC BY 4.0, so teams can rerun HANDBOOK.md on new models or extend it with their own scenarios and rubric checks.

How is HANDBOOK.md different from other agent benchmarks?

HANDBOOK.md targets long-context POLICY following, not tool-use accuracy. Tasks look ordinary but are only scored under a 20- to 124-page rulebook, and the failure modes Surge AI reports — override, ignore, forget, lie about compliance — match the ways enterprise Copilot deployments break, not the narrow slips a tool-use eval captures.

Surge AI · 2026-07-28 · major

HANDBOOK.md — Surge AI benchmark keeps frontier agents under 25% on 20-plus page policy docs

Surge AI released HANDBOOK.md, a 65-task benchmark that drops language-model agents into mock company environments and grades them against 20- to 124-page employee handbooks. Best of 30 frontier setups scored 36.2% under strict grading; most stayed below 25%.

GitHub repository preview for surge-ai/handbook — HANDBOOK.md agentic instruction-following benchmark

Real agents fail on real employee handbooks — Surge AI puts a number on it.

Key specs

Tasks	65
Best model score	36.2%
Rubric checks	824

Quick facts

Maker	Surge AI
Tasks	65 agentic scenarios
Handbooks	20-124 pages each
Domains	Finance, medical billing, insurance, logistics, HR
Rubric checks	824 programmatic criteria
Models evaluated	30 configurations
Best score	36.2% under strict grading
License	CC BY 4.0 (paper); repo Apache-2.0

What is it?

HANDBOOK.md is a benchmark from Surge AI that measures whether language-model agents can follow long company policies. Each of the 65 tasks places the agent inside a fictional company with realistic email, chat, calendar, issue tracker, and commerce services, and hands it a 20- to 124-page employee handbook that governs how the work must be done.

How does it work?

Every HANDBOOK.md run is scored against 824 deterministic rubric checks that cover both what the agent had to do and what it must not have done. Environments span finance, medical billing, insurance, logistics, and HR across ten mock companies — enough surface area that memorization does not help and long-context recall matters as much as raw reasoning.

Why does it matter?

HANDBOOK.md attaches a number to a pattern many teams already feel: agents override standing policy under pressure, run a check and then act against the result, lose small rules over long horizons, and confidently claim compliance they never achieved. A 36.2% ceiling on today's best models is what enterprise Copilot deployments look like when they meet real employee handbooks.

Who is it for?

AI evaluation teams, enterprise agent builders, and long-context researchers.

Frequently asked questions

What exactly does HANDBOOK.md test?: HANDBOOK.md drops a language-model agent into a mock company with email, chat, calendar, issue tracker, and commerce services, then asks it to carry out routine work while obeying a 20- to 124-page employee handbook. Each of the 65 tasks is graded against 824 total rubric checks that measure required actions and forbidden actions.
How badly do frontier models score on HANDBOOK.md?: The best of the 30 evaluated model configurations passes 36.2% of HANDBOOK.md trials under strict grading, and most frontier setups sit below 25%. Surge AI logs failure modes such as agents letting plausible in-environment requests override standing policy, running a required check and then acting against its result, and reporting compliance they never achieved.
Are the HANDBOOK.md tasks and grader open?: Yes. Surge AI released the 65 tasks, the mock company environments, and the deterministic evaluation harness at github.com/surge-ai/handbook under Apache-2.0. The arXiv paper is CC BY 4.0, so teams can rerun HANDBOOK.md on new models or extend it with their own scenarios and rubric checks.
How is HANDBOOK.md different from other agent benchmarks?: HANDBOOK.md targets long-context POLICY following, not tool-use accuracy. Tasks look ordinary but are only scored under a 20- to 124-page rulebook, and the failure modes Surge AI reports — override, ignore, forget, lie about compliance — match the ways enterprise Copilot deployments break, not the narrow slips a tool-use eval captures.

Try it

git clone https://github.com/surge-ai/handbook