Ai2 · 2026-08-07 · notable
TutorMoments — Ai2 benchmark tests when an AI tutor should hold back
TutorMoments is an open benchmark from Ai2 that scores whether a language model knows when to help a struggling student and when to step back. It ships with 462 annotated math tutoring transcripts and Apache-2.0 code.
An open benchmark that grades AI tutors on restraint, not just correctness.
Key specs
| Transcripts | 462 |
|---|---|
| Annotated moments | 1,500+ |
| Teacher annotators | 27 |
What is it?
TutorMoments scores a language model on a judgment call teachers make constantly: give the student more support, or push them to keep thinking. Ai2 built it from 462 de-identified math tutoring transcripts covering U.S. grades 2 through 7, with more than 1,500 key moments marked up by 27 experienced math teachers. The dataset, the code and a technical report are all public.
How does it work?
The benchmark is replay-based. Teachers picked the exact points in a real session where a tutor had to choose between adding a scaffold and pushing for rigor, then recorded what the human tutor did and how well it worked. A model is dropped into those same moments and graded on three axes — appropriate scaffolding, appropriate rigor, and avoiding over-scaffolding — against the human record.
Why does it matter?
Most tutoring evaluations reward a model for being helpful, which is exactly the wrong instinct when a student learns by struggling. Ai2 found that seven tested models, including Claude Opus 4.8, GPT 5.5 and Gemini 2.5 Pro, tend to over-help and rarely push for deeper thinking, and that an evaluation-aware prompt explaining the trade-off improves all of them. Anyone building a study assistant now has a public yardstick for that failure mode.
Who is it for?
education AI teams, eval researchers
Try it
git clone https://github.com/allenai/tutormoments