Peking University, HKU · 2026-04-07 · notable
Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents
A 300-task benchmark that grades agents on the full trajectory — execution traces, audit logs and environment snapshots — not just final outputs. Catches 44% of safety violations final-output grading misses.
Grading AI agents on what they did, not just what they said — and discovering final-output grading misses 44 percent of safety violations.
Key specs
| Tasks | 300 |
|---|---|
| Rubric items | 2,159 |
| Missed safety violations | 44% |
| Models tested | 14 |
What is it?
Claw-Eval is a new evaluation framework for autonomous agents from researchers at Peking University and the University of Hong Kong. It argues the standard way of grading agents — checking only the final output — is 'trajectory-opaque' and systematically unreliable. Instead Claw-Eval records the whole trajectory through three independent channels: execution traces, audit logs and environment snapshots.
How does it work?
The suite contains 300 human-verified tasks in nine categories across three groups — general service orchestration, multimodal perception and generation, and multi-turn professional dialogue — with 2,159 fine-grained rubric items. Each task is graded on completion, safety and robustness across three trials, with the three evidence channels cross-checked against each other. The authors evaluated 14 frontier models and found trajectory-opaque evaluation misses 44% of safety violations and 13% of robustness failures that Claw-Eval's full-trajectory grading catches.
Why does it matter?
Every agent benchmark that only checks the final answer is under-reporting risk. Claw-Eval gives teams a concrete, reproducible way to catch agents that 'get the right answer the wrong way' — especially important as autonomous agents start touching real systems.
Who is it for?
Agent framework authors, red-teamers, anyone shipping agents to production.