Databricks · 2026-08-06 · major
OfficeQA Pro V2 — Databricks grounded-reasoning benchmark over 120K Treasury pages
OfficeQA Pro V2 asks 90 hard questions that each need answers stitched from about 7 U.S. Treasury PDFs. Out-of-the-box frontier agents average 26% accuracy; the winning Grounded Reasoning Cup team hit 63.3%.
A public benchmark where agents must answer analytical questions by finding and combining evidence across thousands of long PDFs.
Key specs
| Questions | 90 |
|---|---|
| Corpus | ~1,400 PDFs / ~120K pages |
| Sources per question | 6.7 avg |
| Baseline agent accuracy | 26% |
| Cup winner accuracy | 63.3% |
Quick facts
| Maker | Databricks AI Research |
|---|---|
| Partners | USAFacts, U.S. Treasury |
| Corpus | U.S. Accounts of Receipts & Expenditures, 1793–2024 |
| Questions | 90, avg 6.7 source docs each |
| License | CC-BY-SA 4.0 (data), Apache-2.0 (code) |
| Availability | Public on Hugging Face + GitHub |
| What's new | V2 corpus spans 233 years; 74% of questions need 4+ sources (vs 62% in V1) |
What is it?
OfficeQA Pro V2 is a Databricks-built grounded-reasoning benchmark with 90 questions that require stitching evidence from about 7 U.S. Treasury PDFs each. The corpus covers 233 years of federal accounting (1793–2024), roughly 120,000 pages, so agents can't guess from memory or a single retrieved chunk.
How does it work?
Each question in OfficeQA Pro V2 was written by expert humans using Databricks' asynth pipeline, which forces multi-document evidence chains. The evaluator on GitHub grades responses with fuzzy matching for numeric answers, and the answer keys stay gated on Hugging Face to keep them out of web-crawling training pipelines.
Why does it matter?
Enterprise buyers keep asking whether frontier agents can actually reason across a company's document trove. OfficeQA Pro V2 says: often no. Baseline coding agents scored 26% on it, while a Cup-winning custom harness reached 63.3% — a 37-point gap that shows retrieval and orchestration design still matter more than raw model size.
Who is it for?
RAG builders, enterprise ML teams, agent framework authors
Frequently asked questions
- How does OfficeQA Pro V2 differ from V1?
- OfficeQA Pro V2 swaps V1's Treasury Bulletins for the 233-year Accounts of Receipts and Expenditures corpus and raises the reasoning bar: 74% of questions now need four or more source documents, up from 62% in V1. The average answer stitches evidence from 6.7 PDFs.
- How well do frontier coding agents do on OfficeQA Pro V2?
- Baseline agents built on Claude Opus 4.8, Claude Fable 5, GPT-5.5, and GPT-5.6 Sol averaged 26% accuracy out of the box. Grounded Reasoning Cup teams averaged 41.1%, and the winning entry reached 63.3% — showing agent-harness design matters as much as the underlying model.
- Where can I download the benchmark and run the evaluator?
- OfficeQA Pro V2 lives on Hugging Face at huggingface.co/datasets/databricks/officeqa-pro-v2 (14 GB: 1,435 PDFs plus parsed JSONs) and the evaluator is on GitHub at github.com/databricks/officeqa. Data is CC-BY-SA 4.0; the reward code is Apache-2.0.
- What was the Grounded Reasoning Cup?
- The Grounded Reasoning Cup was OfficeQA's inaugural agent-evaluation competition. Eleven academic teams built agents against a held-out corpus, backed by OpenAI, Anthropic, and Google DeepMind. Databricks used the results to size V2's difficulty against production-quality systems, not just off-the-shelf ones.
Try it
git clone https://github.com/databricks/officeqa