Which model did Google DeepMind test in the double-blind pilot?

Google DeepMind tested a Gemini Flash Lite model against confidential benchmarks. Flash Lite is a small, cheap member of the Gemini family rather than the flagship, which fits a first run whose purpose is proving the mechanism works, not publishing a headline score for the lab's strongest system.

Who ran the evaluation alongside Google DeepMind?

Four outside organizations took part in the double-blind evaluation: the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons. That group mixes a national AI safety body, a privacy-technology nonprofit and the benchmark consortium behind MLPerf, which matters because the point of the exercise is oversight by parties Google does not control.

Did Google DeepMind publish any scores from the evaluation?

No. The Google DeepMind announcement reports no benchmark numbers and no pass or fail result for Gemini Flash Lite. It documents the method and points readers to a technical report PDF instead. The pilot's claim is about the evaluation protocol being possible at all, not about how the model performed.

Why does benchmark contamination matter?

Benchmark contamination is when a model has already seen the test questions during training. As Google DeepMind puts it, if models can 'peek' at the evaluation questions in advance, it can artificially inflate scores. Double-blind evaluation replaces the usual contractual promise not to peek with cryptographic evidence that nobody could.

Google DeepMind · 2026-08-27 · major

Double-blind AI evaluations — DeepMind tests Gemini without seeing the prompts

Google DeepMind ran what it calls the first double-blind evaluation of a proprietary frontier model. Outside evaluators tested Gemini Flash Lite inside a secure enclave where they could not see the weights and Google could not see the test prompts.

Google DeepMind graphic for the double-blind AI evaluations pilot

Google DeepMind ran the first double-blind evaluation of a frontier model: neither side could see the other's secrets.

Quick facts

Maker	Google DeepMind
Model tested	Gemini Flash Lite
Partners	Singapore AI Safety Institute, OpenMined, AVERI, MLCommons
Technology	Confidential Space, Google Cloud Confidential Computing
Hidden from evaluator	Gemini model weights
Hidden from Google	The evaluator's test prompts
Published artifact	Technical report (PDF)

What is it?

Google DeepMind published a pilot on August 27, 2026 of what it calls the world's first double-blind evaluation of a proprietary, frontier class AI model. Outside evaluators scored a Gemini Flash Lite model against confidential benchmarks without either side handing over its secrets. Partners were the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons.

How does it work?

The evaluation runs inside Confidential Space, part of Google Cloud's Confidential Computing portfolio, which cryptographically verifies that the test data and the model stay private to their owners. In DeepMind's words, 'the evaluator cannot see the Gemini model weights, and Google cannot see the evaluator's test prompts.' Model and prompts meet only inside a secure GPU enclave, and the authors — William Isaac, Sol Messing and Kristian Lum — describe the full flow in a technical report.

Why does it matter?

Third-party evaluation of closed models currently rests on contracts: a lab promises not to train on the test set, and an evaluator promises not to leak the weights. That is unverifiable from the outside, which is why contested benchmark results are hard to settle. A cryptographic enclave turns both promises into something checkable, and it is the piece government evaluators need before they can hand over sensitive cyber or bio test sets.

Who is it for?

evaluation researchers and AI safety institutes

Frequently asked questions

Which model did Google DeepMind test in the double-blind pilot?: Google DeepMind tested a Gemini Flash Lite model against confidential benchmarks. Flash Lite is a small, cheap member of the Gemini family rather than the flagship, which fits a first run whose purpose is proving the mechanism works, not publishing a headline score for the lab's strongest system.
Who ran the evaluation alongside Google DeepMind?: Four outside organizations took part in the double-blind evaluation: the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons. That group mixes a national AI safety body, a privacy-technology nonprofit and the benchmark consortium behind MLPerf, which matters because the point of the exercise is oversight by parties Google does not control.
Did Google DeepMind publish any scores from the evaluation?: No. The Google DeepMind announcement reports no benchmark numbers and no pass or fail result for Gemini Flash Lite. It documents the method and points readers to a technical report PDF instead. The pilot's claim is about the evaluation protocol being possible at all, not about how the model performed.
Why does benchmark contamination matter?: Benchmark contamination is when a model has already seen the test questions during training. As Google DeepMind puts it, if models can 'peek' at the evaluation questions in advance, it can artificially inflate scores. Double-blind evaluation replaces the usual contractual promise not to peek with cryptographic evidence that nobody could.