{"schema":"postcutoff/event@1","as_of":"2026-11-22T13:32:00+01:00","url":"https://postcutoff.com/e/2026-08-27-deepmind-double-blind-ai-evaluations/","md":"https://postcutoff.com/e/2026-08-27-deepmind-double-blind-ai-evaluations/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-08-27-deepmind-double-blind-ai-evaluations","date":"2026-08-27","date_precision":"day","short_title":"Google DeepMind pilots the first 'double-blind' evaluation of a proprietary frontier model with Singapore's AISI and MLCommons","deck":null,"takeaway":"On Aug 27, 2026 Google DeepMind described what it calls the world's first double-blind evaluation of a proprietary frontier-class model.","category":"policy-safety","category_label":"Policy & safety","importance":3,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Google DeepMind: Piloting the world's first double-blind AI evaluations","url":"https://deepmind.google/blog/piloting-the-worlds-first-double-blind-ai-evaluations/","type":"official","group":"primary","domain":"deepmind.google"},{"n":2,"title":"The New Stack: Google found a way to test Gemini without seeing the questions","url":"https://thenewstack.io/google-double-blind-evaluation/","type":"press","group":"press","domain":"thenewstack.io"},{"n":3,"title":"MLQ: Google DeepMind pilots sealed external tests for Gemini model","url":"https://mlq.ai/news/google-deepmind-pilots-sealed-external-tests-for-gemini-model/","type":"press","group":"press","domain":"mlq.ai"}],"official":1,"filed":"2026-09-30","updated":"2026-09-30","orgs":["Google DeepMind","Singapore AI Safety Institute","OpenMined","AVERI","MLCommons"],"title":"Google DeepMind pilots the first 'double-blind' evaluation of a proprietary frontier model with Singapore's AISI and MLCommons","summary":"On Aug 27, 2026 Google DeepMind described what it calls the world's first double-blind evaluation of a proprietary frontier-class model. A Gemini Flash-Lite model was tested inside a cryptographically attested confidential-computing environment on Google Cloud, so the evaluators (Singapore's AI Safety Institute, OpenMined, AVERI and MLCommons) never saw the model weights and Google never saw the test prompts. The aim is to prevent benchmark contamination without forcing labs to hand over weights.","key_facts":["Partners: Singapore AI Safety Institute, OpenMined, AVERI and MLCommons (DeepMind)","Uses Google Cloud Confidential Space to cryptographically verify that the evaluation data and the proprietary model stay private to their owners; The New Stack reports the run used an NVIDIA H100 confidential GPU instance","Model tested: 'a Gemini Flash Lite model' (DeepMind); The New Stack identifies it as Gemini 2.5 Flash Lite, run against reserve prompts from MLCommons' AILuminate safety benchmark and a separate private prompt set for the Singapore context","Removes the old trade-off in which evaluators either hand over test prompts (contamination risk) or labs hand over weights (IP risk); DeepMind says it matters most for sensitive cyber and government evaluations"],"key_numbers":[],"tags":["evaluations","benchmark-contamination","confidential-computing","ai-safety-institutes","third-party-testing","gemini"],"science":null,"body_md":"## What happened\n\nExternal safety testing usually forces a trade-off. Either the evaluator gives the lab its test prompts, which can then leak into\ntraining and inflate scores, or the lab gives the evaluator its model weights, which it does not want to do. Google DeepMind and four\npartners ran an evaluation where neither happened. Inside a hardware-isolated, attested Google Cloud environment, the evaluators' confidential\nbenchmarks ran against a Gemini Flash-Lite model: \"The evaluator cannot see the Gemini model weights, and Google cannot see the\nevaluator's test prompts.\" DeepMind published its methodology and findings with the announcement.\n\n## Why it matters\n\nAs AI safety institutes and outside auditors gain a formal role (e.g. California SB 813's independent assessors, OpenAI's Sept 22\nprinciples for third-party assessments), a trusted way to test closed models on secret benchmarks becomes basic infrastructure. The pilot\nused a small model; it is not yet shown to scale to frontier models.","disputed":[],"related":[{"id":"2026-09-28-openai-safety-cases-frontier-training","url":"https://postcutoff.com/e/2026-09-28-openai-safety-cases-frontier-training/","date":"2026-09-28","date_precision":"day","short_title":"OpenAI publishes early guidelines for 'safety cases' before frontier training runs","deck":null,"takeaway":"It moves the safety gate earlier, to training itself, and fits Altman's stated openness to pausing at new capability levels.","category":"policy-safety","category_label":"Policy & safety","importance":3,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":6,"official":2,"filed":"2026-09-29","updated":"2026-09-29","orgs":["OpenAI"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-09-30","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-09-30","run":null,"sources_read":null,"updated":"2026-09-30","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":88,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":58,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":149,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":88,"in_training_data":false}],"short_url":null}