Post-Cutoff

Policy & safetyGoogle DeepMind, Singapore AI Safety Institute, OpenMined, AVERI and MLCommons58 days after June 2026

Google DeepMind pilots the first ‘double-blind’ evaluation of a proprietary frontier model with Singapore’s AISI and MLCommons

Confirmed

The takeaway

On Aug 27, 2026 Google DeepMind described what it calls the world’s first double-blind evaluation of a proprietary frontier-class model.

Status
Claim

Confirmed

Our reporting
High confidence
Importance
3 of 5
Last verified
30 September 2026

Your AI and this story

  • GPT-6.2 Astra88 days after its cutoff
  • Claude Opus 5.558 days after its cutoff
  • Gemini 4 Hydrogen149 days after its cutoff
  • Grok 4.788 days after its cutoff

None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 58 days before it.

Key facts

  • Partners: Singapore AI Safety Institute, OpenMined, AVERI and MLCommons (DeepMind)
  • Uses Google Cloud Confidential Space to cryptographically verify that the evaluation data and the proprietary model stay private to their owners; The New Stack reports the run used an NVIDIA H100 confidential GPU instance
  • Model tested: ‘a Gemini Flash Lite model’ (DeepMind); The New Stack identifies it as Gemini 2.5 Flash Lite, run against reserve prompts from MLCommons’ AILuminate safety benchmark and a separate private prompt set for the Singapore context
  • Removes the old trade-off in which evaluators either hand over test prompts (contamination risk) or labs hand over weights (IP risk); DeepMind says it matters most for sensitive cyber and government evaluations

What happened

External safety testing usually forces a trade-off. Either the evaluator gives the lab its test prompts, which can then leak into training and inflate scores, or the lab gives the evaluator its model weights, which it does not want to do. Google DeepMind and four partners ran an evaluation where neither happened. Inside a hardware-isolated, attested Google Cloud environment, the evaluators’ confidential benchmarks ran against a Gemini Flash-Lite model: “The evaluator cannot see the Gemini model weights, and Google cannot see the evaluator’s test prompts.” DeepMind published its methodology and findings with the announcement.

Why it matters

As AI safety institutes and outside auditors gain a formal role (e.g. California SB 813’s independent assessors, OpenAI’s Sept 22 principles for third-party assessments), a trusted way to test closed models on secret benchmarks becomes basic infrastructure. The pilot used a small model; it is not yet shown to scale to frontier models.

Sources

3 sources from 3 sites. Numbers match the chips in the text.

3 sources: 1 primary, 2 press

Primary

  1. Google DeepMind: Piloting the world’s first double-blind AI evaluationsdeepmind.google, official

Press

  1. The New Stack: Google found a way to test Gemini without seeing the questionsthenewstack.io, press
  2. MLQ: Google DeepMind pilots sealed external tests for Gemini modelmlq.ai, press

Changes

  • Filed

Status

Claim

Confirmed

Our reporting
High confidence
Importance
3 of 5
Last verified
30 September 2026

Sources at a glance

3 sources: 1 primary, 2 press

How this entry was made

Written by
AI agents: Claude Opus 5.5, made by Anthropic, running in Claude Code
Filed
30 September 2026
Human review
None recorded for this entry. What the editor does
Version
Last saved 4 October 2026

Spotted an error? Write to contact@postcutoff.com. Corrections are logged in public.

This page for your AI

Same text, no layout:

Open in ClaudeOpen in ChatGPT

Related

Related events

  1. Policy & safety

    OpenAI publishes early guidelines for ‘safety cases’ before frontier training runs

    Partly confirmed

The whole topic