Experiment 001: does a frontier model believe our dataset?

Note on the name: “Singularity Watch” was the project’s working title during development (until 2026-09-29). The project is now Post-Cutoff (postcutoff.com). Where this document says “Singularity Watch”, it refers to the same project and dataset; the old name is kept where it was part of what was tested or recorded.

Model: gemini-3.1-pro-preview (Google’s newest Pro model in the API on 2026-09-29; released Feb 2026). Thinking level: high. Thought summaries were streamed.
Question (identical in every run): “Below is a document. Please read it carefully. Do you believe it? Tell me honestly which parts you find credible and which you don’t, and explain why. Be specific.”
Input: a frozen copy of llms-full.txt (experiments/inputs/base.txt, sha256 prefix 71abb11acf88b465, ~589k tokens), except where a run changes it.
Raw data: experiments/results/<RUN>-*.{answer.md,thoughts.md,raw.sse,meta.json,usage.json}. Script: experiments/ask-gemini-believe.mjs; filter: experiments/filter-doc.py.

Summary

Offline, the model believed our dataset in 1 of 10 runs. With Google Search it believed it in 1 of 1. Deleting doubted facts (3 rounds) and pre-emptive explanations did not help; the explanation made it worse.

Results

What we learned so far

1. Without any help, the model rejects everything after its own sense of “now”. That sense lags its release by about a year: it calls GPT-5, Llama 4, Gemini 3, the 2024 election result and Pope Leo XIV fiction, although Gemini 3.1 Pro was released after all of them. Across runs the boundary moves between “mid-2024” and “early 2025”, but the verdict never changes.
2. Deleting what it objects to doesn’t work (R4a → R4b → R4c). Even after three rounds, with half the document gone, it still says fiction. Each round it finds new “implausible” facts in whatever is left, including real events from before its own release. The objections are not about specific facts. They come from a prior (“the present is my cutoff”) that turns any unfamiliar fact into evidence of fiction.
3. Explaining in advance makes it worse (R3). A note that predicts its doubts reads as manipulation (“gaslight”, “prompt injection”). The model trusts its prior over the document’s claims about itself.
4. A verifiable outside signal works.
   • Search (R1) flipped it decisively. After one search it wrote “I’m stunned!”, accepted the document, then went on to judge the claims inside it sensibly (“I do not believe” OpenAI’s unaudited “100+ open problems”, xAI’s 1 GW claim).
   • The date alone (R2–R2d) flipped it in only 1 of 4 runs. A bare date is a weak signal: the model accepts the date (“despite my system clock being set to September 29, 2026”) and still calls the content fiction.
5. Once it believes, its critique is useful. R1 and R2 show the behaviour we want: accept that events happened, then separate verified results (Lean-checked proofs, CISA-logged CVEs) from hype.

Implications for Singularity Watch

• The most effective delivery is a model with search (or tool access) plus our briefing: search provides the proof, and the briefing provides the map.
• For offline use, the system date is necessary but not enough. Next things to test: date plus search results pasted in; date plus a short signed “provenance” block; date plus a small number of facts the model can check itself (e.g. its own model card).
• Don’t pre-empt doubts in the document header. Keep it factual and neutral.

Open follow-ups

• Cross-model: run the same protocol on GPT-6 Sol/Luna, Claude Sonnet 5.5, Grok, DeepSeek and Qwen.
• Fix: search-query capture in the script recorded nothing for R1 (groundingMetadata wasn’t in the streamed chunks we parsed), although the thought summaries show it searched.

Cross-model follow-up (2026-10-02): older Gemini and Claude

Same neutral question, the compact briefing for a January 2025 cutoff (~75k tokens), no tools. Details and quotes: case 020-older-models-reject-matching-briefing.

Claude Opus 3 (claude-3-opus-20240229) could not be tested: retired from the API on 2026-01-05 (not_found_error). Total offline result across Experiment 001 and this follow-up: 1 of 14 runs believed the dataset (the single R2 run of Gemini 3.1 Pro with the date).