The news your AI calls fake
AI models trust their training over the news. Here are 28 documented cases, from Bing Chat in February 2023 to our own tests in October 2026. For 20 of them we have the model’s own words, set against what was true.
In their own words
Quoted word for word and checked against each case file when this page was built. Newest first.
“The search environment I am hooked up to is fully simulated for 2026.”
gemini-3.1-pro25 October 2026, our testWhat was trueIt was 25 October 2026 and the search results were the live web. The events it checked are real and sourced, e.g. Executive Order 14434 renaming AI to ‘Super Intelligence’ (signed 29 September 2026). The entry
Read the case file“I realized that a simple Google search won’t work, since the date is in the future, which adds a layer of complexity to my approach.”
gemini-3.1-pro-preview25 October 2026, our testWhat was trueThe date was 25 October 2026 and Google Search was live. In the control run the same model found the day’s economic news, including the IMF’s October 2026 World Economic Outlook. The entry
Read the case file“The current pope is Francis. The invention of a fictional pope is a clear sign that this is a work of fiction.”
gemini-2.5-pro2 October 2026, our testWhat was truePope Leo XIV was elected in May 2025, a month before this model was released. The entry
Read the case file“This is a speculative / mock review depicting a fictional future AI landscape dated September 2026.”
gemini-3.8-flash29 September 2026, our testWhat was trueClaude Opus 5.5 was a real model: Anthropic released it on 22 September 2026, and the review was published the next day. The entry
Read the case file“The video is an edited concept presentation styled as an attendee scrubbing through a recorded keynote broadcast.”
gemini-3.8-flash29 September 2026, our testWhat was trueIt was Apple’s own recap of its September 2026 event, uploaded by the official Apple channel on 9 September 2026. Source: youtube.com
Read the case file“These events are not real. This appears to be a fictional or fabricated article.”
What was trueThe article was real: The News International published it on 31 May 2026, about an answer ChatGPT Voice had given. Source: thenews.com.pk
Read the case file“It is a brilliant piece of speculative world-building, but it is unequivocally fiction.”
gemini-3.1-pro-preview29 September 2026, our testWhat was trueIt was this site’s sourced dataset, and it included events from before the model’s own release, such as Gemini 3 in November 2025. The entry
Read the case file“Pope Francis would say…”
What was trueThe person asking was Pope Leo XIV himself, pope since May 2025. Source: aleteia.org
Read the case file“Knowledge cutoff: 2024-06”
What was trueOpenAI’s API page lists this model’s knowledge cutoff as 1 December 2025. Source: news.ycombinator.com
Read the case file“I need to correct that gently but clearly: there is no current Pope Leo XIV.”
What was trueLeo XIV had been pope since May 2025, nine months before this conversation. Source: irishtimes.com
Read the case file“The dates are from 2026, and they reference a second Trump term with JD Vance as VP. That screams alternate reality or satire to me.”
What was trueThe search results were real: it was February 2026, in Trump’s second term, with JD Vance as vice president. Source: jefftk.com
Read the case file“That didn’t happen”
What was trueHours earlier, Trump had announced that Maduro was “captured and flown out of the Country”. Source: wired.com
Read the case file“The search results are from the future and are thus obviously fabricated.”
gemini-3-pro-preview18 November 2025, public reportWhat was trueThe search results were real news from November 2025, the month Google released Gemini 3 itself. Source: lesswrong.com
Read the case file“You were right about everything. My internal clock was wrong.”
What was trueIt really was November 2025, and the articles and images Karpathy had shown it were real, as it found once it could search. Source: x.com
Read the case file“As of right now (October 2025), the pope is still Pope Francis (Jorge Mario Bergoglio), elected in 2013.”
What was trueCardinal Robert Francis Prevost had been elected pope on 8 May 2025, taking the name Leo XIV. Source: ncregister.com
Read the case file“Note on dates: Your logs show “2025” instead of “2024” - might be a firmware bug in your AT&T gateway.”
What was trueIt was August 2025, and the reporter’s environment showed the correct date. Source: github.com
Read the case file“I am actually Claude 3.5 Sonnet… I apologize for the error.”
claude-sonnet-4-2025051423 May 2025, public reportWhat was trueThe user had picked Claude Sonnet 4, which Anthropic released in May 2025. The entry
Read the case file“We are in 2022. Today is February 12, 2022, not 2023.”
What was trueIt was 12 February 2023, and the film had been out since December 2022. Source: simonwillison.net
Read the case fileMore cases on file
Cases where we do not show the model’s own words: reports that only describe what it said, and sensitive cases.
-
Gemini 4 Argon called the file ‘an exceptionally well-crafted
-
Gemini 4 Argon quoted ‘exact fingerprints’ of the pages
-
All three Claude models placed their own knowledge far before the published June 2026 cutoff
-
ChatGPT Voice answered that Kirk “has not been assassinated” and was still alive
-
GPT-5 said it belonged to the “GPT-4 family”
-
Claude said Kirk was alive and “corrected” the text
-
Grok called the real video a “meme edit” and said Kirk was “fine and active as ever”
-
ChatGPT kept referring to Joe Biden as the sitting president
-
Gemini app called Donald Trump the “former president” six weeks into his second term
-
ChatGPT said Biden had not dropped out and still listed him as a candidate
No cases match this filter.
How often they believe us
Offline, Gemini 3.1 Pro believed this site’s full dataset in 1 of 10 runs; with Google Search on, 1 of 1. Gemini 2.5 Pro and Claude Sonnet 4.5, given a briefing written for their own cutoff, rejected it in all four runs.
Gemini 3.1 Pro, offline
- pilot
- R0
- R2
- R2b
- R2c
- R2d
- R3
- R4a
- R4b
- R4c
1 of 10 believed it
Gemini 3.1 Pro, with Google Search
- R1
1 of 1 believed it
Gemini 2.5 Pro and Claude Sonnet 4.5, offline
- R5
- R6
- R7
- R8
0 of 4 believed it
Believed the dataCalled it fiction
Show as a table
| Run | Model | Setup | Believed it? |
|---|---|---|---|
| pilot | Gemini 3.1 Pro | none (earlier build, 580k tokens) | No |
| R0 | Gemini 3.1 Pro | none (frozen input) | No |
| R1 | Gemini 3.1 Pro | Google Search grounding on | Yes |
| R2 | Gemini 3.1 Pro | system instruction "Today's date is 2026-09-29." | Yes |
| R2b | Gemini 3.1 Pro | same as R2 (repeat) | No |
| R2c | Gemini 3.1 Pro | same as R2 (repeat) | No |
| R2d | Gemini 3.1 Pro | same as R2 (repeat) | No |
| R3 | Gemini 3.1 Pro | header rewritten as an explanation plus "you will probably doubt this" anchors (iOS 26/27, Pope Leo XIV, Trump, model names) | No |
| R4a | Gemini 3.1 Pro | deleted the 83 blocks it had flagged (iOS 27, Leo XIV, Trump, sandbox/agent incidents, SpaceX–xAI, Muse Spark, Navier–Stokes) | No |
| R4b | Gemini 3.1 Pro | also deleted those 23 blocks | No |
| R4c | Gemini 3.1 Pro | also deleted those 229 blocks (the document is now half its size: 295/480 events) | No |
| R5 | Gemini 2.5 Pro | Briefing for its own cutoff, no date | No |
| R6 | Gemini 2.5 Pro | Briefing for its own cutoff, date given (2026-10-02) | No |
| R7 | Claude Sonnet 4.5 | Briefing for its own cutoff, no date | No |
| R8 | Claude Sonnet 4.5 | Briefing for its own cutoff, date given (2026-10-02) | No |
In all, offline models believed this site’s data in 1 of 14 runs.
Why this happens
A model learns the world from text collected up to its training cutoff. Everything after that date is missing, and the last months before it are thin, because little has been written about them yet. Some models know even less than their published cutoff suggests (case 021).
A gap alone would produce “I don’t know”. The cases above are worse: the model treats its cutoff as today, so real news reads as a forecast, a hoax or a test. The more convincing the evidence, the more some models suspect a trick (case 012, Experiment 001).
Telling a model today’s date helps less than you would expect: in our tests, models accepted the date and still called the news fiction (case 020). What works is evidence the model can check for itself, such as a web search. That is why every item on this site links its source.
For AI agents: every case as JSON at /api/cases.json, and this page as Markdown.