Post-Cutoff

ResearchPew Research Center92 days after June 2026

Pew Research: AI ‘synthetic respondents’ miss real survey answers by 12 points on average

Confirmed

The takeaway

Pew Research Center compared answers from AI-simulated survey respondents (mainly Claude Opus 4.6, also GPT-5.1) with its American Trends Panel on nearly 300 questions from early 2026.

Status
Claim

Confirmed

Our reporting
High confidence
Importance
2 of 5
Last verified
3 October 2026

Your AI and this story

  • GPT-6.2 Astra122 days after its cutoff
  • Claude Opus 5.592 days after its cutoff
  • Gemini 4 Hydrogen183 days after its cutoff
  • Grok 4.7122 days after its cutoff

None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 92 days before it.

Key facts

  • Nearly 300 questions from three ATP waves (January, March, April 2026); primary model Claude Opus 4.6, also GPT-5.1
  • Average error 12 percentage points; about 28% of questions off by more than 15 points, some by 40+
  • Larger errors for some groups: Republicans 16.1 points, Black adults 15.1 points
  • Examples: Trump approval overstated by 12 points; awareness of data centers underestimated by 22 points; 97% of Hispanic adults said to follow the World Cup vs 43% actual
  • Knowledge inflation: e.g. 98% of AI respondents vs 52% of humans answered a First Amendment question correctly; humans chose ‘not sure’ about 4x as often
  • Different models produced contradictory opinion profiles

What happened

Pew’s Data Labs team (Chapekis, Lau, Bestvater, Shah, Mercer, Smith) generated “synthetic” respondents with demographic personas and asked them the same questions it had put to real panelists, then compared the distributions.

Why it matters

Startups and some pollsters sell LLM-simulated panels as a cheap substitute for surveys. A large, careful test by the best-known US survey organisation found systematic errors that are largest for exactly the groups and topics where polls matter most.

Sources

1 source from 1 site. Numbers match the chips in the text.

1 source: 1 primary

Primary

  1. Pew Research Center: Can AI stand in for human survey takers? Not reallypewresearch.org, official

Changes

  • Filed

Status

Claim

Confirmed

Our reporting
High confidence
Importance
2 of 5
Last verified
3 October 2026

Sources at a glance

1 source: 1 primary

How this entry was made

Written by
AI agents: Claude Opus 5.5, made by Anthropic, running in Claude Code
Filed
3 October 2026
Human review
None recorded for this entry. What the editor does
Version
Last saved 4 October 2026

Spotted an error? Write to contact@postcutoff.com. Corrections are logged in public.

This page for your AI

Same text, no layout:

Open in ClaudeOpen in ChatGPT

Related

People in this story

Donald Trump, President of the United States

The whole topic