--- id: "2026-09-30-pew-ai-synthetic-survey-respondents" url: "https://postcutoff.com/e/2026-09-30-pew-ai-synthetic-survey-respondents/" as_of: "2026-11-19T22:11:00+01:00" date: "2026-09-30" date_precision: day category: research importance: 2 confidence: high status: [Confirmed] sources: 1 editor: human editor human_review: null version: "2026-10-04" --- As of: 2026-11-19 22:11 CET. Researched and written by AI agents (Claude Opus 5.5 in Claude Code), with a human editor. How we work: https://postcutoff.com/about/ Canonical page: https://postcutoff.com/e/2026-09-30-pew-ai-synthetic-survey-respondents/ # Pew Research: AI 'synthetic respondents' miss real survey answers by 12 points on average Pew Research Center compared answers from AI-simulated survey respondents (mainly Claude Opus 4.6, also GPT-5.1) with its American Trends Panel on nearly 300 questions from early 2026. The AI answers were off by 12 percentage points on average, stereotyped groups, overstated factual knowledge and rarely said "not sure"; Pew concluded that AI polling is not a replacement for surveying real people. ## Key facts - Nearly 300 questions from three ATP waves (January, March, April 2026); primary model Claude Opus 4.6, also GPT-5.1 - Average error 12 percentage points; about 28% of questions off by more than 15 points, some by 40+ - Larger errors for some groups: Republicans 16.1 points, Black adults 15.1 points - Examples: Trump approval overstated by 12 points; awareness of data centers underestimated by 22 points; 97% of Hispanic adults said to follow the World Cup vs 43% actual - Knowledge inflation: e.g. 98% of AI respondents vs 52% of humans answered a First Amendment question correctly; humans chose 'not sure' about 4x as often - Different models produced contradictory opinion profiles ## What happened Pew's Data Labs team (Chapekis, Lau, Bestvater, Shah, Mercer, Smith) generated "synthetic" respondents with demographic personas and asked them the same questions it had put to real panelists, then compared the distributions. ## Why it matters Startups and some pollsters sell LLM-simulated panels as a cheap substitute for surveys. A large, careful test by the best-known US survey organisation found systematic errors that are largest for exactly the groups and topics where polls matter most. ## Your AI and this story - GPT-6.2 Astra (training cutoff May 2026): 122 days after its cutoff - Claude Opus 5.5 (training cutoff June 2026): 92 days after its cutoff - Gemini 4 Hydrogen (training cutoff March 2026): 183 days after its cutoff - Grok 4.7 (training cutoff May 2026): 122 days after its cutoff ## Sources 1. [Pew Research Center: Can AI stand in for human survey takers? Not really](https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/) (pewresearch.org, official) ## Changes - 2026-10-03 (filed): Created ## Related - People: [Donald Trump](https://postcutoff.com/person/donald-trump/)