{"schema":"postcutoff/event@1","as_of":"2026-11-19T22:11:00+01:00","url":"https://postcutoff.com/e/2026-09-30-pew-ai-synthetic-survey-respondents/","md":"https://postcutoff.com/e/2026-09-30-pew-ai-synthetic-survey-respondents/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-09-30-pew-ai-synthetic-survey-respondents","date":"2026-09-30","date_precision":"day","short_title":"Pew Research: AI 'synthetic respondents' miss real survey answers by 12 points on average","deck":null,"takeaway":"Pew Research Center compared answers from AI-simulated survey respondents (mainly Claude Opus 4.6, also GPT-5.1) with its American Trends Panel on nearly 300 questions from early 2026.","category":"research","category_label":"Research","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Pew Research Center: Can AI stand in for human survey takers? Not really","url":"https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/","type":"official","group":"primary","domain":"pewresearch.org"}],"official":1,"filed":"2026-10-03","updated":"2026-10-03","orgs":["Pew Research Center"],"title":"Pew Research: AI 'synthetic respondents' miss real survey answers by 12 points on average","summary":"Pew Research Center compared answers from AI-simulated survey respondents (mainly Claude Opus 4.6, also GPT-5.1) with its American Trends Panel on nearly 300 questions from early 2026. The AI answers were off by 12 percentage points on average, stereotyped groups, overstated factual knowledge and rarely said \"not sure\"; Pew concluded that AI polling is not a replacement for surveying real people.","key_facts":["Nearly 300 questions from three ATP waves (January, March, April 2026); primary model Claude Opus 4.6, also GPT-5.1","Average error 12 percentage points; about 28% of questions off by more than 15 points, some by 40+","Larger errors for some groups: Republicans 16.1 points, Black adults 15.1 points","Examples: Trump approval overstated by 12 points; awareness of data centers underestimated by 22 points; 97% of Hispanic adults said to follow the World Cup vs 43% actual","Knowledge inflation: e.g. 98% of AI respondents vs 52% of humans answered a First Amendment question correctly; humans chose 'not sure' about 4x as often","Different models produced contradictory opinion profiles"],"key_numbers":[],"tags":["polling","synthetic-respondents","social-science","evaluation","bias"],"science":null,"body_md":"## What happened\n\nPew's Data Labs team (Chapekis, Lau, Bestvater, Shah, Mercer, Smith) generated \"synthetic\" respondents with demographic personas and asked them the same questions it had put to real panelists, then compared the distributions.\n\n## Why it matters\n\nStartups and some pollsters sell LLM-simulated panels as a cheap substitute for surveys. A large, careful test by the best-known US survey organisation found systematic errors that are largest for exactly the groups and topics where polls matter most.","disputed":[],"related":[],"people":[{"id":"donald-trump","name":"Donald Trump","url":"https://postcutoff.com/person/donald-trump/"}],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-03","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-03","run":null,"sources_read":null,"updated":"2026-10-03","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":122,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":92,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":183,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":122,"in_training_data":false}],"short_url":null}