{"schema":"postcutoff/event@1","as_of":"2026-10-30T13:42:00+01:00","url":"https://postcutoff.com/e/2026-10-29-openconjecture-random-conjectures-gpt-models-pnnl/","md":"https://postcutoff.com/e/2026-10-29-openconjecture-random-conjectures-gpt-models-pnnl/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-29-openconjecture-random-conjectures-gpt-models-pnnl","date":"2026-10-29","date_precision":"day","short_title":"OpenConjecture: GPT models on 200 random arXiv conjectures","deck":null,"takeaway":"Reported AI-maths breakthroughs are hand-picked. Researchers at Pacific Northwest National Laboratory sampled 200 conjectures at random from arXiv maths papers posted since February 2026. They ran GPT-5.6 Sol, then GPT-5.6 Sol Pro, then GPT-6 Astra Pro on each, with no problem-specific guidance. The models proposed 70 counterexamples and 55 proofs, and 84% of these passed review by a fresh instance of the same model. No experts checked them. About 60% of the reviewed counterexamples were 'off-the-shelf' and the rest 'standard constructions'. Some exploited missing hypotheses, such as offering the trivial group, and a number of proofs followed closely from known results.","category":"research","category_label":"Research","importance":3,"confidence":"high","status":{"key":"pending","labels":["Event confirmed","Awaiting review"]},"sources":[{"n":1,"title":"arXiv 2610.40382: Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures","url":"https://arxiv.org/abs/2610.40382","type":"paper","group":"primary","domain":"arxiv.org"}],"official":1,"filed":"2026-10-30","updated":"2026-10-30","orgs":["Pacific Northwest National Laboratory","OpenAI"],"title":"OpenConjecture (PNNL): GPT models propose resolutions for 125 of 200 randomly sampled arXiv conjectures, but many counterexamples are 'off-the-shelf' or exploit gaps in the statement","summary":"arXiv 2610.40382 (29 Oct 2026), \"Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures\" by Helen Jenne, Davis Brown, Jesse He and Henry Kvinge (PNNL, UC San Diego, University of Pennsylvania, University of Washington), introduces OpenConjecture, a dataset of conjectures from arXiv maths papers since February 2026. On a random sample of 200, three solver passes produced candidate resolutions for 125. The paper analyses the novelty and fidelity of these candidates rather than their correctness, and warns that the counts are not success rates.","key_facts":["Protocol (paper): up to three sequential passes, 'GPT-5.6 Sol in the first pass, GPT-5.6 Sol Pro in the second, and GPT-6 Astra Pro in the third', reasoning effort xhigh; only conjectures judged 'unresolved' went to the next pass; no problem-specific mathematical guidance","Counts (paper): 'the three passes produced 70 counterexamples and 55 proofs to the 200 conjectures'; counterexamples 49/15/6 and proofs 33/11/11 by pass; '84% of these proposed resolutions hold up to LLM review'","Quality (paper): 'of the proposed counterexamples from the first and second attempts that passed an LLM verification review, roughly 60% were classified as \"off-the-shelf\" and the remainder as \"standard constructions\"'; 'Some of the counterexamples appeared to take advantage of missing hypotheses in the conjecture statement'; the abstract cites 'proposing the trivial group as a counterexample' and says 'a number of proofs follow closely from existing results, but that some resolutions exhibit at least moderate novelty'","Limitation (paper, verbatim): 'Our results discuss candidate resolutions and outcomes of LLM review, not expert-verified solutions. Each proposed resolution was reviewed by a new instance of the same model that produced it … The reported numbers … should not be interpreted as rates' of correct solutions","Framing (introduction): cites the Erdős unit-distance disproof, the Jacobian conjecture counterexample, OpenAI's Navier–Stokes blowup claim, First Proof and the Leipzig benchmarks as the selected breakthroughs it wants to complement"],"key_numbers":[],"tags":["math","evaluation","benchmark","conjectures","gpt-5-6-sol","gpt-6-astra","ai-for-math","methodology"],"science":{"field":"mathematics","subfield":"AI evaluation on open research conjectures","problem":"How often do frontier models resolve randomly chosen open conjectures, and what do their resolutions contribute?","result":"Candidate resolutions for 125 of 200 sampled conjectures (70 counterexamples, 55 proofs), 84% passing same-model review; many trivial, off-the-shelf or exploiting underspecified statements","open_since":"","ai_system":["GPT-5.6 Sol","GPT-5.6 Sol Pro","GPT-6 Astra Pro"],"human_role":"Researchers built the dataset and protocol and classified the outputs; no expert verification of the candidates","verification":{"key":"preprint","label":"Preprint only","text":"Preprint; candidates checked only by LLM review"},"status":"pending","shock":"","kind":"other"},"body_md":"## What happened\n\nMost AI-maths news concerns problems that labs or mathematicians chose because a model succeeded. The PNNL group instead collected\nconjectures stated in recent arXiv papers and drew 200 at random. Three passes of OpenAI models produced claimed answers for 125 of them.\n\nThe authors did not check correctness. They studied what kind of answers came back: how long they were, whether they cited prior work,\nwhether they matched what the conjecture's authors meant, and how novel they looked.\n\n## Why it matters\n\nIt gives a baseline for the flood of AI-assisted conjecture papers we list daily. Many conjectures in recent papers fall to standard\ntools or to loopholes in how they are stated, so a count of \"conjectures resolved\" says little without that classification.","disputed":[{"label":"Verification","text":"Preprint; candidates checked only by LLM review","sources":[]}],"related":[{"id":"2026-10-30-ai-assisted-results-october-30-listing","url":"https://postcutoff.com/e/2026-10-30-ai-assisted-results-october-30-listing/","date":"2026-10-30","date_precision":"day","short_title":"AI-assisted results, 30 Oct: arXiv announcement of 29–30 Oct","deck":null,"takeaway":"The 30 Oct sweep listed 374 arXiv papers that mention AI. We skimmed all their AI statements and read about 35 PDFs. Six results got their own entries: Das–Fox's Murty–Simon proof (GPT-5.6 Sol found the bound), Colbrook–Stepaniants's spectral isometry proof (found by GPT-6 Astra, Lean), the Θ(log k log log k) ratio of greedy for online Steiner forest (GPT Astra, Lean), Gáspár's Laczkovich solution (key proposition by ChatGPT), Ren's pure-semisimplicity counterexample (GPT-6 Astra, Lean) and Petty's conjecture in dimension 5 (Tencent's hyprover agent). PNNL's OpenConjecture study also has one. This listing has the rest, among them a single-author ChatGPT 6 Pro proof of the Alon–Seymour–Thomas separator conjecture, b₄ ≤ 7 for Borsuk's problem and Fiedorowicz's conjecture from a Claude idea.","category":"science","category_label":"Science & math","importance":2,"confidence":"medium","status":{"key":"pending","labels":["Awaiting review"]},"sources":42,"official":42,"filed":"2026-10-30","updated":"2026-10-30","orgs":["OpenAI","Anthropic","Google","Tencent","various researchers"]},{"id":"2026-10-06-openai-math-release-722-manuscripts","url":"https://postcutoff.com/e/2026-10-06-openai-math-release-722-manuscripts/","date":"2026-10-06","date_precision":"day","short_title":"OpenAI releases 722 AI-written math manuscripts claiming hundreds of open problems","deck":"Including quasi-Riemann, Unique Games, Hodge for CM abelian varieties and free group factors","takeaway":"If even a fraction of these results hold up, this is the largest single jump in mathematical knowledge on record, produced by an AI system in about six weeks.","category":"science","category_label":"Science & math","importance":5,"confidence":"high","status":{"key":"pending","labels":["Event confirmed","Awaiting review"]},"sources":79,"official":13,"filed":"2026-10-07","updated":"2026-10-29","orgs":["OpenAI"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-30","type":"filed","text":"Created (science agent; arXiv abstract and PDF read, numbers quoted from the paper)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-30","run":null,"sources_read":"Science agent; arXiv abstract and PDF read, numbers quoted from the paper","updated":"2026-10-30","human_review":null,"version":{"date":"2026-10-30"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":182,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":121,"in_training_data":false},{"model_id":"gemini-4-argon","name":"Gemini 4 Argon","cutoff":"2026-03","days_after":212,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":151,"in_training_data":false}],"short_url":null}