{"schema":"postcutoff/event@1","as_of":"2026-10-12T15:02:00+02:00","url":"https://postcutoff.com/e/2026-10-08-memento-3-arc-agi-3-public-games-frozen-llm/","md":"https://postcutoff.com/e/2026-10-08-memento-3-arc-agi-3-public-games-frozen-llm/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"Adam Bicz","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-10-08-memento-3-arc-agi-3-public-games-frozen-llm","date":"2026-10-08","date_precision":"day","short_title":"Memento 3 (UCL, Huawei Noah's Ark Lab) clears all 25 public ARC-AGI-3 games with a frozen Claude Opus 5 that writes its own world model","deck":null,"takeaway":null,"category":"research","category_label":"Research","importance":3,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":[{"n":1,"title":"arXiv 2610.11794: Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks","url":"https://arxiv.org/abs/2610.11794","type":"paper","group":"primary","domain":"arxiv.org"},{"n":2,"title":"Pandaily: Huawei Noah's Ark Lab's Memento 3 Agent Clears All 25 ARC-AGI-3 Public Games With a Frozen Model (headline only; page text did not load)","url":"https://pandaily.com/huawei-noahs-ark-lab-memento-3-arc-agi-3-public-set-code-world-model-frozen-claude","type":"press","group":"press","domain":"pandaily.com"},{"n":3,"title":"AI Weekly: Memento 3 Posts 100.0 RHAE on ARC-AGI-3 With Frozen LLM Rulebook","url":"https://aiweekly.co/alerts/memento-3-posts-1000-rhae-on-arc-agi-3-with-frozen-llm-rulebook","type":"press","group":"press","domain":"aiweekly.co"}],"official":1,"filed":"2026-10-12","updated":"2026-10-12","orgs":["University College London","Huawei Noah's Ark Lab","University of Liverpool"],"title":"Memento 3 (UCL, Huawei Noah's Ark Lab) clears all 25 public ARC-AGI-3 games with a frozen Claude Opus 5 that writes its own world model","summary":"A preprint posted on arXiv on Oct 8, 2026 by researchers at University College London, Huawei Noah's Ark Lab (UK) and the University of Liverpool describes Memento 3, an agent that keeps its LLM frozen and learns each game's rules as a natural-language rulebook compiled into executable Python. The authors report that it clears every level of all 25 public ARC-AGI-3 games with a mean RHAE of 100.0, using 44% of the human action count. The result is self-reported and covers the public games only.","key_facts":["Paper: 'Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks', arXiv 2610.11794v1, submitted Oct 8, 2026; authors Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, Jun Wang","Method ('Code as Model'): the agent keeps a rulebook of revisable hypotheses about the environment, compiles it into code for prediction and planning, and revises both from prediction errors; new code is accepted only when the LLM judges it faithful to the rulebook and 'cell-exact replay reproduces the observed transitions'","Setup: harness built from the 'baseline1' codebase, run 'in Claude Code with Claude Opus 5 as the backbone model and reasoning effort set to extra-high'; the agent sees only the action interface, not the environment source","Result: all 25 public games cleared, mean RHAE (Relative Human Action Efficiency) 100.0, 7,518 actions against a human baseline of 17,135 (0.44); per-game ratios from 0.20 to 0.67","Comparison in the paper: 59.3 points higher mean RHAE than Claude Opus 5 with the ARC Prize standard harness (which implies 40.7)","The gain over the best earlier harness is small: 'baseline1' also finishes every game, with 8,347 actions (0.49 of the human count) and a mean RHAE of 99.0 in the paper's table; the other four community-leaderboard systems compared (OPINE-World, NOOA, DreamTeam, Continual Harness) leave 5, 6, 19 and 22 games unfinished (mean RHAE 78.4, 85.1, 38.1 and 20.5)","Ablation on three games: the rulebook cuts actions by 9% (617 vs 677) and agent turns by 18% (678 vs 830); a two-model population on game wa30 cuts actions from 899 to 597","Atari Pong case study: a learned feedback controller wins 21:0 in three evaluated episodes without LLM calls during play","Caveat stated by the authors: 'Public-set completion alone therefore cannot isolate the contribution of a world-model architecture'; comparisons must also account for model choice, reasoning budget and held-out evaluation. No semi-private or ARC Prize-verified score is reported, and the paper gives no cost figure beyond 'substantial time and monetary cost'"],"key_numbers":[],"tags":["agents","arc-agi-3","world-model","memory","continual-learning","recursive-self-improvement","harness","claude-opus-5"],"science":null,"body_md":"## What happened\n\nMemento 3 continues the Memento series of agents that learn through external memory instead of weight updates. Earlier versions\nstored experiences and reusable skills; this one learns the \"model side\", the rules of an unfamiliar environment. The agent explores\na game, writes what it believes about the dynamics into a Markdown rulebook, compiles the rulebook into a Python world model, plans\nwith that model, and repairs both when a prediction fails. The authors present this as a model-based route to recursive\nself-improvement in which \"the underlying LLM remains fixed\".\n\nThe numbers come from the paper (read from the PDF). Every figure for other systems is taken by the authors from those systems'\nentries on the ARC-AGI-3 community leaderboard.\n\n## Why it matters\n\nARC-AGI-3's public games are now solved at the efficiency ceiling by two harnesses built on the same frozen model, while that model\nalone scores about 41 with the standard harness. The difference comes from the scaffolding, which is the same pattern as in other\nrecent harness results. The result does not show performance on held-out games, which is what the benchmark's official scores\nmeasure.","disputed":[],"related":[{"id":"2026-10-01-arc-prize-2026-milestone-2-winners","url":"https://postcutoff.com/e/2026-10-01-arc-prize-2026-milestone-2-winners/","date":"2026-10-01","date_precision":"day","short_title":"Open-source ARC-AGI-3 agents top out at 27.9% at ARC Prize 2026 Milestone #2","deck":null,"takeaway":"The Kaggle track runs open models under Kaggle's compute limits, so it measures how far open methods get without frontier APIs.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":8,"official":8,"filed":"2026-10-04","updated":"2026-10-04","orgs":["ARC Prize Foundation","Tufa Labs"]},{"id":"2026-09-28-harness-learning-test-time-adaptation","url":"https://postcutoff.com/e/2026-09-28-harness-learning-test-time-adaptation/","date":"2026-09-28","date_precision":"day","short_title":"Harness learning: a small model trained with RL learns to rewrite an agent's harness from execution feedback","deck":null,"takeaway":"At test time nothing is retrained: the proposer reads the execution report of each run on a new task and rewrites the code again.","category":"research","category_label":"Research","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":1,"official":1,"filed":"2026-10-12","updated":"2026-10-12","orgs":["Carnegie Mellon University","Johns Hopkins University"]},{"id":"2026-09-03-arc-agi-3-gpt-6-astra","url":"https://postcutoff.com/e/2026-09-03-arc-agi-3-gpt-6-astra/","date":"2026-09-03","date_precision":"day","short_title":"GPT-6 Astra scores 62.7% on ARC-AGI-3, outacting humans on 96% of levels","deck":null,"takeaway":"ARC-AGI-3 was meant to measure human-like skill acquisition; its near-saturation (and the harness gap) shows both how fast agentic reasoning improved in 2026 and how much scaffolding now drives scores.","category":"benchmark","category_label":"Benchmarks","importance":5,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":7,"official":4,"filed":"2026-09-29","updated":"2026-10-04","orgs":["ARC Prize Foundation","OpenAI"]},{"id":"2026-07-24-claude-opus-5","url":"https://postcutoff.com/e/2026-07-24-claude-opus-5/","date":"2026-07-24","date_precision":"day","short_title":"Anthropic releases Claude Opus 5","deck":"Near-Fable-5 intelligence at half the price","takeaway":"Opus 5 brought most of Fable 5's capability to half the price.","category":"model-release","category_label":"Model releases","importance":4,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":8,"official":3,"filed":"2026-09-29","updated":"2026-09-29","orgs":["Anthropic"]},{"id":"2026-03-25-arc-agi-3-launch","url":"https://postcutoff.com/e/2026-03-25-arc-agi-3-launch/","date":"2026-03-25","date_precision":"day","short_title":"ARC Prize launches ARC-AGI-3, an interactive game benchmark where frontier AI scored under 1%","deck":null,"takeaway":"It was designed as the hardest-to-game AGI benchmark of 2026; within six months it was largely cracked (see GPT-6 Astra entry), illustrating the pace of agentic progress.","category":"benchmark","category_label":"Benchmarks","importance":4,"confidence":"medium","status":{"key":"partly","labels":["Partly confirmed"]},"sources":4,"official":3,"filed":"2026-09-29","updated":"2026-09-29","orgs":["ARC Prize Foundation"]}],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-10-12","type":"filed","text":"Created (Pandaily Oct 12)"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-10-12","run":null,"sources_read":null,"updated":"2026-10-12","human_review":null,"version":{"date":"2026-10-12"}},"gaps":[{"model_id":"gpt-6-astra","name":"GPT-6 Astra","cutoff":"2026-04","days_after":161,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":100,"in_training_data":false},{"model_id":"gemini-3-8-flash","name":"Gemini 3.8 Flash","cutoff":"2026-03","days_after":191,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":130,"in_training_data":false}],"short_url":null}