Nvidia Dory 1.0: open 4B looped language model
Confirmed
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 3 of 5
- Last verified
- 17 October 2026
Your AI and this story
- GPT-6 Astra169 days after its cutoff
- Claude Opus 5.5108 days after its cutoff
- Gemini 4 Argon199 days after its cutoff
- Grok 4.7138 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 108 days before it.
Key facts
- Architecture (model card): looped normalized-GPT (nGPT) with interleaved sliding-window and full attention; layer stack = Input block (8 layers), shared Recurrent block (20 layers) applied repeatedly with the input reinjected each loop, Output block (8 layers); 4B parameters; up to 256K context; English
- Elastic inference: the number of loops is set at inference time, with no retraining or model swap
- Last-loop KV caching (recurrent_kv_cache_mode = ‘last_loop’): one cache per recurrent attention layer, overwritten each loop, so KV memory equals a non-looped model at any loop count; Nvidia says accuracy stays close to per-loop caching
- Training: pre-trained natively in NVFP4, then continued in BF16; data cutoffs September 2025 (pre-training) and May 2026 (post-training), per comments in the card
- Nvidia’s own benchmarks, 1 / 2 / 4 loops: SWE-bench Verified 50.3 / 59.5 / 61.8; Terminal-Bench 2.1 12.5 / 18.1 / 22.2; SciCode 27.3 / 34.94 / 35.11; GPQA Diamond 69.19 / 72.41 / 72.16; GDPval-AA-V2 Elo 815 / 946 / 967. Qwen3.5-4B in the same table: SWE-bench Verified 54.7, GPQA Diamond 77.71
- Formats: BF16, MXFP8, NVFP4 and GGUF (Q5_K_M); llama.cpp support pending in PR ggml-org/llama.cpp#30306
- License: OpenMDW License Agreement 1.1
What happened
Nvidia’s “Nemotron Labs” line put out Dory as an experimental small model. Instead of stacking more distinct layers, it reuses one block several times, the recurrent-depth idea that OpenAI’s GPT-6 Astra brought to the frontier (see that entry) and that academic work had explored at small scale. Found on Oct 17 in the full run’s Hugging Face check; no Nvidia blog post or press coverage was found yet.
Why it matters
It is an openly downloadable looped LM from a major lab, with a practical fix for the main cost of looping (KV-cache growth) and a dial for inference compute. The benchmark gains from extra loops are Nvidia’s own and flatten between 2 and 4 loops on several tests.
Sources
3 sources from 2 sites. Numbers match the chips in the text.
3 sources: 3 primary
Primary
- Hugging Face: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 (model card, Oct 16)huggingface.co, code
- Hugging Face: Dory 1.0 4B GGUFhuggingface.co, code
- llama.cpp PR #30306: Dory supportgithub.com, code
Changes
- Filed (Hugging Face live check)