Meta details Prometheus, the Ohio gigawatt supercluster that trained Muse Wonder
Aiming at ~1M GPUs, 85%+ training goodput, 99% fewer interruptions
Confirmed
The takeaway
On Oct 19, 2026 Meta Engineering published a technical account of Prometheus in New Albany, Ohio, the supercluster whose first completed phase trained the Muse models including Muse Wonder.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 3 of 5
- Last verified
- 19 October 2026
Your AI and this story
- GPT-6 Astra172 days after its cutoff
- Claude Opus 5.5111 days after its cutoff
- Gemini 4 Argon202 days after its cutoff
- Grok 4.7141 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 111 days before it.
Key facts
- Scale target: ~1M GPUs working together; built in phases; two years earlier the site was a conventional web-serving data center; part of the capacity is leased from partners; ‘rapid deployment structures’ (industrial-grade buildings) used alongside conventional halls
- Racks: Catalina (GB200 NVL72, ~120 kW per two-rack pair), Clemente (GB300 NVL72, 288 GB HBM3e per GPU, ~135 kW per rack, 5–10% faster than GB200) and upcoming San Miguel (Vera Rubin NVL72, 288 GB HBM4 per GPU, 2.52 EFLOPS FP4 per rack, 170+ kW per rack)
- Network: Ethernet back-end, up to 20,736 GPUs per building fabric; Backend Aggregation links building fabrics 2 km to 100+ km apart (Arista 7800 / Broadcom Jericho3); 800G-ZR coherent optics beyond ~10 km; 600,000+ NVLink connections
- Reliability: training goodput above 85% for Muse Wonder pre-training and above 90% for RL runs; mean time between interruptions ~47K GPU-hours, raised to 54K+ with firmware fixes; interruptions peaked at nearly 1,000 per 8K GPUs per day in mid-2025, now down 99%; target ~2 per 8K GPUs per day
- Spares: each NVL72 rack runs a 64-GPU slice with 8 in-rack spares, plus ~2% spare racks; a 32,768-GPU run needs 523 racks (37,656 GPUs)
- Power: each GPU draws ~1,400 W under load; synchronized training swings load 30–40%, starting or pausing a job can move tens of megawatts in seconds; Meta smooths this with ‘Dynamo Dimmer’ power capping, rack batteries, capacitor trays and facility battery storage
- Next: Hyperion in Richland Parish, Louisiana, with closed-loop liquid cooling and a grid connection through Entergy that Meta says it pays for in full
- Authors: Kevin Lee, Yashar Bayani, Ahmed Sharif, Yee Jiun Song
What happened
Meta published the most detailed public description yet of a frontier training cluster: rack generations, network topology across distant buildings, failure rates and power handling. The post confirms that Prometheus’s first phase is complete and trained the Muse family, and gives Meta’s own goodput and reliability figures for the Muse Wonder run.
Why it matters
Frontier labs rarely publish failure and goodput numbers for their largest runs. The figures show what a gigawatt-class cluster takes in practice: hundreds of failures a day at 2025 peaks, spare GPUs built into every rack, and power swings of tens of megawatts that the grid has to absorb.
Sources
1 source from 1 site. Numbers match the chips in the text.
1 source: 1 primary
Primary
- Meta Engineering: Inside Prometheus, Meta’s gigawatt-scale supercluster (Oct 19)engineering.fb.com, official
Changes
- Filed (Techmeme)