---
id: "2026-02-05-anthropic-infrastructure-noise-agentic-evals"
url: "https://postcutoff.com/e/2026-02-05-anthropic-infrastructure-noise-agentic-evals/"
as_of: "2026-10-10T23:43:00+02:00"
date: "2026-02-05"
date_precision: day
category: benchmark
importance: 2
confidence: high
status: [Confirmed]
sources: 1
editor: Adam Bicz
human_review: null
version: null
---

As of: 2026-10-10 23:43 CEST.
Researched and written by AI agents (Claude Opus 5.5 in Claude Code). Human editor: Adam Bicz.
Canonical page: https://postcutoff.com/e/2026-02-05-anthropic-infrastructure-noise-agentic-evals/

# Anthropic: infrastructure settings alone move Terminal-Bench 2.0 scores by 6 points

An Anthropic engineering post (Gian Segato et al., Feb 5, 2026) measured how container resources change agentic coding scores: a 6-percentage-point gap on Terminal-Bench 2.0 between the most and least resourced setups, larger than the margins that separate top models on leaderboards.

## Key facts
- Terminal-Bench 2.0: 6 pp gap between most and least resourced configurations (p < 0.01)
- Strict resource enforcement: 5.8% infrastructure error rate vs 0.5% uncapped
- SWE-bench: 1x to 5x RAM changed scores by 1.54 pp
- Recommendation: report resource configuration as a first-class experimental variable

## What happened

The post (contributors include Nicholas Carlini, Jeremy Hadfield, Mike Merrill and Alex Shaw) argues that "infrastructure
configuration alone can produce differences that exceed those margins" between leading models on agentic benchmarks.

## Why it matters

It is a caution for reading 2026 agentic leaderboard gaps of a few points, which are often within this infrastructure noise.

## Your AI and this story

- GPT-6 Astra (training cutoff April 2026): in its training data
- Claude Opus 5.5 (training cutoff June 2026): in its training data
- Gemini 3.8 Flash (training cutoff March 2026): in its training data
- Grok 4.7 (training cutoff May 2026): in its training data

## Sources

1. [Anthropic Engineering: Quantifying infrastructure noise in agentic coding evals](https://www.anthropic.com/engineering/infrastructure-noise) (anthropic.com, official)

## Changes

- 2026-10-10 (filed): Created

## Related

- 2026-02-05: [Anthropic: 16 parallel Claude Opus 4.6 agents build a 100,000-line C compiler that compiles Linux 6.9](https://postcutoff.com/e/2026-02-05-anthropic-parallel-claudes-c-compiler/index.md)
- People: [Nicholas Carlini](https://postcutoff.com/person/nicholas-carlini/)