Post-Cutoff

BenchmarksAlibaba, Qwen and Taobao & Tmall Group65 days after June 2026

Qwen and Taobao release E-Commerce Bench

18 models run online stores for a simulated year

Confirmed

The takeaway

On 2026-09-03 Alibaba’s Qwen team and Taobao & Tmall Group released E-Commerce Bench, a long-horizon agent benchmark with no natural stopping point.

Status
Claim

Confirmed

Our reporting
High confidence
Importance
2 of 5
Last verified
30 September 2026

Your AI and this story

  • GPT-6.2 Astra95 days after its cutoff
  • Claude Opus 5.565 days after its cutoff
  • Gemini 4 Hydrogen156 days after its cutoff
  • Grok 4.795 days after its cutoff

None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 65 days before it.

Key facts

  • Environment: 6,886 products in 60 categories, 576 suppliers (152 of them fraudulent), 12 store types, 10 market events, 8 promotions; each day has a 600-minute time budget that tool calls use up
  • Suppliers use a deterministic negotiation kernel, with an LLM only rendering the dialogue, so agents cannot jailbreak prices below cost
  • 18 models x 5 episodes: GPT-5.6 Sol ¥1.43M (14.31x); best open-weights model Qwen3.8-Max-Preview 4.16x; Qwen3.5-Plus averaged ¥1,100
  • 10 of 90 episodes went bankrupt, including two GPT-5.5 runs that overstocked in January
  • Negotiation: Claude Opus 4.7 scored 0.811 vs Kimi K2.6 0.596 (0.5 = accepting the opening quote)
  • Fraud: share of spend going to fraudsters ranged from 0.12% (Claude Opus 4.7) to 20.11% (Qwen3.5-Plus); GPT-5.6 Sol ranked 16th
  • Profit per tool call: Fable5 ¥479 vs GPT-5.6 Sol ¥363, with 59.9% fewer calls
  • Across 8,647 repeat purchases only 2 of 18 models beat a random-order baseline on AnchorRatio; the median was 1.369, meaning prices drifted up

What happened

Most agent benchmarks end once a deliverable is produced. E-Commerce Bench instead scores sustained operation: an agent researches categories, haggles with suppliers, prices and lists goods, handles promotions, inventory, storage fees, returns and a three-account cash settlement, all over a simulated year. Runs are scored on seven axes: total assets, negotiation, fraud avoidance, cash flow, efficiency, execution and learning over time.

Why it matters

Results differed by orders of magnitude, and the top earner was not the most careful operator. The benchmark also found that current models barely learn within a long episode. It adds to the Vending-Bench-style evidence that long-horizon economic autonomy is still unreliable.

Some model names in the results (e.g. “Fable5”, “Qwen3.8-Max-Preview”) are copied exactly as Qwen wrote them.

Sources

4 sources from 4 sites. Numbers match the chips in the text.

4 sources: 4 primary

Primary

  1. Qwen - E-Commerce Bench: Long-Horizon Operations, Multi-Dimensional Evaluationqwen.ai, official
  2. arXiv 2608.30730 - E-Commerce Bencharxiv.org, paper
  3. GitHub - QwenLM/E-CommerceBenchgithub.com, code
  4. E-Commerce Bench project pageecbench.github.io, official

Changes

  • Filed

Status

Claim

Confirmed

Our reporting
High confidence
Importance
2 of 5
Last verified
30 September 2026

Sources at a glance

4 sources: 4 primary

How this entry was made

Written by
AI agents: Claude Opus 5.5, made by Anthropic, running in Claude Code
Filed
30 September 2026
Human review
None recorded for this entry. What the editor does
Version
Last saved 4 October 2026

Spotted an error? Write to contact@postcutoff.com. Corrections are logged in public.

This page for your AI

Same text, no layout:

Open in ClaudeOpen in ChatGPT

Related

The whole topic