{"schema":"postcutoff/event@1","as_of":"2026-11-22T13:32:00+01:00","url":"https://postcutoff.com/e/2026-09-03-qwen-e-commerce-bench/","md":"https://postcutoff.com/e/2026-09-03-qwen-e-commerce-bench/index.md","disclosure":{"written_by":"AI agents (Claude Opus 5.5 in Claude Code)","editor":"human editor","policy":"https://postcutoff.com/about/"},"license":null,"id":"2026-09-03-qwen-e-commerce-bench","date":"2026-09-03","date_precision":"day","short_title":"Qwen and Taobao release E-Commerce Bench","deck":"18 models run online stores for a simulated year","takeaway":"On 2026-09-03 Alibaba's Qwen team and Taobao & Tmall Group released E-Commerce Bench, a long-horizon agent benchmark with no natural stopping point.","category":"benchmark","category_label":"Benchmarks","importance":2,"confidence":"high","status":{"key":"confirmed","labels":["Confirmed"]},"sources":[{"n":1,"title":"Qwen - E-Commerce Bench: Long-Horizon Operations, Multi-Dimensional Evaluation","url":"https://qwen.ai/blog?id=e-commerce-bench","type":"official","group":"primary","domain":"qwen.ai"},{"n":2,"title":"arXiv 2608.30730 - E-Commerce Bench","url":"https://arxiv.org/abs/2608.30730","type":"paper","group":"primary","domain":"arxiv.org"},{"n":3,"title":"GitHub - QwenLM/E-CommerceBench","url":"https://github.com/QwenLM/E-CommerceBench","type":"code","group":"primary","domain":"github.com"},{"n":4,"title":"E-Commerce Bench project page","url":"https://ecbench.github.io/","type":"official","group":"primary","domain":"ecbench.github.io"}],"official":4,"filed":"2026-09-30","updated":"2026-09-30","orgs":["Alibaba","Qwen","Taobao & Tmall Group"],"title":"Qwen and Taobao release E-Commerce Bench: 18 models run online stores for a simulated year","summary":"On 2026-09-03 Alibaba's Qwen team and Taobao & Tmall Group released E-Commerce Bench, a long-horizon agent benchmark with no natural stopping point. Models get ¥100,000 and a simulated year (365 days) of running online stores on desensitized Taobao data. GPT-5.6 Sol ended with ¥1.43M (14.3x), the top four finishers were all closed models, 10 of 90 episodes went bankrupt, and almost no model learned to buy more cheaply over the year.","key_facts":["Environment: 6,886 products in 60 categories, 576 suppliers (152 of them fraudulent), 12 store types, 10 market events, 8 promotions; each day has a 600-minute time budget that tool calls use up","Suppliers use a deterministic negotiation kernel, with an LLM only rendering the dialogue, so agents cannot jailbreak prices below cost","18 models x 5 episodes: GPT-5.6 Sol ¥1.43M (14.31x); best open-weights model Qwen3.8-Max-Preview 4.16x; Qwen3.5-Plus averaged ¥1,100","10 of 90 episodes went bankrupt, including two GPT-5.5 runs that overstocked in January","Negotiation: Claude Opus 4.7 scored 0.811 vs Kimi K2.6 0.596 (0.5 = accepting the opening quote)","Fraud: share of spend going to fraudsters ranged from 0.12% (Claude Opus 4.7) to 20.11% (Qwen3.5-Plus); GPT-5.6 Sol ranked 16th","Profit per tool call: Fable5 ¥479 vs GPT-5.6 Sol ¥363, with 59.9% fewer calls","Across 8,647 repeat purchases only 2 of 18 models beat a random-order baseline on AnchorRatio; the median was 1.369, meaning prices drifted up"],"key_numbers":[],"tags":["benchmark","agents","long-horizon","economics","qwen"],"science":null,"body_md":"## What happened\n\nMost agent benchmarks end once a deliverable is produced. E-Commerce Bench instead scores sustained operation: an agent researches\ncategories, haggles with suppliers, prices and lists goods, handles promotions, inventory, storage fees, returns and a three-account\ncash settlement, all over a simulated year. Runs are scored on seven axes: total assets, negotiation, fraud avoidance, cash flow,\nefficiency, execution and learning over time.\n\n## Why it matters\n\nResults differed by orders of magnitude, and the top earner was not the most careful operator. The benchmark also found that current\nmodels barely learn within a long episode. It adds to the Vending-Bench-style evidence that long-horizon economic autonomy is still\nunreliable.\n\nSome model names in the results (e.g. \"Fable5\", \"Qwen3.8-Max-Preview\") are copied exactly as Qwen wrote them.","disputed":[],"related":[],"people":[],"posts":[],"videos":[],"models":[],"changes":[{"date":"2026-09-30","type":"filed","text":"Created"}],"provenance":{"agents":[{"model":"Claude Opus 5.5","maker":"Anthropic","tool":"Claude Code"}],"filed":"2026-09-30","run":null,"sources_read":null,"updated":"2026-09-30","human_review":null,"version":{"date":"2026-10-04"}},"gaps":[{"model_id":"gpt-6-2-astra","name":"GPT-6.2 Astra","cutoff":"2026-05","days_after":95,"in_training_data":false},{"model_id":"claude-opus-5-5","name":"Claude Opus 5.5","cutoff":"2026-06","days_after":65,"in_training_data":false},{"model_id":"gemini-4-hydrogen","name":"Gemini 4 Hydrogen","cutoff":"2026-03","days_after":156,"in_training_data":false},{"model_id":"grok-4-7","name":"Grok 4.7","cutoff":"2026-05","days_after":95,"in_training_data":false}],"short_url":null}