BlogLLM Benchmark Scores as Published, Side by Side
Toolathlon: Results as Published
Every result the catalog holds for Toolathlon: 65 models from 12 developers, read from 27 pages. Each number is as printed by the company named beside it; nothing is ranked or rescored. Models are listed by developer, newest first.
Area Tool Use and WorkflowsMaintainer HKUST NLP groupIntroduced 2025Data version 2026103001, as of 2026-10-30
Toolathlon (The Tool Decathlon) tests whether language agents can finish long, realistic workflows across many software applications. It covers 32 applications and 604 tools, from calendars and notes to databases and cloud consoles, with 108 tasks that take about 20 turns on average. Each task has its own automatic check of the final state, and the environments run in isolated containers.
Toolathlon
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.6 LunaOpenAI, released 2026-07-09 | 53.4% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15 | - |
| GPT-5.6 SolOpenAI, released 2026-07-09 | 58% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15 | - |
| GPT-5.6 TerraOpenAI, released 2026-07-09 | 53.1% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15 | - |
| GPT-5.5OpenAI, released 2026-04-23 | 55.6% | OpenAI, Introducing GPT-5.5checked 2026-10-15; printed on 3 more pages | Effort: xhigh |
| GPT-5.4 miniOpenAI, released 2026-03-17 | 42.9% | OpenAI, Introducing GPT-5.4 mini and nanochecked 2026-10-15 | Effort: xhigh |
| GPT-5.4 nanoOpenAI, released 2026-03-17 | 35.5% | OpenAI, Introducing GPT-5.4 mini and nanochecked 2026-10-15 | Effort: xhigh |
| GPT-5.4OpenAI, released 2026-03-05 | 54.6% | OpenAI, Introducing GPT-5.4checked 2026-10-15; printed on 6 more pages | Effort: xhigh |
| GPT-5.3-CodexOpenAI, released 2026-02-05 | 51.9% | OpenAI, Introducing GPT-5.4checked 2026-10-15 | Effort: xhigh |
| GPT-5.2OpenAI, released 2025-12-11 | 46.3% | OpenAI, Introducing GPT-5.2checked 2026-10-15; printed on 3 more pages | Effort: xhigh |
| GPT-5.1OpenAI, released 2025-11-12 | 36.1% | OpenAI, Introducing GPT-5.2checked 2026-10-15 | Effort: high |
| GPT-5OpenAI, released 2025-08-07 | 29% | DeepSeek, DeepSeek-V3.2 model cardchecked 2026-10-15 | Effort: high |
| GPT-5 miniOpenAI, released 2025-08-07 | 26.9% | OpenAI, Introducing GPT-5.4 mini and nanochecked 2026-10-15 | Effort: high |
| Claude Fable 5Anthropic, released 2026-06-09 | 61.7% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15 | - |
| Claude Mythos 5Anthropic, released 2026-06-09 | 61.7% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15 | - |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 59.9% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15; printed on 1 more page | - |
| Claude Mythos PreviewAnthropic, released 2026-04-07 | 61.1% | OpenAI, GPT-5.6: Frontier intelligence that scales with your ambitionchecked 2026-10-15 | - |
| Claude Sonnet 4.6Anthropic, released 2026-02-17 | 44.8% | MiniMax, MiniMax-M2.7 model cardchecked 2026-10-15 | - |
| Claude Opus 4.6Anthropic, released 2026-02-05 | 47.2% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15; printed on 3 more pages | Effort: max |
| Claude Opus 4.5Anthropic, released 2025-11-24 | 43.5% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| Claude Sonnet 4.5Anthropic, released 2025-09-29 | 38.6% | DeepSeek, DeepSeek-V3.2 model cardchecked 2026-10-15 | - |
| Gemini 3.5 FlashGoogle, released 2026-05-19 | 56.5% | Google, Gemini 3.5 Flash model cardchecked 2026-10-15 | - |
| Gemma 4 26B A4BGoogle, released 2026-04-02 | 12% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15 | - |
| Gemma 4 31BGoogle, released 2026-04-02 | 21.2% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15 | - |
| Gemini 3.1 Pro PreviewGoogle, released 2026-02-19 | 48.8% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15; printed on 6 more pages | Effort: high |
| Gemini 3 FlashGoogle, released 2025-12-17 | 49.4% | Google, Gemini 3.5 Flash model cardchecked 2026-10-15 | - |
| Gemini 3 ProGoogle, released 2025-11-18 | 36.4% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| DeepSeek-V4-Flash (Preview)DeepSeek, released 2026-04-24 | 47.8% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: max |
| 40.7% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: non-think | |
| 43.5% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: high | |
| DeepSeek-V4-Pro (Preview)DeepSeek, released 2026-04-24 | 51.8% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15; printed on 1 more page | Effort: max |
| 49% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: high | |
| 46.3% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: non-think | |
| DeepSeek-V3.2DeepSeek, released 2025-12-01 | 35.2% | DeepSeek, DeepSeek-V3.2 model cardchecked 2026-10-15; printed on 2 more pages | Effort: thinking |
| Qwen3.6-35B-A3BAlibaba, released 2026-04-16 | 26.9% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15 | - |
| Qwen3.6-PlusAlibaba, released 2026-04-02 | 39.8% | Z.ai, GLM-5.1 model cardchecked 2026-10-15 | - |
| Qwen3.5-27BAlibaba, released 2026-02-24 | 31.5% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15 | - |
| Qwen3.5-35B-A3BAlibaba, released 2026-02-24 | 28.7% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15 | - |
| Qwen3.5-397B-A17BAlibaba, released 2026-02-16 | 38.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| Qwen3-Max-ThinkingAlibaba | 18.8% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| Kimi K2.6Moonshot AI, released 2026-04-20 | 50% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15; printed on 1 more page | - |
| Kimi K2.5Moonshot AI, released 2026-01-27 | 27.8% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15; printed on 3 more pages | - |
| GLM-5.2Z.ai, released 2026-06-16 | 48.2% | Z.ai, GLM-5.2 model cardchecked 2026-10-15 | - |
| GLM-5.1Z.ai, released 2026-04-07 | 40.7% | Z.ai, GLM-5.1 model cardchecked 2026-10-15; printed on 2 more pages | - |
| GLM-5Z.ai, released 2026-02-12 | 38% | Z.ai, GLM-5 model cardchecked 2026-10-15; printed on 1 more page | - |
| GLM-4.7Z.ai, released 2025-12-22 | 23.8% | Z.ai, GLM-5 model cardchecked 2026-10-15 | - |
| MiniMax M2.7MiniMax, released 2026-03-18 | 46.3% | MiniMax, MiniMax-M2.7 model cardchecked 2026-10-15; printed on 1 more page | - |
| MiniMax M2.5MiniMax, released 2026-02-12 | 38.3% | MiniMax, MiniMax-M2.7 model cardchecked 2026-10-15 | - |
Toolathlon Verified
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.6 LunaOpenAI, released 2026-07-09 | 67.5% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15 | Effort: max |
| GPT-5.6 SolOpenAI, released 2026-07-09 | 74.9% | Alibaba, Qwen3.8-Max comparisonchecked 2026-10-14; printed on 3 more pages | Effort: max |
| GPT-5.6 TerraOpenAI, released 2026-07-09 | 64.8% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15 | Effort: high |
| GPT-5.5OpenAI, released 2026-04-23 | 73.5% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: xhigh |
| Claude Sonnet 5.5Anthropic, released 2026-09-28 | 77.8% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19 | Effort: max; Harness: Internal harness mirroring Toolathlon-Verified definitions |
| Claude Opus 5.5Anthropic, released 2026-09-22 | 77.8% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Fable 5.1Anthropic, released 2026-09-01 | 77.8% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Opus 5Anthropic, released 2026-07-24 | 80.6% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 3 more pages | Effort: max |
| Claude Sonnet 5Anthropic, released 2026-06-30 | 74.7% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19; printed on 3 more pages | Effort: max; Harness: Internal harness mirroring Toolathlon-Verified definitions |
| Claude Fable 5Anthropic, released 2026-06-09 | 77.9% | Alibaba, Qwen3.8-Max comparisonchecked 2026-10-14; printed on 4 more pages | - |
| Claude Mythos 5Anthropic, released 2026-06-09 | 79.3% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 79.9% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 7 more pages | Effort: max |
| Muse Spark OpenMeta, released 2026-10-15 | 72.8% | Meta, Meta Muse Spark Open 816B-A42B model cardchecked 2026-10-19 | Effort: xhigh |
| DeepSeek-V4-Flash (Preview)DeepSeek, released 2026-04-24 | 49.7% | DeepSeek, DeepSeek-V4-Flash-0731 model cardchecked 2026-10-15; printed on 1 more page | - |
| DeepSeek-V4-Pro (Preview)DeepSeek, released 2026-04-24 | 55.9% | DeepSeek, DeepSeek-V4-Flash-0731 model cardchecked 2026-10-15; printed on 1 more page | - |
| DeepSeek-V4-Flash-0731DeepSeek | 70.3% | DeepSeek, DeepSeek-V4-Flash-0731 model cardchecked 2026-10-15; printed on 2 more pages | - |
| DeepSeek-V4-Pro-0813DeepSeek | 74.1% | DeepSeek, DeepSeek-V4-Pro-0813 model cardchecked 2026-10-15; printed on 1 more page | - |
| Qwen3.8-MaxAlibaba, released 2026-08-03 | 72.5% | Alibaba, Qwen3.8-Max comparisonchecked 2026-10-14; printed on 1 more page | - |
| Qwen3.7-PlusAlibaba, released 2026-06-01 | 50.6% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | - |
| Qwen3.7-MaxAlibaba | 49.7% | Alibaba, Qwen3.8-Max comparisonchecked 2026-10-14 | - |
| Qwen3.8-2.4T-A95BAlibaba | 72.5% | Meta, Meta Muse Spark Open 816B-A42B model cardchecked 2026-10-19 | Effort: xhigh |
| Qwen3.8-27BAlibaba | 67.1% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | - |
| Qwen3.8-Flash-NextAlibaba | 73.5% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | - |
| Kimi K3Moonshot AI, released 2026-07-16 | 76.5% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15; printed on 3 more pages | Effort: max |
| GLM-5.3Z.ai, released 2026-08-18 | 73% | Z.ai, GLM-5.3 model cardchecked 2026-10-15; printed on 1 more page | Effort: max |
| GLM-5.2Z.ai, released 2026-06-16 | 59.9% | Z.ai, GLM-5.3 model cardchecked 2026-10-15; printed on 4 more pages | - |
| MiniMax M3MiniMax, released 2026-06-01 | 53.7% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15 | - |
| Nemotron 3 UltraNVIDIA, released 2026-06-04 | 34.3% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15; printed on 1 more page | - |
| MiMo-V2.6-FlashXiaomi, released 2026-09-21 | 73.6% | Xiaomi, Xiaomi MiMo-V2.6-Pro-RL model cardchecked 2026-10-19 | - |
| MiMo-V2.6-ProXiaomi, released 2026-09-21 | 76.9% | Xiaomi, Xiaomi MiMo-V2.6-Pro-RL model cardchecked 2026-10-19 | - |
| MiMo-V2.5-ProXiaomi, released 2026-04-27 | 49.1% | Xiaomi, Xiaomi MiMo-V2.6-Pro-RL model cardchecked 2026-10-19 | - |
| InklingThinking Machines, released 2026-07-14 | 45.5% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15; printed on 1 more page | Effort: xhigh |
Toolathlon Verified Pass@3
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| Claude Sonnet 5.5Anthropic, released 2026-09-28 | 85.2% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19 | Effort: max; Harness: Internal harness mirroring Toolathlon-Verified definitions |
| Claude Opus 5.5Anthropic, released 2026-09-22 | 82.4% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Fable 5.1Anthropic, released 2026-09-01 | 81.5% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Opus 5Anthropic, released 2026-07-24 | 87% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Sonnet 5Anthropic, released 2026-06-30 | 84.3% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19; printed on 2 more pages | Effort: max; Harness: Internal harness mirroring Toolathlon-Verified definitions |
| Claude Mythos 5Anthropic, released 2026-06-09 | 86.1% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 88% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
Toolathlon Verified Pass^3
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| Claude Sonnet 5.5Anthropic, released 2026-09-28 | 68.5% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19 | Effort: max; Harness: Internal harness mirroring Toolathlon-Verified definitions |
| Claude Opus 5.5Anthropic, released 2026-09-22 | 72.2% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Fable 5.1Anthropic, released 2026-09-01 | 73.1% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Opus 5Anthropic, released 2026-07-24 | 73.1% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Sonnet 5Anthropic, released 2026-06-30 | 65.7% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19; printed on 2 more pages | Effort: max; Harness: Internal harness mirroring Toolathlon-Verified definitions |
| Claude Mythos 5Anthropic, released 2026-06-09 | 73.1% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 71.3% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 2 more pages | Effort: max |
Toolathlon Verified average assistant turns
lower is better
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| Claude Opus 5.5Anthropic, released 2026-09-22 | 26.9 | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25 | Effort: max |
| Claude Fable 5.1Anthropic, released 2026-09-01 | 23.7 | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Opus 5Anthropic, released 2026-07-24 | 23.5 | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Sonnet 5Anthropic, released 2026-06-30 | 24.5 | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Mythos 5Anthropic, released 2026-06-09 | 19.8 | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 20.4 | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 1 more page | Effort: max |
Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.