BlogLLM Benchmark Scores as Published, Side by Side
Qwen3.5-397B-A17B: Benchmark Scores as Published
Every result the catalog holds for Qwen3.5-397B-A17B: 116 tests in 14 areas, from 5 pages. 95 printed by Alibaba itself, the rest by the company named beside each number. Each number is as printed; nothing is ranked or rescored.
Developer AlibabaReleased 2026-02-16Context window 262KInput Text, image, videoOutput TextWeights Open (Apache 2.0)Data version 2026103001, as of 2026-10-30
Knowledge Work
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| Vals Finance Agent v1.1 | 61.3% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | Tools: none |
| 59% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | Tools: web search | |
| GDPval (metric not stated) | 34.6% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
| ProfBench (Search) | 53% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | Tools: web search |
Agentic Coding
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| SWE-bench Verified | 76.4% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 3 more pages | - |
| SWE-Bench Pro | 50.9% | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | Harness: internal scaffold; Context: 200000 |
| Terminal-Bench 2.1 | 49.9% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
| SWE-bench Multilingual | 69.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| Terminal-Bench 2.0 | 52.5% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| NL2Repo-Bench | 32.2 | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | Harness: Claude Code (baselines) |
Science and Math
Long Context
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| AA-LCR | 68.7% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| LongBench v2 | 63.2% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| RULER at 1M | 90.1% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | Context: 1048576 |
Computer Use
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| OSWorld-Verified | 62.2% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| ScreenSpot-Pro | 65.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| AndroidWorld | 66.8% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
Multimodal Understanding
Cybersecurity
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| SecCodeBench | 68.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
General Knowledge and Reasoning
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| Humanity's Last Exam (no tools) | 28.7% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| MMLU-Pro | 87.8% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| Humanity's Last Exam (with tools) | 48.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | Tools: enabled; search; Context: 256000 |
| MMLU-Redux | 94.9% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| C-Eval | 93% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| AA-Omniscience Accuracy | 35.9% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
| SuperGPQA | 70.4% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| AA-Omniscience Non-Hallucination | 7.4% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
| Humanity's Last Exam (Verified) | 37.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
Code Generation
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| LiveCodeBench v6 | 83.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| QwenWebBench rating | 1186 | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | - |
| IOI 2025 (total points) | 441.3 | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
Tool Use and Workflows
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| Toolathlon | 38.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| τ³-bench Banking | 9.8% | Mistral AI, Mistral Medium 3.5 model cardchecked 2026-10-15; printed on 1 more page | Effort: max; Harness: gpt-5.2 user simulator (low), 4 trials; terminal- or embedding-based agentic search retrieval, highest score |
| MCPMark | 46.1% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| τ²-bench (domain not stated) | 86.7% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| Claw-Eval pass^3 | 48.1% | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | - |
| DeepPlanning | 34.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| PinchBench | 86.6% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
| VitaBench | 49.7% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| τ³-bench Airline | 81.5% | Mistral AI, Mistral Medium 3.5 model cardchecked 2026-10-15; printed on 1 more page | Effort: max; Harness: gpt-5.2 user simulator (low), 4 trials; Sierra-reported for Claude Sonnet 4.5 and Qwen3.5 |
| τ³-bench Retail | 84.4% | Mistral AI, Mistral Medium 3.5 model cardchecked 2026-10-15; printed on 1 more page | Effort: max; Harness: gpt-5.2 user simulator (low), 4 trials; Sierra-reported for Claude Sonnet 4.5 and Qwen3.5 |
| τ³-bench Telecom | 97.8% | Mistral AI, Mistral Medium 3.5 model cardchecked 2026-10-15; printed on 1 more page | Effort: max; Harness: gpt-5.2 user simulator (low), 4 trials; Sierra-reported for Claude Sonnet 4.5 and Qwen3.5 |
| BFCL v4 | 72.9% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| SkillsBench | 30 | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | Harness: OpenCode |
| Claw-Eval average score | 70.7 | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | - |
| QwenClawBench | 51.8 | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | Context: 256000 |
| τ³-bench (average) | 71% | NVIDIA, NVIDIA Nemotron 3 Ultra 550B-A55B model cardchecked 2026-10-15 | - |
Web Search and Research
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| BrowseComp | 78.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | Harness: discard-all context strategy; Tools: enabled; search; Context: 256000 |
| 69% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | Harness: simple context-folding, 256K; Tools: enabled; search; Context: 256000 | |
| BrowseComp-ZH | 70.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | Tools: enabled; search; Context: 256000 |
| WideSearch (metric not stated) | 74 | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | Tools: enabled; search; Context: 256000 |
| Seal-0 | 46.9% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | Tools: enabled; search; Context: 256000 |
Instruction Following
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| IFBench | 76.5 | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| MultiChallenge | 67.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| IFEval | 92.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| COLLIE | 88.9% | Mistral AI, Mistral Medium 3.5 model cardchecked 2026-10-15 | Effort: max |
Multilingual
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| MMMLU | 88.5% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| MMLU-ProX | 84.7% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| WMT24++ (XCOMET-XXL) | 78.9 | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| Global PIQA | 89.8% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| INCLUDE | 85.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| MaXIFE | 88.2% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| NOVA-63 | 59.1% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
Healthcare
| Test | Value | Printed by | Setup as printed |
|---|---|---|---|
| MedXpertQA MM | 70% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| PMC-VQA | 64.2% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| SLAKE | 79.9% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
Price
API list prices in US dollars per one million tokens, as printed on the developer's pricing page on the day checked.
| Price | Value | Printed by | Setup as printed |
|---|---|---|---|
| Input | $0.172 | Alibaba, Alibaba Model Studio pricingchecked 2026-10-15 | Context: up to 128,000 tokens; Long context: $0.43 per 1M tokens above 128,000 tokens |
| Output | $1.032 | Alibaba, Alibaba Model Studio pricingchecked 2026-10-15 | Context: up to 128,000 tokens; Long context: $2.58 per 1M tokens above 128,000 tokens |
More from Alibaba
- Qwen3.8-Max
- Qwen3.7-Plus
- Qwen3.6-27B
- Qwen3.6-35B-A3B
- Qwen3.6-Plus
- Qwen3.5-122B-A10B
- Qwen3.5-27B
- Qwen3.5-35B-A3B
- Qwen3-VL-32B-Instruct
- Qwen3-VL-32B-Thinking
- Qwen3-VL-8B-Instruct
- Qwen3-VL-8B-Thinking
- Qwen3-VL-235B-A22B
- Qwen3-VL-235B-A22B-Thinking
- Qwen3-Next-80B-A3B
- Qwen3-30B-A3B-Instruct-2507
- Qwen3-30B-A3B-Thinking-2507
- Qwen3-Coder-Plus
- Qwen3-235B-A22B
- Qwen3-32B
- Qwen3-8B
- QwQ-32B-Preview
- Qwen-Max
- Qwen3-Max-Thinking
- Qwen3.7-Max
- Qwen3.8-2.4T-A95B
- Qwen3.8-27B
- Qwen3.8-Flash-Next
Every number here is read from the same catalog file as the benchmark table, data version 2026103001. Where one page printed the model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.