BlogLLM Benchmark Scores as Published, Side by Side
MathVision: Results as Published
Every result the catalog holds for MathVision: 43 models from 9 developers, read from 10 pages. Each number is as printed by the company named beside it; nothing is ranked or rescored. Models are listed by developer, newest first.
Area Multimodal UnderstandingMaintainer Ke Wang (lead author) and co-authors of the MATH-Vision work; project page at mathllm.github.io/mathvisionIntroduced 2024Data version 2026103001, as of 2026-10-30
MathVision tests whether a multimodal model can solve competition mathematics problems that come with visual contexts. The set holds 3,040 problems sourced from real math competitions, spread across 16 mathematical disciplines and graded into 5 difficulty levels. The project page reports accuracy as a percentage, overall and per subject, on the full 3,040-example test set.
MathVision
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.6 SolOpenAI, released 2026-07-09 | 95.8% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: none |
| GPT-5.5OpenAI, released 2026-04-23 | 92.2% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: xhigh; Tools: none |
| GPT-5.4OpenAI, released 2026-03-05 | 92% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Effort: xhigh; Tools: none |
| GPT-5.2OpenAI, released 2025-12-11 | 83% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| GPT-5 miniOpenAI, released 2025-08-07 | 71.9% | Alibaba, Qwen3.5-122B-A10B model cardchecked 2026-10-15 | - |
| Claude Fable 5Anthropic, released 2026-06-09 | 94.8% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: none; Fallback: w/ fallback |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 86.7% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: none |
| Claude Opus 4.6Anthropic, released 2026-02-05 | 65.5% | Alibaba, Qwen3.8-27B model cardchecked 2026-10-15; printed on 2 more pages | Effort: max; Tools: none |
| Claude Opus 4.5Anthropic, released 2025-11-24 | 74.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| Claude Sonnet 4.5Anthropic, released 2025-09-29 | 71.1% | Alibaba, Qwen3.5-122B-A10B model cardchecked 2026-10-15 | - |
| Gemma 4 12BGoogle, released 2026-04-02 | 79.7% | Google, Gemma 4 model cardchecked 2026-10-15 | - |
| Gemma 4 26B A4BGoogle, released 2026-04-02 | 82.4% | Google, Gemma 4 model cardchecked 2026-10-15 | - |
| Gemma 4 31BGoogle, released 2026-04-02 | 85.6% | Google, Gemma 4 model cardchecked 2026-10-15 | - |
| Gemma 4 E2BGoogle, released 2026-04-02 | 52.4% | Google, Gemma 4 model cardchecked 2026-10-15 | - |
| Gemma 4 E4BGoogle, released 2026-04-02 | 59.5% | Google, Gemma 4 model cardchecked 2026-10-15 | - |
| Gemini 3.1 Pro PreviewGoogle, released 2026-02-19 | 89.8% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Effort: high; Tools: none |
| Gemini 3 ProGoogle, released 2025-11-18 | 86.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 1 more page | - |
| Gemma 3 27BGoogle, released 2025-03-12 | 46% | Google, Gemma 4 model cardchecked 2026-10-15 | Effort: no think |
| Qwen3.7-PlusAlibaba, released 2026-06-01 | 90.3% | Alibaba, Qwen3.8-27B model cardchecked 2026-10-15; printed on 1 more page | Tools: none |
| Qwen3.6-27BAlibaba, released 2026-04-22 | 85.1% | Alibaba, Qwen3.8-27B model cardchecked 2026-10-15 | Tools: none |
| Qwen3.5-122B-A10BAlibaba, released 2026-02-24 | 86.2% | Alibaba, Qwen3.5-122B-A10B model cardchecked 2026-10-15; printed on 1 more page | - |
| Qwen3.5-27BAlibaba, released 2026-02-24 | 86% | Alibaba, Qwen3.5-122B-A10B model cardchecked 2026-10-15 | - |
| Qwen3.5-35B-A3BAlibaba, released 2026-02-24 | 83.9% | Alibaba, Qwen3.5-122B-A10B model cardchecked 2026-10-15 | - |
| Qwen3.5-397B-A17BAlibaba, released 2026-02-16 | 88.6% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| Qwen3-VL-235B-A22BAlibaba, released 2025-09-23 | 74.6% | Alibaba, Qwen3.5-122B-A10B model cardchecked 2026-10-15; printed on 1 more page | - |
| Qwen3-VL-235B-A22B-ThinkingAlibaba, released 2025-09-23 | 74.6% | Moonshot AI, Kimi K2.5 model cardchecked 2026-10-15 | Tools: none |
| Qwen3.8-27BAlibaba | 90% | Alibaba, Qwen3.8-27B model cardchecked 2026-10-15; printed on 1 more page | Tools: none |
| Qwen3.8-Flash-NextAlibaba | 90.6% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15; printed on 1 more page | Tools: none |
| Kimi K3Moonshot AI, released 2026-07-16 | 94.3% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: none |
| Kimi K2.6Moonshot AI, released 2026-04-20 | 87.4% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Tools: none |
| Kimi K2.5Moonshot AI, released 2026-01-27 | 84.2% | Moonshot AI, Kimi K2.5 model cardchecked 2026-10-15; printed on 2 more pages | Tools: none |
| GLM-5.3-FlashZ.ai, released 2026-08-26 | 82.5% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
| Nemotron 3.5 Super VL 120B-A12BNVIDIA, released 2026-10-15 | 78.16% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
| Inkling-SmallThinking Machines | 78.8% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
MathVision (with tools)
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.6 SolOpenAI, released 2026-07-09 | 97.8% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: enabled; Python |
| GPT-5.5OpenAI, released 2026-04-23 | 96.8% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: xhigh; Tools: enabled; Python |
| GPT-5.4OpenAI, released 2026-03-05 | 96.1% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Effort: xhigh; Tools: python |
| Claude Fable 5Anthropic, released 2026-06-09 | 98.6% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: enabled; Python; Fallback: w/ fallback |
| Claude Opus 4.8Anthropic, released 2026-05-28 | 97.1% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: enabled; Python |
| Claude Opus 4.6Anthropic, released 2026-02-05 | 84.6% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Effort: max; Tools: python |
| Gemini 3.1 Pro PreviewGoogle, released 2026-02-19 | 95.7% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Effort: high; Tools: python |
| Qwen3.7-PlusAlibaba, released 2026-06-01 | 88.7% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | Tools: enabled; code interpreter |
| Qwen3.8-27BAlibaba | 94.6% | Alibaba, Qwen3.8-27B model cardchecked 2026-10-15; printed on 1 more page | Tools: enabled; code interpreter |
| Qwen3.8-Flash-NextAlibaba | 95.7% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | Tools: enabled; code interpreter |
| Kimi K3Moonshot AI, released 2026-07-16 | 97.8% | Moonshot AI, Kimi K3 model cardchecked 2026-10-15 | Effort: max; Tools: enabled; Python |
| Kimi K2.6Moonshot AI, released 2026-04-20 | 93.2% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Tools: python |
| Kimi K2.5Moonshot AI, released 2026-01-27 | 85% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Tools: python |
MathVision (mini)
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| Gemma 3 12BGoogle, released 2025-03-12 | 31.9% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit |
| Qwen3-VL-32B-InstructAlibaba, released 2025-10-21 | 54.3% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit; Context: 4096 |
| 60.5% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit; Context: 32768 | |
| Qwen3-VL-32B-ThinkingAlibaba, released 2025-10-21 | 60.9% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: thinking; Harness: VLMEvalKit; Context: 4096 |
| 58.6% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: thinking; Harness: VLMEvalKit; Context: 40960 | |
| Qwen3-VL-8B-InstructAlibaba, released 2025-10-15 | 45.7% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit; Context: 4096 |
| 50% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit; Context: 32768 | |
| Qwen3-VL-8B-ThinkingAlibaba, released 2025-10-15 | 43.1% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: thinking; Harness: VLMEvalKit; Context: 4096 |
| 50.7% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: thinking; Harness: VLMEvalKit; Context: 40960 | |
| Kimi-VL-A3B-InstructMoonshot AI, released 2025-04-10 | 28.3% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit |
| Kimi-VL-A3B-ThinkingMoonshot AI, released 2025-04-10 | 50.3% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: thinking; Harness: VLMEvalKit |
| Phi-4-reasoning-vision-15BMicrosoft, released 2026-03-04 | 34.2% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: force nothink; Harness: VLMEvalKit |
| 36.2% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Effort: force thinking; Harness: VLMEvalKit | |
| 36.2% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit | |
| Phi-4-multimodal-instructMicrosoft, released 2025-02-26 | 20% | Microsoft, Microsoft Phi-4-reasoning-vision-15B model cardchecked 2026-10-15 | Harness: VLMEvalKit |
Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.