BlogLLM Benchmark Scores as Published, Side by Side
SWE-bench Multilingual: Results as Published
Every result the catalog holds for SWE-bench Multilingual: 56 models from 10 developers, read from 21 pages. Each number is as printed by the company named beside it; nothing is ranked or rescored. Models are listed by developer, newest first.
Area Agentic CodingMaintainer SWE-bench team (Kabir Khandpur, Kilian Lieret, Carlos E. Jimenez, Ofir Press, John Yang)Introduced 2025Data version 2026103001, as of 2026-10-30
SWE-bench Multilingual tests agentic software engineering outside Python: 300 issue-fixing tasks from 42 repositories in nine languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust). A task is solved when the agent's patch makes the failing tests pass without breaking the passing ones; the score is the share of tasks resolved.
SWE-bench Multilingual
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.2OpenAI, released 2025-12-11 | 72% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| GPT-5OpenAI, released 2025-08-07 | 55.3% | Moonshot AI, Kimi K2 Thinking release pagechecked 2026-10-15; printed on 1 more page | Effort: high; Harness: in-house harness derived from SWE-agent; Tools: enabled; bash and edit tools |
| gpt-oss-20bOpenAI, released 2025-08-05 | 41.93% | NVIDIA, NVIDIA Nemotron 3.5 Lightning 30B-A3B model cardchecked 2026-10-19 | - |
| GPT-4.1OpenAI, released 2025-04-14 | 31.5% | Moonshot AI, Kimi K2 release pagechecked 2026-10-15 | Harness: agentic, single attempt |
| Claude Haiku 5.5Anthropic, released 2026-10-07 | 83.7% | Anthropic, Claude Haiku 5.5 System Cardchecked 2026-10-19 | - |
| Claude Sonnet 5.5Anthropic, released 2026-09-28 | 90.3% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19; printed on 1 more page | Effort: max |
| Claude Opus 5.5Anthropic, released 2026-09-22 | 93.9% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Fable 5.1Anthropic, released 2026-09-01 | 89.1% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Opus 5Anthropic, released 2026-07-24 | 89.5% | Anthropic, Claude Opus 5.5 System Cardchecked 2026-10-25; printed on 1 more page | Effort: max |
| Claude Sonnet 5Anthropic, released 2026-06-30 | 78.3% | Anthropic, Claude Sonnet 5.5 System Cardchecked 2026-10-19 | - |
| Claude Fable 5Anthropic, released 2026-06-09 | 86.6% | Anthropic, System Card: Claude Fable 5.1 & Claude Mythos 5.1checked 2026-10-25 | Effort: max |
| Claude Opus 4.6Anthropic, released 2026-02-05 | 77.5% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15; printed on 2 more pages | Effort: max; Harness: mini-SWE-agent; Context: 256000 |
| Claude Opus 4.5Anthropic, released 2025-11-24 | 77.5% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 3 more pages | - |
| Claude Haiku 4.5Anthropic, released 2025-10-15 | 67.4% | Anthropic, Claude Haiku 5.5 System Cardchecked 2026-10-19 | - |
| Claude Sonnet 4.5Anthropic, released 2025-09-29 | 68% | Moonshot AI, Kimi K2 Thinking release pagechecked 2026-10-15; printed on 1 more page | Effort: thinking; Harness: in-house harness derived from SWE-agent; Tools: enabled; bash and edit tools |
| Claude Sonnet 4Anthropic, released 2025-05-22 | 51% | Moonshot AI, Kimi K2 release pagechecked 2026-10-15 | Effort: no extended thinking; Harness: agentic, single attempt |
| Gemma 4 26B A4BGoogle, released 2026-04-02 | 17.3% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15; printed on 1 more page | Harness: internal scaffold; Context: 200000 |
| Gemma 4 31BGoogle, released 2026-04-02 | 51.7% | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15; printed on 1 more page | Harness: internal scaffold; Context: 200000 |
| Gemini 3.1 Pro PreviewGoogle, released 2026-02-19 | 76.9% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15 | Effort: high; Harness: in-house SWE-agent variant |
| Gemini 3 ProGoogle, released 2025-11-18 | 65% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| DeepSeek-V4-Flash (Preview)DeepSeek, released 2026-04-24 | 73.3% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15; printed on 1 more page | Effort: max |
| 70.2% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: high | |
| 69.7% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: non-think | |
| DeepSeek-V4-Pro (Preview)DeepSeek, released 2026-04-24 | 69.8% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15; printed on 1 more page | Effort: non-think |
| 76.2% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: max | |
| 74.1% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15 | Effort: high | |
| DeepSeek-V3.2DeepSeek, released 2025-12-01 | 70.2% | Moonshot AI, Kimi K2.5 model cardchecked 2026-10-15; printed on 2 more pages | Harness: in-house SWE-agent variant |
| DeepSeek-V3.2-ExpDeepSeek, released 2025-09-29 | 57.9% | DeepSeek, DeepSeek-V3.2-Exp model cardchecked 2026-10-15; printed on 1 more page | Tools: enabled; exact tools in source |
| DeepSeek-V3.1-TerminusDeepSeek, released 2025-09-22 | 57.8% | DeepSeek, DeepSeek-V3.1-Terminus model cardchecked 2026-10-15; printed on 1 more page | Tools: enabled; exact tools in source |
| DeepSeek-V3.1DeepSeek, released 2025-08-21 | 54.5% | DeepSeek, DeepSeek-V3.1-Terminus model cardchecked 2026-10-15 | Tools: enabled; exact tools in source |
| DeepSeek-V3-0324DeepSeek, released 2025-03-24 | 25.8% | Moonshot AI, Kimi K2 release pagechecked 2026-10-15 | Harness: agentic, single attempt |
| DeepSeek-V4-Flash-0731DeepSeek | 72% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
| Qwen3.7-PlusAlibaba, released 2026-06-01 | 75.8% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | Harness: mini-SWE-agent; Context: 256000 |
| Qwen3.6-27BAlibaba, released 2026-04-22 | 71.3% | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15 | Harness: internal scaffold; Context: 200000 |
| Qwen3.6-35B-A3BAlibaba, released 2026-04-16 | 67.2% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15; printed on 2 more pages | Harness: internal scaffold; Context: 200000 |
| Qwen3.5-122B-A10BAlibaba, released 2026-02-24 | 62% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
| Qwen3.5-27BAlibaba, released 2026-02-24 | 69.3% | Alibaba, Qwen3.6-27B model cardchecked 2026-10-15; printed on 1 more page | Harness: internal scaffold; Context: 200000 |
| Qwen3.5-35B-A3BAlibaba, released 2026-02-24 | 60.3% | Alibaba, Qwen3.6-35B-A3B model cardchecked 2026-10-15 | Harness: internal scaffold; Context: 200000 |
| Qwen3.5-397B-A17BAlibaba, released 2026-02-16 | 69.3% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15; printed on 2 more pages | - |
| Qwen3-235B-A22BAlibaba, released 2025-04-29 | 20.9% | Moonshot AI, Kimi K2 release pagechecked 2026-10-15 | Effort: non-thinking; Harness: agentic, single attempt |
| Qwen3-Max-ThinkingAlibaba | 66.7% | Alibaba, Qwen3.5-397B-A17B model cardchecked 2026-10-15 | - |
| Qwen3.8-27BAlibaba | 73.8% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15 | Harness: mini-SWE-agent; Context: 256000 |
| Qwen3.8-Flash-NextAlibaba | 81% | Alibaba, Qwen3.8-Flash-Next model cardchecked 2026-10-15; printed on 1 more page | Harness: mini-SWE-agent; Context: 256000 |
| Kimi K2.6Moonshot AI, released 2026-04-20 | 76.7% | Moonshot AI, Kimi K2.6 model cardchecked 2026-10-15; printed on 2 more pages | Harness: in-house SWE-agent variant |
| Kimi K2.5Moonshot AI, released 2026-01-27 | 73% | Moonshot AI, Kimi K2.5 model cardchecked 2026-10-15; printed on 3 more pages | Harness: in-house SWE-agent variant |
| Kimi K2 ThinkingMoonshot AI, released 2025-11-06 | 61.1% | Moonshot AI, Kimi K2 Thinking release pagechecked 2026-10-15; printed on 1 more page | Effort: thinking; Harness: in-house harness derived from SWE-agent; Tools: enabled; bash and edit tools |
| Kimi K2 Instruct 0905Moonshot AI, released 2025-09-05 | 55.9% | Moonshot AI, Kimi K2 Thinking release pagechecked 2026-10-15 | Harness: in-house harness derived from SWE-agent; Tools: enabled; bash and edit tools |
| Kimi K2 InstructMoonshot AI, released 2025-07-11 | 47.3% | Moonshot AI, Kimi K2 release pagechecked 2026-10-15 | Harness: agentic, single attempt |
| GLM-5.3-FlashZ.ai, released 2026-08-26 | 78.3% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
| GLM-5.1Z.ai, released 2026-04-07 | 73.3% | DeepSeek, DeepSeek-V4-Pro (preview) model cardchecked 2026-10-15; printed on 1 more page | Effort: thinking |
| GLM-5Z.ai, released 2026-02-12 | 73.3% | Z.ai, GLM-5 model cardchecked 2026-10-15 | Harness: OpenHands; Context: 200000 |
| GLM-4.7Z.ai, released 2025-12-22 | 66.7% | Z.ai, GLM-4.7 model cardchecked 2026-10-15; printed on 1 more page | - |
| GLM-4.6Z.ai, released 2025-09-30 | 53.8% | Z.ai, GLM-4.7 model cardchecked 2026-10-15 | - |
| MiniMax M2.7MiniMax, released 2026-03-18 | 76.5% | MiniMax, MiniMax-M2.7 model cardchecked 2026-10-15; printed on 1 more page | - |
| Nemotron 3.5 Super VL 120B-A12BNVIDIA, released 2026-10-15 | 67.72% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
| Nemotron 3.5 Lightning 30B-A3BNVIDIA, released 2026-08-11 | 39.33% | NVIDIA, NVIDIA Nemotron 3.5 Lightning 30B-A3B model cardchecked 2026-10-19 | - |
| Nemotron 3 UltraNVIDIA, released 2026-06-04 | 68.8% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19; printed on 1 more page | - |
| Nemotron 3 SuperNVIDIA, released 2026-03-11 | 49.8% | NVIDIA, NVIDIA Nemotron 3.5 Lightning 30B-A3B model cardchecked 2026-10-19 | - |
| Nemotron 3 Nano 30B-A3BNVIDIA, released 2025-12-15 | 14.07% | NVIDIA, NVIDIA Nemotron 3.5 Lightning 30B-A3B model cardchecked 2026-10-19 | - |
| Inkling-SmallThinking Machines | 71.4% | NVIDIA, NVIDIA Nemotron 3.5 Super VL 120B-A12B model cardchecked 2026-10-19 | - |
Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.