BlogLLM Benchmark Scores as Published, Side by Side
SWE Atlas QnA: Results as Published
Every result the catalog holds for SWE Atlas QnA: 15 models from 11 developers, read from 4 pages. Each number is as printed by the company named beside it; nothing is ranked or rescored. Models are listed by developer, newest first.
Area Agentic CodingMaintainer Scale AIIntroduced 2026Data version 2026103001, as of 2026-10-30
SWE Atlas Codebase QnA asks a coding agent to answer technical questions about a real repository that it can build and run inside a container. The leaderboard has 124 tasks drawn from 11 production repositories in Go, Python, C, and TypeScript. Human experts write a rubric of factual claims for each question, and a model judge marks each claim as met or not met. The task resolve rate is the share of answers that meet every rubric item, and a task also fails if the agent changes the source code.
SWE-Atlas-QnA (setting not stated)
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.5OpenAI, released 2026-04-23 | 45.43% | MiniMax, MiniMax M3checked 2026-10-15 | - |
| Claude Opus 4.7Anthropic, released 2026-04-16 | 45.16% | MiniMax, MiniMax M3checked 2026-10-15 | - |
| Claude Sonnet 4.6Anthropic, released 2026-02-17 | 31.2% | MiniMax, MiniMax M3checked 2026-10-15 | - |
| Gemini 3.1 Pro PreviewGoogle, released 2026-02-19 | 13.5% | MiniMax, MiniMax M3checked 2026-10-15 | - |
| Muse Spark OpenMeta, released 2026-10-15 | 58% | Meta, Meta Muse Spark Open 816B-A42B model cardchecked 2026-10-19 | Effort: xhigh; Harness: preferred harness of each model |
| DeepSeek-V4-Pro-0813DeepSeek | 66% | Mistral AI, Mistral Large 4 (preview)checked 2026-10-15 | - |
| Qwen3.8-MaxAlibaba, released 2026-08-03 | 62% | Mistral AI, Mistral Large 4 (preview)checked 2026-10-15 | - |
| Qwen3.8-2.4T-A95BAlibaba | 62% | Meta, Meta Muse Spark Open 816B-A42B model cardchecked 2026-10-19 | Effort: xhigh; Harness: preferred harness of each model |
| Kimi K3Moonshot AI, released 2026-07-16 | 66% | Meta, Meta Muse Spark Open 816B-A42B model cardchecked 2026-10-19; printed on 1 more page | Effort: max; Harness: preferred harness of each model |
| GLM-5.3Z.ai, released 2026-08-18 | 59% | Meta, Meta Muse Spark Open 816B-A42B model cardchecked 2026-10-19; printed on 1 more page | Effort: max; Harness: preferred harness of each model |
| Mistral Large 4 (preview)Mistral AI, released 2026-10-06 | 59.4% | Mistral AI, Mistral Large 4 (preview)checked 2026-10-15; printed on 1 more page | - |
| MiniMax M3MiniMax, released 2026-06-01 | 37.9% | MiniMax, MiniMax M3checked 2026-10-15 | - |
| MiniMax M2.7MiniMax, released 2026-03-18 | 11.29% | MiniMax, MiniMax M3checked 2026-10-15 | - |
SWE-Atlas-QnA (strict)
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GLM-5.2Z.ai, released 2026-06-16 | 46.4% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15 | Effort: max; Tools: no internet (strict) |
| MiniMax M3MiniMax, released 2026-06-01 | 42.3% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15 | Tools: no internet (strict) |
| InklingThinking Machines, released 2026-07-14 | 25.5% | IFM, IFM K2-Horizon-375B-A23B model cardchecked 2026-10-15 | Effort: xhigh; Tools: no internet (strict) |
Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.