sugra.blog

BlogLLM Benchmark Scores as Published, Side by Side

DeepSeek-V4.1-Flash-1017: Benchmark Scores as Published

Every result the catalog holds for DeepSeek-V4.1-Flash-1017: 50 tests in 11 areas, from 6 pages. 11 printed by DeepSeek itself, the rest by the company named beside each number. Each number is as printed; nothing is ranked or rescored.

Developer DeepSeekReleased 2026-10-17Context window 1MInput Text, imageOutput TextWeights Open (MIT)Data version 2026103001, as of 2026-10-30

Compare in the table

Knowledge Work

TestValuePrinted bySetup as printed
AutomationBench60.5%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 2 more pagesDataset revision: v1.0.6 (printed in the label)
APEX-Agents63.4%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
OfficeQA Pro62.4%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code
GDPval-AA v2.11746Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
AA-Briefcase v1.11570Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
GDP.pdf w/ python (mean rubric pass rate)73%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: python
Apex Bench v1.172.1%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18-
Business Operation (Internal)8.9%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
FrontierFinance54.5%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Finance Agent v2
OfficeQA Pro v240.4%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code
DECK-Arena (Internal, Elo)1231Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
Harvey Legal Agent (public set, all-pass rate)21%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Stirrup

Agentic Coding

TestValuePrinted bySetup as printed
DeepSWE v1.177%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 2 more pagesHarness: Mini-SWE-V2
Terminal-Bench 4.042.3%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 2 more pagesHarness: Claude Code
NL2Repo-Bench68.2DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 1 more page-
ProgramBench Almost@129.1%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 1 more page-
SWE-Marathon40%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code; Dataset revision: v1.1
Kimi Code Bench 2.076.1%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
Kimi Code Bench 3.0 Long-horizon51.5%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max

ML Engineering

TestValuePrinted bySetup as printed
PostTrainBench v1.230.5%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code; Dataset revision: v1.2

Science and Math

TestValuePrinted bySetup as printed
GPQA Diamond92.2%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
Terminal-Bench Science 0.19.3%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Model as printed: DeepSeek-V4.1-Flash-1017
15.7%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Model as printed: DeepSeek-V4.1-Flash-1017 (max)
FrontierMath Tier 4 (v2)46.3%Epoch AI, Epoch AI Benchmarking Hubchecked 2026-10-30Effort: max
FrontierMath Tier 1-3 (v2)71.2%Epoch AI, Epoch AI Benchmarking Hubchecked 2026-10-30Effort: max
BioMysteryBench80%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code

Computer Use

TestValuePrinted bySetup as printed
OSWorld 2.1 partial69.3%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max

Multimodal Understanding

TestValuePrinted bySetup as printed
CharXiv Reasoning (no tools)77%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none
MMMU-Pro80.6%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none
GDP.pdf17.2%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none
CharXiv Reasoning (with tools)85.6%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: python
BabyVision (with tools)92.5%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 2 more pagesTools: enabled; exact tools in source
MMMU-Pro (with tools)82.5%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: python
Chartography (with tools)83%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 1 more pageTools: enabled; exact tools in source
ZeroBench (with tools, pass@5)51%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 1 more pageTools: enabled; exact tools in source
PerceptionBench47.7%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none
ZeroBench (pass@5)23%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none
ZeroBench_main (avg@5)11.8%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none

Cybersecurity

TestValuePrinted bySetup as printed
xdev-bench-hard, problems solved (pass@3)85.7%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max

General Knowledge and Reasoning

TestValuePrinted bySetup as printed
Humanity's Last Exam (no tools)42.6%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 2 more pages-
44%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18Dataset revision: text-only subset of HLE (dagger on the card)
Humanity's Last Exam (with tools)58.8%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: with tools
SimpleQA Verified42.5%Epoch AI, Epoch AI Benchmarking Hubchecked 2026-10-30Effort: max
Humanity's Last Exam Diamond (no tools)21.6%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none
Kimi Hallu Bench Hard w/ tools (lower is better)lower is better23.3%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: with tools
Kimi Hallu Bench w/o tools (lower is better)lower is better13.1%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none

Tool Use and Workflows

TestValuePrinted bySetup as printed
Agents' Last Exam (pass rate)35.4%DeepSeek, DeepSeek-V4.1-Flash-1017 model cardchecked 2026-10-18; printed on 1 more page-
Agents' Last Exam (CLI)36%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code; Dataset revision: v1.1

Instruction Following

TestValuePrinted bySetup as printed
HANDBOOK.md28.9%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: OpenHands

Engineering and Design

TestValuePrinted bySetup as printed
BenchCAD81.6%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Harness: Claude Code; Tools: with tools
Design Bench (Internal, Elo)863Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max
BenchCAD w/o tools41.2%Moonshot AI, Introducing Kimi K3.1 Prochecked 2026-10-24Effort: max; Tools: none

Price

API list prices in US dollars per one million tokens, as printed on the developer's pricing page on the day checked. Cost per task is measured by Vals AI.

PriceValuePrinted bySetup as printed
Input$0.3DeepSeek, DeepSeek API pricingchecked 2026-10-18-
Cached input$0.006DeepSeek, DeepSeek API pricingchecked 2026-10-18-
Output$1.2DeepSeek, DeepSeek API pricingchecked 2026-10-18-
Input, off-peak hours$0.15DeepSeek, DeepSeek API pricingchecked 2026-10-18-
Cached input, off-peak hours$0.003DeepSeek, DeepSeek API pricingchecked 2026-10-18-
Output, off-peak hours$0.6DeepSeek, DeepSeek API pricingchecked 2026-10-18-
Cost per taskVals AI, USD, lower is better$0.58Vals AI, Vibe Code Bench v1.1 leaderboardchecked 2026-10-27Harness: OpenHands

Side by side

More from DeepSeek

Every number here is read from the same catalog file as the benchmark table, data version 2026103001. Where one page printed the model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.