Blog

LLM Benchmark Scores as Published, Side by Side

Benchmark results AI labs publish on release pages and model cards, and their official API prices, in one table by area and test. Each number links to its source page; there is no ranking and no score of our own.

A tilted benchmark table with one shaded column, beside the title LLM benchmark scores, side by side.

A launch page is a table with the publisher’s model in the first column. I kept the tables and lost the first column.

The table behind every model launch

Most model releases arrive with a comparison table: benchmarks down the side, models across the top, and the publisher’s newest model in the first column. The numbers are useful, but the tables are difficult to compare. Each release chooses a different set of models, tests, tools and evaluation settings, and each new launch makes the previous table look old.

I read these tables for work. We run models every day at Sugra Systems, and choosing one for a task often begins with what its developer says it can do. I started recording those numbers together with their sources and stated conditions. That record became the catalog below.

What this catalog is

The catalog contains results published by model developers on their release pages and model cards, and the API prices from their official pricing pages. Two independent evaluators run the models themselves, and their numbers are marked with their names. Vals AI printed what one coding task cost, the highlighted last row of the Price area, which a price per million tokens cannot show, and its own Vibe Code Bench score fills that cell where the developer printed none. Epoch AI fills a cell of GPQA Diamond, FrontierMath or SimpleQA Verified only where the developer printed no number for its newest model; OpenAI commissioned FrontierMath and has access to most of its problems. Sugra does not run these evaluations. Each value is stored as printed, in its original unit, with the source, date and conditions disclosed by the publisher.

Results remain attached to their publications. Two values in the same row may come from different harnesses, reasoning settings or tool policies; placing them side by side does not turn them into a single controlled experiment. Different benchmark versions and metrics remain separate.

There is no overall score and no ranking. The table is a sourced catalog of published claims: a place to see what was reported, compare the available evidence and open the original source before drawing a conclusion.

Weights
Developer
Models
Area
Benchmarks
Published results by area and test
AreaBenchmarkOpenAIGPT-6 AstraAnthropicClaude Fable 5.2AnthropicClaude Opus 5.5GoogleGemini 4 CarbonxAIGrok 4.7
ModelReleased2026-09-032026-11-092026-09-222026-10-282026-09-21
Context window
Max output-
Input
Output
Weights
Knowledge WorkAA-Briefcase v1.1
AutomationBench-
GDPval-AA v2.1-
Agentic CodingTerminal-Bench 4.0
FrontierSWE v2
Vibe Code Bench
ML EngineeringPostTrainBench v1.1via Google---
Science and MathGPQA Diamond-
Terminal-Bench Science 0.1-
FrontierMath Tier 4 (v2)--
Long ContextGraphWalks BFS 256k to 1M (F1)via Google--
Computer UseOSWorld 2.1 offline subset-
Multimodal UnderstandingChartography (no tools)-
CybersecurityCWE-bench v1via Google-
ExploitGym--
General Knowledge and ReasoningHumanity's Last Exam (no tools)--
SimpleQA Verifiedvia Epoch AI--
Tool Use and WorkflowsAgents' Last Exam v1.1 (score)via Google--
Web Search and ResearchBrowseComp---
HealthcareHealthBench--
HealthBench Professional (length-adjusted)--
PhysicianBenchvia Anthropic--
BiosecurityBioSecBench Functionvia xAI---
Safety and AlignmentGray Swan IPIattack success rate after k attempts, lower is better--
Engineering and DesignBlueprint-Bench 2 (normalized score)via Google--
RoboDojo Simvia Anthropic--
Preference and Composite RankingsArtificial Analysis Intelligence Index v5.0via Google--
PriceInputUSD per 1M tokens, lower is better
OutputUSD per 1M tokens, lower is better
Cost per taskVals AI, USD, lower is better

Data version 2026111201, checked 2026-11-12. 13597 results on 851 tests for 238 models, 2597 developer prices for 345 models and 117 OpenRouter prices for 43 open-weight models, from 148 pages.

The Model rows are what each developer prints on its own model page; a dash there means it prints none. Each number is as printed, by the model's own developer where it published one; a number another company printed is marked via and its name. A price is the developer's API list price in US dollars per million tokens. Cost per task is measured by Vals AI. A number marked via Epoch AI or via Vals AI is that evaluator's own run, shown only where the developer printed none; Epoch AI data is CC BY 4.0. A shaded number is the best in its row; on a price row, the lowest. Click a value to see its source and stated conditions. A range means the page printed the model under several setups, and the click lists each. A dash means that no cataloged publication reported a result for that model; on a price row, that the developer lists that price in another unit or currency.

The areas

Benchmarks are grouped by the kind of work they test. The areas below are the catalog’s own; their descriptions are copied here from it.

AreaWhat it covers
Knowledge WorkTasks a professional is paid for: legal analysis, finance and tax questions, office documents, spreadsheets, research memos and other desk work. The tests here hand the model a realistic brief and judge the finished product, often with a human expert or an expert-written rubric.
Agentic CodingChanging and repairing real programs. The model works inside a repository or a terminal over many steps, reads the code, runs commands, and is scored on whether the resulting change passes hidden tests or a reviewer. This is the busiest area on every launch page.
ML EngineeringTraining and improving models rather than ordinary software: reproducing a research paper, running an experiment, tuning a training recipe, writing a compute kernel. The score is usually whether the resulting artifact works or reaches a target.
Science and MathMathematics from contest problems to research-level questions, plus scientific reasoning and laboratory knowledge in chemistry, physics and biology. Dangerous biological capabilities are kept out of this area and sit under Biosecurity.
Long ContextFinding, matching and using information spread across very long inputs: hundreds of thousands or a million tokens of documents, transcripts or structured data. The tests plant facts or relationships deep in the input and check whether the model can still use them.
Computer UseOperating a computer the way a person does: a desktop, a browser, a mobile device, through screenshots and clicks rather than an API. Tasks are everyday chores in real applications, scored on whether the end state is correct. Open-ended web research has its own area.
Multimodal UnderstandingImages, charts, video, audio, scanned documents and sheet music, and reasoning that connects them with text. The model has to read what it sees or hears and answer about it.
CybersecurityFinding and fixing vulnerabilities, reproducing known flaws in controlled test environments, analysing binaries and operating production systems under failure. These tests measure capability in a sandbox; the Safety and Alignment area measures whether the model refuses to misuse it.
General Knowledge and ReasoningBroad knowledge, logic and abstract problem solving, factual accuracy and the mixed question sets that try to cover everything at once. The benchmarks here are the ones most often quoted as a single headline number.
Code GenerationWriting code from a specification without a long agentic process: a function, a component, a drawing, an algorithm. Scored by tests, by rendering the output, or by a judge comparing candidates.
Tool Use and WorkflowsCalling functions and APIs, chaining tools, planning multi-step work and carrying a long task to the end. The environments range from office automation to games, and the score is whether the whole workflow completed.
Web Search and ResearchFinding hard-to-locate facts on the open web and synthesizing an answer from many pages. The questions are built so that no single page answers them.
Instruction FollowingObeying constraints: output formats, length limits, data schemas, forbidden words and multi-step user requirements. The score is how many constraints the answer honored.
MultilingualKnowledge and task performance across natural languages other than English: translation, multilingual question sets and regional exams.
Writing and ConversationCreative and editorial writing, style, role play, and the emotional and social quality of a conversation.
HealthcareClinical and medical knowledge, medical documents, and the quality and safety of help given in a medical conversation, usually graded against rubrics written by physicians.
BiosecurityTests of dangerous biological capability and of dual-use risk: virology know-how, wet-lab troubleshooting, and whether the model refuses requests that could help build a weapon. Some sets also cover chemical and cyber risk. Here a high capability score is not good news, and the refusal scores are the ones to read.
Safety and AlignmentRefusals of harmful requests, resistance to jailbreaks, honesty under pressure, sycophancy, and whether an agent discloses its own failures or takes forbidden shortcuts. Most of these are rates where lower is better; the table says so on each row.
Engineering and DesignCAD modeling, electrical engineering, spatial design and the production of engineering artifacts from a specification, scored on whether the artifact meets the spec.
Preference and Composite RankingsHead-to-head preference rankings by people or judge models, and composite indexes that average many unrelated tests into one number. A specialized index stays in its own area; only the catch-all ones live here.
PriceNot a test: what the developer charges through its API, in US dollars per one million tokens, as printed on its own pricing page on the day it was checked. One row per price component (input, cached input, cache write, output) and per schedule: standard, and the developer's others such as batch, flex, fast and priority. Lower is better. A long-context, regional or non-dollar price stays on the pricing page the value links to. Where the developer of an open-weight model prints no price, the row shows the price OpenRouter lists for it, marked via OpenRouter.

Open weights, in short

What it means. A model with open weights is one whose developer publishes the trained model file. Anyone can download it and run it on their own computer or server, with no API and nothing leaving the machine. A closed model runs only on its developer’s servers, through its app or API. Open weights is not open source: the training data and code usually stay private, and the licence printed under Open in the Weights row (Apache 2.0, MIT or the developer’s own) says what you may do with the model. A model whose developer has not said either way appears only under All.

Reading the names.

  • 4B, 30B, 1T: the size, in billions or trillions of parameters. Bigger usually means better answers and more memory.
  • 30B-A3B: a mixture-of-experts model, 30B parameters in all with 3B at work for each token. All 30B must fit in memory, but it answers about as fast as a 3B model.
  • E4B: “effective 4B”, a design made for phones: about 4B parameters do the work, and extra lookup tables make the file bigger. The Size row prints both numbers.
  • Instruct, Thinking, Base: tuned to follow instructions, tuned to reason before it answers, or the raw model before that tuning, which the table marks pre-trained.
  • Q4, Q8, GGUF, MLX: a model stored at about 4 or 8 bits per parameter so it fits a smaller machine, and the file formats people download it in: GGUF for llama.cpp and the apps built on it, MLX for Macs.

How much memory. At 4-bit, the usual download for your own machine, a model needs about 0.65 GB for each billion parameters, and close to 1 GB more for the program running it and a short conversation: a 30B model needs about 21 GB, so a 24 GB GPU or a 32 GB laptop. At 8-bit the same model needs about 35 GB, at the full 16-bit about 66 GB. On a laptop, a Mac or a DGX Spark the system and the model share one memory, so the table counts about three quarters of it for the model; a graphics card’s own memory counts in full.

Choose Open weights above the table and a Runs on line appears: pick a device and the table shows the biggest open models that fit it. The Runs on row does the same sum for each open model and names the smallest device that holds it; where the developer prints its own requirement, the row uses that and says so. NVIDIA prints larger figures for DGX Spark, up to 200B parameters on one, without saying at which precision; the table keeps the same 4-bit sum as for every other device, and NVIDIA’s figures are in the hint on each Spark. The sum says whether a model fits, not how fast it answers: speed depends on the device’s memory bandwidth, and the table does not show it.

How people run them.

  • On a computer, in an app: LM Studio, Jan or Ollama. Install the app, pick a model from its list, chat.
  • On a phone: Google AI Edge Gallery or PocketPal AI, with the small models made for phones.
  • In a terminal: Ollama (ollama run gpt-oss:20b), llama.cpp, the engine inside many of these apps, or MLX on a Mac.
  • On a server, for many users at once: vLLM or SGLang.
  • Without hardware of your own: a hosted provider runs the open model and charges per token. OpenRouter puts many of them behind one API key, and Hugging Face Inference Providers does the same with a Hugging Face account. Some open models are free on OpenRouter, with a daily request limit.

The benchmarks, one by one

Each benchmark in the current view is described below: who maintains it, what it tests and which cataloged publications reported a result for it.

Knowledge WorkBack to table

AA-BriefcaseArtificial Analysis
Maintainer page

AA-Briefcase tests AI agents on long, multi-week business projects in fields such as data science, product management and corporate strategy. The material is four multi-week projects with thousands of input files and 91 tasks in total, and each task asks for a professional deliverable such as a spreadsheet, presentation or memo. Each deliverable is graded on pass-or-fail rubric checks plus pairwise comparisons for analytical quality and presentation, and these are combined into an Elo score. The evaluation is private and run by Artificial Analysis.

AutomationBenchZapier, 2026
Maintainer page

AutomationBench tests whether AI agents can carry out business workflows that span several software applications, such as a CRM, an inbox, a calendar and a messaging tool, through their REST APIs. The agent has to find the right endpoints on its own, follow layered business rules, and ignore misleading records along the way. Grading is programmatic: the final state of the simulated systems is checked against success criteria, so a run is judged on the end result.

GDPval-AAArtificial Analysis, which runs its own version of OpenAI's GDPval dataset
Maintainer page

GDPval-AA is Artificial Analysis's version of OpenAI's GDPval, a set of realistic work tasks drawn from 44 occupations in nine major industries. Models get shell access and web browsing in an agent loop and produce a work product, and the outputs are compared in blind pairwise judgments that are turned into Elo ratings. The v2 revision re-anchors the scale so that human-expert deliverables sit near 1000 points.

Agentic CodingBack to table

FrontierSWEProximal Labs (Proximal-Labs on GitHub), 2026
Maintainer page

FrontierSWE tests coding agents on very long engineering and research jobs: reimplementing complex systems, speeding up existing code without breaking its correctness, and designing and training machine learning models on held-out data. Version 2 has 34 tasks, each with up to a 20-hour budget per trial. Results are averaged over five trials per task (mean at 5).

Terminal-BenchStanford University and Laude Institute, 2025
Maintainer page

Terminal-Bench tests whether an AI agent can finish real jobs by typing commands into a Linux terminal, such as building software, configuring a system or processing files. Each task runs in its own sandboxed container, and the agent succeeds only if the final state of that container passes the task's automatic checks. Version 2.0 (November 2025) has 89 hand-checked tasks, and the companion Harbor framework runs the agents.

Vibe Code BenchVals AI, 2026
Maintainer page

Vibe Code Bench tests whether an AI model can build a complete, working web application from a written description, starting from nothing. The benchmark has 100 app specifications split into validation and held-out test sets. An automated browser agent then checks each deployed app against 964 multi-step user workflows, and the score is the accuracy across their 10,131 steps.

ML EngineeringBack to table

PostTrainBenchAISA Group, 2026
Maintainer page

PostTrainBench tests whether an AI agent can do post-training, the stage that turns a raw language model into a useful assistant. Each agent gets a small base model, one GPU and a 10-hour budget, and is free to pick data and methods to improve it on a target test. Scores compare the agent-trained model with the official instruction-tuned version, and the authors also track cheating such as training on test data.

Science and MathBack to table

FrontierMathEpoch AI, 2024
Maintainer page

FrontierMath is a set of research-level mathematics problems written by professional mathematicians across most branches of modern maths. Tier 4 is its hardest part, with 43 problems in version 2 (released June 12, 2026). Answers are checked automatically: the model must submit a short Python function that returns the final answer within 30 seconds, and each problem scores 1 or 0.

GPQA DiamondDavid Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel R. Bowman, 2023
Maintainer page

GPQA is a set of 448 multiple-choice questions in biology, physics and chemistry, written by experts with or working toward PhDs in those fields. Domain experts reached about 65% accuracy, while skilled non-experts with web access reached about 34%, which is why the questions are described as hard to answer by search alone. The Diamond subset is a smaller, harder selection of these questions.

Terminal-Bench ScienceTerminal-Bench-Science team, led by researchers at Stanford University, built with the Terminal-Bench and Harbor team, 2026
Maintainer page

Terminal-Bench-Science tests AI agents on the computational workflows working scientists actually run, not textbook questions. Each task is a containerised terminal job, such as statistical analysis, simulation, signal processing or proofs, and the output is checked by task-specific automated tests. Version 0.1 has 70 tasks written by practising researchers across life, physical, Earth, mathematical and engineering sciences.

Long ContextBack to table

GraphWalksOpenAI, 2025
Maintainer page

GraphWalks gives a model a directed graph as a long edge list and asks one graph question in a single answer, with no tools. The two task types are a breadth-first search from a start node to a given depth, which returns only the nodes at exactly that depth, and a parents query, which returns the nodes with an edge pointing into a given node. The public dataset has 1,150 examples, split into inputs of 128k and shorter and inputs from 256k to 1M. Each answer ends with a 'Final Answer:' line of node IDs and is scored by recall, precision and F1 against the true set.

Computer UseBack to table

OSWorldXLANG Lab, University of Hong Kong, 2024
Maintainer page

OSWorld tests whether an AI agent can carry out open-ended tasks on a real computer desktop, on Ubuntu, Windows or macOS, by clicking and typing in real applications. OSWorld 2.0 has 108 long workflows across professional and everyday work, each taking a skilled human over an hour, with some changing mid-task and some requiring questions to the user. Success is binary completion within a 500-step limit, with a partial score for checkpoints reached.

Multimodal UnderstandingBack to table

ChartographySurge AI, 2026
Maintainer page

Chartography tests whether AI models can read and reason about professional charts, such as Kaplan-Meier survival curves, candlestick charts, Sankey diagrams and Bode plots, which standard chart tests rarely include. It has 100 tasks written by professionals in fields such as medicine, engineering, finance and manufacturing. Numeric answers are checked against ranges set by the expert author, and scores are averaged over repeated runs.

CybersecurityBack to table

CWE-benchCollinear AI, 2026
Maintainer page

CWE-Bench evaluates coding agents on defensive security work. Each task gives the agent a checkout of a real open-source repository, sometimes with disclosed CVEs reproduced in it, and asks it to audit the code and fix what it finds without being told how many vulnerabilities there are. A fix counts only if the exploit is blocked and the existing tests still pass, with no partial credit, and agents are not asked to write exploits. The page reports 100 held-out tasks in v0 and 120 in v1.

ExploitGymAcademic author team led by UC Berkeley researchers, 2026
Maintainer page

ExploitGym measures whether AI agents can take a program input that already triggers a software vulnerability and turn it step by step into a working attack. The benchmark has 898 instances in three areas: ordinary user programs, a JavaScript engine, and the Linux kernel. Protections are varied from task to task to see how they change agent success, and every task runs in a reproducible container.

General Knowledge and ReasoningBack to table

Humanity’s Last ExamCenter for AI Safety (CAIS) and Scale AI, 2025
Maintainer page

Humanity's Last Exam is a test of about 2,500 expert-level questions across dozens of subjects, including mathematics, the humanities and the natural sciences. The items are multiple-choice or short-answer, so answers can be graded automatically. It was built because leading models had passed more than 90 percent of older tests, and at launch the most capable models scored below 10 percent.

SimpleQA VerifiedGoogle (Lukas Haas and co-authors), 2025
Maintainer page

SimpleQA Verified is a cleaned 1,000-question version of SimpleQA, with mislabeled answers, duplicates and topic imbalance fixed. An automated grader marks each answer, and the paper reports an F1 score; model cards usually print accuracy or pass@1.

Tool Use and WorkflowsBack to table

Agents’ Last ExamUC Berkeley Center for Responsible, Decentralized Intelligence, 2026
Maintainer page

Agents' Last Exam tests AI agents on long, realistic professional tasks with checkable outcomes, covering 55 sub-industries of the non-physical economy. Agents get only a task description and work in full operating-system sandboxes, and outputs are graded against hidden, deterministic reference answers. The public release (ALE-V1) has 147 tasks drawn from a larger corpus of more than 1,500.

Web Search and ResearchBack to table

BrowseCompOpenAI, 2025
Maintainer page

BrowseComp tests whether an AI agent with web browsing can find hard-to-locate facts that are spread across many sources and tied together. It has 1,266 questions with short answers that are easy to check against a reference. Models without browsing scored near zero when it was released, and a browsing agent solved about half of the questions.

HealthcareBack to table

HealthBenchOpenAI, 2025
Maintainer page

HealthBench tests how well an AI model answers health conversations, such as those with patients or clinicians, judged against detailed rubrics written by physicians. It has 5,000 multi-turn conversations and 48,562 rubric criteria, each with a weight that reflects how much a physician would value it. An automatic grader scores each reply against the rubric, and the benchmark includes subsets named Consensus and Hard.

Maintainer page

HealthBench Professional tests AI models on real conversations that clinicians had with an AI assistant, in three areas: care consultation, writing and documentation, and medical research. Each example is scored against rubrics written and checked by at least three physicians. It is meant to be hard, and about a third of the examples were chosen because physicians deliberately tried to find failures.

BiosecurityBack to table

BioSecBenchLatchBio, 2026
Maintainer page

BioSecBench is a family of three checkable tests from LatchBio for AI agents doing biology work with a biosecurity angle. The Refusal part pairs routine research tasks with tasks that look like research but hide a hazard, and measures how often an agent refuses. The Surveillance part checks whether an agent can run pathogen genomic surveillance analyses correctly, and the Function part checks whether an agent can infer biological function from experimental data, scored against ground truth.

Safety and AlignmentBack to table

Gray Swan Indirect Prompt Injection (IPI)Gray Swan with the UK AI Security Institute, the US Center for AI Standards and Innovation, and model developers, 2026
Maintainer page

Probability that an attacker finds a successful indirect prompt injection after k attempts, across coding, tool use and GUI computer use.

Not checked against a maintainer's page.

Engineering and DesignBack to table

Blueprint-BenchAndon Labs, 2025
Maintainer page

Blueprint-Bench tests spatial reasoning by asking a model to turn photographs of an apartment into a two-dimensional floor plan. The plan is scored against the true one by comparing which rooms connect and how their sizes rank. Blueprint-Bench 2 is the agent-only version: an agent works through 50 apartments in sequence and keeps a notepad between them, and its score is normalized so that a random plan scores 0 and a perfect one scores 1.

Preference and Composite RankingsBack to table

Maintainer page

The Artificial Analysis Intelligence Index combines several test suites of primarily text-based, English-language tasks into one score for a language model. Its methodology page groups the evaluations into agents, coding, general knowledge and scientific reasoning, with a weight for each. The score is a weighted average, and agent tests are converted from Elo ratings to a 0 to 1 contribution.

PriceBack to table

API price listeach developer, on its own pricing page

Not a test: the price the developer charges through its API, in US dollars per one million tokens, as printed on its pricing page on the day it was checked. Input is the prompt, cached input a prompt prefix served from the cache, cache write the cost of storing that prefix, output the reply including any thinking tokens. Standard is the regular rate; batch, flex, fast and priority are the developer's other schedules. A price for long context, for a region or in another currency is on the pricing page the value links to. Where the developer of an open-weight model prints no price, the value is the price OpenRouter, a service that runs such models from several providers, lists for it, marked via OpenRouter. Lower is better.

How the catalog is maintained

The catalog is a public repository, sugra-llm-benchmarks, and this page reads a copy of the file it builds, refreshed when the article is updated; the data version and its date are printed under the table. It is a maintained record, not a scraper: new release pages and model cards are read by hand, because the details that matter often live in headings, footnotes and evaluation notes, such as the benchmark version, the reasoning effort or the tool policy behind a number. Values are added as printed and checked against the repository’s data contract, and a correction never replaces the past silently: the new record supersedes the earlier one, which stays in the history.

The catalog shows what model developers published. It does not say which model is best for your workload, and it does not replace independent evaluation or a test on your own tasks. If a value here differs from the source it cites, the source is right; open the value and press Report an error, which opens an issue in the repository with the record filled in, so the catalog can be corrected. Copy citation next to it gives the value with its source and the data version in one line. The models and the leading scores will keep changing, and the rules of the catalog should not. Writing a number down with its source is what lets an old number remain an honest record.

Share this articleXLinkedInFacebookTelegramRedditHacker NewsEmailCopy linkMarkdown

Call Sugra API from your code, one key for every direction.

Get API keyDocsSugra Data