LLM Benchmark Scores as Published, Side by Side
Benchmark results AI labs publish on release pages and model cards, and their official API prices, in one table by area and test. Each number links to its source page; there is no ranking and no score of our own.
PublishedUpdated

What this catalog is
The table holds benchmark results as model developers printed them on their release pages and model cards, and the API prices from their own pricing pages, each number linked to its source. A number measured by Vals AI or Epoch AI instead is marked with their name.
There is no overall score and no ranking: each number is a developer’s claim, with a link to where it was made.
| Area | Benchmark | OpenAIGPT-6.2 Astra | AnthropicClaude Fable 5.2 | AnthropicClaude Opus 5.5 | GoogleGemini 4 Carbon | xAIGrok 4.7 |
|---|---|---|---|---|---|---|
| Model | Released | 2026-11-17 | 2026-11-09 | 2026-09-22 | 2026-10-28 | 2026-09-21 |
| Context window | ||||||
| Max output | - | |||||
| Input | ||||||
| Output | ||||||
| Weights | ||||||
| Knowledge Work | AA-Briefcase v1.1 | - | ||||
| AutomationBench | - | - | ||||
| GDPval-AA v2.1 | - | - | ||||
| Agentic Coding | Vibe Code Bench | |||||
| Terminal-Bench 4.0 | - | |||||
| FrontierSWE v2 | - | |||||
| Science and Math | GPQA Diamond | |||||
| FrontierMath Tier 1-3 (v2)via Epoch AI | ||||||
| Terminal-Bench Science 0.1 | - | - | ||||
| Long Context | GraphWalks BFS 256k to 1M (F1)via Google | - | - | - | ||
| Computer Use | OSWorld 2.1 offline subset | - | - | |||
| Multimodal Understanding | Chartography (no tools) | - | - | |||
| Cybersecurity | CWE-bench v2via Google | |||||
| AutoBaxBench (hard tier)via Google | - | - | ||||
| General Knowledge and Reasoning | SimpleQA Verifiedvia Epoch AI | |||||
| Tool Use and Workflows | Agents' Last Exam v1.1 (score)via Google | - | - | - | ||
| Healthcare | HealthBench Professional (length-adjusted) | - | - | |||
| Safety and Alignment | Gray Swan IPIattack success rate after k attempts, lower is better | - | ||||
| Engineering and Design | Blueprint-Bench 2 (normalized score)via Google | - | - | - | ||
| Preference and Composite Rankings | Artificial Analysis Intelligence Index v5.0via Google | - | - | - | ||
| Price | InputUSD per 1M tokens, lower is better | |||||
| OutputUSD per 1M tokens, lower is better | ||||||
| Cost per taskVals AI, USD, lower is better |
A shaded number is the best in its row; on a price row, the lowest. Click a value to see its source. A dash means no data. Epoch AI data is CC BY 4.0.
The areas
| Area | What it covers |
|---|---|
| Knowledge Work | Tasks a professional is paid for: legal analysis, finance and tax questions, office documents, spreadsheets, research memos and other desk work. The tests here hand the model a realistic brief and judge the finished product, often with a human expert or an expert-written rubric. |
| Agentic Coding | Changing and repairing real programs. The model works inside a repository or a terminal over many steps, reads the code, runs commands, and is scored on whether the resulting change passes hidden tests or a reviewer. |
| ML Engineering | Training and improving models rather than ordinary software: reproducing a research paper, running an experiment, tuning a training recipe, writing a compute kernel. The score is usually whether the resulting artifact works or reaches a target. |
| Science and Math | Mathematics from contest problems to research-level questions, plus scientific reasoning and laboratory knowledge in chemistry, physics and biology. Dangerous biological capabilities are kept out of this area and sit under Biosecurity. |
| Long Context | Finding, matching and using information spread across very long inputs: hundreds of thousands or a million tokens of documents, transcripts or structured data. The tests plant facts or relationships deep in the input and check whether the model can still use them. |
| Computer Use | Operating a computer the way a person does: a desktop, a browser, a mobile device, through screenshots and clicks rather than an API. Tasks are everyday chores in real applications, scored on whether the end state is correct. Open-ended web research has its own area. |
| Multimodal Understanding | Images, charts, video, audio, scanned documents and sheet music, and reasoning that connects them with text. The model has to read what it sees or hears and answer about it. |
| Cybersecurity | Finding and fixing vulnerabilities, reproducing known flaws in controlled test environments, analysing binaries and operating production systems under failure. These tests measure capability in a sandbox; the Safety and Alignment area measures whether the model refuses to misuse it. |
| General Knowledge and Reasoning | Broad knowledge, logic and abstract problem solving, factual accuracy and the mixed question sets that try to cover everything at once. The benchmarks here are the ones most often quoted as a single headline number. |
| Code Generation | Writing code from a specification without a long agentic process: a function, a component, a drawing, an algorithm. Scored by tests, by rendering the output, or by a judge comparing candidates. |
| Tool Use and Workflows | Calling functions and APIs, chaining tools, planning multi-step work and carrying a long task to the end. The environments range from office automation to games, and the score is whether the whole workflow completed. |
| Web Search and Research | Finding hard-to-locate facts on the open web and synthesizing an answer from many pages. The questions are built so that no single page answers them. |
| Instruction Following | Obeying constraints: output formats, length limits, data schemas, forbidden words and multi-step user requirements. The score is how many constraints the answer honored. |
| Multilingual | Knowledge and task performance across natural languages other than English: translation, multilingual question sets and regional exams. |
| Writing and Conversation | Creative and editorial writing, style, role play, and the emotional and social quality of a conversation. |
| Healthcare | Clinical and medical knowledge, medical documents, and the quality and safety of help given in a medical conversation, usually graded against rubrics written by physicians. |
| Biosecurity | Tests of dangerous biological capability and of dual-use risk: virology know-how, wet-lab troubleshooting, and whether the model refuses requests that could help build a weapon. Some sets also cover chemical and cyber risk. Here a high capability score is not good news, and the refusal scores are the ones to read. |
| Safety and Alignment | Refusals of harmful requests, resistance to jailbreaks, honesty under pressure, sycophancy, and whether an agent discloses its own failures or takes forbidden shortcuts. Most of these are rates where lower is better; the table says so on each row. |
| Engineering and Design | CAD modeling, electrical engineering, spatial design and the production of engineering artifacts from a specification, scored on whether the artifact meets the spec. |
| Preference and Composite Rankings | Head-to-head preference rankings by people or judge models, and composite indexes that average many unrelated tests into one number. A specialized index stays in its own area; only the catch-all ones live here. |
| Price | Not a test: what the developer charges through its API, in US dollars per one million tokens, from its own pricing page. Lower is better. Where the developer of an open-weight model prints no price, the row shows the price OpenRouter lists for it, marked via OpenRouter. |
Open weights, in short
What it means. A model with open weights is one whose developer publishes the trained model file. Anyone can download it and run it on their own computer or server, with no API and nothing leaving the machine. A closed model runs only on its developer’s servers, through its app or API. Open weights is not open source: the training data and code usually stay private, and the licence printed under Open in the Weights row (Apache 2.0, MIT or the developer’s own) says what you may do with the model. A model whose developer has not said either way appears only under All.
Reading the names.
4B,30B,1T: the size, in billions or trillions of parameters. Bigger usually means better answers and more memory.30B-A3B: a mixture-of-experts model, 30B parameters in all with 3B at work for each token. All 30B must fit in memory, but it answers about as fast as a 3B model.E4B: “effective 4B”, a design made for phones: about 4B parameters do the work, and extra lookup tables make the file bigger. The Size row prints both numbers.Instruct,Thinking,Base: tuned to follow instructions, tuned to reason before it answers, or the raw model before that tuning, which the table marks pre-trained.Q4,Q8,GGUF,MLX: a model stored at about 4 or 8 bits per parameter so it fits a smaller machine, and the file formats people download it in: GGUF for llama.cpp and the apps built on it, MLX for Macs.
How much memory. At 4-bit, the usual download for your own machine, a model needs about 0.7 GB for each billion parameters: a 30B model needs about 21 GB, so a 24 GB GPU or a 32 GB laptop. Choose Open weights above the table and pick a device on the Runs on line: the table shows the open models that fit it.
How people run them.
- On a computer, in an app: LM Studio, Jan or Ollama. Install the app, pick a model from its list, chat.
- On a phone: Google AI Edge Gallery or PocketPal AI, with the small models made for phones.
- In a terminal: Ollama (
ollama run gpt-oss:20b), llama.cpp, the engine inside many of these apps, or MLX on a Mac. - On a server, for many users at once: vLLM or SGLang.
- Without hardware of your own: a hosted provider runs the open model and charges per token. OpenRouter puts many of them behind one API key, and Hugging Face Inference Providers does the same with a Hugging Face account. Some open models are free on OpenRouter, with a daily request limit.
The benchmarks, one by one
Each benchmark in the current view is described below: who maintains it, what it tests and which cataloged publications reported a result for it.
Knowledge WorkBack to table
AA-Briefcase tests AI agents on long, multi-week business projects in fields such as data science, product management and corporate strategy. The material is four multi-week projects with thousands of input files and 91 tasks in total, and each task asks for a professional deliverable such as a spreadsheet, presentation or memo. Each deliverable is graded on pass-or-fail rubric checks plus pairwise comparisons for analytical quality and presentation, and these are combined into an Elo score. The evaluation is private and run by Artificial Analysis.
AutomationBench tests whether AI agents can carry out business workflows that span several software applications, such as a CRM, an inbox, a calendar and a messaging tool, through their REST APIs. The agent has to find the right endpoints on its own, follow layered business rules, and ignore misleading records along the way. Grading is programmatic: the final state of the simulated systems is checked against success criteria, so a run is judged on the end result.
GDPval-AA is Artificial Analysis's version of OpenAI's GDPval, a set of realistic work tasks drawn from 44 occupations in nine major industries. Models get shell access and web browsing in an agent loop and produce a work product, and the outputs are compared in blind pairwise judgments that are turned into Elo ratings. The v2 revision re-anchors the scale so that human-expert deliverables sit near 1000 points.
Agentic CodingBack to table
FrontierSWE tests coding agents on very long engineering and research jobs: reimplementing complex systems, speeding up existing code without breaking its correctness, and designing and training machine learning models on held-out data. Version 2 has 34 tasks, each with up to a 20-hour budget per trial. Results are averaged over five trials per task (mean at 5).
Terminal-Bench tests whether an AI agent can finish real jobs by typing commands into a Linux terminal, such as building software, configuring a system or processing files. Each task runs in its own sandboxed container, and the agent succeeds only if the final state of that container passes the task's automatic checks. Version 2.0 (November 2025) has 89 hand-checked tasks, and the companion Harbor framework runs the agents.
Vibe Code Bench tests whether an AI model can build a complete, working web application from a written description, starting from nothing. The benchmark has 100 app specifications split into validation and held-out test sets. An automated browser agent then checks each deployed app against 964 multi-step user workflows, and the score is the accuracy across their 10,131 steps.
Science and MathBack to table
FrontierMath is a set of research-level mathematics problems written by professional mathematicians across most branches of modern maths. Tier 4 is its hardest part, with 43 problems in version 2 (released June 12, 2026). Answers are checked automatically: the model must submit a short Python function that returns the final answer within 30 seconds, and each problem scores 1 or 0.
GPQA is a set of 448 multiple-choice questions in biology, physics and chemistry, written by experts with or working toward PhDs in those fields. Domain experts reached about 65% accuracy, while skilled non-experts with web access reached about 34%, which is why the questions are described as hard to answer by search alone. The Diamond subset is a smaller, harder selection of these questions.
Terminal-Bench-Science tests AI agents on the computational workflows working scientists actually run, not textbook questions. Each task is a containerised terminal job, such as statistical analysis, simulation, signal processing or proofs, and the output is checked by task-specific automated tests. Version 0.1 has 70 tasks written by practising researchers across life, physical, Earth, mathematical and engineering sciences.
Long ContextBack to table
GraphWalks gives a model a directed graph as a long edge list and asks one graph question in a single answer, with no tools. The two task types are a breadth-first search from a start node to a given depth, which returns only the nodes at exactly that depth, and a parents query, which returns the nodes with an edge pointing into a given node. The public dataset has 1,150 examples, split into inputs of 128k and shorter and inputs from 256k to 1M. Each answer ends with a 'Final Answer:' line of node IDs and is scored by recall, precision and F1 against the true set.
Computer UseBack to table
OSWorld tests whether an AI agent can carry out open-ended tasks on a real computer desktop, on Ubuntu, Windows or macOS, by clicking and typing in real applications. OSWorld 2.0 has 108 long workflows across professional and everyday work, each taking a skilled human over an hour, with some changing mid-task and some requiring questions to the user. Success is binary completion within a 500-step limit, with a partial score for checkpoints reached.
Multimodal UnderstandingBack to table
Chartography tests whether AI models can read and reason about professional charts, such as Kaplan-Meier survival curves, candlestick charts, Sankey diagrams and Bode plots, which standard chart tests rarely include. It has 100 tasks written by professionals in fields such as medicine, engineering, finance and manufacturing. Numeric answers are checked against ranges set by the expert author, and scores are averaged over repeated runs.
CybersecurityBack to table
AutoBaxBench tests whether a model writes backend code that both works and is secure. Its tasks are generated by AutoBaxBuilder, an LLM pipeline that writes a scenario, functional tests and end-to-end exploits from scratch. The model implements a web backend from an OpenAPI specification in a single call, with no security instructions in the prompt; fixed functional tests and black-box exploits aimed at specific CWEs grade it, and an application counts as secure only if it first passes every functional test. Tasks are split into easy, medium and hard subsets with 1, 3 and 5 API endpoints to implement.
CWE-Bench evaluates coding agents on defensive security work. Each task gives the agent a checkout of a real open-source repository, sometimes with disclosed CVEs reproduced in it, and asks it to audit the code and fix what it finds without being told how many vulnerabilities there are. A fix counts only if the exploit is blocked and the existing tests still pass, with no partial credit, and agents are not asked to write exploits. The page reports 100 held-out tasks in v0 and 120 in v1.
General Knowledge and ReasoningBack to table
SimpleQA Verified is a cleaned 1,000-question version of SimpleQA, with mislabeled answers, duplicates and topic imbalance fixed. An automated grader marks each answer, and the paper reports an F1 score; model cards usually print accuracy or pass@1.
Tool Use and WorkflowsBack to table
Agents' Last Exam tests AI agents on long, realistic professional tasks with checkable outcomes, covering 55 sub-industries of the non-physical economy. Agents get only a task description and work in full operating-system sandboxes, and outputs are graded against hidden, deterministic reference answers. The public release (ALE-V1) has 147 tasks drawn from a larger corpus of more than 1,500.
HealthcareBack to table
HealthBench Professional tests AI models on real conversations that clinicians had with an AI assistant, in three areas: care consultation, writing and documentation, and medical research. Each example is scored against rubrics written and checked by at least three physicians. It is meant to be hard, and about a third of the examples were chosen because physicians deliberately tried to find failures.
Safety and AlignmentBack to table
Probability that an attacker finds a successful indirect prompt injection after k attempts, across coding, tool use and GUI computer use.
Engineering and DesignBack to table
Blueprint-Bench tests spatial reasoning by asking a model to turn photographs of an apartment into a two-dimensional floor plan. The plan is scored against the true one by comparing which rooms connect and how their sizes rank. Blueprint-Bench 2 is the agent-only version: an agent works through 50 apartments in sequence and keeps a notepad between them, and its score is normalized so that a random plan scores 0 and a perfect one scores 1.
Preference and Composite RankingsBack to table
The Artificial Analysis Intelligence Index combines several test suites of primarily text-based, English-language tasks into one score for a language model. Its methodology page groups the evaluations into agents, coding, general knowledge and scientific reasoning, with a weight for each. The score is a weighted average, and agent tests are converted from Elo ratings to a 0 to 1 contribution.
PriceBack to table
Not a test: the price the developer charges through its API, in US dollars per one million tokens, as printed on its pricing page on the day it was checked. Input is the prompt, cached input a prompt prefix served from the cache, cache write the cost of storing that prefix, output the reply including any thinking tokens. Standard is the regular rate; batch, flex, fast and priority are the developer's other schedules. A price for long context, for a region or in another currency is on the pricing page the value links to. Where the developer of an open-weight model prints no price, the value is the price OpenRouter, a service that runs such models from several providers, lists for it, marked via OpenRouter. Lower is better.
How the catalog is maintained
The catalog lives in a public repository, sugra-llm-benchmarks, and this table reads the file it builds.
If a value here differs from the source it cites, the source is right: open the value and press Report an error.