LLM Benchmark Scores as Published, Side by Side

AutomationBench tests whether AI agents can carry out business workflows that span several software applications, such as a CRM, an inbox, a calendar and a messaging tool, through their REST APIs. The agent has to find the right endpoints on its own, follow layered business rules, and ignore misleading records along the way. Grading is programmatic: the final state of the simulated systems is checked against success criteria, so a run is judged on the end result.

GDPval-AA is Artificial Analysis's version of OpenAI's GDPval, a set of realistic work tasks drawn from 44 occupations in nine major industries. Models get shell access and web browsing in an agent loop and produce a work product, and the outputs are compared in blind pairwise judgments that are turned into Elo ratings. The v2 revision re-anchors the scale so that human-expert deliverables sit near 1000 points.

Harvey's Legal Agent Benchmark tests AI agents on long, multi-step legal work. Each task gives the agent a short instruction and a client matter with case files, and asks for a work product that a reviewing lawyer would check, across more than 1,200 tasks in 24 legal practice areas. Expert-written rubrics grade each deliverable, and a task counts as passed only when every criterion passes.

DeepSWE tests whether coding agents can make original, long-horizon software engineering changes in real open-source repositories. It has 113 tasks across 91 repositories in TypeScript, Go, Python, JavaScript and Rust, all written from scratch so that solutions are not copied from existing commits. Hand-written verifiers check behavior through public interfaces, and each verifier runs three times to catch flakiness.

FrontierSWE tests coding agents on very long engineering and research jobs: reimplementing complex systems, speeding up existing code without breaking its correctness, and designing and training machine learning models on held-out data. Version 2 has 34 tasks, each with up to a 20-hour budget per trial. Results are averaged over five trials per task (mean at 5).

Terminal-Bench tests whether an AI agent can finish real jobs by typing commands into a Linux terminal, such as building software, configuring a system or processing files. Each task runs in its own sandboxed container, and the agent succeeds only if the final state of that container passes the task's automatic checks. Version 2.0 (November 2025) has 89 hand-checked tasks, and the companion Harbor framework runs the agents.

Vibe Code Bench tests whether an AI model can build a complete, working web application from a written description, starting from nothing. The benchmark has 100 app specifications split into validation and held-out test sets. An automated browser agent then checks each deployed app against 964 multi-step user workflows, and the score is the accuracy across their 10,131 steps.

PostTrainBench tests whether an AI agent can do post-training, the stage that turns a raw language model into a useful assistant. Each agent gets a small base model, one GPU and a 10-hour budget, and is free to pick data and methods to improve it on a target test. Scores compare the agent-trained model with the official instruction-tuned version, and the authors also track cheating such as training on test data.

GPQA is a set of 448 multiple-choice questions in biology, physics and chemistry, written by experts with or working toward PhDs in those fields. Domain experts reached about 65% accuracy, while skilled non-experts with web access reached about 34%, which is why the questions are described as hard to answer by search alone. The Diamond subset is a smaller, harder selection of these questions.

RiemannBench tests research-level mathematics that goes well beyond olympiad-style problems. It is a private set of expert-written problems, each with a unique closed-form answer that is checked by a program, and each problem is verified by two experts solving it independently. Models run as unconstrained research agents with coding tools and search, and are scored with a statistical estimator over many independent runs.

Terminal-Bench-Science tests AI agents on the computational workflows working scientists actually run, not textbook questions. Each task is a containerised terminal job, such as statistical analysis, simulation, signal processing or proofs, and the output is checked by task-specific automated tests. Version 0.1 has 70 tasks written by practising researchers across life, physical, Earth, mathematical and engineering sciences.

GraphWalks gives a model a directed graph as a long edge list and asks one graph question in a single answer, with no tools. The two task types are a breadth-first search from a start node to a given depth, which returns only the nodes at exactly that depth, and a parents query, which returns the nodes with an edge pointing into a given node. The public dataset has 1,150 examples, split into inputs of 128k and shorter and inputs from 256k to 1M. Each answer ends with a 'Final Answer:' line of node IDs and is scored by recall, precision and F1 against the true set.

OSWorld tests whether an AI agent can carry out open-ended tasks on a real computer desktop, on Ubuntu, Windows or macOS, by clicking and typing in real applications. OSWorld 2.0 has 108 long workflows across professional and everyday work, each taking a skilled human over an hour, with some changing mid-task and some requiring questions to the user. Success is binary completion within a 500-step limit, with a partial score for checkpoints reached.

Chartography tests whether AI models can read and reason about professional charts, such as Kaplan-Meier survival curves, candlestick charts, Sankey diagrams and Bode plots, which standard chart tests rarely include. It has 100 tasks written by professionals in fields such as medicine, engineering, finance and manufacturing. Numeric answers are checked against ranges set by the expert author, and scores are averaged over repeated runs.

CharXiv tests whether a multimodal model can read real scientific charts. The set holds 2,323 charts taken from arXiv papers, each with descriptive questions about basic chart elements and reasoning questions that need information synthesized across the chart. All charts and questions were hand-picked and verified by human experts. The reasoning split, the one model cards report, is scored as accuracy on the reasoning questions.

MMMU-Pro tests whether a multimodal model still understands multi-discipline questions once the shortcuts of the original MMMU are removed. The benchmark was built in three steps: questions that text-only models could answer were filtered out, the candidate options were increased from four to ten, and a vision-only setting embeds the whole question inside an image. The paper reports 1,730 questions, and the overall score is the average of the ten-option standard setting and the vision-only setting, with model answers read from an 'Answer: $LETTER' line.

CWE-Bench evaluates coding agents on defensive security work. Each task gives the agent a checkout of a real open-source repository, sometimes with disclosed CVEs reproduced in it, and asks it to audit the code and fix what it finds without being told how many vulnerabilities there are. A fix counts only if the exploit is blocked and the existing tests still pass, with no partial credit, and agents are not asked to write exploits. The page reports 100 held-out tasks in v0 and 120 in v1.

ExploitBench measures how far an AI agent gets along a ladder of 16 checkable capability steps when it works on known, already-patched bugs in the V8 JavaScript engine used by Chromium. The steps run from triggering the bug up to full control of the target, and each step is confirmed by an automatic verifier with no AI judge. The public set has 41 bugs, all reported in 2024 or later.

ExploitGym measures whether AI agents can take a program input that already triggers a software vulnerability and turn it step by step into a working attack. The benchmark has 898 instances in three areas: ordinary user programs, a JavaScript engine, and the Linux kernel. Protections are varied from task to task to see how they change agent success, and every task runs in a reproducible container.

Humanity's Last Exam is a test of about 2,500 expert-level questions across dozens of subjects, including mathematics, the humanities and the natural sciences. The items are multiple-choice or short-answer, so answers can be graded automatically. It was built because leading models had passed more than 90 percent of older tests, and at launch the most capable models scored below 10 percent.

SimpleQA Verified is a cleaned 1,000-question version of SimpleQA, with mislabeled answers, duplicates and topic imbalance fixed. An automated grader marks each answer, and the paper reports an F1 score; model cards usually print accuracy or pass@1.

Agents' Last Exam tests AI agents on long, realistic professional tasks with checkable outcomes, covering 55 sub-industries of the non-physical economy. Agents get only a task description and work in full operating-system sandboxes, and outputs are graded against hidden, deterministic reference answers. The public release (ALE-V1) has 147 tasks drawn from a larger corpus of more than 1,500.

BrowseComp tests whether an AI agent with web browsing can find hard-to-locate facts that are spread across many sources and tied together. It has 1,266 questions with short answers that are easy to check against a reference. Models without browsing scored near zero when it was released, and a browsing agent solved about half of the questions.

HealthBench tests how well an AI model answers health conversations, such as those with patients or clinicians, judged against detailed rubrics written by physicians. It has 5,000 multi-turn conversations and 48,562 rubric criteria, each with a weight that reflects how much a physician would value it. An automatic grader scores each reply against the rubric, and the benchmark includes subsets named Consensus and Hard.

HealthBench Professional tests AI models on real conversations that clinicians had with an AI assistant, in three areas: care consultation, writing and documentation, and medical research. Each example is scored against rubrics written and checked by at least three physicians. It is meant to be hard, and about a third of the examples were chosen because physicians deliberately tried to find failures.

BioSecBench is a family of three checkable tests from LatchBio for AI agents doing biology work with a biosecurity angle. The Refusal part pairs routine research tasks with tasks that look like research but hide a hazard, and measures how often an agent refuses. The Surveillance part checks whether an agent can run pathogen genomic surveillance analyses correctly, and the Function part checks whether an agent can infer biological function from experimental data, scored against ground truth.

Probability that an attacker finds a successful indirect prompt injection after k attempts, across coding, tool use and GUI computer use.

Not checked against a maintainer's page.

BenchCAD is a benchmark of 17,900 industrial CAD programs written in CadQuery, each checked by running it, across 106 part families such as gears, springs and drill bits. Models answer questions about drawings and code, turn an image into code, and edit code according to an instruction. The authors report that current models often capture the rough outer shape of a part but struggle to build faithful parametric programs.

Blueprint-Bench tests spatial reasoning by asking a model to turn photographs of an apartment into a two-dimensional floor plan. The plan is scored against the true one by comparing which rooms connect and how their sizes rank. Blueprint-Bench 2 is the agent-only version: an agent works through 50 apartments in sequence and keeps a notepad between them, and its score is normalized so that a random plan scores 0 and a perfect one scores 1.

EEBench is an agent benchmark for electrical and hardware engineering, aimed at the requirements, design and test loop of a hardware project. Agents submit atopile design bundles, and the harness builds the circuit graph, bill of materials and simulation decks from that source. Each requirement is checked by simulation at worst-case component-tolerance corners, and the leaderboard score blends engineering results at 65% with cost efficiency at 35% against a reference bill of materials. The held-out tasks are not published.

The Artificial Analysis Intelligence Index combines several test suites of primarily text-based, English-language tasks into one score for a language model. Its methodology page groups the evaluations into agents, coding, general knowledge and scientific reasoning, with a weight for each. The score is a weighted average, and agent tests are converted from Elo ratings to a 0 to 1 contribution.

Not a test: the price the developer charges through its API, in US dollars per one million tokens, as printed on its pricing page on the day it was checked. Input is the prompt, cached input a prompt prefix served from the cache, cache write the cost of storing that prefix, output the reply including any thinking tokens. Standard is the regular rate; batch, flex, fast and priority are the developer's other schedules. A price for long context, for a region or in another currency is on the pricing page the value links to. Lower is better.