BlogLLM Benchmark Scores as Published, Side by Side
SWE-Lancer: Results as Published
Every result the catalog holds for SWE-Lancer: 4 models from 1 developer, read from 2 pages. Each number is as printed by the company named beside it; nothing is ranked or rescored. Models are listed by developer, newest first.
Area Agentic CodingMaintainer OpenAIIntroduced 2025Data version 2026103001, as of 2026-10-30
SWE-Lancer tests whether a model can earn money doing real freelance software engineering. Its tasks are more than 1,400 jobs posted on Upwork for the Expensify codebase, worth one million US dollars in payouts together, from small bug fixes to feature builds. Individual contributor (IC) tasks are graded by end-to-end tests written by professional engineers; manager tasks ask the model to pick the best of several proposals. The public subset is called Diamond.
SWE-Lancer IC Diamond
| Model | Value | Printed by | Setup as printed |
|---|---|---|---|
| GPT-5.3-CodexOpenAI, released 2026-02-05 | 81.4% | OpenAI, Introducing GPT-5.3-Codexchecked 2026-10-15 | Effort: xhigh |
| GPT-5.2-CodexOpenAI, released 2025-12-18 | 76% | OpenAI, Introducing GPT-5.3-Codexchecked 2026-10-15 | Effort: xhigh |
| GPT-5.2OpenAI, released 2025-12-11 | 74.6% | OpenAI, Introducing GPT-5.2checked 2026-10-15; printed on 1 more page | Effort: xhigh; Dataset revision: 197 of 237 IC Diamond problems (40 omitted) |
| GPT-5.1OpenAI, released 2025-11-12 | 69.7% | OpenAI, Introducing GPT-5.2checked 2026-10-15 | Effort: high; Dataset revision: 197 of 237 IC Diamond problems (40 omitted) |
Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.