SWE-bench Pro: Results as Published

SWE-Bench Pro tests AI agents on realistic, long-horizon software engineering tasks in real codebases, often requiring multi-file patches that a professional engineer might take hours or days to finish. It has 1,865 problems from 41 actively maintained repositories, split into a public set, a held-out set and a commercial set from startup partners. Results are reported as Pass@1 under a unified scaffold, and a patch counts only if it passes the verifying tests.

Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.