SciCode: Results as Published

SciCode tests whether language models can write code that solves real research problems in the natural sciences. The benchmark has 80 main problems split into 338 subproblems, each combining knowledge recall, reasoning and code synthesis, across 16 natural-science subfields including mathematics, physics, chemistry, biology and materials science. The problems were curated with scientists, and scientist-annotated gold-standard solutions and test cases are used for evaluation. The metric is the percentage of problems solved, and the best model tested in the paper's most realistic setting solved 4.6% of problems.

Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.