MBPP: Results as Published

MBPP, Mostly Basic Programming Problems, asks a model to write a short Python program from a natural language description. The dataset has 974 crowd-sourced tasks meant to be solvable by entry-level programmers. Each task includes test cases, and the prompts consistently contain three input-output examples written as assert statements. A task counts as solved when the synthesized program passes those tests, and the paper reports the share of problems solved.

Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.