ProgramBench: Results as Published

ProgramBench tests whether a coding agent can rebuild a program from scratch. Each of its 200 tasks, drawn from open-source GitHub repositories and ranging from small command-line tools to projects such as SQLite and FFmpeg, gives the agent a reference executable and its documentation and asks for a codebase that matches the reference behaviour, checked by hidden behavioural tests. A task is resolved only when every test passes; a near-perfect category counts implementations that pass most tests, which model cards print as "Almost".

Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.