MCP Atlas: Results as Published

MCP Atlas tests whether a language model can use real Model Context Protocol servers to answer multi-step questions. The prompts do not name the servers, tools or parameters, so the model has to find the right tools itself, call them correctly, recover from errors and synthesize a final answer; the paper describes 36 real MCP servers and 220 tools, and tasks span several servers. The benchmark has 1,000 human-authored tasks, and Scale publishes a 500-task public split while holding the other 500 back to protect leaderboard integrity. Each ground-truth answer is split into factual claims, an LLM judge (a Google model at temperature 0.0) scores each claim as 1, 0.5 or 0, and a task passes at 75% claim coverage. The metric that ranks models is the pass rate.

Every number here is read from the same catalog file as the benchmark table, data version 2026103001, and is the number the table shows for that model and test. Where one page printed a model under several setups, each value is listed. The catalog is public in armanobosyan/sugra-llm-benchmarks; a wrong number can be reported from the table, with Report an error under the value, or as an issue in that repository.