ProgramBench

What it measures

ProgramBench was introduced by researchers at Meta Superintelligence Labs, Stanford University and Harvard University to test whether coding agents can design and implement complete programs from scratch. Its 200 tasks range from CLI utilities such as jq and ripgrep to SQLite, FFmpeg and PHP, with more than 248,000 behavioral tests in total.

Think of receiving a machine without its blueprints: you can press buttons, observe the outputs and read the manual, but cannot look inside; the agent must build a replacement with matching observable behavior. Agents can execute the reference binary but cannot access its source, decompile it or retrieve implementations from the internet; they must choose the language, architecture and module boundaries and submit source code with a build script.

Hidden tests compare the candidate with the original, and a task counts as resolved only when every behavioral test passes. Unlike DeepSWE tasks that modify an existing repository, this benchmark emphasizes starting from a blank slate, discovering requirements and verifying completeness; the official leaderboard currently uses mini-SWE-agent, so results reflect both the model and that scaffold.

Reading the scores

Resolved is the share of tasks passing every hidden behavioral test, not the average test pass rate. Resolving 9 out of 200 programs gives 4.5%, even if many other programs pass numerous individual tests. Almost is the share of tasks passing at least 95% of tests and includes fully resolved tasks, so the two percentages must not be added; it is an auxiliary measure of partial progress. This snapshot contains all 21 model configurations on the official September 9, 2026 leaderboard, retaining published effort settings; the default order is Resolved, then Almost, then average test pass rate. Low scores reflect the strict requirement to recreate complete programs, and 0% does not mean no coding ability; finite tests cannot prove equivalence for every possible input. Factory’s study in our selected readings uses a curated subset of 24 tasks and different execution setups, so its results must not be mixed with this full 200-task leaderboard.

SourceProgramBench · Meta / Stanford / Harvard

Related reading: Factory on completing large software tasks