Ongoing
Runs are still going. The loop's numbers below are a checkpoint, not a final result, and they are Humanfia-reported: they are not on the ProgramBench leaderboard.
ProgramBench hands an agent a compiled program and its documentation and asks it to rebuild the program from scratch, with no source, no decompiler and no internet. A task counts as resolved only when the rebuild passes every hidden behavioral test.The leaderboard also reports "almost resolved", at least 95% of the tests passed. Every number on this page is the strict figure, fully resolved. Across its 200 tasks, single-shot numbers sit close to the floor. That makes it an unusually clean instrument for measuring a loop.
Together · 3.5%
7Tasks of 200 resolved by Opus-4.8 building and GPT-5.5 reviewing.
GPT-5.5 alone · 0.5%
1Alone, as on its official leaderboard entry at xhigh effort.
Opus-4.8 alone · 0%
0Alone, as on its official leaderboard entry at xhigh effort.
Two models that solve almost nothing separately solve seven times the better one's share when they work as a builder and an independent reviewer, round after round. Across tasks, the loop averages a 14.2% improvement over the single-shot baseline.
The loop
The arrangement is a Ralph loop with a reviewer in it, the shape RLCR Flow was built around: one agent builds, a second model reads what it built, and the review goes back as the next round's work. Neither model is changed. Only who is asked what, and in what order, is.
Swipe the diagram sideways to see all of it →
1/0 Opus-4.8 builds: a program, from a binary and its documentation.
Where 3.5% would sit
The official leaderboard runs every model alone, in the same scaffold, mini-SWE-agent. Our loop is a different setup and is not on it. Set beside the board for scale, 3.5% falls between the top entry and everything else.
- 1Claude Opus 5 (xhigh)4.5%
- 2Opus-4.8 ⇄ GPT-5.5 loopHumanfia-reported3.5%
- 3Muse Spark 1.3 (max)2.5%
- 4GPT-5.6 Sol (xhigh)1.0%
- 4Muse Spark 1.3 (xhigh)1.0%
- 6GPT 5.5 (xhigh)the reviewer, alone0.5%
- 7Claude Opus 4.8 (xhigh)the builder, alone0.0%
The part that bothers us
In the later rounds, performance was still rising at roughly 1.2% per round when the runs were cut off.
That is not a result. It is a missing one. The runs stop at a four-hour time budget, not at convergence, so what we have measured is partly the budget rather than the method. The honest statement is this: the number at four hours is 3.5%, the slope at four hours is positive, and we do not yet know where it goes.
Finding that out, rather than guessing, is what FlowBench is for.