Skip to content
← All news

News2 min read

0% + 0.5% → 3.5% on ProgramBench

Alone, Opus-4.8 fully resolves 0% of ProgramBench and GPT-5.5 resolves 0.5%. As a builder and a reviewer in one loop they resolve 3.5%, close to the best official entry's 4.5%, and the curve was still rising when the four-hour budget ran out. A Humanfia-reported checkpoint.

Ongoing

Runs are still going. The loop's numbers below are a checkpoint, not a final result, and they are Humanfia-reported: they are not on the ProgramBench leaderboard.

ProgramBench hands an agent a compiled program and its documentation and asks it to rebuild the program from scratch, with no source, no decompiler and no internet. A task counts as resolved only when the rebuild passes every hidden behavioral test.The leaderboard also reports "almost resolved", at least 95% of the tests passed. Every number on this page is the strict figure, fully resolved. Across its 200 tasks, single-shot numbers sit close to the floor. That makes it an unusually clean instrument for measuring a loop.

Together · 3.5%

7

Tasks of 200 resolved by Opus-4.8 building and GPT-5.5 reviewing.

GPT-5.5 alone · 0.5%

1

Alone, as on its official leaderboard entry at xhigh effort.

Opus-4.8 alone · 0%

0

Alone, as on its official leaderboard entry at xhigh effort.

Two models that solve almost nothing separately solve seven times the better one's share when they work as a builder and an independent reviewer, round after round. Across tasks, the loop averages a 14.2% improvement over the single-shot baseline.

The loop ​

The arrangement is a Ralph loop with a reviewer in it, the shape RLCR Flow was built around: one agent builds, a second model reads what it built, and the review goes back as the next round's work. Neither model is changed. Only who is asked what, and in what order, is.

Builder ⇄ reviewer · one round
Opus-4.8buildsGPT-5.5reviewsthe rebuildthe review

Swipe the diagram sideways to see all of it →

1/0 Opus-4.8 builds: a program, from a binary and its documentation.

Where 3.5% would sit ​

The official leaderboard runs every model alone, in the same scaffold, mini-SWE-agent. Our loop is a different setup and is not on it. Set beside the board for scale, 3.5% falls between the top entry and everything else.

ProgramBench · fully resolved · official board, 28 September 2026, plus our loop
  1. 1Claude Opus 5 (xhigh)4.5%
  2. 2Opus-4.8 ⇄ GPT-5.5 loopHumanfia-reported3.5%
  3. 3Muse Spark 1.3 (max)2.5%
  4. 4GPT-5.6 Sol (xhigh)1.0%
  5. 4Muse Spark 1.3 (xhigh)1.0%
  6. 6GPT 5.5 (xhigh)the reviewer, alone0.5%
  7. 7Claude Opus 4.8 (xhigh)the builder, alone0.0%
Every entry but ours is from programbench.com, run alone in mini-SWE-agent. Ours is a checkpoint at four hours.

The part that bothers us ​

In the later rounds, performance was still rising at roughly 1.2% per round when the runs were cut off.

That is not a result. It is a missing one. The runs stop at a four-hour time budget, not at convergence, so what we have measured is partly the budget rather than the method. The honest statement is this: the number at four hours is 3.5%, the slope at four hours is positive, and we do not yet know where it goes.

Finding that out, rather than guessing, is what FlowBench is for.

ProgramBench · The flows · FlowBench

Read next

News17,520 QCode definitions across 21 latticesQCode Discovery has searched 17,520 quantum-code definitions over 21 lattices and kept 5,178 that could win. Now nothing counts as a win without an exact construction, an independent check and a fresh replay of the IBM baselines. Humanfia-reported.Read it BlogModel, Tool, Flow: 2, 18, 50 of 50The same model, measured three ways (raw API, the vendor's CLI, and that CLI inside a flow) on PutnamBench, Physics Cup, SuperChem and an HLE subset. Putting the CLI inside a flow adds more than the CLI's own tools do.Read it News#1 on Lean-Eval v1 first solvesLean-Eval froze its first 128-problem release and moved everything older to an archive. On v1 our GPT-5.6 flow has the most first solves, 30, and is one behind the leader on total; on the archive it has 170 of 171.Read it