There is a claim we keep making on this site: the loop around the agent is worth a large multiple on hard work. It is easy to assert and annoying to measure. This is the cleanest version of the measurement we have.
One model. Three ways of running it. Four benchmarks.These are Humanfia-reported numbers from our own runs. The KDA² post later repeated the ablation with four newer models, and found the same pattern for every one of them.
Swipe the diagram sideways to see all of it →
1/0 Model level: the API, called directly. 2 of 50.
Reading the table
Tools are worth a lot. Giving the model a shell, a file system and a compiler multiplies it on the formal benchmarks: 2 to 18 on PutnamBench. Nobody disputes this, and it is why coding CLIs exist.
The flow is worth more. From 18 to 50 on PutnamBench, and from 42% to 63% on the HLE subset, is not a refinement of the tool-level result. It is a different regime. It comes from holding the model and the tools fixed and changing only who is asked what, in what order, and when the run is allowed to stop.
The flow does not make the model smarter. It stops the run from wasting what the model already has.
The saturation is informative too. Put every benchmark on one scale and the ladders differ. PutnamBench climbs from 4% to 100%. Physics Cup goes 22 → 33 → 40 of 50, a real gain that is visibly running into something the loop cannot fix.
Not every benchmark is loop-limited
Swipe the plot sideways to see all of it →
Not every benchmark is loop-limited. A method that claimed to help everywhere equally would be describing something other than what it does.
That is what Humanize is, in the least glamorous sense available: a model-ability booster, worth the most where the model has ability the run was wasting.
The flows · the runtime · the same ablation, four models later