Skip to content
← All posts

Blog1 min read

Model, Tool, Flow2, 18, 50 of 50

The same model, measured three ways (raw API, the vendor's CLI, and that CLI inside a flow) on PutnamBench, Physics Cup, SuperChem and an HLE subset. Putting the CLI inside a flow adds more than the CLI's own tools do.

There is a claim we keep making on this site: the loop around the agent is worth a large multiple on hard work. It is easy to assert and annoying to measure. This is the cleanest version of the measurement we have.

One model. Three ways of running it. Four benchmarks.These are Humanfia-reported numbers from our own runs. The KDA² post later repeated the ablation with four newer models, and found the same pattern for every one of them.

Three levels · one model
Flow50Tool18Model2solved of 50

Swipe the diagram sideways to see all of it →

1/0 Model level: the API, called directly. 2 of 50.

One GPT model · three levels of scaffolding
2−16
18
50+32
PutnamBench
Switch benchmarks, and choose a baseline to read the gain over it. Tool level on PutnamBench was measured without the comparator gate.

Reading the table ​

Tools are worth a lot. Giving the model a shell, a file system and a compiler multiplies it on the formal benchmarks: 2 to 18 on PutnamBench. Nobody disputes this, and it is why coding CLIs exist.

The flow is worth more. From 18 to 50 on PutnamBench, and from 42% to 63% on the HLE subset, is not a refinement of the tool-level result. It is a different regime. It comes from holding the model and the tools fixed and changing only who is asked what, in what order, and when the run is allowed to stop.

The flow does not make the model smarter. It stops the run from wasting what the model already has.

The saturation is informative too. Put every benchmark on one scale and the ladders differ. PutnamBench climbs from 4% to 100%. Physics Cup goes 22 → 33 → 40 of 50, a real gain that is visibly running into something the loop cannot fix.

Share of each benchmark solved, by level

Not every benchmark is loop-limited

Swipe the plot sideways to see all of it →

Not every benchmark is loop-limited. A method that claimed to help everywhere equally would be describing something other than what it does.

That is what Humanize is, in the least glamorous sense available: a model-ability booster, worth the most where the model has ability the run was wasting.

The flows · the runtime · the same ablation, four models later

Read next

News0% + 0.5% → 3.5% on ProgramBenchAlone, Opus-4.8 fully resolves 0% of ProgramBench and GPT-5.5 resolves 0.5%. As a builder and a reviewer in one loop they resolve 3.5%, close to the best official entry's 4.5%, and the curve was still rising when the four-hour budget ran out. A Humanfia-reported checkpoint.Read it News17,520 QCode definitions across 21 latticesQCode Discovery has searched 17,520 quantum-code definitions over 21 lattices and kept 5,178 that could win. Now nothing counts as a win without an exact construction, an independent check and a fresh replay of the IBM baselines. Humanfia-reported.Read it BlogFlow Science: A New Dimension of Scaling AgentsModels scale with parameters and data, and agents scale with tools; the next dimension is the flow. Why one flow cannot solve every problem, flow-level ablations on four benchmarks with the models held fixed, four findings we did not expect, and whether a flow is engineering or a science.Read it