IMR R&D · harness · A/B & regression · July 2026

When more scaffolding made it worse

The board always showed each model's best score across every harness version — a fair "capability ceiling", but one that quietly hides the thing we most need to see: when a harness change makes a model worse. So we built an A/B view. It caught a real regression on day one.

The blind spot

what we showed
Best-across-harness
A bench counts as solved if it ever passed on any harness. Great for "what's the ceiling" — but a pass on the old harness papers over a fail on the new one.
what it hid
Silent regressions
Ship a harness change that lifts most models but breaks one bench on one model, and the board never flinches. The union absorbs it.
what we needed
A clean A/B
Hold everything constant, change one harness version, and count what moved — up and down — on the benches that ran on both.

The A/B view — only what ran on both counts

Harness A e.g. v1.6.1 Harness B e.g. v1.7.1 shared benches only fail → pass = improved pass → fail = regressed no-signal runs excluded net delta, per model & bench the exact benches that flipped, both ways

The catch — qwen3.6-35b-a3b on SupportFlow

v1.6.1 (base)
18 / 29
Solid. The plain harness, no injected scaffolding.
v1.7.1 (agents+skills)
7 / 30
Eleven benches it used to solve now fail. Not infra noise — real agentic runs, real tokens.
the signature
Circling
The failing runs burn 17–40 iterations, several hitting the 40-step ceiling. The extra scaffolding made it thrash instead of converge.
The honest caveat: this isn't yet a clean verdict. The v1.6.1 numbers are from a run nine days older than the v1.7.1 run, so engine and tooling changes are tangled up with the harness change. Before we call it, we re-run base v1.6.1 on today's engine — if it still scores ~18, the agents+skills bundle is genuinely to blame; if it also drops, it was drift. Measure, don't assume.
Why it matters: the whole project rests on the idea that scaffolding lifts weak models. This is the counter-case made visible — more scaffolding can make a model worse, and now we can see exactly when, on which bench, by how much. The A/B view turns "did we improve?" from a hunch into a signed number, with the regressions we used to hide surfaced in red.

IMR R&D · harness note · the A/B comparison view + the first regression it caught, July 2026.