The coin-flip that wasn’t

The symptom

A single SupportFlow run wasn’t a score — it was a noisy sample. Re-running the same model on the same bench flipped pass↔fail with no code change. Averaged across the suite it sat around 44%, but the average hid the real problem: you couldn’t hand a model a task and trust the result. That’s useless for a benchmark, and worse for the actual goal — giving these models real bugs to fix.

The investigation — retries didn’t help

If a bench is a genuine 50/50 coin, best-of-3 should solve it ~88% of the time. We ran exactly that experiment on six coin-flip benches. Best-of-3 solved one of six. That result is only possible if the failures are correlated — the model makes the same mistake every attempt. So the fix was never “retry more.” The transcripts showed why: four of six failures ended with “abandoned: explored but never attempted an edit” — the model kept trying to write its fix, and every write bounced.

The root cause — a path the harness threw away

The model addressed files the way the codebase itself does: /app/app/Services/Mailer.php — a Laravel/Docker absolute path where /app is the container root. Our sandbox joined that against the repo:

Every /app/-style edit was silently refused as “outside the repo,” so the agent explored, got told “you haven’t edited anything,” tried the same path again, and eventually gave up. The one bench that did solve? On its lucky attempt the model happened to emit a relative path — which the harness accepted. The “coin-flip” was a path-format lottery the harness itself created.

The fix — harness v1.7.2

Crucially: no timeouts loosened, no step caps raised, no hard benches scoped out, no model-specific hacks. Every change removes harness friction and lifts every model equally. v1.7.2 is its own frozen engine version — the old v1.6.1 baseline stays untouched for comparability.

The payoff — the whole board moved

Same models, same 32 benches, previous harness (v1.7.1-both) versus the fixed one (v1.7.2-both). Grey = before, blue = after.

Models that scored zero on SupportFlow weren’t incapable — they were failing on the path bug. Once it was gone, qwen3.6 (all three variants) cleared ~78%, devstral 56%, and a spread opened up that actually reflects capability. The two smallest models still score zero: that’s a real floor, not friction.

How we found it, and why it matters

The lesson wasn’t just the bug — it was the method. A single agentic run is a noisy sample, so we stopped trusting single runs, ran a deliberate best-of-N experiment to quantify the retry lever, read the actual transcripts instead of guessing, and let the failure mode tell us what to build. The fix that followed was small, surgical, and it moved everything.