The bench grows up

The board, today

The capability bench is the gauntlet: 33 real bugs from a production Laravel ticketing app, each graded mechanically by the app’s own PHPUnit suite — no soft 0–100 judge, just pass or fail. Twelve models have run it on the frozen v1.6.1 harness. The lead is a lean 35B mixture-of-experts; the tail is still filling in as runs complete.

The result that keeps proving itself

The headline isn’t the leader — it’s the 4-billion-parameter model sitting third, ahead of a 26B and a 80B. With the right scaffolding, capability stops tracking parameter count as tightly as you’d expect. The harness lifts everyone; the model still matters, but the floor moved, and it moved most for the smallest models.

The hardware frontier — and its ceiling

The obvious next question: if scaffolding lifts small models, does a second GPU let us run a much bigger one? We linked two GX10 superchips over a 200 GbE fabric — roughly 256 GB pooled — and tried. The answer was a clean, useful no. Distributing a 35B across both boxes runs at full single-box speed (no penalty), but frontier-size weights don’t fit the way the headline number suggests: a 235B at 8-bit wedges the cluster, its 4-bit shrink loads but sits on the ragged edge of memory and hard-hangs under load, and a 128B contender stalls the same way.

A second box buys context, throughput and redundancy — not “run a model twice the size.” The realistic target is the biggest model that fits one box at 8-bit with room for its context.

The gate got teeth

Before any model reaches the gauntlet it has to clear the gate: 16 deterministic, temperature-zero probes run five times each, scored strict. They’re harness-invariant — pure capability, no scaffolding — covering basic vitals, tool use, output fidelity, and a new fifth group this month: agentic discipline, the habits that decide whether a model can survive an autonomous loop at all.

Closing the loop

And the harness no longer just benches in a clean room — it runs inside the production app. The same engine that scores the gauntlet now drives a development agent live in Paperclip: it picks up a real issue, fetches the context itself, makes the surgical edits, runs the suite, and only reports done when it’s green. The bench and the product finally share one brain.