IMR R&D · project journal · the climb · June 2026

From a flat board to a climb

A day’s work that changed how the bench reads itself. The gauntlet stopped being a flat pass/fail grid and became a climb — gate, then ladders, up to the real-world flagship — with one quiet realisation baked in: a high score can just mean a model ran more, not that it’s better. Here’s what shipped.

6
stages in the climb — gate → coding → agentic → C# → PHP → supportflow
79
benches in the full stack a model can attempt across all stages
6
lazy model aliases merged into proper names — 2,288 results repointed
2×
GX10 cluster grinding the 122B run live, with a freshness-gated pulse

1 · The climb

The old gauntlet page was a wall of coloured squares — every model against every bench, no sense of direction. The new one tells the story of the project itself: as we build the stack, each model climbs it. First the deterministic gate (vitals, fidelity, tool-use — harness-invariant). Then the ladders, easiest first: a handful of self-contained coding fixes, then multi-step agentic plans, then the owned C# and PHP ladders. At the top sits supportflow — 33 real merged pull requests from a live SaaS codebase, graded by the app’s own test suite. The further right a model gets, the more it’s doing real engineering rather than toy puzzles.

It’s built to grow: a new ladder (Laravel and IDMS2025 are next) or extra gate probes appear in the climb automatically — no code change, just a row of config. Each stage shows the score on the latest harness a model has fully run, version-labelled, so a half-finished new harness never masquerades as a regression.

The climb — six stages, lead model fill THE CLIMB · qwen3.6-35b-a3b, SOLVED / STAGE SIZE Each rung is harder & more real than the last. The leader clears the early rungs, then the flagship bites. 11/11 Gate 5/6 Coding 9/9 Agentic 15/15 C# 5/5 PHP 19/33 Supportflow the real world — where it stalls

2 · Score rewards coverage. Capability is the hit-rate.

Ranking by raw benches-solved hides a trap: the model at the top might simply have run the most. So the climb now carries two more numbers next to the score — Cov (how much of the 79-bench stack a model has attempted) and Rate (solved ÷ attempted, its capability once it’s actually had its shot). The picture that fell out is the whole point of the bench.

The board leader, qwen3.6-35b, posts 65/79 — impressive, but on 100% coverage. Meanwhile gpt-oss-120b sits mid-table on score, yet has run only 57% of the stack and passes 93% of what it touches. llama-3.3-70b: 51% coverage, 90% rate. The two models that look ordinary are the sharpest once you account for how little they’ve been given.

Coverage vs capability COVERAGE × HIT-RATE · supportflow-stack low coverage ran the whole stack → 100% rate low rate COVERAGE — share of the 79-bench stack attempted under-run, high hit-rate — the ones to watch gpt-oss-120b llama-3.3-70b qwen3.6-35b leader · 100% / 82% the full-coverage pack — 95–100% cov, 54–71% rate gemma-4-26b · qwen3.5-4b ★ · devstral-small-2 · gpt-oss-20b · qwen3-next-80b · granite-4.1-30b · mistral-small-3.1

3 · Names, fixed at the source

Early runs were logged under lazy shorthand — qwen, gemma, devstral — while later runs used proper names like qwen3.6-35b-a3b. The board had been papering over the split with an alias roll-up. Today it was fixed in the data itself: six lazy keys merged into their canonical models, 2,288 results repointed, zero collisions, with a full table backup taken first. The bench now means exactly one thing by each name.

4 · The cluster, and a pulse that tells the truth

Two NVIDIA GX10 boxes are pooled over 200GbE into one ~240 GB memory space, running a 122B model through the same v1.6.1 harness the small models use — same logic, bigger brain. The live indicator is back too, but rebuilt: it only pulses when a run’s agent-loop feed has moved in the last 30 minutes, so a dead or stalled run goes quiet instead of lying. You can watch the cluster work a bench in real time, both on the climb and down in the full matrix.

Shipped
The climb
Gate → ladders → supportflow, per-model progress, auto-extends with new ladders.
Shipped
Coverage & rate
Score for breadth, hit-rate for capability — the under-run sharpshooters now visible.
Shipped
Info pages
Every bench and harness has its own page — runs, turns, tok/s, descriptions.
Still climbing. Next: a proper Sonnet yardstick run so the ceiling line is real (it’s suppressed today, on purpose, while its coverage is thin), plus new Laravel and IDMS2025 ladders — which the climb will pick up on its own. The bench keeps the harness as the only variable, and now it keeps itself honest about how much each model has actually been asked to do.

IMR R&D · Touchstone project journal · generated 29 June 2026 · touchstone.imrrd.com