A lot landed in a short stretch and barely got written down. This is the catch-up: the platform rebuild, the run queue, the harness tooling, the data clean-up, and the hardware experiments — grouped so nothing gets lost. The headline (a one-line harness bug that was capping every model) has its own note; this is everything around it.
1 · The platform — a real v2
The bench went from a single-file dashboard to a production stack: Postgres 16 as the source of truth, a FastAPI ingest service, a Next.js front end, HTTPS via Caddy. Every tunable now lives in the database, never a file on disk.
data
SQLite → Postgres
Migrated with parity checks, single /ingest route, scoring weights live-editable from the UI.
front end
The whole site
Landing, leaderboard (3 boards), the gauntlet-as-a-journey, benches & harness detail pages, a theme switcher and a 3D timeline.
models
Profiles & census
Per-model pages with a capability “starfish” radar, alias roll-up, and a weekly census that tracks what changed.
2 · The run queue & orchestration
Benches are no longer hand-launched. A request goes into a queue; a serial, lock-gated worker loads the model and runs it, one at a time, and reports when done.
Request form · model + bench + harness pickers
Serial worker · one model at a time, box-busy aware
ETA per run + queue total
Reorder / pause / resume
Scheduling (run at a set time)
Email + summary on finish
Admin token · manage from mobile
Watchdog v4 · prod/bench modes, OOM guard
3 · Harness tooling & comparison
The bench’s core question is “is the model the only thing that changed?” — so the harness became a first-class, versioned, comparable object.
A/B
Harness impact view
Pick two harness versions, see exactly which benches improved, regressed, or held — so a “fix” can’t quietly cost you elsewhere.
injections
Bundles from the DB
Agent/skill injections live in Postgres and are materialised in front of the runner — the harness itself stays pure.
reasoning
Thinking on/off
Handling for reasoning models (thinking toggles, domain split) and a planned Compliance/Steerability axis — how hard a model is to actually deploy.
4 · Data integrity & board hygiene
A benchmark is only as good as its data. Several classes of noise were found and removed — reversibly, with backups.
ingest guard
Infra errors ≠ capability
Server-down, model-unloaded and 5xx responses no longer count as bench fails — 455 bad rows purged, and strong models hit the ceilings you’d expect.
quant
Each quant is its own row
Q8 vs BF16 of the same model are separate board entries with a coloured badge — you’re comparing like with like.
context
Board = 64k & 128k only
Sub-64k runs archived off the live board so context isn’t a hidden variable.
phantoms
Mislabels swept
A degenerate model auto-blocked at the ingest door; a mislabelled “phi-4” (263 rows that were really reasoning-plus) and a duplicate removed.
5 · Hardware — the two-box experiment
We linked two GX10s into one ~256 GB pooled brain to see whether frontier-size models were within reach, ran the giants on it, learned the ceiling, and tore it back down.
cluster
Two boxes, one brain
TCP over 200GbE (no RDMA), documented bring-up runbook, benched a 235B-Q4 and a 128B dense model across both boxes.
the ceiling
Fit-one-box wins
235B-Q8 wedges a 2-box cluster; the pragmatic target is models that fit one GX10 at Q8 with context. Cluster = optional headroom, not the plan.
6 · Documentation & reports
The Reports section was reorganised into a card index on a monthly timeline, and a run of infographics was produced — the cluster build, DB-driven injections, the harness A/B, the v2 integrity story, the request queue, and the reasoning/compliance thinking. This note is part of that catch-up.
The through-line of the week: make the bench trustworthy — one variable at a time, clean data, versioned harnesses, reproducible runs — so that when a number moves, it means something.