The bench outgrew its first skin. We rebuilt it on Postgres + FastAPI + Next.js around a single ingest door — then spent the real effort making sure only trustworthy numbers ever get through it. A benchmark is only worth the discipline of the data behind it.
The v2 stack — one door in
Guards on the door — what never gets stored
infra ≠ capability
Server errors dropped
Model unloaded, connection refused, 5xx — never counted as a bench fail.
zero-output
No-signal guard
A run that returns no tokens, no steps, nothing — dropped, not stored as a failure.
block-list
_DROP_MODELS
Known-bad models bounce at the door, so they can’t creep back from any replay source.
controlled variable
Context 64k / 128k
One fixed context window across the board; anything lower is quarantined out.
The phantoms we hunted down
mislabel
phi-4 → reasoning-plus
Its config lms_id pointed at the reasoning-plus weights, so 263 "phi-4" results were actually a different model wearing the wrong name. Removed.
degenerate
glm-4.7-flash
A broken GGUF that reasons past its budget and never answers. It kept resurrecting from a replay source — so we blocked it at the ingest door for good.
duplicate
deepseek-r1-32b
An older key sharing one model’s weights, carrying 215 stale rows. Consolidated and removed.
A wrong lms_id silently benches the wrong model under the right name — the most dangerous kind of bad data, because every number looks plausible. The lesson: don’t chase bad rows one by one, guard the single door they all have to pass through, and audit the config that decides what actually runs.
Where it left the board: zero no-signal rows anywhere, every result at the standard context, phantoms gone and unable to return. A benchmark’s credibility isn’t its leaderboard — it’s whether you can trust that each cell means what it says. That’s the work that doesn’t show up in a score but underwrites all of them.
IMR R&D · platform note · the Postgres/FastAPI/Next.js rebuild and the data-integrity guards, July 2026.