IMR R&D · local model evaluation harness

localharness

turning small, locally-hosted models into agents that fix real production bugs — on hardware you already own.

345
steps to fix a real bug, once the harness hands the model the spec (same model)
5/8
real production bugs solved by a 4-billion-param model on the live-app bench
$0
per-token cost — runs entirely on owned local hardware

how it works — the agent loop

The agent loop task + tools the job + the failing test model acts one tool call harness runs it + hands back result tests pass → done finish observation feeds the next action

the scalpel toolkit

grep — find code
read — ranged + full
edit — surgical, not rewrite
write — new files only
test — the real suite
finish — only when green

the feedback that makes weak models work

v1.6 · give it the spec
seed the failing test
hand the model the exact test it must pass — so it edits instead of guessing. (qwen3.6/195: 34-step loop → 5-step fix.)
v1.4 · learn from failure
evidence-carrying retries
a failed attempt's diff + test output is handed to the next try — retries build on what didn't work.
v1.5 · stop the loops
repeat awareness
re-reading a file you already have returns "you've got this" — not a silent duplicate.
v1.1 · progress, not punishment
progress-gated rails
nudge a stuck model back on track; only abandon when there's genuinely no progress.

measured, versioned, never guessed

v1.0
scalpel tools
v1.1
progress rails
v1.2
nudge — rejected
v1.3
best-of-N
v1.4
evidence retries
v1.5
repeat aware
v1.6
seed the spec
Every change is git-tagged, frozen, and scored on a live Laravel app graded by its own test suite — adopted only if it lifts the board. Capability, not crutches.