IMR R&D · the origin · May 2026

A home lab that builds its own software

Before Touchstone was a capability bench, it was a wager: could a team of local language models run an autonomous development shop — finding their own weak spots, writing the fixes, and opening the pull requests — on a single GPU sitting on a desk? This is where it started.

14
local models evaluated — Qwen3, Gemma3, Mistral, Phi4, DeepSeek, Llama, Falcon
1 GPU
all inference on one GB10 superchip — 128 GB unified memory, no cloud
100
GitHub issues worked in the first autonomous sprint
0
lines of human code — agents owned the fix → test → PR loop end to end

The bet

The whole system was designed around a single, deliberately harsh constraint: everything runs locally, on hardware we already own. No frontier API doing the real work, no rented H100s. One consumer superchip, one orchestration box, and a stack of small open models. Every architectural decision — the load watchdog, the concurrency throttle, the one-PR-per-issue rule, the 128K context cap — falls out of that one constraint.

If a weak model on cheap hardware could be made to do real engineering, the win wouldn’t be a bigger bill — it would be a blueprint anyone could run.

Two machines, one job

The pipeline lives across two boxes on a home LAN. A fanless Intel NUC is the brain — it owns orchestration, the database, and the scheduler, but has no GPU. It delegates every token of inference over HTTP to the GX10: NVIDIA’s Grace Blackwell GB10, an ARM CPU and a Blackwell GPU sharing one 128 GB pool of unified memory.

Hardware architecture ORCHESTRATION NODE Intel NUC Ubuntu 22.04 · no GPU · 192.168.0.5 • bench_api — drives every benchmark run • PostgreSQL — issues, scores, agent memory • Paperclip scheduler — dispatches agent runs • pf_wrapper — timeout + progress streaming • SSH gateway — all ops route through here INFERENCE NODE NVIDIA GX10 · GB10 Grace Blackwell · 192.168.0.7 128 GB unified memory — CPU + GPU, one pool • LM Studio :1234 — OpenAI-compatible API • lm_watchdog :7777 — the load gate • models up to ~70B fit — OOM = full halt HTTP · LAN prompts out · tokens back

The most dangerous failure mode

Unified memory has a sting in the tail: there is no separate VRAM to watch, so memory pressure is invisible until it isn’t. Load two large models at once, exceed the pool, and the machine doesn’t throw an error — it freezes solid and needs a physical reboot. With agents firing autonomously, that risk is constant.

The answer was a tiny gatekeeper: lm_watchdog, a lightweight HTTP server that allows exactly one model load at a time. Everything that wants to load a model has to ask it first.

The OOM watchdog load gate load request from bench_api gate open? one load only 200 — load it tell LM Studio to load 409 — busy retry with backoff no double-load no OOM halt

The recover endpoint that clears a stuck gate has its own guard: if a benchmark is mid-flight it refuses, so a runaway retry can never clear the gate and let a second model in while inference is still running.

The pipeline — five steps, fully automated

A benchmark run is a fixed five-step loop. The orchestrator drives it; the watchdog gates the load; and the final step closes the loop by feeding results back into the agents that decide what to fix next.

The benchmark pipeline 1 · Load gated by watchdog 2 · Run prompts via chat API 3 · Score Scorer rates 0–100 4 · Write Postgres + leaderboard 5 · Improve agents pick the fixes weak spots become the next sprint’s issues

The crew — Paperclip Forge

The agents wake when an issue lands in their queue, read their instruction file, do the work, and go back to sleep. Those instruction files are the only memory they carry between runs. Three roles, each a different model picked for the job:

plans & reviews
CEO
Reads the scores, finds the weak spots, writes the issues, reviews every PR — and never merges with CI red. Writes no code itself.
Claude Sonnet · on the NUC
implements
CTO
Takes one issue, clones the repo, branches, writes the fix, opens the PR, comments the link. One branch, one PR, one issue — no batching.
Qwen3-next-80B · on the GX10
grades
Scorer
Reads each model response against the rubric and returns a 0–100 score with a one-line rationale. Full history kept, per model, per bench.
lightweight model · per run

The war story: gridlock

Early on, the CEO did the obvious thing and queued all 100 issues at once. The result was instructive: 20-plus CTO runs hit the single GPU together, every process stuck waiting for its turn, each timing out before it ever saw a first token. Nothing shipped. The fix was unglamorous and decisive — raise the per-run timeout, and stop queuing everything. Issues now sit blocked until the CEO releases them, three at a time, as slots free up.

Concurrency throttle BEFORE — 20+ AT ONCE 1 GPU 128 GB all waiting · all timing out · nothing ships AFTER — 3 IN FLIGHT 1 GPU 128 GB blocked — released one at a time

What it proved — and what came next

The first sprint shipped real pull requests against a real codebase with no human writing code — the loop closed. But the benchmarks that drove it (Hello World, Statistics, a CSV bugfix, a rate limiter) were toy problems, and a 0–100 judge is a soft ruler.

B1 Hello World
B2 Statistics
B3 CSV bugfix
B4 Rate limiter
This was the starting point: an autonomous dev pipeline on one consumer GPU. The lesson it handed us — that the bottleneck is rarely the model and usually the scaffolding around it — reframed the whole project. The toy benchmarks gave way to a real production codebase graded by its own test suite, the soft 0–100 judge gave way to mechanical pass/fail, and the question narrowed from “can a crew of models run a shop” to “how good can we make one weak model with engineering alone.” That is the next chapter — and the bench you see today.

IMR R&D · project journal · origin chapter, May 2026 · re-skinned from the original LMEH writeup.