Harnesses
The scaffolding around the model — the only variable that moves on the supportflow gauntlet. Columns run in ladder order; cells are coloured by lift vs baseline with the vs-previous-rung gain inline (▲/▼); each model's best cell is ringed. Everything recomputes live as we bench more.
Harness impact — did it improve or regress?
Pick two harnesses. Only benches a model ran on both count, so this is a clean A/B: fail→pass is a real improvement, pass→fail is a regression the best-across-harness board silently absorbs.
compare→
Main harnesses
The model-leaderboard harness — light, single-attempt scaffolding.
why: The control line: minimal help so the leaderboard reflects the model, not the harness.
v1.6 + the big-write parser fix — the current frozen harness; all clean comparisons run here.
why: A stable, frozen reference so model×harness numbers stay comparable.
v1.6.1 against the 2×GX10 pooled-memory cluster (llama.cpp) — same logic, bigger models.
why: Bench frontier-size models that don't fit one box, without changing the harness.
Top models × main harnesses (top 10 by best-anywhere; Sonnet = ceiling)
Sonnet ceiling pending — its gauntlet baseline is incomplete (0/22 run), so the ceiling row + gap column stay hidden to avoid a false "local beats Sonnet" read. Both appear automatically once the Sonnet baseline run lands.
All harnesses
Current line
The model-leaderboard harness — light, single-attempt scaffolding.
why: The control line: minimal help so the leaderboard reflects the model, not the harness.
The agentic loop — explore the repo, make surgical edits, run the failing test, iterate.
why: The core strategy every localharness version builds on (grep / ranged+full read / surgical edit).
v1.6 + the big-write parser fix — the current frozen harness; all clean comparisons run here.
why: A stable, frozen reference so model×harness numbers stay comparable.
v1.6.1 against the 2×GX10 pooled-memory cluster (llama.cpp) — same logic, bigger models.
why: Bench frontier-size models that don't fit one box, without changing the harness.
v1.7 with no injection — the control for the mdskills experiment.
why: The 'none' bundle baseline to measure injection lift against.
v1.7 + the AGENTS repo-map injection.
why: Test whether a repo map alone lifts the model.
v1.7 + skill-recipe injection (mdskills) — procedural scaffolding for HOW to approach the fix.
why: Measure whether injected procedure pushes models past the 40-step circling.
Per-domain thinking control: reasoning models run NO-THINK (prefilled empty think block) on gate + coding/C#/PHP ladders, and keep FULL thinking on SupportFlow.
why: Reasoning models over-think toy benches and time out; thinking earns its keep only on the hard agentic domain. Split measures real capability.
v1.8 with agents+skills bundle (both)
repo_agent engine with _safe_path recovery for absolute/container paths, write-to-existing overwrite escape hatch, and a 12k per-call output cap.
why: SupportFlow 'variance' was largely a path-resolution bug: models used /app/... absolute paths the harness rejected, so edits bounced and the agent gave up. Fixing it turns coin-flips into reliable first-pass solves.
Superseded rungs
No tools, no scaffolding — one-shot answer on the task.
why: The absolute floor: what a model does with zero help.
retired: Showed the bottleneck was the toolkit, not the model — replaced by the scalpel tools. → v1.1
An early plain harness variant.
why: Legacy baseline.
retired: Replaced by the canonical baseline. → baseline
Single attempt on the real repo — no iterate/test loop.
why: Baseline for repo tasks before the agentic loop.
retired: Superseded by repo_agent's explore→edit→test loop. → repo_agent
Early dev-cycle strategy (clone→branch→fix→PR) from the Paperclip Forge era.
why: First autonomous fix loop, pre-supportflow-gauntlet.
retired: Superseded by the repo_agent gauntlet loop. → repo_agent
Early structured test-feedback experiment.
why: Precursor to evidence-carrying retries.
retired: Folded into v1.4. → v1.4
Progress-gated rails — nudge a stuck model, only abandon on genuine no-progress.
why: Stop weak models defeating themselves without punishing exploration.
retired: Rolled forward into the current line. → v1.6.1
A stronger nudge to push exploration → committing an edit sooner.
why: Tried to cut the explore-without-acting loop.
retired: Rejected — the nudge hurt as much as it helped; reverted. → v1.3-bo3
Best-of-3 — run up to 3 attempts, a test oracle picks the winner.
why: Give weak models more shots at a passing diff.
retired: Folded into the current line. → v1.6.1
Evidence-carrying retries — a failed try's diff + test output is handed to the next try.
why: Retries should build on what failed, not restart blind.
retired: Folded into v1.6.1. → v1.6.1
Repeat awareness — re-reading a file you already hold returns 'you've got this', not a silent dup.
why: Kill wasteful repeat-actions / circling.
retired: Folded into v1.6.1. → v1.6.1
Seed the failing-test SOURCE into the prompt — the exact spec, so the model edits instead of guessing.
why: Don't make the model re-read source to guess the target; hand it the target.
retired: Superseded by v1.6.1 (parser fix). → v1.6.1
External calibration
OpenHands — an external third-party agent scaffold.
why: Calibration: how our harness compares to a known external agent.
mini-swe-agent, blind variant — external reference run.
why: Calibration against a known minimal agent.
mini-swe-agent, default variant — external reference run.
why: Calibration against a known minimal agent.
Legacy
Full harness × model matrix (every harness × every model)
| Model | hermes | best |
|---|---|---|
| qwen3.6-35b-a3b | 6/22 | 6 |
| hermes-4.3-36b | 4/22 | 4 |
| gemma-4-26b-a4b-it | 3/22 | 3 |
| qwen3-next-80b | 3/15 | 3 |
| devstral-small-2-2512 | 2/3 | 2 |