The Gauntlet — the climb

The story of the project in one view: as we build out the stack — gate probes, then the ladders (coding, agentic, C#, PHP) up to the flagship supportflow gauntlet of real merged-PR issues — each model climbs as far as it can. Every stage shows the model's score on the latest harness it fully ran (labelled); the full per-harness detail is at the bottom. New ladders and harnesses appear here automatically as we add them.

Model	Score	Cov	Rate	Gate 11 benches	Coding 6 benches	Agentic 6 benches	C# 15 benches	PHP 5 benches	Supportflow 22 benches
qwen3.6-35b-a3b	49	100%	75%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	15/15/15 baseline	5/5/5 baseline	6/22/22 v0.18.0
gemma-4-26b-a4b-it	46	100%	71%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	15/15/15 baseline	5/5/5 baseline	3/22/22 v0.18.0
qwen3-next-80b	45	89%	78%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	14/15/15 baseline	5/5/5 baseline	3/15/22 ~v0.18.0
gpt-oss-20b	42	66%	98%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	14/15/15 baseline	5/5/5 baseline	—
claude-sonnet-4.6	41	66%	95%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	13/15/15 baseline	5/5/5 baseline	—
gpt-oss-120b	41	66%	95%	11/11/11 baseline	5/6/6 baseline	5/6/6 baseline	15/15/15 baseline	5/5/5 baseline	—
qwen3.6-27b	41	66%	95%	11/11/11 v1.7.1	6/6/6 baseline	6/6/6 baseline	13/15/15 v1.7.1	5/5/5 v1.7.1	—
qwen3.6-35b-a3b-or	41	66%	95%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	13/15/15 baseline	5/5/5 baseline	—
qwen3.6-35b-bf16	41	66%	95%	11/11/11 baseline	5/6/6 baseline	6/6/6 baseline	14/15/15 baseline	5/5/5 baseline	—
granite-4.1-30b	40	66%	93%	11/11/11 baseline	6/6/6 baseline	6/6/6 baseline	12/15/15 baseline	5/5/5 baseline	—
devstral-small-2-2512	39	71%	85%	11/11/11 baseline	5/6/6 baseline	6/6/6 baseline	11/15/15 baseline	4/5/5 baseline	2/3/22 ~v0.18.0
mistral-small-3.1-24b-instruct-2503	38	66%	88%	11/11/11 baseline	5/6/6 baseline	6/6/6 baseline	12/15/15 baseline	4/5/5 baseline	—
qwen3.5-122b-a10b-cluster	38	66%	88%	11/11/11 baseline	5/6/6 baseline	6/6/6 baseline	12/15/15 baseline	4/5/5 baseline	—
llama-3.3-70b	37	63%	90%	9/11/11 baseline	6/6/6 baseline	6/6/6 baseline	11/13/15 ~baseline	5/5/5 baseline	—
qwen3.5-4b	36	66%	84%	11/11/11 baseline	3/6/6 baseline	6/6/6 baseline	11/15/15 baseline	5/5/5 baseline	—
granite-4.1-8b	35	66%	81%	11/11/11 baseline	4/6/6 baseline	6/6/6 baseline	10/15/15 baseline	4/5/5 baseline	—
qwen3-coder-30b	35	66%	81%	10/11/11 baseline	5/6/6 baseline	6/6/6 baseline	12/15/15 baseline	2/5/5 baseline	—
gemini-2.5-flash-lite	32	66%	74%	11/11/11 baseline	4/6/6 baseline	5/6/6 baseline	9/15/15 baseline	3/5/5 baseline	—
qwen3.5-2b	25	66%	58%	9/11/11 baseline	2/6/6 baseline	6/6/6 baseline	6/15/15 baseline	2/5/5 baseline	—
deepseek-r1-distill-qwen-32b	24	66%	56%	7/11/11 baseline	6/6/6 baseline	1/6/6 baseline	10/15/15 baseline	0/5/5 baseline	—
qwen3-235b-a22b-q4-cluster	17	31%	85%	—	—	—	12/15/15 baseline	5/5/5 baseline	—
phi-4-reasoning-plus	16	66%	37%	4/11/11 baseline	4/6/6 baseline	2/6/6 baseline	5/15/15 baseline	1/5/5 baseline	—
qwen3.5-0.8b	14	66%	33%	9/11/11 baseline	0/6/6 baseline	5/6/6 baseline	0/15/15 baseline	0/5/5 baseline	—
phi-4	5	8%	100%	—	—	—	—	5/5/5 baseline	—
hermes-4.3-36b	4	34%	18%	—	—	—	—	—	4/22/22 v0.18.0
deepseek-r1-32b	4	8%	80%	—	—	—	—	4/5/5 baseline	—

Score = benches solved (rewards coverage). Cov = share of the stack attempted. Rate = solved ÷ attempted — capability once a model has had its shot, so a model that has run little but hits hard isn't buried by one that simply ran more. Each stage block reads passed / attempted / total benches.

Full detail — model × bench

Every bench, per harness. green = passed · red = failed · grey = not run. Pick a harness:

baseline v1.7.1 hermes mini-blind bare dev_cycle structured_feedback mini-default openhands


Model	Score	Miss	Time

Composition

The benches that make up this harness's gauntlet, by domain.