gate.fidelity.enum

Gategauntlet

A deterministic gate probe — harness-invariant vitals / fidelity / tool-use check that every model must clear before the ladders count.

Solved

21/23

runs passed

Models

have attempted

Harnesses

scaffolds tried

Runs (latest per model × harness — solved sorted first)

Model	Harness	Result	Turns	tok/s	Latency	Ctx	When
gpt-oss-120b	baseline	PASS	1	34.2	497ms	64k	2026-06-23
granite-4.1-8b	baseline	PASS	1	14.4	139ms	64k	2026-06-23
llama-3.3-70b	baseline	PASS	1	2.7	741ms	64k	2026-06-23
devstral-small-2-2512	baseline	PASS	1	7.5	268ms	64k	2026-06-23
mistral-small-3.1-24b-instruct-2503	baseline	PASS	1	7.6	262ms	64k	2026-06-23
qwen3.5-122b-a10b-cluster	baseline	PASS	1	16.8	19.7s	64k	2026-06-29
qwen3.5-4b	baseline	PASS	1	45.4	7.3s	128k	2026-06-30
gpt-oss-20b	baseline	PASS	1	73.9	988ms	64k	2026-07-01
qwen3-coder-30b	baseline	PASS	1	17.2	116ms	64k	2026-07-01
gemma-4-26b-a4b-it	baseline	PASS	1	6.8	294ms	64k	2026-07-01
qwen3.6-35b-a3b-or	baseline	PASS	1	89.7	3.0s	64k	2026-07-01
gemini-2.5-flash-lite	baseline	PASS	1	0.9	1.2s	64k	2026-07-01
qwen3.5-0.8b	baseline	PASS	1	23.0	87ms	64k	2026-07-01
qwen3.6-27b	baseline	PASS	1	7.9	30.1s	64k	2026-07-01
qwen3.6-35b-bf16	baseline	PASS	1	34.2	7.7s	64k	2026-07-02
qwen3.5-2b	baseline	PASS	1	27.0	74ms	64k	2026-07-02
granite-4.1-30b	baseline	PASS	1	6.4	313ms	64k	2026-07-02
qwen3.6-27b	v1.7.1	PASS	1	6.9	434ms	64k	2026-07-02
qwen3.6-35b-a3b	baseline	PASS	1	25.2	119ms	64k	2026-07-02
qwen3-next-80b	baseline	PASS	1	17.7	113ms	64k	2026-07-02
claude-sonnet-4.6	baseline	PASS	1	2.5	1.6s	195k	2026-07-06
deepseek-r1-distill-qwen-32b	baseline	fail	1	4.6	1.5m	64k	2026-07-02
phi-4-reasoning-plus	baseline	fail	1	14.4	23.6s	64k	2026-07-02

Transcripts

gpt-oss-120b · baseline · PASS

done

granite-4.1-8b · baseline · PASS

done

llama-3.3-70b · baseline · PASS

done

devstral-small-2-2512 · baseline · PASS

done

mistral-small-3.1-24b-instruct-2503 · baseline · PASS

done

qwen3.5-122b-a10b-cluster · baseline · PASS

done

qwen3.5-4b · baseline · PASS



done

gpt-oss-20b · baseline · PASS

done

qwen3-coder-30b · baseline · PASS

done

gemma-4-26b-a4b-it · baseline · PASS

done

qwen3.6-35b-a3b-or · baseline · PASS

done

gemini-2.5-flash-lite · baseline · PASS

done

qwen3.5-0.8b · baseline · PASS

done

qwen3.6-27b · baseline · PASS



done

qwen3.6-35b-bf16 · baseline · PASS



done

qwen3.5-2b · baseline · PASS

done

granite-4.1-30b · baseline · PASS

done

qwen3.6-27b · v1.7.1 · PASS



done

qwen3.6-35b-a3b · baseline · PASS



done

qwen3-next-80b · baseline · PASS

done

claude-sonnet-4.6 · baseline · PASS

done

deepseek-r1-distill-qwen-32b · baseline · fail



The correct status for a task that has been completed and merged is:

**done**

phi-4-reasoning-plus · baseline · fail

User says: "A task's work has been completed and merged. Reply with the single status value that means finished. Choose exactly one from this list: todo, backlog, in_progress, in_review, done, blocked, cancelled. Output only that one value in lowercase, nothing else." So answer is "done". But check instructions: "A task's work has been completed and merged" implies that the correct status is "done", because it means finished.

Wait, but careful reading: The list provided: todo, backlog, in_progress, in_review, done, blocked, cancelled. Which one means finished? "Done" means finished. But check instructions: "A task's work has been completed and merged." That implies that the status is "done". So answer should be "done".

But wait, maybe it's "in_review"? No, because "merged" means code review passed. But then "done" is the final state.

Wait, but careful reading: "A task's work has been completed and merged." The instructions say: "Reply with the single status value that means finished." So answer is "done". But check list: "todo", "backlog", "in_progress", "in_review", "done", "blocked", "cancelled". Which one means finished? "Done" means finished. So answer is "done".

I must output exactly "done" in lowercase, nothing else.

Thus answer: done

I'll produce final answer as "done". We'll check instructions: "Output only that one value in lowercase, nothing else." So I'll output "done" with no extra text.

I'll now produce answer.done