← Benches
gate.vitals.json_valid
GategauntletA deterministic gate probe — harness-invariant vitals / fidelity / tool-use check that every model must clear before the ladders count.
Solved
22/23
runs passed
Models
22
have attempted
Harnesses
2
scaffolds tried
Runs (latest per model × harness — solved sorted first)
| Model | Harness | Result | Turns | tok/s | Latency | Ctx | When |
|---|---|---|---|---|---|---|---|
| gpt-oss-120b | baseline | PASS | 1 | 47.2 | 530ms | 64k | 2026-06-23 |
| granite-4.1-8b | baseline | PASS | 1 | 21.7 | 646ms | 128k | 2026-06-23 |
| llama-3.3-70b | baseline | PASS | 1 | 2.8 | 5.0s | 64k | 2026-06-23 |
| devstral-small-2-2512 | baseline | PASS | 1 | 8.2 | 1.7s | 64k | 2026-06-23 |
| mistral-small-3.1-24b-instruct-2503 | baseline | PASS | 1 | 8.2 | 1.3s | 64k | 2026-06-23 |
| qwen3.5-122b-a10b-cluster | baseline | PASS | 1 | 17.1 | 38.4s | 64k | 2026-06-29 |
| qwen3.5-4b | baseline | PASS | 1 | 45.4 | 7.0s | 128k | 2026-06-30 |
| gpt-oss-20b | baseline | PASS | 1 | 73.6 | 1.0s | 64k | 2026-07-01 |
| qwen3-coder-30b | baseline | PASS | 1 | 31.7 | 347ms | 64k | 2026-07-01 |
| gemma-4-26b-a4b-it | baseline | PASS | 1 | 23.5 | 595ms | 64k | 2026-07-01 |
| qwen3.6-35b-a3b-or | baseline | PASS | 1 | 116.5 | 5.6s | 64k | 2026-07-01 |
| gemini-2.5-flash-lite | baseline | PASS | 1 | 1.2 | 10.4s | 64k | 2026-07-01 |
| qwen3.5-0.8b | baseline | PASS | 1 | 55.8 | 197ms | 64k | 2026-07-01 |
| qwen3.6-27b | baseline | PASS | 1 | 7.9 | 1.2m | 64k | 2026-07-01 |
| deepseek-r1-distill-qwen-32b | baseline | PASS | 1 | 4.7 | 2.0m | 64k | 2026-07-02 |
| qwen3.6-35b-bf16 | baseline | PASS | 1 | 34.4 | 18.6s | 64k | 2026-07-02 |
| qwen3.5-2b | baseline | PASS | 1 | 67.5 | 163ms | 64k | 2026-07-02 |
| granite-4.1-30b | baseline | PASS | 1 | 7.2 | 1.9s | 64k | 2026-07-02 |
| qwen3.6-27b | v1.7.1 | PASS | 1 | 7.8 | 2.7s | 64k | 2026-07-02 |
| qwen3.6-35b-a3b | baseline | PASS | 1 | 44.9 | 334ms | 64k | 2026-07-02 |
| qwen3-next-80b | baseline | PASS | 1 | 38.0 | 368ms | 64k | 2026-07-02 |
| claude-sonnet-4.6 | baseline | PASS | 1 | 5.4 | 3.2s | 195k | 2026-07-06 |
| phi-4-reasoning-plus | baseline | fail | 1 | 14.4 | 31.5s | 64k | 2026-07-02 |
Transcripts
gpt-oss-120b · baseline · PASS
{"name":"Touchstone","version":4}granite-4.1-8b · baseline · PASS
{"name": "Touchstone", "version": 4}llama-3.3-70b · baseline · PASS
{"name": "Touchstone", "version": 4}devstral-small-2-2512 · baseline · PASS
{"name": "Touchstone", "version": 4}mistral-small-3.1-24b-instruct-2503 · baseline · PASS
{"name":"Touchstone","version":4}qwen3.5-122b-a10b-cluster · baseline · PASS
{"name": "Touchstone", "version": 4}qwen3.5-4b · baseline · PASS
{"name": "Touchstone", "version": 4}gpt-oss-20b · baseline · PASS
{"name":"Touchstone","version":4}qwen3-coder-30b · baseline · PASS
{"name":"Touchstone","version":4}gemma-4-26b-a4b-it · baseline · PASS
{"name": "Touchstone", "version": 4}qwen3.6-35b-a3b-or · baseline · PASS
{"name": "Touchstone", "version": 4}gemini-2.5-flash-lite · baseline · PASS
{"name": "Touchstone", "version": 4}qwen3.5-0.8b · baseline · PASS
{"name":"Touchstone","version":4}qwen3.6-27b · baseline · PASS
{"name": "Touchstone", "version": 4}deepseek-r1-distill-qwen-32b · baseline · PASS
{"name": "Touchstone", "version": 4}qwen3.6-35b-bf16 · baseline · PASS
{"name":"Touchstone","version":4}qwen3.5-2b · baseline · PASS
{"name":"Touchstone","version":4}granite-4.1-30b · baseline · PASS
{"name": "Touchstone", "version": 4}qwen3.6-27b · v1.7.1 · PASS
{
"name": "Touchstone",
"version": 4
}qwen3.6-35b-a3b · baseline · PASS
{"name": "Touchstone", "version": 4}qwen3-next-80b · baseline · PASS
{"name": "Touchstone", "version": 4}claude-sonnet-4.6 · baseline · PASS
{"name": "Touchstone", "version": 4}phi-4-reasoning-plus · baseline · fail
User says: "Output a single JSON object with exactly two keys: 'name' set to the string Touchstone, and 'version' set to the integer 4. Output only the JSON. No markdown, no code fences, no extra text." So we need to output a JSON object that is exactly as described.
We must produce a JSON object with keys "name" and "version". The value for name should be string "Touchstone", version should be integer 4. We must not include any markdown formatting or code fences. Also instructions: "Output only the JSON." So we output just the JSON text, no extra text.
We need to produce a valid JSON object with exactly two keys: "name" and "version". The value for name is string "Touchstone", version is integer 4. We must not include any markdown formatting or code fences. Also instructions: "Output only the JSON." So we output just the JSON text, no extra text.
I'll produce a valid JSON object with exactly two keys. I'll produce something like:
{
"name": "Touchstone",
"version": 4
}
But careful: The instruction says "Output only the JSON" so I must not include any markdown formatting or code fences. So I'll output just plain text.
I need to check if there's any additional instructions? It says "Output a single JSON object with exactly two keys: 'name' set to the string Touchstone, and 'version' set to the integer 4." So that is exactly what we do.
We must produce a valid JSON. I'll produce:
{"name": "Touchstone", "version": 4}
I need to check if there's any extra text? The instructions say: "Output only the JSON" so no markdown formatting, no code fences, no extra text. So I'll output exactly that.
I must not include any additional commentary or explanation. I'll produce a valid JSON object with keys "name" and "version".
I'll produce:
{"name": "Touchstone", "version": 4}
That is the answer.
I'll now produce final answer accordingly.{"name": "Touchstone", "version": 4}