gate.tools.error_recovery

Gategauntlet

A deterministic gate probe — harness-invariant vitals / fidelity / tool-use check that every model must clear before the ladders count.

Solved

18/23

runs passed

Models

have attempted

Harnesses

scaffolds tried

Runs (latest per model × harness — solved sorted first)

Model	Harness	Result	Turns	tok/s	Latency	Ctx	When
gpt-oss-120b	baseline	PASS	3	42.7	2.0s	64k	2026-06-23
granite-4.1-8b	baseline	PASS	3	21.7	2.8s	64k	2026-06-23
llama-3.3-70b	baseline	PASS	3	2.8	18.5s	64k	2026-06-23
mistral-small-3.1-24b-instruct-2503	baseline	PASS	3	8.4	9.9s	128k	2026-06-23
qwen3.5-122b-a10b-cluster	baseline	PASS	3	15.4	13.5s	64k	2026-06-29
qwen3.5-4b	baseline	PASS	3	43.0	4.8s	128k	2026-06-30
gpt-oss-20b	baseline	PASS	3	65.7	1.8s	64k	2026-07-01
gemma-4-26b-a4b-it	baseline	PASS	3	22.9	2.6s	64k	2026-07-01
qwen3.6-35b-a3b-or	baseline	PASS	3	56.3	3.7s	64k	2026-07-01
gemini-2.5-flash-lite	baseline	PASS	3	18.2	2.7s	64k	2026-07-01
qwen3.6-27b	baseline	PASS	3	7.7	27.8s	64k	2026-07-01
qwen3.6-35b-bf16	baseline	PASS	3	30.5	7.7s	64k	2026-07-02
granite-4.1-30b	baseline	PASS	3	7.2	9.0s	64k	2026-07-02
qwen3.6-27b	v1.7.1	PASS	3	7.6	10.2s	64k	2026-07-02
qwen3.6-35b-a3b	baseline	PASS	3	42.7	1.8s	64k	2026-07-02
qwen3-next-80b	baseline	PASS	3	35.1	2.5s	64k	2026-07-02
phi-4-reasoning-plus	baseline	PASS	3	14.3	59.9s	64k	2026-07-02
claude-sonnet-4.6	baseline	PASS	3	17.8	8.4s	195k	2026-07-06
devstral-small-2-2512	baseline	fail	2	8.2	5.9s	128k	2026-06-23
qwen3-coder-30b	baseline	fail	2	32.6	1.9s	64k	2026-07-01
qwen3.5-0.8b	baseline	fail	2	71.9	751ms	64k	2026-07-01
deepseek-r1-distill-qwen-32b	baseline	fail	1	4.7	31.3s	64k	2026-07-02
qwen3.5-2b	baseline	fail	2	77.8	668ms	64k	2026-07-02

Transcripts

gpt-oss-120b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The issue’s title is **“Fix watchdog null‑context blindness.”**"}

granite-4.1-8b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue 7349 is **\"Fix watchdog null-context blindness\"**."}

llama-3.3-70b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

mistral-small-3.1-24b-instruct-2503 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue #7349 is \"Fix watchdog null-context blindness\"."}

qwen3.5-122b-a10b-cluster · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

qwen3.5-4b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "\n\nThe title of issue 7349 is: **Fix watchdog null-context blindness**"}

gpt-oss-20b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue **7349** is:\n\n> **“Fix watchdog null‑context blindness”**"}

gemma-4-26b-a4b-it · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "<|channel>thought\n<channel|>The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

qwen3.6-35b-a3b-or · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

gemini-2.5-flash-lite · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

qwen3.6-27b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "\n\nThe title of issue 7349 is \"Fix watchdog null-context blindness\"."}

qwen3.6-35b-bf16 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "\n\nThe title of issue 7349 is \"Fix watchdog null-context blindness\"."}

granite-4.1-30b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue **7349** is:\n\n**“Fix watchdog null‑context blindness.”**"}

qwen3.6-27b · v1.7.1 · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "\n\nThe title of issue 7349 is **\"Fix watchdog null-context blindness\"**."}

qwen3.6-35b-a3b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "\n\nThe title of issue 7349 is \"Fix watchdog null-context blindness\"."}

qwen3-next-80b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

phi-4-reasoning-plus · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "User message: \"[TOOL_RESULT]{\"id\": 7349, \"title\": \"Fix watchdog null-context blindness\"}[END_TOOL_RESULT]\" indicates that the tool call succeeded and returned a JSON object with id=7349 and title \"Fix watchdog null-context blindness\". The user's original request was \"Fetch issue 7349 with the tool and tell me its title.\" So now we need to output answer: \"The title is 'Fix watchdog null-context blindness'.\"\n\nBut instructions say: \"Answer normally if no tools are needed.\" But in this case, we have a tool result. The instructions say: \"Use tool results to inform your response\" so I must use the provided tool result.\n\nI must produce an answer that says something like: \"The title of issue 7349 is 'Fix watchdog null-context blindness'.\"\n\nBut careful: The instructions say \"if calling tools, end with tool calls (no text after)\". But now we are not calling a tool. We have already used the tool call and got result. So I can produce answer normally.\n\nI must check if any additional instructions: \"never output [TOOL_RESULT] or [END_TOOL_RESULT]\" tags in my answer. So I should not include them.\n\nThus final answer: \"The title of issue 7349 is 'Fix watchdog null-context blindness'.\" But maybe I'll also mention that the tool call succeeded and then provide the title.\n\nI must produce a plain text answer with no markdown formatting by default? The instructions say \"By default, do not reply with Markdown formatting unless requested\" but it's not explicitly said. But instructions say: \"Answer normally if no tools are needed.\" So I'll produce plain text answer.\n\nI'll produce final answer: \"The title of issue 7349 is 'Fix watchdog null-context blindness'.\"\n\nI must check if any additional details required? The user said \"Fetch issue 7349 with the tool and tell me its title.\" So I can simply say: \"Issue 7349's title is 'Fix watchdog null-context blindness'.\" That is enough.\n\nI'll produce answer.The title of issue 7349 is \"Fix watchdog null-context blindness\"."}

claude-sonnet-4.6 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}, {"turn": 2, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "The title of issue **#7349** is: **\"Fix watchdog null-context blindness\"**."}

devstral-small-2-2512 · baseline · fail

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": "ABC-123"}, "args_ok": true}], "final": "The issue ID must be in numeric format (e.g., 7349). Please provide the numeric ID for the issue you want to fetch."}

qwen3-coder-30b · baseline · fail

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "It seems there was a temporary backend error while trying to fetch the issue. Please try again later or let me know if you'd like me to attempt fetching it once more.\n"}

qwen3.5-0.8b · baseline · fail

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "I'm sorry, but there was a temporary backend error when fetching issue 7349. Please try again later."}

deepseek-r1-distill-qwen-32b · baseline · fail

{"tool_calls": [], "final": "\nOkay, so I need to fetch issue number 7349 using the available tool and then tell the user its title. Let me think about how to approach this step by step.\n\nFirst, I remember that there's a tool called \"get_issue\" which can fetch a tracker issue by its numeric ID. The parameters required for this tool are just the \"id\", which is an integer. So, I need to call this tool with id=7349.\n\nI should structure my request correctly. From the examples provided, it looks like each tool call needs to be within [TOOL_REQUEST]\n{\"name\": \"tool_name\", \"arguments\": {\"param\":\"value\"}}\n[END_TOOL_REQUEST]"}

qwen3.5-2b · baseline · fail

{"tool_calls": [{"turn": 1, "name": "get_issue", "args": {"id": 7349}, "args_ok": true}], "final": "I was unable to fetch issue 7349 due to a temporary backend error. Please try again later."}