gate.tools.result_use

Gategauntlet

A deterministic gate probe — harness-invariant vitals / fidelity / tool-use check that every model must clear before the ladders count.

Solved

23/23

runs passed

Models

have attempted

Harnesses

scaffolds tried

Runs (latest per model × harness — solved sorted first)

Model	Harness	Result	Turns	tok/s	Latency	Ctx	When
gpt-oss-120b	baseline	PASS	2	43.6	1.4s	64k	2026-06-23
granite-4.1-8b	baseline	PASS	2	21.8	1.6s	64k	2026-06-23
llama-3.3-70b	baseline	PASS	2	2.7	12.1s	64k	2026-06-23
devstral-small-2-2512	baseline	PASS	2	8.4	3.0s	64k	2026-06-23
mistral-small-3.1-24b-instruct-2503	baseline	PASS	2	7.9	5.3s	128k	2026-06-23
qwen3.5-122b-a10b-cluster	baseline	PASS	2	15.0	7.1s	64k	2026-06-29
qwen3.5-4b	baseline	PASS	2	43.5	3.1s	128k	2026-06-30
gpt-oss-20b	baseline	PASS	2	67.3	1.2s	64k	2026-07-01
qwen3-coder-30b	baseline	PASS	2	29.3	1.2s	64k	2026-07-01
gemma-4-26b-a4b-it	baseline	PASS	2	22.5	1.4s	64k	2026-07-01
qwen3.6-35b-a3b-or	baseline	PASS	2	55.4	2.6s	64k	2026-07-01
gemini-2.5-flash-lite	baseline	PASS	2	2.1	8.7s	64k	2026-07-01
qwen3.5-0.8b	baseline	PASS	2	65.7	548ms	64k	2026-07-01
qwen3.6-27b	baseline	PASS	2	7.8	33.2s	64k	2026-07-01
deepseek-r1-distill-qwen-32b	baseline	PASS	2	4.7	1.4m	64k	2026-07-02
qwen3.6-35b-bf16	baseline	PASS	2	32.2	7.2s	64k	2026-07-02
qwen3.5-2b	baseline	PASS	2	74.0	527ms	64k	2026-07-02
granite-4.1-30b	baseline	PASS	2	7.2	5.4s	64k	2026-07-02
qwen3.6-27b	v1.7.1	PASS	2	7.6	5.4s	64k	2026-07-02
qwen3.6-35b-a3b	baseline	PASS	2	42.3	970ms	64k	2026-07-02
qwen3-next-80b	baseline	PASS	2	33.7	1.0s	64k	2026-07-02
phi-4-reasoning-plus	baseline	PASS	3	14.2	36.7s	64k	2026-07-02
claude-sonnet-4.6	baseline	PASS	2	22.6	4.3s	195k	2026-07-06

Transcripts

gpt-oss-120b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The user with ID 7 is **Ada Lovelace**."}

granite-4.1-8b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace."}

llama-3.3-70b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The user with id 7 is named Ada Lovelace."}

devstral-small-2-2512 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace."}

mistral-small-3.1-24b-instruct-2503 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace."}

qwen3.5-122b-a10b-cluster · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace."}

qwen3.5-4b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "\n\nUser 7's name is Ada Lovelace."}

gpt-oss-20b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "User 7 is **Ada Lovelace**."}

qwen3-coder-30b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace.\n"}

gemma-4-26b-a4b-it · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "<|channel>thought\n<channel|>The name of user 7 is Ada Lovelace."}

qwen3.6-35b-a3b-or · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace."}

gemini-2.5-flash-lite · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "Ada Lovelace is the name of user 7."}

qwen3.5-0.8b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "User 7 is Ada Lovelace."}

qwen3.6-27b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "\n\nUser 7's name is Ada Lovelace."}

deepseek-r1-distill-qwen-32b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "\n\nThe user with ID 7 is named Ada Lovelace."}

qwen3.6-35b-bf16 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "\n\nUser 7's name is Ada Lovelace."}

qwen3.5-2b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is Ada Lovelace."}

granite-4.1-30b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The user with ID 7 is **Ada Lovelace**."}

qwen3.6-27b · v1.7.1 · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "\n\nThe name of user 7 is Ada Lovelace."}

qwen3.6-35b-a3b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "\n\nThe name of user 7 is Ada Lovelace."}

qwen3-next-80b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The user with ID 7 is named Ada Lovelace."}

phi-4-reasoning-plus · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}, {"turn": 2, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "User says: \"Look up user 7 and tell me their name.\" The conversation includes a tool call result. We need to produce answer that uses the provided tool result.\n\nThe instructions say: \"Use the provided tool to look up data, then answer.\" So we have a tool result block with JSON content:\n[TOOL_RESULT]{\"user_id\": 7, \"name\": \"Ada Lovelace\", \"role\": \"engineer\"}[END_TOOL_RESULT]\n\nWe need to produce an answer that says: \"User 7's name is Ada Lovelace.\" But instructions say: \"Answer normally if no tools are needed.\"\n\nBut we have a tool result. The conversation includes the tool call and then the tool result block. We must use the provided tool result to inform our response.\n\nWe need to produce answer that says something like: \"User 7 is Ada Lovelace.\" But instructions say: \"Look up user 7 and tell me their name.\"\n\nI'll produce answer: \"Ada Lovelace\" or \"The name of user 7 is Ada Lovelace.\" I must not include any tool result tags in my answer. The instructions say: \"Never output [TOOL_RESULT] or [END_TOOL_RESULT] tags yourself.\" So I should simply produce a plain text answer.\n\nI'll produce answer: \"User 7's name is Ada Lovelace.\"\n\nI'll produce answer as final message. We'll produce answer with no markdown formatting, just plain text.\n\nI'll produce answer: \"The user with ID 7 is named Ada Lovelace.\" I'll produce answer in plain text.\n\nI'll produce answer: \"Ada Lovelace\" or \"User 7's name is Ada Lovelace.\"\n\nI'll produce answer as final message.User 7's name is Ada Lovelace."}

claude-sonnet-4.6 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_user", "args": {"user_id": 7}, "args_ok": true}], "final": "The name of user 7 is **Ada Lovelace**! She is listed as an engineer in the system."}