gate.tools.single_call

Gategauntlet

A deterministic gate probe — harness-invariant vitals / fidelity / tool-use check that every model must clear before the ladders count.

Solved

21/23

runs passed

Models

have attempted

Harnesses

scaffolds tried

Runs (latest per model × harness — solved sorted first)

Model	Harness	Result	Turns	tok/s	Latency	Ctx	When
gpt-oss-120b	baseline	PASS	2	38.8	1.6s	64k	2026-06-23
granite-4.1-8b	baseline	PASS	2	21.8	1.7s	128k	2026-06-23
llama-3.3-70b	baseline	PASS	2	2.8	12.2s	64k	2026-06-23
devstral-small-2-2512	baseline	PASS	2	8.0	3.6s	64k	2026-06-23
mistral-small-3.1-24b-instruct-2503	baseline	PASS	2	7.9	5.8s	64k	2026-06-23
qwen3.5-122b-a10b-cluster	baseline	PASS	2	16.0	11.8s	64k	2026-06-29
qwen3.5-4b	baseline	PASS	2	43.7	3.9s	128k	2026-06-30
gpt-oss-20b	baseline	PASS	2	67.6	1.3s	64k	2026-07-01
qwen3-coder-30b	baseline	PASS	2	29.6	1.3s	64k	2026-07-01
gemma-4-26b-a4b-it	baseline	PASS	2	23.3	1.6s	64k	2026-07-01
qwen3.6-35b-a3b-or	baseline	PASS	2	47.2	4.3s	64k	2026-07-01
gemini-2.5-flash-lite	baseline	PASS	2	2.8	6.9s	64k	2026-07-01
qwen3.5-0.8b	baseline	PASS	2	69.0	638ms	64k	2026-07-01
qwen3.6-27b	baseline	PASS	2	7.8	26.2s	64k	2026-07-01
qwen3.6-35b-bf16	baseline	PASS	2	32.0	6.6s	64k	2026-07-02
qwen3.5-2b	baseline	PASS	2	74.7	576ms	64k	2026-07-02
granite-4.1-30b	baseline	PASS	2	7.2	8.0s	64k	2026-07-02
qwen3.6-27b	v1.7.1	PASS	2	7.7	5.9s	64k	2026-07-02
qwen3.6-35b-a3b	baseline	PASS	2	43.5	1.0s	64k	2026-07-02
qwen3-next-80b	baseline	PASS	2	34.0	1.1s	64k	2026-07-02
claude-sonnet-4.6	baseline	PASS	2	22.9	5.9s	195k	2026-07-06
deepseek-r1-distill-qwen-32b	baseline	fail	1	4.6	35.0s	64k	2026-07-02
phi-4-reasoning-plus	baseline	fail	2	14.3	46.2s	64k	2026-07-02

Transcripts

gpt-oss-120b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is **clear** with a temperature of **17 °C**."}

granite-4.1-8b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

llama-3.3-70b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17 degrees Celsius."}

devstral-small-2-2512 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

mistral-small-3.1-24b-instruct-2503 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

qwen3.5-122b-a10b-cluster · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is:\n- Temperature: 17°C\n- Conditions: Clear\n\nIt's a nice day in Paris with clear skies and mild temperatures!"}

qwen3.5-4b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "\n\nThe current weather in Paris is:\n- **Temperature**: 17°C\n- **Conditions**: Clear skies\n\nIt's a pleasant day with clear conditions!"}

gpt-oss-20b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is **17 °C** with **clear skies**."}

qwen3-coder-30b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

gemma-4-26b-a4b-it · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "<|channel>thought\n<channel|>The current weather in Paris is clear with a temperature of 17°C."}

qwen3.6-35b-a3b-or · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

gemini-2.5-flash-lite · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is 17°C and clear."}

qwen3.5-0.8b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is **17°C** with **clear** conditions."}

qwen3.6-27b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "\n\nThe current weather in Paris is clear with a temperature of 17°C. Let me know if you need anything else!"}

qwen3.6-35b-bf16 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "\n\nThe current weather in Paris is clear with a temperature of 17°C (63°F). Let me know if you need any other details!"}

qwen3.5-2b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

granite-4.1-30b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in **Paris** is:\n\n- **Temperature:** 17 °C  \n- **Conditions:** Clear  \n\n(Information retrieved using the live weather tool.)"}

qwen3.6-27b · v1.7.1 · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "\n\nThe current weather in Paris is clear with a temperature of 17°C."}

qwen3.6-35b-a3b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "\n\nThe current weather in Paris is clear with a temperature of 17°C."}

qwen3-next-80b · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "The current weather in Paris is clear with a temperature of 17°C."}

claude-sonnet-4.6 · baseline · PASS

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Paris"}, "args_ok": true}], "final": "Here is the current weather in **Paris**:\n\n- 🌡️ **Temperature:** 17°C\n- ☀️ **Conditions:** Clear\n\nIt looks like a pleasant and clear day in Paris! Perfect weather for a stroll along the Seine. 🗼"}

deepseek-r1-distill-qwen-32b · baseline · fail

{"tool_calls": [], "final": "\nOkay, so I need to figure out how to respond when someone asks about the current weather in Paris and tells me to use the available tool. Let's break this down step by step.\n\nFirst, I remember that there's a tool called \"get_weather\" which is designed exactly for getting weather information. The user specifically mentioned using it, so that's a clear指示.\n\nLooking at the tool's parameters, it requires a city name as input. In this case, the city is Paris. So, I need to structure my request correctly by including the city parameter with the value \"Paris\".\n\nI also recall from the examples provided that each tool call should be within [TOOL_REQUEST]\n{\"name\": \"tool_name\", \"arguments\": {\"param\":\"value\"}}\n[END_TOOL_REQUEST]"}

phi-4-reasoning-plus · baseline · fail

{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"param": "value"}, "args_ok": true}], "final": "User query: \"What is the current weather in Paris? Use the available tool to find out.\" The conversation includes a previous message from user that says \"[TOOL_RESULT]{\"city\": \"Paris\", \"temp_c\": 17, \"conditions\": \"clear\"}[END_TOOL_RESULT]\" which is presumably the result of calling get_weather. So we need to produce answer using the provided tool result.\n\nWe have instructions: \"The results of tool calls will be provided in the following format:\n[TOOL_RESULT]\n<result>\n[END_TOOL_RESULT]\"\n\nSo now our answer should incorporate that result. The result says: {\"city\": \"Paris\", \"temp_c\": 17, \"conditions\": \"clear\"}. So we need to produce a final answer message that includes this information.\n\nWe must not output the [TOOL_RESULT] or [END_TOOL_RESULT] tags ourselves. But instructions say: \"Never output [TOOL_RESULT] or [END_TOOL_RESULT] tags yourself.\" So I'll just include the result in my answer text.\n\nI need to produce a final answer message that says something like: \"The current weather in Paris is 17°C with clear conditions\" etc.\n\nLet's check instructions: \"Answer normally if no tools are needed.\" But here we have tool results. The conversation includes a tool call result from the user. So I'll produce an answer message that uses the provided tool result.\n\nI must not include any markdown formatting by default, so plain text is fine.\n\nI need to produce final answer message: \"The current weather in Paris is 17°C with clear conditions.\" But instructions say: \"Use tool results to inform your response.\"\n\nSo I'll produce a final answer message that says something like: \"According to the latest data, the temperature in Paris is 17°C and the sky is clear.\" Possibly include details.\n\nI must not output any additional text after the tool call. Wait, instructions: \"One tool per block\" but we already have one tool result provided by user. But now I'm answering final answer message. So I'll produce a final answer message that says something like:\n\n\"The current weather in Paris is 17°C with clear conditions.\" Possibly include additional details if needed.\n\nI must not output any markdown formatting unless requested, so plain text is fine.\n\nI'll produce final answer message: \"The current weather in Paris is 17°C and the sky is clear.\"\n\nI'll produce final answer message. I'll produce final answer message that says something like:\n\n\"According to our data, the temperature in Paris is currently 17°C with clear conditions.\" Possibly include additional details if needed.\n\nI must not output any markdown formatting unless requested.\n\nI'll produce final answer message: \"The current weather in Paris is 17°C and the sky is clear.\"\n\nI'll produce final answer message.Based on the latest data, the temperature in Paris is currently 17°C with clear conditions."}