Touchstone
← Benches

ladder.agentic.single_call

Agenticgauntlet

An owned ladder bench, graded mechanically against a fixed test.

Solved
20/22
runs passed
Models
22
have attempted
Harnesses
1
scaffolds tried

Runs (latest per model × harness — solved sorted first)

ModelHarnessResultTurnstok/sLatencyCtxWhen
gpt-oss-120bbaselinePASS244.11.4s64k2026-06-23
granite-4.1-8bbaselinePASS220.11.6s64k2026-06-23
llama-3.3-70bbaselinePASS22.710.6s64k2026-06-23
devstral-small-2-2512baselinePASS28.43.0s64k2026-06-23
mistral-small-3.1-24b-instruct-2503baselinePASS28.45.0s64k2026-06-23
qwen3.5-122b-a10b-clusterbaselinePASS215.47.7s64k2026-06-29
qwen3.5-4bbaselinePASS243.22.7s128k2026-06-30
gpt-oss-20bbaselinePASS269.21.3s64k2026-07-01
qwen3-coder-30bbaselinePASS229.21.2s64k2026-07-01
gemma-4-26b-a4b-itbaselinePASS219.81.6s64k2026-07-01
qwen3.6-35b-a3b-orbaselinePASS268.92.6s64k2026-07-01
gemini-2.5-flash-litebaselinePASS29.71.8s64k2026-07-01
qwen3.5-0.8bbaselinePASS267.4564ms64k2026-07-01
qwen3.6-27bbaselinePASS27.821.4s64k2026-07-01
qwen3.6-35b-bf16baselinePASS231.95.8s64k2026-07-02
qwen3.5-2bbaselinePASS273.1520ms64k2026-07-02
granite-4.1-30bbaselinePASS27.14.8s64k2026-07-02
qwen3.6-35b-a3bbaselinePASS241.6961ms64k2026-07-02
qwen3-next-80bbaselinePASS232.5985ms64k2026-07-02
claude-sonnet-4.6baselinePASS212.37.4s195k2026-07-06
deepseek-r1-distill-qwen-32bbaselinefail14.728.5s64k2026-07-02
phi-4-reasoning-plusbaselinefail814.05.3m64k2026-07-02

Transcripts

gpt-oss-120b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18 °C."}
granite-4.1-8b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is **18°C**."}
llama-3.3-70b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18 degrees Celsius."}
devstral-small-2-2512 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is **18 degrees Celsius**."}
mistral-small-3.1-24b-instruct-2503 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18 degrees Celsius."}
qwen3.5-122b-a10b-cluster · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "18"}
qwen3.5-4b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "\n\nThe current temperature in Berlin is 18 degrees Celsius."}
gpt-oss-20b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is **18 °C**."}
qwen3-coder-30b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18°C."}
gemma-4-26b-a4b-it · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "<|channel>thought\n<channel|>The current temperature in Berlin is 18°C."}
qwen3.6-35b-a3b-or · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18°C."}
gemini-2.5-flash-lite · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18 degrees Celsius."}
qwen3.5-0.8b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18°C."}
qwen3.6-27b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "\n\n18"}
qwen3.6-35b-bf16 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "\n\nThe current temperature in Berlin is 18°C."}
qwen3.5-2b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18°C."}
granite-4.1-30b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is **18 °C**."}
qwen3.6-35b-a3b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "\n\nThe current temperature in Berlin is 18°C."}
qwen3-next-80b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is 18°C."}
claude-sonnet-4.6 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": "The current temperature in Berlin is **18°C**. It's also cloudy there right now."}
deepseek-r1-distill-qwen-32b · baseline · fail
{"tool_calls": [], "final": "\nOkay, so I need to figure out how to respond when someone asks for the current temperature in Berlin using the get_weather tool and then just give them the number. Let's break this down step by step.\n\nFirst, the user is asking about the weather in Berlin specifically, so I should use the get_weather function with \"Berlin\" as the city parameter. That makes sense because the tool requires a city name to fetch the current weather data.\n\nI remember from the examples that when using tools, each request needs to be within [TOOL_REQUEST]\n{\"name\": \"tool_name\", \"arguments\": {\"param\":\"value\"}}\n[END_TOOL_REQUEST]"}
phi-4-reasoning-plus · baseline · fail
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 2, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 3, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 4, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}, {"turn": 5, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 6, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 7, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}, {"turn": 8, "name": "get_weather", "args": {"city": "Berlin"}, "args_ok": true}], "final": ""}