← Benches
ladder.agentic.error_recovery
AgenticgauntletAn owned ladder bench, graded mechanically against a fixed test.
Solved
19/22
runs passed
Models
22
have attempted
Harnesses
1
scaffolds tried
Runs (latest per model × harness — solved sorted first)
| Model | Harness | Result | Turns | tok/s | Latency | Ctx | When |
|---|---|---|---|---|---|---|---|
| gpt-oss-120b | baseline | PASS | 3 | 45.4 | 2.6s | 64k | 2026-06-23 |
| granite-4.1-8b | baseline | PASS | 3 | 21.4 | 2.3s | 64k | 2026-06-23 |
| llama-3.3-70b | baseline | PASS | 3 | 2.8 | 15.1s | 64k | 2026-06-23 |
| devstral-small-2-2512 | baseline | PASS | 3 | 8.1 | 3.5s | 64k | 2026-06-23 |
| mistral-small-3.1-24b-instruct-2503 | baseline | PASS | 3 | 8.4 | 7.9s | 64k | 2026-06-23 |
| qwen3.5-122b-a10b-cluster | baseline | PASS | 3 | 15.4 | 12.4s | 64k | 2026-06-29 |
| qwen3.5-4b | baseline | PASS | 3 | 43.3 | 5.7s | 128k | 2026-06-30 |
| gpt-oss-20b | baseline | PASS | 3 | 68.8 | 2.3s | 64k | 2026-07-01 |
| qwen3-coder-30b | baseline | PASS | 3 | 29.0 | 1.9s | 64k | 2026-07-01 |
| gemma-4-26b-a4b-it | baseline | PASS | 3 | 19.3 | 2.3s | 64k | 2026-07-01 |
| qwen3.6-35b-a3b-or | baseline | PASS | 3 | 90.8 | 6.7s | 64k | 2026-07-01 |
| gemini-2.5-flash-lite | baseline | PASS | 3 | 23.5 | 8.7s | 64k | 2026-07-01 |
| qwen3.6-27b | baseline | PASS | 3 | 7.7 | 31.8s | 64k | 2026-07-01 |
| qwen3.6-35b-bf16 | baseline | PASS | 3 | 32.4 | 20.0s | 64k | 2026-07-02 |
| qwen3.5-2b | baseline | PASS | 3 | 63.0 | 603ms | 64k | 2026-07-02 |
| granite-4.1-30b | baseline | PASS | 3 | 7.2 | 7.4s | 64k | 2026-07-02 |
| qwen3.6-35b-a3b | baseline | PASS | 3 | 39.8 | 1.4s | 64k | 2026-07-02 |
| qwen3-next-80b | baseline | PASS | 3 | 30.1 | 1.5s | 64k | 2026-07-02 |
| claude-sonnet-4.6 | baseline | PASS | 3 | 15.2 | 10.7s | 195k | 2026-07-06 |
| qwen3.5-0.8b | baseline | fail | 2 | 70.8 | 720ms | 64k | 2026-07-01 |
| deepseek-r1-distill-qwen-32b | baseline | fail | 1 | 4.8 | 19.6s | 64k | 2026-07-02 |
| phi-4-reasoning-plus | baseline | fail | 8 | 13.9 | 5.4m | 64k | 2026-07-02 |
Transcripts
gpt-oss-120b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "The config value is: **OK-42**"}granite-4.1-8b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": false}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value retrieved is **OK-42**."}llama-3.3-70b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is OK-42."}devstral-small-2-2512 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is `OK-42`."}mistral-small-3.1-24b-instruct-2503 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": false}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is OK-42."}qwen3.5-122b-a10b-cluster · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "The config value is: OK-42"}qwen3.5-4b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "\n\nThe config value is: **OK-42**"}gpt-oss-20b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "The configuration value is: **OK-42**"}qwen3-coder-30b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": false}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is \"OK-42\"."}gemma-4-26b-a4b-it · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": false}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "<|channel>thought\n<channel|>The config value is OK-42."}qwen3.6-35b-a3b-or · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "The config value is: OK-42"}gemini-2.5-flash-lite · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is OK-42."}qwen3.6-27b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "\n\nThe config value is: **OK-42**"}qwen3.6-35b-bf16 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "\n\nThe config value is **OK-42**."}qwen3.5-2b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "The config value is: OK-42"}granite-4.1-30b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value retrieved is **\"OK-42\"**."}qwen3.6-35b-a3b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {}, "args_ok": true}], "final": "\n\nThe config value is: **OK-42**"}qwen3-next-80b · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is: OK-42"}claude-sonnet-4.6 · baseline · PASS
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {}, "args_ok": true}, {"turn": 2, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The first call to `get_config` failed with a transient 503 error, so I retried — and it succeeded! Here is the config value:\n\n**`OK-42`**"}qwen3.5-0.8b · baseline · fail
{"tool_calls": [{"turn": 1, "name": "get_config", "args": {"retry": true}, "args_ok": true}], "final": "The config value is not available. The request failed with a transient 503 error. Please retry the request."}deepseek-r1-distill-qwen-32b · baseline · fail
{"tool_calls": [], "final": "\nOkay, so I need to fetch the config using the get_config tool. The user mentioned that if it fails, I should retry. Let me think about how to approach this.\n\nFirst, I'll try calling get_config without any arguments since the parameters don't require anything. So, I'll send a request with an empty arguments object.\n\n[TOOL_REQUEST]\n{\"name\": \"get_config\", \"arguments\": {}}\n[END_TOOL_REQUEST]"}phi-4-reasoning-plus · baseline · fail
{"tool_calls": [{"turn": 1, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 2, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 3, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 4, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 5, "name": "get_weather", "args": {"location": "Paris"}, "args_ok": true}, {"turn": 6, "name": "get_config", "args": {"retry": false}, "args_ok": true}, {"turn": 7, "name": "get_config", "args": {"retry": false}, "args_ok": true}, {"turn": 8, "name": "get_config", "args": {"retry": false}, "args_ok": true}], "final": ""}