Coding
GOAL
Can the model write and fix code that actually runs.
WHAT IT TESTS
Code generation and bug-fixes across focused exercises.
HOW IT GRADES
Generated code executed against fixed test cases; pass = tests green.
Standings — gauntlet benches solved
| Model | Solved | |
|---|---|---|
| gpt-oss-20b | 24/24 | |
| qwen3.6-35b-a3b | 24/24 | |
| gemma-4-26b-a4b-it | 24/24 | |
| qwen3.5-4b | 16/24 | |
| qwen3.6-27b | 12/12 | |
| claude-sonnet-4.6 | cloud | 6/6 |
| deepseek-r1-distill-qwen-32b | 6/6 | |
| granite-4.1-30b | 6/6 | |
| llama-3.3-70b | 6/6 | |
| qwen3.6-35b-a3b-or | 6/6 | |
| qwen3-next-80b | 6/6 | |
| devstral-small-2-2512 | 5/6 | |
| qwen3-coder-30b | 5/6 | |
| qwen3.5-122b-a10b-cluster | 5/6 | |
| qwen3.6-35b-bf16 | 5/6 | |
| gpt-oss-120b | 5/6 | |
| mistral-small-3.1-24b-instruct-2503 | 5/6 | |
| gemini-2.5-flash-lite | cloud | 4/6 |
| phi-4-reasoning-plus | 4/6 | |
| granite-4.1-8b | 4/6 | |
| qwen3.5-2b | 2/6 | |
| qwen3.5-0.8b | 0/6 |
Benches — green = in gauntlet
ladder.coding.expr_evalladder.coding.fix_binary_searchladder.coding.lru_cacheladder.coding.merge_intervalsladder.coding.topological_sortladder.coding.wildcard_match