Gate
GOAL
Prove a model can be trusted with the basics before harder tiers count.
WHAT IT TESTS
Instruction-following, tool calls without hallucination, faithful structured output, error recovery.
HOW IT GRADES
Deterministic single- and multi-step prompts graded mechanically (vitals, tools, fidelity).
Standings — gauntlet benches solved
| Model | Solved | |
|---|---|---|
| qwen3.6-27b | 22/22 | |
| devstral-small-2-2512 | 11/11 | |
| gemini-2.5-flash-lite | cloud | 11/11 |
| gemma-4-26b-a4b-it | 11/11 | |
| gpt-oss-120b | 11/11 | |
| gpt-oss-20b | 11/11 | |
| granite-4.1-30b | 11/11 | |
| granite-4.1-8b | 11/11 | |
| claude-sonnet-4.6 | cloud | 11/11 |
| qwen3.5-122b-a10b-cluster | 11/11 | |
| qwen3.5-4b | 11/11 | |
| qwen3.6-35b-a3b | 11/11 | |
| qwen3.6-35b-a3b-or | 11/11 | |
| qwen3.6-35b-bf16 | 11/11 | |
| qwen3-next-80b | 11/11 | |
| mistral-small-3.1-24b-instruct-2503 | 11/11 | |
| qwen3-coder-30b | 10/11 | |
| llama-3.3-70b | 9/11 | |
| qwen3.5-0.8b | 9/11 | |
| qwen3.5-2b | 9/11 | |
| deepseek-r1-distill-qwen-32b | 7/11 | |
| phi-4-reasoning-plus | 4/11 |
Benches — green = in gauntlet
gate.fidelity.csvgate.fidelity.enumgate.fidelity.longctxgate.fidelity.no_commasgate.tools.error_recoverygate.tools.multi_selectgate.tools.no_hallucinationgate.tools.result_usegate.tools.single_callgate.vitals.json_validgate.vitals.readygate.fidelity.multistepgate.fidelity.transformgate.vitals.answer_only