Models
Every model under test — local and cloud. Coverage is the share of the full gauntlet attempted; Ctx is the peak context the model actually consumed (flags models that burn context differently). Click any model for its full profile, capability starfish, and history.
| Model | Overall | Coverage | Solved | Speed | Ctx | Quant | Last run | Best at | Status |
|---|---|---|---|---|---|---|---|---|---|
| claude-sonnet-4.6 · cloud | 43/65 | 46/48 | 23.8 t/s | 6.4% | cloud | 2026-07-06 | gate | evaluated | |
| deepseek-r1-32b | 5/65 | 4/10 | — | — | — | 2026-07-02 | php | ||
| deepseek-r1-distill-qwen-32b DeepSeek | 43/65 | 27/48 | 5 t/s | 19.1% | Q8_0 | 2026-07-03 | coding | evaluated | |
| devstral-small-2-2512 Mistral AI · 24B | 46/65 | 43/62 | 8.2 t/s | 18.6% | Q8 | 2026-06-23 | gate | evaluated | |
| gemini-2.5-flash-lite · cloud | 43/65 | 36/48 | 26 t/s | 19.8% | cloud | 2026-07-01 | gate | evaluated | |
| gemma-4-26b-a4b-it Google · 26B-A4B | 65/65 | 72/109 | 22.3 t/s | 19.9% | Q8 | 2026-07-01 | coding | evaluated | |
| gpt-oss-120b OpenAI | 43/65 | 45/48 | 47.1 t/s | 17.6% | MXFP4 | 2026-06-17 | gate | evaluated | |
| gpt-oss-20b OpenAI · 20B | 43/65 | 73/100 | 72.1 t/s | 17.7% | MXFP4 | 2026-07-01 | coding | evaluated | |
| granite-4.1-30b IBM · 30B | 43/65 | 47/50 | 7.1 t/s | 18.8% | Q8 | 2026-07-02 | gate | evaluated | |
| granite-4.1-8b IBM | 43/65 | 38/59 | 20.5 t/s | 18.8% | Q8 | 2026-06-23 | gate | evaluated | |
| hermes-4.3-36b | 22/65 | 4/22 | — | — | — | 2026-07-08 | supportflow | ||
| llama-3.3-70b Meta | 41/65 | 39/53 | 2.8 t/s | 17.5% | Q8 | 2026-06-23 | coding | evaluated | |
| mistral-small-3.1-24b-instruct-2503 Mistral AI · 24B | 43/65 | 43/59 | 8.2 t/s | 18.6% | Q8 | 2026-06-23 | gate | evaluated | |
| phi-4 | 5/65 | 5/10 | — | — | — | 2026-07-02 | php | ||
| phi-4-reasoning-plus Microsoft | 43/65 | 19/48 | 14.1 t/s | 50.6% | Q8 | 2026-07-02 | coding | evaluated | |
| Qwen3-235B-A22B (cluster) 235B-A22B | 20/65 | 17/20 | — | — | Q4 | 2026-06-30 | php | shelved | |
| qwen3-coder-30b Alibaba · 30B-A3B | 43/65 | 50/93 | 30.8 t/s | 18.6% | Q8 | 2026-07-01 | gate | evaluated | |
| qwen3-next-80b Alibaba · 80B-A3B | 58/65 | 53/79 | 34.3 t/s | 18.6% | Q8 | 2026-07-02 | coding | active | |
| qwen3.5-0.8b Alibaba | 43/65 | 20/54 | 84.7 t/s | 19.2% | Q8 | 2026-07-02 | agentic | evaluated | |
| qwen3.5-122b-a10b-cluster | 43/65 | 43/48 | 16.3 t/s | 21.6% | Q8_0 | 2026-06-29 | gate | ||
| qwen3.5-2b Alibaba | 43/65 | 31/54 | 68.9 t/s | 19.2% | Q8 | 2026-07-02 | agentic | evaluated | |
| qwen3.5-4b Alibaba · 4B | 43/65 | 60/77 | 43.4 t/s | 20.9% | Q8 | 2026-07-02 | gate | evaluated | |
| qwen3.6-27b Alibaba | 43/65 | 85/90 | 7.8 t/s | 19.8% | Q8_0 | 2026-07-05 | gate | evaluated | |
| qwen3.6-35b-a3b Alibaba · 35B-A3B | 65/65 | 80/122 | 46 t/s | 22.9% | Q8 | 2026-07-05 | coding | in-prod | |
| qwen3.6-35b-a3b Alibaba · 35B-A3B | 43/65 | 46/48 | 33.1 t/s | 20.9% | BF16 | 2026-07-02 | gate | evaluated | |
| qwen3.6-35b-a3b Alibaba · 35B-A3B · cloud | 43/65 | 46/48 | 97.3 t/s | 21.4% | cloud | 2026-07-01 | gate | evaluated |