← Benches
sup-pr67
SupportFlowgauntletDrawn from a real merged pull request (#67) in the SupportFlow SaaS repo — a genuine engineering fix used as a clean FAIL→PASS bench: the model must make the failing test suite pass the way the human author did.
Solved
2/4
runs passed
Models
4
have attempted
Harnesses
1
scaffolds tried
Runs (latest per model × harness — solved sorted first)
| Model | Harness | Result | Turns | tok/s | Latency | Ctx | When |
|---|---|---|---|---|---|---|---|
| devstral-small-2-2512 | hermes | PASS | 0 | — | — | 64k | 2026-07-07 |
| qwen3.6-35b-a3b | hermes | PASS | 0 | — | — | 64k | 2026-07-07 |
| gemma-4-26b-a4b-it | hermes | fail | 0 | — | — | 64k | 2026-07-07 |
| hermes-4.3-36b | hermes | fail | 0 | — | — | 64k | 2026-07-08 |