SWE
GOAL
Calibration against the public SWE-bench style.
WHAT IT TESTS
Real GitHub issues fixed in real repositories.
HOW IT GRADES
FAIL->PASS: apply a fix, the repo's own tests must go green. Calibration sidebar.
Standings — gauntlet benches solved
| Model | Solved |
|---|
Benches — green = in gauntlet
ladder.swe.django-11620ladder.swe.django-11848ladder.swe.django-12589ladder.swe.django-13447ladder.swe.django-15388ladder.swe.flask-4045ladder.swe.flask-4992ladder.swe.flask-5063ladder.swe.requests-3362ladder.swe.requests-863ladder.swe.scikit-learn-11040ladder.swe.sphinx-8474ladder.swe.sympy-14024ladder.swe.sympy-18835ladder.swe.sympy-19007ladder.swe.sympy-24102ladder.swe.xarray-4493