Benchmarks · coding
DeepSWE v1.1
Pass@1 on 113 original long-horizon software engineering tasks across 91 repos and 5 languages, run with mini-swe-agent in isolated environments (v1.1 fixes dependency drift and flaky tasks).
28 models · 70 results · data as of 2026-10-29 · deepswe.datacurve.ai ↗
Leaderboard
GPT-6 Astra xhigh
More from this benchmark
3 published measuresAgent steps per task70 results · lower is better
Time per task70 results · lower is better
Output tokens per task70 results · lower is better
All results
70 rows| # | Model | Effort | DeepSWE v1.1 | Subscription $ / task | API $ / task | Agent steps | Time | Output tokens | Harness |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | xhigh | 74.1% ±2.9 | $0.38 | $4.43 | 29 | 19 min | 30k | mini-swe-agent |
| 2 | Gemini 3.8 Flash | high | 73.8% ±1.4 | $0.096 | $2.36 | 166 | 11 min | 143k | mini-swe-agent |
| 3 | Opus 5 | max | 73.6% ±3.9 | $0.21 | $12 | 99 | 32 min | 118k | mini-swe-agent |
| 4 | GPT-6 Astra | high | 73.2% ±3.4 | $0.34 | $3.92 | 27 | 17 min | 27k | mini-swe-agent |
| 5 | GPT-6 Astra | max | 73.2% ±0.8 | $0.64 | $7.50 | 28 | 33 min | 61k | mini-swe-agent |
| 6 | Opus 5 | xhigh | 73.2% ±3.1 | $0.16 | $9.07 | 89 | 26 min | 92k | mini-swe-agent |
| 7 | Opus 5 | high | 72.8% ±1.9 | $0.11 | $6.08 | 73 | 19 min | 64k | mini-swe-agent |
| 8 | GPT-6 Astra | medium | 72.8% ±2.6 | $0.26 | $3.08 | 26 | 15 min | 20k | mini-swe-agent |
| 9 | GPT-5.6 Sol | max | 72.7% ±2.8 | $0.46 | $8.39 | 61 | 19 min | 60k | mini-swe-agent |
| 10 | Gemini 3.8 Flash | medium | 71.0% ±2.3 | $0.080 | $1.97 | 147 | 14 min | 125k | mini-swe-agent |
| 11 | GPT-5.6 Sol | xhigh | 70.7% ±0.8 | $0.26 | $4.70 | 44 | 13 min | 41k | mini-swe-agent |
| 12 | Fable 5 | xhigh | 69.9% ±3.2 | $1.51 | $13 | 68 | 24 min | 80k | mini-swe-agent |
| 13 | Fable 5 | max | 69.7% ±4.0 | $2.44 | $22 | 88 | 35 min | 119k | mini-swe-agent |
| 14 | GPT-5.6 Terra | max | 69.6% ±2.6 | $0.27 | $4.95 | 76 | 17 min | 72k | mini-swe-agent |
| 15 | GPT-5.6 Sol | high | 69.4% ±1.4 | $0.19 | $3.47 | 37 | 9.9 min | 28k | mini-swe-agent |
| 16 | GLM 5.3 | max | 69.0% ±3.0 | $0.41 | $3.99 | 124 | 35 min | 80k | mini-swe-agent |
| 17 | Opus 5 | medium | 68.9% ±1.2 | $0.057 | $3.29 | 52 | 13 min | 37k | mini-swe-agent |
| 18 | Fable 5 | high | 68.6% ±1.1 | $1.03 | $9.18 | 59 | 18 min | 57k | mini-swe-agent |
| 19 | Kimi K3 | max | 68.5% ±4.5 | $0.36 | $4.65 | 98 | 1.3 h | 82k | mini-swe-agent |
| 20 | Grok 4.6 | medium | 67.5% ±2.3 | $0.35 | $3.45 | 70 | 15 min | 50k | mini-swe-agent |
| 21 | GPT-5.6 Luna | max | 67.2% ±4.0 | $0.28 | $3.03 | 102 | 19 min | 73k | mini-swe-agent |
| 22 | GPT-5.5 | xhigh | 67.0% ±6.5 | $0.40 | $7.23 | 82 | 30 min | 46k | mini-swe-agent |
| 23 | GPT-6 Astra | low | 67.0% ±1.3 | $0.14 | $1.60 | 20 | 10 min | 11k | mini-swe-agent |
| 24 | Grok 4.6 | xhigh | 66.7% ±2.2 | $0.57 | $5.50 | 87 | 21 min | 71k | mini-swe-agent |
| 25 | Gemini 3.7 Flash | medium | 65.5% ±3.1 | $0.26 | $2.03 | 117 | 21 min | 94k | mini-swe-agent |
| 26 | Fable 5 | medium | 65.4% ±4.4 | $0.69 | $6.09 | 48 | 14 min | 40k | mini-swe-agent |
| 27 | Gemini 3.7 Flash | high | 65.3% ±1.8 | $0.28 | $2.18 | 125 | 21 min | 107k | mini-swe-agent |
| 28 | Grok 4.6 | high | 65.2% ±1.5 | $0.45 | $4.38 | 79 | 18 min | 61k | mini-swe-agent |
| 29 | GPT-5.5 | high | 64.4% ±3.1 | $0.28 | $5.10 | 62 | 34 min | 31k | mini-swe-agent |
| 30 | GLM 5.3 Flash | max | 63.4% ±4.4 | free | $0.48 | 123 | 26 min | 73k | mini-swe-agent |