Benchmarks · Coding · Agents & computer use
Ramp SWE-Bench v2
500 software tasks built from work Ramp's engineers shipped to production, each model running in its lab's own coding harness at high reasoning effort. Share resolved in a single attempt, with web tools off.
Category
16 models · 16 results · updated 2026-11-17 · labs.ramp.com ↗
Leaderboard
GPT-6.2 Astra high
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's most subsidized plan, or API price with no plan data. 2 more models under All models in settings.
More from this benchmark
2 published measuresTime per task16 results · lower is better
3 more models under All models in settings.
Output tokens per task16 results · lower is better
All results
16 rows| # | Model | Effort | Ramp SWE-Bench v2 | Subscription $ / task | API $ / task | Time | Output tokens | Harness |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6.2 Astra | high | 76.4% ±3.7 | $0.22 | $3.31 | 6.6 min | 9.6k | codex |
| 2 | Gemini 4 Carbon | high | 75.4% ±3.8 | $0.26 | $4.25 | 56 min | 72k | opencode |
| 3 | Fable 5.2 | high | 73.4% ±3.9 | $0.14 | $3.79 | 20 min | 31k | claude-code |
| 4 | GPT-6.2 Sol | high | 73.4% ±3.9 | $0.035 | $1.05 | 7.5 min | 11k | codex |
| 5 | Gemini 4 Hydrogen | high | 71.6% ±3.9 | $0.086 | $2.11 | 24 min | 48k | opencode |
| 6 | GPT-6.1 Terra | high | 70.4% ±4.0 | $0.016 | $0.46 | 8.3 min | 16k | codex |
| 7 | Opus 5.5 | high | 67.8% ±4.1 | $0.020 | $1.53 | 15 min | 16k | claude-code |
| 8 | Grok 4.7 | high | 64.4% ±4.2 | $0.30 | $4.32 | 20 min | 51k | grok-build |
| 9 | GPT-6 Astra | high | 63.4% ±4.2 | $0.19 | $5.22 | 8.3 min | 15k | codex |
| 10 | Sonnet 5.5 | high | 63.2% ±4.2 | $0.006 | $0.47 | 10 min | 11k | claude-code |
| 11 | DeepSeek V4.1 Flash 1017 | high | 60.8% ±4.3 | free | $0.22 | 19 min | 40k | deepseek-harness |
| 12 | GPT-6.1 Sol | high | 59.6% ±4.3 | $0.025 | $0.74 | 8.0 min | 16k | codex |
| 13 | Muse Wonder | high | 58.6% ±4.3 | $3.06 | $4.59 | 17 min | 39k | muse-code |
| 14 | Haiku 5.5 | high | 57.4% ±4.3 | $0.005 | $0.28 | 14 min | 36k | claude-code |
| 15 | GPT-6 Luna | high | 52.4% ±4.4 | $0.005 | $0.071 | 9.5 min | 28k | codex |
| 16 | Muse Ember | high | 51.8% ±4.4 | $0.11 | $0.11 | 14 min | 42k | muse-code |