Benchmarks · Coding · Agents & computer use
LongTail-SWE
Can a coding agent fix real GitHub issues in languages most benchmarks skip? 202 tasks across 13 languages, such as Elixir, OCaml and Haskell, three attempts each, with every model in the same agent setup. The share of attempts that pass the tests.
11 models · 11 results · data as of 2026-11-18 · longtail-swe.com ↗
Leaderboard
Opus 5.5 high
More from this benchmark
5 published measuresAgent steps per task11 results · lower is better
9 more models under All models in settings.
Score vs any published measure4 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | |||
|---|---|---|---|
| Opus 5.5 high | 89.6 | 74.3 | 1.2 |
| GPT-6.1 Sol high | 85.6 | 76.7 | 0.7 |
| Kimi K3.1 max | 87.1 | 64.4 | 1.5 |
| GPT-5.6 Sol high | 87.6 | 63.9 | 1.3 |
| DeepSeek V4.1 Flash 1017 max | 86.1 | 64.4 | 0.9 |
| Kimi K3 max | 82.2 | 60.4 | 1.4 |
| GPT-5.6 Luna xhigh | 84.2 | 57.4 | 0.6 |
| DeepSeek V4 Flash high | 72.8 | 39.6 | 1.6 |
| GLM 5.3 Flash max | 70.3 | 23.8 | 0.8 |
| Qwen 3.8 27B xhigh | 62.9 | 20.8 | 0.8 |
| Qwen 3.6 27B max | 23.8 | 9.41 | 0.7 |
All results
11 rows| # | Model | Effort | LongTail-SWE | Agent steps | Harness |
|---|---|---|---|---|---|
| 1 | Opus 5.5 | high | 82.0% | 53 | SWE-agent-like scaffold |
| 2 | GPT-6.1 Sol | high | 81.8% | 27 | SWE-agent-like scaffold |
| 3 | Kimi K3.1 | max | 77.2% | 50 | SWE-agent-like scaffold |
| 4 | GPT-5.6 Sol | high | 76.2% | 34 | SWE-agent-like scaffold |
| 5 | DeepSeek V4.1 Flash 1017 | max | 75.9% | 159 | SWE-agent-like scaffold |
| 6 | Kimi K3 | max | 72.1% | 154 | SWE-agent-like scaffold |
| 7 | GPT-5.6 Luna | xhigh | 71.8% | 77 | SWE-agent-like scaffold |
| 8 | DeepSeek V4 Flash | high | 56.8% | 200 | SWE-agent-like scaffold |
| 9 | GLM 5.3 Flash | max | 48.4% | 314 | SWE-agent-like scaffold |
| 10 | Qwen 3.8 27B | xhigh | 42.9% | 214 | SWE-agent-like scaffold |
| 11 | Qwen 3.6 27B | max | 16.5% | 113 | SWE-agent-like scaffold |