Benchmarks · coding
Real-SWE (Specific Labs)
Resolution rate of coding agents on private, licensed enterprise codebases, published as pass@1 over eight runs per task on a ten-task sample. Maintained by Specific Labs. Score: Resolution rate = pass@1 correctness averaged over eight independent runs per task, with a published 95% confidence interval.
8 models · 8 results · data as of 2026-10-29 · withspecific.com ↗
Leaderboard
Fable 5.1 max
All results
8 rows| # | Model | Effort | Real-SWE (Specific Labs) | Harness |
|---|---|---|---|---|
| 1 | Fable 5.1 | max | 38.8% | Claude Code |
| 2 | GPT-6 Astra | max | 33.8% | Codex CLI |
| 3 | Gemini 3.8 Flash | max | 31.3% | Gemini CLI |
| 4 | GLM 5.3 | max | 28.8% | Claude Code |
| 5 | Grok 4.6 | max | 23.8% | Grok Build |
| 6 | Muse Spark 1.3 | max | 23.8% | Muse Code |
| 7 | Kimi K3 | max | 18.8% | Kimi Code |
| 8 | GPT-5.6 Sol | max | 16.3% | Codex CLI |