Benchmarks · coding
ProgramBench v1 Almost Resolved
Share of public ProgramBench v1 tasks where the implementation passes at least 95% of hidden behavioral tests.
51 models · 52 results · updated 2026-10-20 · vals.ai ↗
Leaderboard
Opus 5.5 max
More from this benchmark
47 published measuresScore vs any published measure47 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | |||||
|---|---|---|---|---|---|
| Opus 5.5 max, fallback | 25655.3M | 124.2M | 25469.8M | 185.4M | |
| GPT-6 Astra max | 751.2M | 19.6M | 9.2M | 693.9M | |
| Sonnet 5.5 max, fallback | 26746.1M | 138.9M | 26556.7M | 189.3M | |
| GPT-6.1 Sol max | 1004.1M | 30.4M | 18.1M | 950.1M | 54.0M |
| Gemini 4 Argon high | 5363.0M | 40.8M | 18.7M | 4930.4M | 0 |
| GPT-6 Sol max | 6137.6M | 31.9M | 13.9M | 6073.5M | 63.2M |
| Grok 4.7 xhigh | 4016.8M | 40.4M | 21.7M | 1901.6M | 0 |
| Grok 4.6 high | 2164.9M | 62.7M | 44.3M | 902.6M | 0 |
| Muse Spark 1.3 max | 10898.5M | 59.1M | 24.7M | 2731.6M | 0 |
| Kimi K3 max | 22310.4M | 87.2M | 54.3M | ||
| MiMo V2.6 Flash high | 15388.9M | 107.7M | 80.1M | 15236.8M | 0 |
| DeepSeek V4.1 Flash max | 14371.7M | 70.1M | 46.7M | 14340.4M | 0 |
| GLM 5.3 max | 15021.5M | 68.4M | 45.5M | 14859.3M | 0 |
| GPT-5.6 Terra max | 3001.7M | 22.2M | 9.5M | 2888.8M | |
| Gemini 3.8 Flash high | 8981.1M | 33.8M | 15.1M | 8555.7M | 0 |
| DeepSeek V4 Pro 0813 max | 11180.5M | 36.7M | 17.1M | 11150.3M | 0 |
| MiMo V2.6 Pro high | 8121.2M | 81.7M | 62.8M | 8016.9M | 0 |
| Gemini 3.7 Flash high | 5272.5M | 33.0M | 9.6M | 5006.3M | 0 |
| DeepSeek V4 Flash 0731 high | 11633.5M | 60.4M | 37.8M | 11608.8M | 0 |
| GPT-6 Luna max | 1373.3M | 27.1M | 16.7M | 1324.5M | 47.9M |
| GLM 5.3 Flash max | 8901.5M | 41.7M | 22.3M | 8651.9M | 0 |
| Muse Spark 1.2 xhigh | 2012.7M | 28.0M | 14.1M | 1931.2M | 0 |
| Qwen 3.8 27B xhigh | 6949.8M | 41.1M | 29.5M | 0 | |
| Inkling max | 4102.8M | 18.6M | 4065.5M | 0 |
All results
52 rows| # | Model | Effort | Score | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | 65.0% | $1.03 | $68 | mini-SWE-agent |
| 2 | Fable 5.1 | max | 53.5% | $9.12 | $59 | mini-SWE-agent |
| 3 | GPT-6 Astra | max | 50.0% | $0.99 | $12 | mini-SWE-agent |
| 4 | Sonnet 5.5 | max | 49.5% | $0.54 | $36 | mini-SWE-agent |
| 5 | GPT-6.1 Sol | max | 46.0% | $0.33 | $2.87 | mini-SWE-agent |
| 6 | Opus 5 | max | 41.5% | $1.04 | $60 | mini-SWE-agent |
| 7 | Gemini 4 Argon | high | 41.0% | $6.29 | $18 | mini-SWE-agent |
| 8 | Fable 5 | max | 33.0% | $8.53 | $76 | mini-SWE-agent |
| 9 | GPT-6 Sol | max | 32.5% | $0.72 | $12 | mini-SWE-agent |
| 10 | Haiku 5.5 | max | 29.5% | $0.14 | $6.83 | mini-SWE-agent |
| 11 | GPT-5.6 Sol | max | 23.0% | $0.84 | $15 | mini-SWE-agent |
| 12 | Grok 4.7 | xhigh | 21.5% | $4.32 | $46 | mini-SWE-agent |
| 13 | Grok 4.6 | high | 19.0% | $1.72 | $17 | mini-SWE-agent |
| 14 | Muse Spark 1.3 | max | 19.0% | $27 | $54 | mini-SWE-agent |
| 15 | Kimi K3 | max | 17.5% | $5.45 | $70 | mini-SWE-agent |
| 16 | MiMo V2.6 Flash | high | 17.5% | free | $0.47 | mini-SWE-agent |
| 17 | DeepSeek V4.1 Flash | max | 16.5% | $0.056 | $0.90 | mini-SWE-agent |
| 18 | GLM 5.3 | max | 16.5% | $2.28 | $22 | mini-SWE-agent |
| 19 | Opus 4.8 | max | 15.5% | $0.54 | $31 | mini-SWE-agent |
| 20 | GPT-5.6 Terra | max | 15.0% | $0.30 | $5.38 | mini-SWE-agent |
| 21 | Gemini 3.8 Flash | high | 14.0% | $0.44 | $11 | mini-SWE-agent |
| 22 | Sonnet 5 | max | 13.5% | $0.42 | $24 | mini-SWE-agent |
| 23 | DeepSeek V4 Pro 0813 | max | 13.5% | $3.41 | $3.41 | mini-SWE-agent |
| 24 | Muse Spark 1.3 | xhigh | 13.0% | $5.78 | $12 | mini-SWE-agent |
| 25 | MiMo V2.6 Pro | high | 12.5% | $0.61 | $0.73 | mini-SWE-agent |
| 26 | Qwen 3.8 Max | max | 10.5% | $12 | $12 | mini-SWE-agent |
| 27 | Opus 4.7 | max | 9.5% | mini-SWE-agent | ||
| 28 | Gemini 3.7 Flash | high | 9.5% | $0.92 | $7.13 | mini-SWE-agent |
| 29 | GLM 5.2 | max | 9.5% | $1.10 | $13 | mini-SWE-agent |
| 30 | GPT-5.5 | xhigh | 9.5% | $0.38 | $6.95 | mini-SWE-agent |