Benchmarks · coding
ProgramBench v1 Fully Resolved
Share of public ProgramBench v1 tasks where the implementation passes every hidden behavioral test.
51 models · 52 results · updated 2026-10-20 · vals.ai ↗
Leaderboard
Opus 5.5 max
More from this benchmark
46 published measuresScore vs any published measure46 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | |||||
|---|---|---|---|---|---|
| Opus 5.5 max, fallback | 25655.3M | 124.2M | 25469.8M | 185.4M | |
| Sonnet 5.5 max, fallback | 26746.1M | 138.9M | 26556.7M | 189.3M | |
| GPT-6 Astra max | 751.2M | 19.6M | 9.2M | 693.9M | |
| GPT-6.1 Sol max | 1004.1M | 30.4M | 18.1M | 950.1M | 54.0M |
| Gemini 4 Argon high | 5363.0M | 40.8M | 18.7M | 4930.4M | 0 |
| Muse Spark 1.3 max | 10898.5M | 59.1M | 24.7M | 2731.6M | 0 |
| GPT-6 Sol max | 6137.6M | 31.9M | 13.9M | 6073.5M | 63.2M |
| Kimi K3 max | 22310.4M | 87.2M | 54.3M | ||
| GLM 5.3 max | 15021.5M | 68.4M | 45.5M | 14859.3M | 0 |
| Gemini 3.8 Flash high | 8981.1M | 33.8M | 15.1M | 8555.7M | 0 |
| Grok 4.6 high | 2164.9M | 62.7M | 44.3M | 902.6M | 0 |
| DeepSeek V4.1 Flash max | 14371.7M | 70.1M | 46.7M | 14340.4M | 0 |
| GPT-5.6 Terra max | 3001.7M | 22.2M | 9.5M | 2888.8M | |
| GPT-6 Luna max | 1373.3M | 27.1M | 16.7M | 1324.5M | 47.9M |
| Grok 4.7 xhigh | 4016.8M | 40.4M | 21.7M | 1901.6M | 0 |
| MiMo V2.6 Flash high | 15388.9M | 107.7M | 80.1M | 15236.8M | 0 |
| MiMo V2.6 Pro high | 8121.2M | 81.7M | 62.8M | 8016.9M | 0 |
| Muse Spark 1.2 xhigh | 2012.7M | 28.0M | 14.1M | 1931.2M | 0 |
| DeepSeek V4 Flash 0731 high | 11633.5M | 60.4M | 37.8M | 11608.8M | 0 |
| DeepSeek V4 Pro 0813 max | 11180.5M | 36.7M | 17.1M | 11150.3M | 0 |
| Gemini 3.7 Flash high | 5272.5M | 33.0M | 9.6M | 5006.3M | 0 |
| GLM 5.3 Flash max | 8901.5M | 41.7M | 22.3M | 8651.9M | 0 |
| Inkling max | 4102.8M | 18.6M | 4065.5M | 0 | |
| Qwen 3.8 27B xhigh | 6949.8M | 41.1M | 29.5M | 0 |
All results
52 rows| # | Model | Effort | Score | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | 18.5% | $1.03 | $68 | mini-SWE-agent |
| 2 | Fable 5.1 | max | 7.0% | $9.12 | $59 | mini-SWE-agent |
| 3 | Sonnet 5.5 | max | 6.5% | $0.54 | $36 | mini-SWE-agent |
| 4 | GPT-6 Astra | max | 5.5% | $0.99 | $12 | mini-SWE-agent |
| 5 | Opus 5 | max | 3.0% | $1.04 | $60 | mini-SWE-agent |
| 6 | GPT-6.1 Sol | max | 3.0% | $0.33 | $2.87 | mini-SWE-agent |
| 7 | Haiku 5.5 | max | 2.5% | $0.14 | $6.83 | mini-SWE-agent |
| 8 | Gemini 4 Argon | high | 2.5% | $6.29 | $18 | mini-SWE-agent |
| 9 | Muse Spark 1.3 | max | 2.5% | $27 | $54 | mini-SWE-agent |
| 10 | Fable 5 | max | 2.0% | $8.53 | $76 | mini-SWE-agent |
| 11 | GPT-6 Sol | max | 2.0% | $0.72 | $12 | mini-SWE-agent |
| 12 | Kimi K3 | max | 2.0% | $5.45 | $70 | mini-SWE-agent |
| 13 | GLM 5.3 | max | 1.5% | $2.28 | $22 | mini-SWE-agent |
| 14 | GPT-5.6 Sol | max | 1.5% | $0.84 | $15 | mini-SWE-agent |
| 15 | Opus 4.8 | max | 1.0% | $0.54 | $31 | mini-SWE-agent |
| 16 | Gemini 3.8 Flash | high | 1.0% | $0.44 | $11 | mini-SWE-agent |
| 17 | Grok 4.6 | high | 1.0% | $1.72 | $17 | mini-SWE-agent |
| 18 | Sonnet 4.6 | max | 0.5% | mini-SWE-agent | ||
| 19 | DeepSeek V4.1 Flash | max | 0.5% | $0.056 | $0.90 | mini-SWE-agent |
| 20 | GLM 5.2 | max | 0.5% | $1.10 | $13 | mini-SWE-agent |
| 21 | GPT-5.5 | xhigh | 0.5% | $0.38 | $6.95 | mini-SWE-agent |
| 22 | GPT-5.6 Terra | max | 0.5% | $0.30 | $5.38 | mini-SWE-agent |
| 23 | GPT-6 Luna | max | 0.5% | $0.027 | $0.18 | mini-SWE-agent |
| 24 | Grok 4.7 | xhigh | 0.5% | $4.32 | $46 | mini-SWE-agent |
| 25 | Inkling Small | max | 0.5% | free | $0.48 | mini-SWE-agent |
| 26 | MiMo V2.6 Flash | high | 0.5% | free | $0.47 | mini-SWE-agent |
| 27 | MiMo V2.6 Pro | high | 0.5% | $0.61 | $0.73 | mini-SWE-agent |
| 28 | Muse Spark 1.2 | xhigh | 0.5% | $1.23 | $2.47 | mini-SWE-agent |
| 29 | Muse Spark 1.3 | xhigh | 0.5% | $5.78 | $12 | mini-SWE-agent |
| 30 | Opus 4.7 | max | 0.0% | mini-SWE-agent |