Benchmarks · coding
ProgramBench v1 Raw Pass Rate
Average percentage of hidden behavioral tests passed per public ProgramBench v1 task.
51 models · 52 results · updated 2026-10-20 · vals.ai ↗
Leaderboard
Opus 5.5 max
More from this benchmark
48 published measuresScore vs any published measure48 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | |||||
|---|---|---|---|---|---|
| Opus 5.5 max, fallback | 25655.3M | 124.2M | 25469.8M | 185.4M | |
| GPT-6 Astra max | 751.2M | 19.6M | 9.2M | 693.9M | |
| GPT-6.1 Sol max | 1004.1M | 30.4M | 18.1M | 950.1M | 54.0M |
| Gemini 4 Argon high | 5363.0M | 40.8M | 18.7M | 4930.4M | 0 |
| GPT-6 Sol max | 6137.6M | 31.9M | 13.9M | 6073.5M | 63.2M |
| Sonnet 5.5 max, fallback | 26746.1M | 138.9M | 26556.7M | 189.3M | |
| Grok 4.7 xhigh | 4016.8M | 40.4M | 21.7M | 1901.6M | 0 |
| Grok 4.6 high | 2164.9M | 62.7M | 44.3M | 902.6M | 0 |
| GPT-5.6 Terra max | 3001.7M | 22.2M | 9.5M | 2888.8M | |
| DeepSeek V4.1 Flash max | 14371.7M | 70.1M | 46.7M | 14340.4M | 0 |
| Gemini 3.8 Flash high | 8981.1M | 33.8M | 15.1M | 8555.7M | 0 |
| Muse Spark 1.3 xhigh | 8950.5M | 45.5M | 19.5M | 8236.6M | 0 |
| MiMo V2.6 Flash high | 15388.9M | 107.7M | 80.1M | 15236.8M | 0 |
| DeepSeek V4 Pro 0813 max | 11180.5M | 36.7M | 17.1M | 11150.3M | 0 |
| MiMo V2.6 Pro high | 8121.2M | 81.7M | 62.8M | 8016.9M | 0 |
| Gemini 3.7 Flash high | 5272.5M | 33.0M | 9.6M | 5006.3M | 0 |
| GLM 5.3 max | 15021.5M | 68.4M | 45.5M | 14859.3M | 0 |
| GPT-6 Luna max | 1373.3M | 27.1M | 16.7M | 1324.5M | 47.9M |
| Kimi K3 max | 22310.4M | 87.2M | 54.3M | ||
| Muse Spark 1.2 xhigh | 2012.7M | 28.0M | 14.1M | 1931.2M | 0 |
| DeepSeek V4 Flash 0731 high | 11633.5M | 60.4M | 37.8M | 11608.8M | 0 |
| GLM 5.3 Flash max | 8901.5M | 41.7M | 22.3M | 8651.9M | 0 |
| Inkling max | 4102.8M | 18.6M | 4065.5M | 0 | |
| Qwen 3.8 27B xhigh | 6949.8M | 41.1M | 29.5M | 0 |
All results
52 rows| # | Model | Effort | Score | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | 87.0% | $1.03 | $68 | mini-SWE-agent |
| 2 | GPT-6 Astra | max | 85.4% | $0.99 | $12 | mini-SWE-agent |
| 3 | GPT-6.1 Sol | max | 85.3% | $0.33 | $2.87 | mini-SWE-agent |
| 4 | Gemini 4 Argon | high | 83.7% | $6.29 | $18 | mini-SWE-agent |
| 5 | Fable 5.1 | max | 82.7% | $9.12 | $59 | mini-SWE-agent |
| 6 | Opus 5 | max | 82.3% | $1.04 | $60 | mini-SWE-agent |
| 7 | GPT-6 Sol | max | 81.9% | $0.72 | $12 | mini-SWE-agent |
| 8 | Haiku 5.5 | max | 78.7% | $0.14 | $6.83 | mini-SWE-agent |
| 9 | GPT-5.6 Sol | max | 77.6% | $0.84 | $15 | mini-SWE-agent |
| 10 | Fable 5 | max | 76.8% | $8.53 | $76 | mini-SWE-agent |
| 11 | Sonnet 5.5 | max | 76.3% | $0.54 | $36 | mini-SWE-agent |
| 12 | Grok 4.7 | xhigh | 75.7% | $4.32 | $46 | mini-SWE-agent |
| 13 | Grok 4.6 | high | 74.0% | $1.72 | $17 | mini-SWE-agent |
| 14 | GPT-5.6 Terra | max | 72.3% | $0.30 | $5.38 | mini-SWE-agent |
| 15 | Sonnet 5 | max | 72.1% | $0.42 | $24 | mini-SWE-agent |
| 16 | DeepSeek V4.1 Flash | max | 72.0% | $0.056 | $0.90 | mini-SWE-agent |
| 17 | Opus 4.8 | max | 71.9% | $0.54 | $31 | mini-SWE-agent |
| 18 | Gemini 3.8 Flash | high | 71.9% | $0.44 | $11 | mini-SWE-agent |
| 19 | Muse Spark 1.3 | xhigh | 71.7% | $5.78 | $12 | mini-SWE-agent |
| 20 | MiMo V2.6 Flash | high | 71.7% | free | $0.47 | mini-SWE-agent |
| 21 | GPT-5.5 | xhigh | 70.8% | $0.38 | $6.95 | mini-SWE-agent |
| 22 | DeepSeek V4 Pro 0813 | max | 70.1% | $3.41 | $3.41 | mini-SWE-agent |
| 23 | MiMo V2.6 Pro | high | 69.8% | $0.61 | $0.73 | mini-SWE-agent |
| 24 | Muse Spark 1.3 | max | 68.7% | $27 | $54 | mini-SWE-agent |
| 25 | Gemini 3.7 Flash | high | 68.7% | $0.92 | $7.13 | mini-SWE-agent |
| 26 | GPT-5.6 Luna | max | 68.3% | $0.088 | $0.94 | mini-SWE-agent |
| 27 | GLM 5.3 | max | 66.3% | $2.28 | $22 | mini-SWE-agent |
| 28 | GPT-6 Luna | max | 66.2% | $0.027 | $0.18 | mini-SWE-agent |
| 29 | Grok 4.5 | high | 65.2% | $0.65 | $4.12 | mini-SWE-agent |
| 30 | Opus 4.7 | max | 64.2% | mini-SWE-agent |