Benchmarks · reasoning
CritPt v2 Hard Subset
The private 40-question hard subset of CritPt v2: the 70 challenges minus the 30 that seven frontier models solved in all five attempts at launch. Code-execution tool in an offline sandbox; accuracy.
Category
28 models · 71 results · data as of 2026-10-29 · artificialanalysis.ai ↗
Leaderboard
GPT-6 Astra high
More from this benchmark
4 published measuresScore vs any published measure4 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | ||||
|---|---|---|---|---|
| GPT-6 Astra high | 1,514,181 | 151,287 | 298,641 | 1,125,385 |
| GPT-6.1 Sol max | 2,077,656 | 156,223 | 597,534 | 1,480,931 |
| GPT-6.1 Terra max | 10,774,559 | 203,652 | 1,331,553 | 9,331,077 |
| Opus 5.5 xhigh, fallback | 9,857,857 | 266,643 | 1,198,905 | 8,355,136 |
| Gemini 4 Carbon high | 73,733,125 | 1,175,506 | 4,060,423 | 67,661,911 |
| Fable 5.1 high, fallback | 8,586,844 | 248,297 | 1,245,272 | 7,100,786 |
| Sonnet 5.5 max, fallback | 59,677,774 | 494,212 | 5,143,006 | 54,052,155 |
| GPT-5.6 Sol max | 10,515,506 | 198,032 | 1,031,028 | 9,350,938 |
| Gemini 4 Argon high | 41,985,516 | 609,862 | 2,250,466 | 38,702,215 |
| Muse Spark 1.3 max | 45,639,342 | 638,644 | 2,929,271 | 41,983,634 |
| Haiku 5.5 xhigh | 25,088,885 | 319,679 | 3,124,863 | 21,799,143 |
| Gemini 3.8 Flash medium | 48,396,639 | 661,256 | 1,344,240 | 46,124,377 |
| GPT-6 Luna max | 16,104,278 | 179,065 | 1,772,017 | 14,793,738 |
| Grok 4.7 low | 29,470,202 | 421,977 | 1,387,075 | 27,422,031 |
| Kimi K3 max | 23,567,341 | 267,772 | 1,495,257 | 21,667,352 |
| MiMo V2.6 Pro | 67,497,157 | 450,009 | 4,201,659 | 62,535,477 |
| GLM 5.3 max | 98,920,363 | 553,517 | 5,825,702 | 92,412,450 |
| MiMo V2.6 Flash | 70,256,160 | 452,410 | 4,741,652 | 64,751,934 |
| MiniMax M3 | 84,624,728 | 493,328 | 4,505,758 | 79,414,452 |
| DeepSeek V4.1 Flash 1017 max | 71,771,264 | 437,542 | 4,331,245 | 66,861,235 |
| GLM 5.3 Flash max | 156,742,537 | 623,937 | 14,825,083 | 142,433,173 |
| Inkling Small | 25,518,250 | 385,350 | 2,280,085 | 23,013,837 |
| Inkling xhigh | 34,639,650 | 409,727 | 2,860,218 | 31,500,765 |
| Qwen 3.8 27B medium | 75,865,686 | 731,867 | 3,935,909 | 70,857,059 |
| GPT-5 high | 3,823,129 | 207,182 | 909,338 | 3,182,366 |
All results
71 rows| # | Model | Effort | CritPt v2 Hard Subset | Subscription $ / task | API $ / task | Harness |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | high | 71.5% | $0.061 | $0.71 | Stirrup |
| 2 | GPT-6 Astra | medium | 71.0% | $0.035 | $0.41 | Stirrup |
| 3 | GPT-6 Astra | max | 70.5% | $0.097 | $1.14 | Stirrup |
| 4 | GPT-6 Astra | xhigh | 70.5% | $0.073 | $0.85 | Stirrup |
| 5 | GPT-6.1 Sol | max | 68.5% | $0.027 | $0.23 | Stirrup |
| 6 | GPT-6.1 Sol | xhigh | 68.0% | $0.019 | $0.17 | Stirrup |
| 7 | GPT-6.1 Sol | high | 67.5% | $0.016 | $0.13 | Stirrup |
| 8 | GPT-6.1 Sol | medium | 65.0% | $0.008 | $0.071 | Stirrup |
| 9 | GPT-6.1 Terra | max | 64.5% | $0.25 | $0.25 | Stirrup |
| 10 | Opus 5.5 | xhigh | 63.0% | $0.015 | $0.97 | Stirrup |
| 11 | Gemini 4 Carbon | high | 63.0% | $3.91 | $3.91 | Stirrup |
| 12 | Opus 5.5 | max | 61.0% | $0.039 | $2.58 | Stirrup |
| 13 | Fable 5.1 | high | 60.5% | $0.37 | $2.39 | Stirrup |
| 14 | GPT-6.1 Terra | high | 60.5% | $0.069 | $0.069 | Stirrup |
| 15 | GPT-6 Astra | low | 60.0% | $0.020 | $0.24 | Stirrup |
| 16 | GPT-6.1 Sol | low | 59.5% | $0.005 | $0.043 | Stirrup |
| 17 | Opus 5.5 | high | 58.0% | $0.006 | $0.38 | Stirrup |
| 18 | GPT-6.1 Terra | xhigh | 57.5% | $0.12 | $0.12 | Stirrup |
| 19 | Fable 5.1 | medium | 56.5% | $0.22 | $1.38 | Stirrup |
| 20 | Fable 5.1 | xhigh | 56.5% | $0.68 | $4.33 | Stirrup |
| 21 | Sonnet 5.5 | max | 56.0% | $0.029 | $1.93 | Stirrup |
| 22 | GPT-5.6 Sol | max | 56.0% | Stirrup | ||
| 23 | GPT-6.1 Terra | medium | 56.0% | $0.040 | $0.040 | Stirrup |
| 24 | Sonnet 5.5 | xhigh | 54.0% | $0.007 | $0.45 | Stirrup |
| 25 | Opus 5.5 | medium | 52.5% | $0.004 | $0.25 | Stirrup |
| 26 | Gemini 4 Argon | high | 52.5% | $0.77 | $2.15 | Stirrup |
| 27 | Fable 5.1 | max | 51.0% | $0.71 | $4.55 | Stirrup |
| 28 | GPT-6.1 Terra | low | 48.5% | $0.018 | $0.018 | Stirrup |
| 29 | Fable 5.1 | low | 47.5% | $0.14 | $0.89 | Stirrup |
| 30 | Sonnet 5.5 | high | 47.5% | $0.003 | $0.18 | Stirrup |