Benchmarks · coding
WeirdML v2
17 ML tasks where models write Python pipelines from scratch (shape inference, chess winners, Kolmogorov shuffles); mean accuracy across tasks.
111 models · 127 results · data as of 2026-10-29 · htihle.github.io ↗
Leaderboard
GPT-6 Astra Pro xhigh
All results
127 rows| # | Model | Effort | WeirdML v2 | Subscription $ / task | API $ / task |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra Pro | xhigh | 93.6% | $29 | $29 |
| 2 | GPT-6 Astra | xhigh | 93.3% | $0.66 | $7.75 |
| 3 | Fable 5.1 | xhigh | 92.9% | $1.82 | $12 |
| 4 | GPT-6 Astra | high | 92.9% | $0.25 | $2.98 |
| 5 | Fable 5.1 | high | 92.3% | $0.50 | $3.20 |
| 6 | Fable 5 | xhigh | 91.9% | $0.95 | $8.42 |
| 7 | Opus 5 | xhigh | 91.8% | $0.082 | $4.74 |
| 8 | Opus 5 | high | 91.6% | $0.038 | $2.19 |
| 9 | GPT-5.6 Sol Pro | xhigh | 89.4% | $3.87 | $3.87 |
| 10 | GPT-5.6 Sol | max | 88.8% | $0.069 | $1.25 |
| 11 | Fable 5 | max | 87.8% | $0.35 | $3.11 |
| 12 | GPT-5.6 Sol | xhigh | 87.0% | $0.13 | $2.31 |
| 13 | Opus 5 | max | 86.3% | $0.025 | $1.42 |
| 14 | GPT-5.5 | xhigh | 84.9% | $0.14 | $2.57 |
| 15 | Gemini 3.8 Flash | max | 84.8% | $0.51 | $0.51 |
| 16 | Opus 4.8 | xhigh | 82.9% | $0.041 | $2.35 |
| 17 | Kimi K3 | max | 82.6% | $0.16 | $2.05 |
| 18 | GPT-5.6 Terra | max | 78.3% | $0.014 | $0.25 |
| 19 | Opus 4.6 | max | 77.9% | $5.78 | $5.78 |
| 20 | GPT-5.3 Codex | xhigh | 77.9% | $2.45 | $2.45 |
| 21 | GPT-5.4 | xhigh | 77.7% | $5.72 | $5.72 |
| 22 | Opus 4.7 | max | 76.4% | $1.82 | $1.82 |
| 23 | Opus 4.8 | medium | 76.0% | $0.015 | $0.85 |
| 24 | Opus 4.7 | xhigh | 75.4% | $7.95 | $7.95 |
| 25 | GLM 5.3 | max | 75.4% | $0.056 | $0.54 |
| 26 | Qwen 3.8 | xhigh | 75.2% | $1.42 | $1.42 |
| 27 | GPT-5.2 | xhigh | 72.2% | $2.05 | $2.05 |
| 28 | Gemini 3.1 Pro | high | 72.1% | $1.36 | $1.36 |
| 29 | Opus 4.8 | max | 70.5% | $0.008 | $0.48 |
| 30 | GLM 5.2 | xhigh | 70.1% | $0.044 | $0.51 |