Benchmarks · agentic
WeirdML v3
Fully agentic: 11 hand-made ML and data-analysis tasks run in coding-agent harnesses, where models must explore unfamiliar data and build pipelines with limited data, unspecified goals or little feedback. Overall effective score (95% interval).
25 models · 25 results · data as of 2026-10-29 · htihle.github.io ↗
Leaderboard
GPT-6 Astra xhigh
More from this benchmark
1 published measuresOutput tokens per task25 results · lower is better
All results
25 rows| # | Model | Effort | WeirdML v3 | Subscription $ / task | API $ / task | Output tokens | Harness |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | xhigh | 42.2% ±1.9 | $2.38 | $28 | 153k | codex_cli 0.154.0 |
| 2 | GPT-6.1 Sol | xhigh | 38.3% ±2.1 | $0.62 | $5.29 | 216k | codex_cli 0.156.0 |
| 3 | Opus 5.5 | xhigh | 31.2% ±1.2 | $0.16 | $10 | 181k | claude_code 2.1.280 |
| 4 | GPT-6.1 Terra | xhigh | 28.1% ±1.5 | $2.86 | $2.86 | 147k | codex_cli 0.164.0 |
| 5 | Fable 5.1 | xhigh | 26.0% ±1.4 | $4.98 | $32 | 251k | claude_code 2.1.270 |
| 6 | Gemini 4 Argon | high | 20.3% ±1.5 | $5.78 | $16 | 310k | gemini_cli 0.62.0 |
| 7 | Sonnet 5.5 | xhigh | 19.9% ±1.6 | $0.14 | $9.27 | 279k | claude_code 2.1.280 |
| 8 | GPT-6 Sol | xhigh | 19.7% ±1.3 | $0.43 | $6.91 | 144k | codex_cli 0.156.0 |
| 9 | Opus 5 | xhigh | 19.0% ±1.7 | $0.44 | $25 | 288k | claude_code 2.1.270 |
| 10 | Fable 5 | xhigh | 15.6% ±2.0 | $5.95 | $53 | 300k | claude_code 2.1.270 |
| 11 | Muse Wonder (Max) | xhigh | 15.2% ±1.1 | $7.32 | $11 | 169k | codex_cli 0.162.1 |
| 12 | GPT-5.6 Sol | xhigh | 15.0% ±1.4 | $0.35 | $6.28 | 116k | codex_cli 0.154.0 |
| 13 | Haiku 5.5 | xhigh | 11.7% ±1.6 | $0.049 | $2.30 | 315k | claude_code 2.1.280 |
| 14 | DeepSeek V4.1 Flash 1017 | high | 9.2% ±1.2 | free | $0.88 | 830k | codex_cli 0.162.1 |
| 15 | Grok 4.7 | xhigh | 8.5% ±1.2 | $2.50 | $27 | 275k | codex_cli 0.156.0 |
| 16 | GPT-6 Luna | xhigh | 7.9% ±1.0 | $0.048 | $0.32 | 181k | codex_cli 0.156.0 |
| 17 | Kimi K3 | high | 7.5% ±1.4 | $1.54 | $20 | 209k | kimi_code 2.0.0 |
| 18 | Gemini 3.8 Flash | high | 7.4% ±1.0 | $0.16 | $3.94 | 156k | gemini_cli 0.59.0 |
| 19 | Muse Spark 1.3 | xhigh | 6.7% ±1.3 | $2.78 | $5.56 | 280k | codex_cli 0.156.0 |
| 20 | Opus 4.5 | high | 6.5% ±1.5 | $8.70 | $8.70 | 83k | claude_code 2.1.270 |
| 21 | DeepSeek V4.1 Flash | high | 5.9% ±1.8 | $0.046 | $0.74 | 323k | opencode 1.18.30 |
| 22 | GPT-5.6 Luna | xhigh | 5.2% ±1.4 | $0.069 | $0.74 | 99k | codex_cli 0.154.0 |
| 23 | GPT-5.2 | xhigh | 5.0% ±1.0 | $9.29 | $9.29 | 265k | codex_cli 0.156.0 |
| 24 | Mistral Large 4 | high | 4.4% ±1.4 | $29 | $29 | 1.1M | opencode 1.18.30 |
| 25 | Inkling | xhigh | 3.9% ±1.0 | free | $1.33 | 20k | opencode 1.18.30 |