Benchmarks · Coding · Agents & computer use
Bug Hunt Bench
Finding and fixing real bugs in real code: 105 bugs replanted in two production repos with their tests still passing. One round per repo in the model's own coding CLI; a blind judge checks fixes against a withheld answer key. Share of the 105 fixed. By Pawel Huryn.
48 models · 130 results · updated 2026-11-13 · bughunt.productcompass.pm ↗
Leaderboard
Fable 5.2 max
Heavy line: best score at each price. Thin lines join one model's effort levels. Cost uses each model's median subscription plan, or API price with no plan data. 30 more models under All models in settings.
More from this benchmark
9 published measuresTime per task130 results · lower is better
31 more models under All models in settings.
Score vs any published measure8 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | |||||||
|---|---|---|---|---|---|---|---|
| Fable 5.2 max | 58.0 | 23.0 | 35.0 | 2.3 | 2.3 | 24.7 | 3 |
| Gemini 4 Carbon high | 57.3 | 22.3 | 35.0 | 2.7 | 0.6 | 25.0 | 3 |
| Sonnet 5.5 max | 51.3 | 22.7 | 28.7 | 3 | 1 | 26.3 | 3 |
| GPT-6 Astra max | 45.0 | 22.0 | 23.0 | 1 | 0 | 55.0 | 3 |
| GPT-6.1 Sol max | 44.3 | 22.3 | 22.0 | 3 | 0.3 | 65.3 | 3 |
| GPT-5.6 Sol max | 43.5 | 19.0 | 24.5 | 2 | 0.5 | 48.5 | 2 |
| Fable 5.1 max | 43.0 | 19.0 | 24.0 | 2 | 4 | 11.0 | 1 |
| DeepSeek V4.1 Flash 1017 max | 42.7 | 19.0 | 23.7 | 3.5 | 1 | 44.0 | 7 |
| Muse Wonder Full max | 42.0 | 21.5 | 20.5 | 2.5 | 0 | 36.5 | 2 |
| Opus 5.5 max | 41.7 | 19.0 | 22.7 | 2.4 | 2.6 | 9 | 3 |
| Gemini 4 Argon high | 40.7 | 13.0 | 27.7 | 1.7 | 1 | 7.3 | 3 |
| GPT-6.1 Terra max | 40.7 | 18.0 | 22.7 | 2 | 0.3 | 48.0 | 3 |
| Kimi K3.1 max | 36.0 | 17.0 | 19.0 | 2.3 | 1 | 29.3 | 3 |
| Gemini 4 Hydrogen xhigh | 32.7 | 9.3 | 23.3 | 2.3 | 2 | 34.3 | 3 |
| Muse Spark 1.3 max | 32.2 | 14.2 | 18.0 | 1.6 | 1.2 | 24.0 | 5 |
| GPT-5.6 Terra max | 32.0 | 16.0 | 16.0 | 1 | 0 | 45.0 | 1 |
| Muse Wonder max | 31.7 | 12.0 | 19.7 | 3 | 1.6 | 13.3 | 3 |
| GPT-5.6 Luna max | 31.3 | 16.0 | 15.3 | 3.3 | 0 | 54.7 | 3 |
| GPT-6 Sol max | 29.3 | 15.3 | 14.0 | 2 | 0.3 | 41.3 | 3 |
| Fable 5 max | 29.0 | 12.0 | 17.0 | 0 | 0 | 5 | 1 |
| Grok 4.7 xhigh | 29.0 | 13.0 | 16.0 | 1.4 | 0.2 | 32.2 | 5 |
| Grok 4.6 xhigh | 28.7 | 11.7 | 17.0 | 2.7 | 0.7 | 18.0 | 3 |
| Opus 5 max | 27.0 | 13.0 | 14.0 | 2 | 5 | 2 | 1 |
| Qwen 3.8 Flash max | 26.0 | 13.0 | 13.0 | 3 | 3 | 7 | 1 |
| Qwen 3.8 Max | 25.7 | 12.3 | 13.3 | 1.4 | 2 | 7 | 3 |
All results
130 rows| # | Model | Effort | Bug Hunt Bench | Subscription $ / task | API $ / task | Time | Harness |
|---|---|---|---|---|---|---|---|
| 1 | Fable 5.2 | max | 55.2% | $5.13 | $140 | 2.4 h | Claude Code |
| 2 | Gemini 4 Carbon | high | 54.6% | $22 | $31 | 1.9 h | Antigravity CLI |
| 3 | Gemini 4 Carbon | medium | 53.3% | $18 | $25 | 1.4 h | Antigravity CLI |
| 4 | Sonnet 5.5 | max | 48.9% | $2.33 | $154 | 4.8 h | Claude Code |
| 5 | Fable 5.2 | xhigh | 46.7% | $2.24 | $61 | 57 min | Claude Code |
| 6 | Gemini 4 Carbon | low | 43.8% | $5.47 | $7.67 | 1.0 h | Antigravity CLI |
| 7 | GPT-6 Astra | max | 42.9% | $1.41 | $33 | 1.5 h | Codex CLI |
| 8 | GPT-6.1 Sol | max | 42.2% | $0.20 | $5.88 | 2.3 h | Codex CLI |
| 9 | GPT-5.6 Sol | max | 41.4% | $2.63 | $95 | 4.2 h | Codex CLI |
| 10 | Fable 5.1 | max | 41.0% | $14 | $87 | 1.2 h | Claude Code |
| 11 | Fable 5.2 | high | 41.0% | $1.71 | $47 | 40 min | Claude Code |
| 12 | GPT-6 Astra | xhigh | 41.0% | $1.04 | $24 | 59 min | Codex CLI |
| 13 | DeepSeek V4.1 Flash 1017 | max | 40.7% | free | $1.84 | 1.0 h | DeepSeek Harness / DeepSeek API |
| 14 | GPT-6.1 Sol | xhigh | 40.7% | $0.15 | $4.35 | 1.6 h | Codex CLI |
| 15 | Muse Wonder Full | max | 40.0% | 2.4 h | Muse Code / Meta API | ||
| 16 | Opus 5.5 | max | 39.7% | $0.89 | $59 | 1.1 h | Claude Code |
| 17 | Muse Wonder Full | xhigh | 39.5% | 1.5 h | Muse Code / Meta API | ||
| 18 | Fable 5.2 | medium | 39.5% | $0.98 | $27 | 20 min | Claude Code |
| 19 | Fable 5.2 | low | 39.0% | $0.89 | $24 | 19 min | Claude Code |
| 20 | Gemini 4 Argon | high | 38.8% | $5.10 | $7.16 | 57 min | Antigravity CLI |
| 21 | GPT-6.1 Terra | max | 38.8% | $0.15 | $3.88 | 37 min | Codex CLI |
| 22 | Gemini 4 Argon | medium | 37.4% | $5.34 | $7.50 | 60 min | Antigravity CLI |
| 23 | DeepSeek V4.1 Flash 1017 | high | 37.3% | free | $1.27 | 53 min | DeepSeek Harness / DeepSeek API |
| 24 | GPT-5.6 Sol | xhigh | 37.1% | $1.46 | $53 | 2.1 h | Codex CLI |
| 25 | GPT-6.1 Sol | high | 34.8% | $0.11 | $3.26 | 1.2 h | Codex CLI |
| 26 | Opus 5.5 | xhigh | 34.3% | $0.53 | $35 | 43 min | Claude Code |
| 27 | Sonnet 5.5 | xhigh | 34.3% | $0.65 | $43 | 1.2 h | Claude Code |
| 28 | Kimi K3.1 | max | 34.3% | $14 | $14 | 1.1 h | Claude Code / Moonshot API |
| 29 | Gemini 4 Argon | low | 33.6% | $4.61 | $6.47 | 60 min | Antigravity CLI |
| 30 | GPT-6 Astra | high | 33.3% | $0.88 | $21 | 41 min | Codex CLI |