Benchmarks · agentic
AutomationBench (Zapier)
Zapier's agent benchmark (v1.0.6): end-to-end business workflows across 47 real tools in six functions, graded strictly on the final environment state (every assertion must pass). Private held-out set.
40 models · 118 results · data as of 2026-10-29 · zapier.com ↗
Leaderboard
Gemini 4 Carbon high
All results
118 rows| # | Model | Effort | AutomationBench (Zapier) | Subscription $ / task | API $ / task |
|---|---|---|---|---|---|
| 1 | Gemini 4 Carbon | high | 53.9% | $2.12 | $2.12 |
| 2 | Gemini 4 Carbon | medium | 53.0% | $1.65 | $1.65 |
| 3 | Gemini 4 Argon | high | 51.3% | $0.61 | $1.70 |
| 4 | Gemini 4 Argon | medium | 50.1% | $0.55 | $1.54 |
| 5 | Sonnet 5.5 | max | 44.8% | $0.017 | $1.14 |
| 6 | Opus 5.5 | max | 42.5% | $0.022 | $1.44 |
| 7 | GPT-6 Astra | max | 41.4% | $0.15 | $1.73 |
| 8 | GPT-6 Astra | xhigh | 39.0% | $0.13 | $1.50 |
| 9 | GPT-6 Astra | high | 37.1% | $0.12 | $1.44 |
| 10 | Sonnet 5.5 | xhigh | 36.8% | $0.007 | $0.48 |
| 11 | Opus 5.5 | xhigh | 35.8% | $0.013 | $0.89 |
| 12 | GPT-6 Astra | medium | 34.1% | $0.11 | $1.27 |
| 13 | GPT-6 Sol | xhigh | 33.2% | $0.017 | $0.27 |
| 14 | Opus 5.5 | high | 33.0% | $0.011 | $0.71 |
| 15 | GPT-6 Sol | max | 32.0% | $0.021 | $0.34 |
| 16 | Gemini 4 Carbon | low | 31.5% | $1.02 | $1.02 |
| 17 | GPT-6 Sol | high | 31.2% | $0.015 | $0.24 |
| 18 | Gemini 3.7 Flash | high | 30.4% | $0.079 | $0.61 |
| 19 | GPT-6 Astra | low | 30.3% | $0.092 | $1.08 |
| 20 | Gemini 3.8 Flash | high | 29.7% | $0.025 | $0.62 |
| 21 | Gemini 3.8 Flash | medium | 29.7% | $0.022 | $0.55 |
| 22 | Opus 5.5 | medium | 29.5% | $0.010 | $0.65 |
| 23 | Sonnet 5.5 | high | 29.4% | $0.005 | $0.36 |
| 24 | GPT-5.6 Sol | max | 28.8% | $0.037 | $0.67 |
| 25 | Opus 5 | max | 26.9% | $0.053 | $3.05 |
| 26 | GPT-6 Sol | medium | 26.9% | $0.013 | $0.21 |
| 27 | GPT-5.6 Sol | xhigh | 26.3% | $0.041 | $0.74 |
| 28 | Opus 5 | xhigh | 25.3% | $0.020 | $1.15 |
| 29 | GPT-6 Astra | none | 25.3% | $0.11 | $1.28 |
| 30 | GPT-5.6 Sol | high | 24.8% | $0.035 | $0.64 |