Benchmarks · agentic
Ramp Accounting Bench
Long-horizon accounting work across realistic financial systems: 137 tasks, 3 attempts each in one shared agent harness (Mercor Archipelago), graded by GPT-5.6 Luna against per-task criteria. Value is the mean share of criteria met, best-of-3 as Ramp publishes it. Cost is per attempt.
16 models · 16 results · updated 2026-10-14T19:15:50.422493Z · labs.ramp.com ↗
Leaderboard
Gemini 4 Argon max
More from this benchmark
10 published measuresTime per task16 results · lower is better
Output tokens per task16 results · lower is better
Score vs any published measure8 measures: sub-scores, splits, counts
Breakdown
darker is better within each column · click a column to sort| Model | |||||||
|---|---|---|---|---|---|---|---|
| Gemini 4 Argon max | 23.4 | 26.3 | 16.8 | 2.92 | 9.49 | 76.4 | 39.7 |
| Opus 5.5 max, fallback | 10.9 | 19.7 | 8.03 | 8.76 | 11.7 | 31.4 | 20.2 |
| Fable 5.1 max, fallback | 12.4 | 19.7 | 6.57 | 7.3 | 13.1 | 34.3 | 18.5 |
| GPT-6 Astra max | 11.7 | 19.0 | 8.03 | 7.3 | 10.9 | 33.3 | 16.8 |
| GPT-6.1 Sol max | 10.9 | 16.8 | 5.84 | 5.84 | 10.9 | 36.1 | 17.1 |
| Gemini 3.8 Flash max | 10.9 | 14.6 | 5.84 | 3.65 | 8.76 | 95.3 | 94.3 |
| Grok 4.7 max | 9.49 | 19.0 | 5.84 | 9.49 | 13.1 | 90.8 | 35.6 |
| GLM 5.3 max | 11.7 | 21.2 | 2.92 | 9.49 | 18.3 | 56.4 | 33.7 |
| Sonnet 5.5 max, fallback | 8.03 | 12.4 | 2.92 | 4.38 | 9.49 | 31.3 | 20.5 |
| Gemini 3.7 Flash max | 9.49 | 15.3 | 6.57 | 5.84 | 8.76 | 57.3 | 56.5 |
| Haiku 5.5 max | 7.3 | 13.9 | 1.46 | 6.57 | 12.4 | 49.4 | 22.5 |
| Qwen 3.8 Max | 12.4 | 19.7 | 1.46 | 7.3 | 18.3 | 62.0 | 39.3 |
| DeepSeek V4.1 Flash max | 12.4 | 16.8 | 2.19 | 4.38 | 14.6 | 70.7 | 35.4 |
| Kimi K3 max | 10.2 | 13.9 | 1.46 | 3.65 | 12.4 | 50.2 | 27.0 |
| GLM 5.3 Flash max | 7.3 | 13.9 | 1.46 | 6.57 | 12.4 | 43.9 | 29.2 |
| GPT-6 Luna max | 7.3 | 10.9 | 0.73 | 3.65 | 10.2 | 51.3 | 23.5 |
All results
16 rows| # | Model | Effort | Ramp Accounting Bench | Subscription $ / task | API $ / task | Time | Output tokens | Harness |
|---|---|---|---|---|---|---|---|---|
| 1 | Gemini 4 Argon | max | 61.1% | $3.03 | $3.03 | 17 min | 86k | Mercor Archipelago |
| 2 | Opus 5.5 | max | 50.5% | $0.020 | $1.34 | 3.4 min | 20k | Mercor Archipelago |
| 3 | Fable 5.1 | max | 49.7% | $0.54 | $3.48 | 5.7 min | 23k | Mercor Archipelago |
| 4 | GPT-6 Astra | max | 48.7% | $0.20 | $2.31 | 5.0 min | 12k | Mercor Archipelago |
| 5 | GPT-6.1 Sol | max | 46.4% | $0.049 | $0.42 | 4.9 min | 11k | Mercor Archipelago |
| 6 | Gemini 3.8 Flash | max | 45.7% | $1.47 | $1.47 | 8.9 min | 83k | Mercor Archipelago |
| 7 | Grok 4.7 | max | 44.1% | $0.29 | $3.11 | 15 min | 55k | Mercor Archipelago |
| 8 | GLM 5.3 | max | 43.1% | $0.22 | $2.16 | 37 min | 140k | Mercor Archipelago |
| 9 | Sonnet 5.5 | max | 42.5% | $0.010 | $0.67 | 2.6 min | 20k | Mercor Archipelago |
| 10 | Gemini 3.7 Flash | max | 42.3% | $0.11 | $0.85 | 5.9 min | 44k | Mercor Archipelago |
| 11 | Haiku 5.5 | max | 42.3% | $0.004 | $0.21 | 4.2 min | 52k | Mercor Archipelago |
| 12 | Qwen 3.8 Max | max | 41.2% | $2.85 | $2.85 | 15 min | 63k | Mercor Archipelago |
| 13 | DeepSeek V4.1 Flash | max | 40.8% | $0.008 | $0.12 | 12 min | 62k | Mercor Archipelago |
| 14 | Kimi K3 | max | 39.2% | $0.12 | $1.55 | 11 min | 31k | Mercor Archipelago |
| 15 | GLM 5.3 Flash | max | 35.7% | free | $0.18 | 17 min | 112k | Mercor Archipelago |
| 16 | GPT-6 Luna | max | 34.0% | $0.008 | $0.051 | 5.4 min | 30k | Mercor Archipelago |