Benchmarks · agentic
VitaBench
Can an agent finish a real-life errand that spans food delivery, restaurant bookings and travel — 66 tools, a customer whose wishes shift mid-conversation, and no domain policy to lean on? Maintained by Meituan LongCat team. Score: Cross-Scenarios Avg @4 on the 100 cross-scenario tasks (the paper's main result), graded by the benchmark's rubric-based sliding-window LLM evaluator; the page does not define @4 in words — by the column name it is the average over four trials.
12 models · 14 results · data as of 2026-10-29 · vitabench.github.io ↗
Leaderboard
Gemini 3 Flash high
All results
14 rows| # | Model | Effort | VitaBench |
|---|---|---|---|
| 1 | Gemini 3 Flash | high | 32.5% |
| 2 | Gemini 3 Pro | high | 31.5% |
| 3 | Gemini 3 Pro | low | 30.0% |
| 4 | o3 | high | 26.3% |
| 5 | GPT-5.2 | xhigh | 24.3% |
| 6 | DeepSeek V3.2 | high | 24.0% |
| 7 | o4 mini | high | 19.5% |
| 8 | GLM 4.7 | none | 18.3% |
| 9 | Qwen 3 235B-A22B | xhigh | 14.5% |
| 10 | Qwen 3 Max | max | 14.3% |
| 11 | Kimi K2 | medium | 12.8% |
| 12 | Kimi K2.0905 | 11.5% | |
| 13 | Qwen 3 32B | none | 5.3% |
| 14 | GPT-5.2 | none | 0.8% |