Benchmarks · creative
Japanese-RP-Bench (Aratako)
Benchmark measuring LLM Japanese roleplay ability via 10-turn two-model dialogues scored by a four-judge ensemble across eight criteria. Maintained by Aratako. Score: Overall Average = mean evaluation value across the 8 criteria, averaged over 4 judge models (gpt-4o-2024-08-06, o1-mini-2024-09-12, Claude 3.5 Sonnet, Gemini 1.5 Pro).
20 models · 20 results · data as of 2026-10-29 · raw.githubusercontent.com ↗
Leaderboard
Opus 3 none
All results
20 rows| # | Model | Effort | Score |
|---|---|---|---|
| 1 | Opus 3 | none | 4.4 |
| 2 | Sonnet 3.5 | medium | 4.4 |
| 3 | GPT-4o mini | high | 4.3 |
| 4 | Gemini 1.5 Pro | none | 4.3 |
| 5 | Mistral | none | 4.3 |
| 6 | GPT-4o | high | 4.2 |
| 7 | Qwen 2.5 72B | none | 4.2 |
| 8 | o1 | high | 4.2 |
| 9 | Gemini 1.5 Flash | none | 4.2 |
| 10 | Haiku 3 | medium | 4.2 |
| 11 | Qwen 2.5 32B | none | 4.1 |
| 12 | o1 mini | high | 4.1 |
| 13 | Mistral Large 2407 | 4.1 | |
| 14 | Gemma 2 27B IT | 4.1 | |
| 15 | Llama 3.1 70b | none | 4.1 |
| 16 | Llama 3.1 405b | none | 4.0 |
| 17 | Mistral Small 2409 | 3.7 | |
| 18 | Llama 3 | 3.7 | |
| 19 | Qwen 2.5 7B | 3.7 | |
| 20 | Llama 3.1 8b | medium | 3.0 |