Benchmarks · long-context
RULER
NVIDIA synthetic benchmark that evaluates long-context LLMs with configurable sequence length and task complexity across 13 tasks in 4 categories. Maintained by NVIDIA. Score: Per-length scores at 4K-128K plus Avg. across the 13 RULER tasks (wAvg. inc/dec also reported); performance above the 4K Llama-2-7b threshold (85.6%) is underlined.
21 models · 21 results · data as of 2026-10-29 · raw.githubusercontent.com ↗
Leaderboard
Gemini 1.5 Pro none
All results
21 rows| # | Model | Effort | RULER |
|---|---|---|---|
| 1 | Gemini 1.5 Pro | none | 95.8% |
| 2 | Qwen 2.5 14B | 95.7% | |
| 3 | Qwen 3 235B-A22B | xhigh | 95.0% |
| 4 | Qwen 3 14B | none | 94.6% |
| 5 | Qwen 3 32B | none | 93.7% |
| 6 | Qwen 2.5 7B | 91.8% | |
| 7 | GPT-4.1106 | 91.6% | |
| 8 | Qwen 3 30B-A3B | none | 91.6% |
| 9 | GLM 4 | 89.9% | |
| 10 | Llama 3.1 70b | none | 89.6% |
| 11 | Mistral | none | 89.1% |
| 12 | Qwen 3 8B | none | 89.1% |
| 13 | Llama 3.1 8b | medium | 88.3% |
| 14 | Llama 3 70b | none | 86.5% |
| 15 | Mistral Large 2411 | 86.0% | |
| 16 | Qwen 2 72B | none | 85.9% |
| 17 | Qwen 3 4B | none | 85.2% |
| 18 | Llama 3 8b | none | 82.4% |
| 19 | Mistral Large 2407 | 80.5% | |
| 20 | GLM 3 | 69.6% | |
| 21 | Qwen 1.5 72B | 55.7% |