LetTokenBurnVideo-Index: A Curated Meta-Benchmark for Video Understanding
A curated meta-benchmark of 840 verified video understanding items selected from a pool of over half a million across 115 benchmarks, designed to resist shortcut attacks.
$ npm i -g lettokenburn-cli$ ltb bench info video-index
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 56.8 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Molmo2-8B | 19.0 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| InternVL3.5-8B-Instruct | 15.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| LLaVA-OneVision-2-8B | 14.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| InternVideo2.5-8B | 12.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Video-XL-2 | 12.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| VideoLLAMA3-7B | 12.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Cambrian-S-7B | 11.3 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen3-VL-8B-Instruct | 9.5 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| VideoChat3-4B | 9.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The benchmark is constructed by auditing 115 existing video multiple-choice benchmarks using an 'attack pyramid' protocol that tests for exploits at five levels (options-only up to shuffled frames). Items are retained only if they remain hard even when attackers have access to degraded visual views or temporal perturbations.
Scoring uses accuracy on the selected items, with a primary focus on verifying capabilities against strong open-source and proprietary models. The key finding is that many existing benchmarks break under simple shortcuts (e.g., reading options), so Video-Index represents only the 'hardest' verified subset.