Bench Gallery
Abstract
Recent studies suggest that video generative models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible.
To this end, we introduce VGI-Bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generative models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria.
Our analysis further explores the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors.
Data Construction
Comparison with Related Works
Comparison with related video benchmarks. Each pie encodes the fraction of tasks that satisfy a desideratum versus not.
| Benchmark | Reasoning Demand |
Realistic Appearance |
Process- Sensitive |
Difficulty Control |
|---|---|---|---|---|
| PhysGenBench | Low / Mid | ⍻ | ||
| WorldSimBench | Low / Mid | ✓ | ||
| TiVi-Bench† | High | ⍻ | ||
| V-ReasonBench | High | ✗ | ||
| VBVR-Bench | High | ✗ | ||
| VGI-Bench (Ours) | High | ✓ |
† Data has not been publicly released; statistics are inferred from the paper. Realistic Appearance: photorealistic-style inputs vs. synthetic (line-art / schematic) inputs.
Construction Pipeline
1Task Proposal
Reasoning-intensive, process-sensitive tasks expressible as video — success hinges on the trajectory, not just the final frame. 3 difficulty levels, ~10 instances each.
2Task Material Collection (image)
Input images from web / datasets, or generated by GPT-Image-2 & Nano Banana Pro (direct or schematic → photorealistic). Human-in-the-loop, standardised to 16:9.
3Task Material Collection (text)
Text prompt (goal, object / action rules, generation controls), a reference solution (image or target state), and per-task evaluation rubrics.
4Quality Control
Pre-generation: probe 2 easy instances on SOTA models; keep a task only if ≥1 solves and ≥1 fails — challenging yet partly feasible. Manual review: goal fidelity, 5–10s feasibility, prompt clarity.
See the paper for full construction details.
Eval Suite
Every generated video is scored by two complementary VLM-as-judge metrics; their per-instance product is the Final Score reported on the leaderboard.
Completeness global
Captures the global progress toward the task goal. A task-specific tiered standard maps each run to one of three levels — complete / partial / failed — and the VLM-judge, given uniformly sampled frames (2 fps) plus that standard, returns the tier, mapped to {1, 0.5, 0}. complete allows minor visual imperfections; partial means meaningful progress without the correct final state; failed covers little progress or severe drift from the input.
Rubric Score local
Measures local process validity throughout the video against a fine-grained per-task checklist of explicit rules and constraints. A coarse-to-fine adaptive pass (coarse sampling at 4 fps, flagged intervals resampled at 8 fps) with a 10-frame sliding focus window catches transient violations. Each item violated x times scores 1/(x+1) (inverse-decay penalty), averaged over all items.
Final Score aggregation
Final Score = Completeness × Rubric Score
We combine the two metrics multiplicatively rather than by an arithmetic mean, because they are jointly necessary conditions — not interchangeable contributions. A near-static clip preserves most local rules (high Rubric) yet makes little progress toward the goal (low Completeness); conversely, a clip can reach the target state (high Completeness) while violating the rules along the way (low Rubric). An average would let one strong factor mask the other's failure, whereas the product requires the model to both reach the goal and respect the process — exactly when neither aspect alone should be allowed to dominate.
Image-generation outputs have no temporal axis, so they instead use a single binary Success Rate against the ground-truth image.
Leaderboard
| Model | Overall | Visual Organization | Spatiotemporal Dynamics | Structured Puzzles | Physical Manipulation | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Easy | Mid | Hard | Avg. | Easy | Mid | Hard | Avg. | Easy | Mid | Hard | Avg. | Easy | Mid | Hard | Avg. | ||
| Commercial | |||||||||||||||||
| Seedance 2.0 | 51.0 | 72.6 | 56.4 | 53.5 | 60.8 | 54.6 | 47.4 | 33.4 | 45.3 | 46.9 | 45.1 | 41.8 | 44.6 | 64.4 | 59.2 | 44.4 | 56.0 |
| Kling 3.0 | 44.0 | 67.6 | 47.5 | 43.7 | 52.9 | 45.2 | 35.2 | 28.2 | 36.5 | 45.3 | 38.3 | 28.9 | 37.5 | 60.6 | 51.2 | 45.6 | 52.5 |
| Sora 2 | 36.7 | 55.6 | 37.0 | 39.1 | 43.9 | 45.3 | 34.6 | 24.4 | 34.8 | 40.5 | 25.8 | 22.4 | 29.5 | 47.9 | 41.8 | 32.6 | 40.8 |
| Gen-4.5 | 36.6 | 60.5 | 38.3 | 41.9 | 46.9 | 39.6 | 34.6 | 29.5 | 34.8 | 29.2 | 20.7 | 18.7 | 22.9 | 52.1 | 43.4 | 39.5 | 45.0 |
| Wan 2.7 | 35.7 | 37.0 | 32.2 | 32.5 | 33.9 | 38.9 | 42.6 | 28.9 | 36.8 | 25.6 | 23.3 | 24.7 | 24.5 | 55.8 | 48.2 | 37.1 | 47.1 |
| Veo 3.1 | 32.0 | 50.3 | 43.3 | 44.0 | 45.9 | 28.1 | 21.2 | 16.9 | 22.3 | 31.8 | 21.8 | 13.5 | 22.4 | 49.6 | 40.4 | 37.3 | 42.5 |
| Open Source | |||||||||||||||||
| MiniMax-H3 | 44.4 | 42.8 | 42.3 | 39.8 | 41.6 | 44.3 | 40.7 | 35.9 | 40.3 | 51.9 | 52.3 | 47.6 | 50.6 | 58.6 | 41.5 | 33.0 | 44.4 |
| VBVR-Wan 2.2 | 40.2 | 46.1 | 32.4 | 34.3 | 37.6 | 37.0 | 21.6 | 28.2 | 29.0 | 57.9 | 45.0 | 50.0 | 51.0 | 46.7 | 43.1 | 37.9 | 42.6 |
| Wan 2.2 | 21.6 | 30.4 | 15.3 | 18.5 | 21.4 | 38.7 | 32.0 | 19.9 | 30.2 | 17.1 | 8.4 | 5.6 | 10.4 | 33.7 | 22.1 | 17.5 | 24.4 |
| HunyuanVideo-1.5 | 19.1 | 26.6 | 17.9 | 24.7 | 23.1 | 34.7 | 27.2 | 23.9 | 28.7 | 9.9 | 8.8 | 7.9 | 8.9 | 22.5 | 16.7 | 11.9 | 17.0 |
| Model | Avg. | Easy | Mid | Hard |
|---|---|---|---|---|
| Commercial | ||||
| Nano-Banana-Pro | 55.0 | 62.5 | 50.0 | 52.5 |
| Qwen-Image-3-Pro | 42.7 | 51.9 | 43.8 | 32.5 |
| GPT-Image-2 | 41.2 | 48.8 | 38.8 | 36.2 |
| MAI-Image-2.5-Pro | 36.8 | 44.9 | 35.1 | 30.3 |
| Grok-Imagine | 34.6 | 31.2 | 41.2 | 31.2 |
| Seedream 4.5 | 23.8 | 27.5 | 21.2 | 22.5 |
| Flux.2 Max Edit | 20.6 | 23.8 | 20.3 | 17.7 |
| Open Source | ||||
| SenseNova-U1.5-8B-MoT | 22.5 | 32.5 | 21.2 | 13.8 |
| SenseNova-U1 | 16.3 | 22.2 | 14.4 | 12.2 |
| JoyAI-Image | 10.4 | 11.2 | 10.0 | 10.0 |
| Qwen-Image-Edit | 7.5 | 11.2 | 5.0 | 6.2 |
| BAGEL | 0.8 | 1.2 | 1.2 | 0.0 |
| Step1X-Edit | 0.8 | 1.2 | 0.0 | 1.2 |
Success Rate (%) against the ground-truth image (different metric from the video branch); best and second-best per row highlighted.
Eval Results
Pick a category, choose a task, then tick the models to compare their generated results. Video models cover all tasks; image models only cover the adapted subset (greyed-out where unavailable).
BibTeX
@misc{he2026vgibenchprobingvisualintelligence,
title={VGI-Bench: Probing Visual Intelligence in Video Generation Models},
author={Xuan He and Cong Wei and Yuhao Cheng and Linrui Ma and Yuxuan Zhang and Zuojun Li and Yuhao Wen and Jize Jiang and Zeyi Liu and Yuren Hao and Songcheng Cai and Keming Wu and Penghui Du and Kai Zou and Rui Yang and Chenkai Sun and Ke Yang and Ping Nie and Kelsey R Allen and Chenglong Wang and Michel Galley and Jianfeng Gao and ChengXiang Zhai},
year={2026},
eprint={2608.19583},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.19583},
}