SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/texaser/svi-bench
Project: https://svi-bench.github.io
Area: Multimodal VLM
Keywords: strategic video intelligence, multi-agent video benchmark, causal reasoning, video simulation, agentic retrieval
TL;DR¶
SVI-Bench is the first large-scale benchmark evaluating the full Strategic Video Intelligence (SVI) stack—from perception through causal reasoning and simulation to agentic synthesis—across 35K hours of multi-agent team sports videos, revealing a severe capability cliff where state-of-the-art models drop from ~74% on fine-grained perception to just 4.6% on autonomous cross-corpus synthesis.
Background & Motivation¶
True video intelligence demands far more than recognizing visible surface primitives—it requires understanding why events unfold, predicting what would change under counterfactual conditions, and deciding optimal goal-directed actions. This full progression forms an integrated cognitive hierarchy: low-level spatiotemporal perception, causal and strategic reasoning, counterfactual simulation, and multi-step agentic synthesis. However, state-of-the-art vision-language models remain largely confined to descriptive perception, unable to trace multi-agent tactical breakdowns, simulate what-if alternatives, or formulate strategic plans in complex dynamic environments.
Progress has been bottlenecked by a fundamental trade-off in existing benchmarks. In-the-wild video datasets offer natural visual richness but lack objective, verifiable ground truth for causal attribution and strategic intent. Conversely, synthetic simulation environments provide mathematically verifiable physical ground truth but drastically oversimplify the complexity, non-stationarity, and adversarial coordination inherent to real-world multi-agent systems. As a result, research has lacked a unified domain capable of simultaneously offering real visual complexity and definitive, verifiable outcome signals.
Team sports (basketball, soccer, and hockey) provide an ideal dynamic microworld to bridge this divide. They naturally feature 10 to 22 coordinated agents executing interdependent decisions under tight physical and temporal adversarial constraints. Furthermore, tactical setups (screens, off-ball cuts, pressing schemes) generate delayed causal consequences seconds or minutes later, while unambiguous scoring events, fouls, and match outcomes provide definitive, leak-free ground truth. The core idea is to establish team sports as a dynamic microworld and introduce SVI-Bench, the first large-scale benchmark evaluating the complete four-pillar SVI stack across 9 progressive tasks to systematically diagnose and quantify the capability cliff in multi-agent video intelligence.
Method¶
Overall Architecture¶
SVI-Bench formalizes Strategic Video Intelligence into a data engine and a four-pillar evaluation hierarchy spanning 9 distinct tasks. The data engine ingests ~35K hours of multi-season broadcast video, ~15M timestamped play-by-play logs, ~15K hours of expert commentary, ~23K journalist game reports, and ~103K box-score statistical records across basketball, soccer, and hockey. Through game-clock temporal alignment and identity graph resolution, it produces a densely cross-referenced corpus powering evaluations from localized 10-second perception to multi-hour causal reasoning, generative simulation, and corpus-wide agentic tool use.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Multimodal Sources<br/>Video / Play-by-Play / ASR Audio / Reports / Box Scores"] --> B["Multimodal Data Engine<br/>Game-Clock Alignment & Entity Identity Graphs"]
B --> C["Pillar 1: Dynamic Scene Understanding<br/>Play Description T1 / Fine-Grained QA T2 / Video Retrieval T3"]
C --> D["Pillar 2: Causal Reasoning<br/>Strategic QA T4 / Outcome Forecasting T5 / Narrative Synthesis T6"]
D --> E["Pillar 3: Strategic Simulation<br/>Motion-Conditioned Video T7 / Goal-Conditioned Video T8"]
E --> F["Pillar 4: Agentic Synthesis<br/>Multi-Step Tool-Assisted Search & Evidence Integration T9"]
Key Designs¶
1. Multimodal Data Engine: Game-Clock Temporal Grounding and Cross-Modal Identity Resolution To resolve temporal drift and entity ambiguities across uncurated broadcast feeds, the data engine establishes official league play-by-play game clocks as the primary temporal backbone. Broadcast ASR transcripts and journalist post-game recaps are temporally anchored to video segments via timestamp matching and semantic anchor verification. Cross-modal entity resolution then constructs relational identity graphs connecting player jersey numbers, full names, roster positions, and team affiliations across visual boxes and textual event records. Finally, pillar-guided LLM synthesis generates difficulty-calibrated QA pairs, plausible distractors, and narrative targets, which undergo rigorous rule-based validation against event logs and stratified expert human quality audits.
2. Four-Pillar Cognitive Stack: A Progressive Evaluation Progression Rather than treating video understanding as isolated benchmarks, SVI-Bench structures its 9 tasks into a strictly dependent cognitive progression: - Pillar 1: Dynamic Scene Understanding (T1–T3) evaluates foundational spatiotemporal perception over 10-second clips, comprising structured dense play description (T1, evaluated along 6 Likert axes), multi-agent fine-grained action QA across 31 question types (T2), and compositional video retrieval against 5,000 fine-grained distractors within the same sport (T3). - Pillar 2: Causal Reasoning (T4–T6) probes long-horizon understanding across 3 to 150 minutes, requiring models to explain strategic failures and tactical shifts (T4), forecast upcoming game events and statistical evolution beyond the input window (T5), and synthesize full-game visual evidence into salient 500-word tactical narratives (T6). - Pillar 3: Strategic Simulation (T7–T8) tests counterfactual generative capability over 5–10 second horizons, challenging video diffusion backbones to generate multi-player dynamics following prescribed bounding-box trajectories on player-removed backgrounds (T7), or autonomously plan intermediate multi-agent actions to achieve specified spatial target goals (T8).
3. Cross-Corpus Agentic Synthesis: Hard-to-Find, Easy-to-Verify Scaled Tool Orchestration (T9) Addressing the limitations of single-video QA or synthetic API benchmarks, T9 introduces a corpus-scale agentic environment spanning 7,430 games, ~1.8M video clips (~5,670 hours), and ~33K technical documents. Agents are equipped with structured search, document retrieval, and video inspection tools to resolve multi-hop strategic queries (e.g., identifying a specific late-game defensive rotation under distinct score margins). Queries incorporate multi-constraint narrative cues that make brute-force browsing infeasible, forcing autonomous query formulation, conditional branching, and backtracking. Answers converge to unambiguous, short factual tokens (e.g., player numbers or shot zones), enabling fully automated, objective verification.
Key Experimental Results¶
Main Results¶
SVI-Bench was evaluated across proprietary frontier models (GPT-5.2, Gemini 3.0/3.1 Pro), open-source vision-language models (Qwen3-VL, Molmo 2, BIMBA, LLaVA-Video), and video generation baselines (fine-tuned Wan 2.1, MagicMotion, ATI).
| Pillar & Task ID | Task Name | Format / Metric | Baseline Model | Best Model / Fine-Tuned | Human Expert |
|---|---|---|---|---|---|
| Pillar 1: T1 | Structured Play Description | Open / Avg. Score (0–5) | GPT-5.2: 1.61 | LLaVA-Video-7B (FT): 2.17 | — |
| Pillar 1: T2 | Fine-Grained Action QA | 5-way MCQ / Accuracy (%) | GPT-5.2: 52.91% | LLaVA-Video-7B (FT): 73.91% | 75.78% |
| Pillar 1: T3 | Compositional Video Retrieval | 1-of-5001 / R@1 (%) | InternVideo2 (Zero-shot): 0.8% | InternVideo2 (FT): 3.0% (R@10: 13.3%) | — |
| Pillar 2: T4 | Strategic Reasoning QA | Open / Score (0–5) | Molmo 2-8B: 1.82 | Gemini 3.1 Pro: 2.17 (Soccer: 2.49) | 4.20 |
| Pillar 2: T5 | Outcome Forecasting | MCQ / Top-1 Accuracy (%) | Qwen3-VL-8B: 36.92% | Qwen3-VL-8B (FT): 44.82% | 58.90% |
| Pillar 2: T6 | Long-Form Narrative Synthesis | Open / Saliency Coverage (%) | Qwen3-VL-8B: 5.81% | Gemini 3.1 Pro: 7.33% (Factuality: 73.01%) | — |
| Pillar 3: T7 | Motion-Conditioned Generation | Video Gen / Video mIoU (B-ball) | ATI: 0.397 | Wan 2.1 (FT): 0.513 (Soccer: 0.611) | — |
| Pillar 3: T8 | Goal-Conditioned Action Gen. | Video Gen / Goal Accuracy (%) | MagicMotion: 31.4% | Wan 2.1 (FT): 50.2% (Final mIoU: 0.344) | — |
| Pillar 4: T9 | Cross-Corpus Agentic Reasoning | Agent Search / Accuracy (%) | Qwen3-Omni-30B: 2.1% | GPT-5.2: 4.6% (21 tool calls/query) | — |
Ablation & Diagnostic Analysis: Video vs. Oracle Ground-Truth Text¶
To determine whether failure in reasoning and agentic tasks stems from perceptual failure or deficiencies in high-level multi-step planning and causal reasoning, the authors conducted Oracle experiments replacing raw video with ground-truth textual descriptions derived from official event logs.
| Evaluated Task | Default Video Mode | Oracle Text Mode | Absolute Gain (\(\Delta\)) | Diagnostic Bottleneck Analysis |
|---|---|---|---|---|
| T4: Strategic Reasoning QA (0–5 Score) | 2.06 | 2.46 | +0.40 | Modest gain indicates strategic causal attribution is fundamentally bottlenecked by reasoning, not visual perception. |
| T5: Outcome Forecasting (Accuracy %) | 38.2% | 41.9% | +3.7% | Minimal gain demonstrates that predicting game evolution requires probabilistic multi-agent modeling beyond event tracking. |
| T6: Factuality Accuracy (%) | 71.99% | 87.19% | +15.20% | Oracle text substantially reduces fine-grained factual hallucinations. |
| T6: Saliency Coverage (%) | 7.10% | 20.60% | +13.50% | Even with exhaustive textual events, models struggle to identify pivotal turning-point moments. |
| T9: Cross-Corpus Agentic Reasoning (Acc %) | 4.6% | 54.0% | +49.4% | Dramatic gain shows visual retrieval noise compounds severely in tool loops; however, 54% confirms multi-step planning remains difficult. |
Key Findings¶
- The Performance Cliff: Models exhibit a dramatic performance decline across the hierarchy. While fine-tuned models near human performance on low-level perception (T2: 73.91% vs. 75.78% human), performance collapses to 2.17/5 on strategic QA (T4), 44.82% on forecasting (T5), and a staggering 4.6% on agentic synthesis (T9).
- Domain Fine-Tuning Does Not Transfer Upward: Supervised fine-tuning provides large improvements on perceptual tasks (+36.9% on T2), but produces only marginal benefits on temporal causal prediction (+7.9% on T5) and fails to resolve high-level strategic reasoning or agentic planning.
- Severe Confidence Miscalibration: On outcome forecasting (T5), human expert accuracy scales reliably with self-reported confidence (from 50% to 100%), whereas frontier models like GPT-5.2 exhibit a 28-point gap between confidence and empirical accuracy, assigning near-identical confidence to correct and incorrect predictions.
Highlights & Insights¶
- Team Sports as a Principled Dynamic Microworld: SVI-Bench elegantly resolves the longstanding dilemma between visual realism (in-the-wild video) and verifiable ground truth (synthetic physics engines), providing a sandbox featuring adversarial multi-agent dynamics governed by strict, measurable outcomes.
- Unified Four-Pillar Cognitive Stack: Rather than assessing isolated capabilities, the benchmark formalizes video intelligence as an end-to-end continuum, uniquely integrating video counterfactual generation (T7/T8) alongside corpus-scale agentic tool use (T9).
- Perception Is Not the Sole Bottleneck: The Oracle experiments conclusively prove that multi-agent video intelligence cannot be solved solely by training larger visual encoders; causal reasoning, strategic forecasting, and tool-augmented evidence gathering present distinct, independent cognitive bottlenecks.
Limitations & Future Work¶
- Domain Specificity of Team Sports: While team sports serve as an effective proxy, broadcast camera habits, fixed boundaries, and artificial game rules may limit zero-shot transfer to unstructured domains such as surgical procedures, disaster response, or autonomous driving.
- Reliance on LLM Evaluators: Tasks T1, T4, and T6 rely on LLM-as-a-judge protocols. Despite strong human agreement correlations (mean absolute differences of 0.12–0.40 on a 0–5 scale), residual stylistic or verbosity biases may persist.
- Generative Physics and Artifacts: In Pillar 3, even fine-tuned video diffusion models achieve a Video mIoU of only 0.513 on basketball, frequently suffering from limb distortion, player merging, and trajectory drift under physical contact.
Related Work & Insights¶
- vs. Video-MME / EgoSchema: While existing long-video benchmarks emphasize perceptual retrieval ("who did what"), SVI-Bench tests why decisions were made, how game states evolve over hours, and how counterfactual scenarios unfold.
- vs. CLEVRER / PHYRE: Unlike synthetic collision microworlds limited to rigid balls and blocks, SVI-Bench benchmarks dynamic, deceptive, and cooperative multi-human multi-agent interactions.
- vs. SoccerNet / SPORTU: Traditional sports video benchmarks concentrate primarily on player tracking and action spotting; SVI-Bench is the first to bridge sports video into counterfactual video simulation and multi-million-clip agentic reasoning.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the four-pillar Strategic Video Intelligence framework and establishes a rigorous dynamic microworld benchmark.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Massive corpus (35K hours), 9 comprehensive tasks, rigorous multi-model benchmarking, Oracle ablation, and expert human calibration.
- Writing Quality: ⭐⭐⭐⭐⭐ Beautifully structured, conceptually precise, and transparent about empirical model failures.
- Value: ⭐⭐⭐⭐⭐ Sets a pivotal standard guiding the vision-language community from passive perception to active causal reasoning, simulation, and agency.