Evidence-Backed Video Question Answering¶
Conference: ECCV 2026
Paper: CVF / ECCV Poster
Code: https://github.com/SalesforceAIResearch/EVQA
Area: Segmentation
Keywords: Evidence-Backed Video QA, Spatio-Temporal Segmentation, Dense Masklet Tracking, Video LLM, Instruction Tuning
TL;DR¶
Addressing the black-box nature of current Video LLMs, this work introduces Evidence-Backed Video Question Answering (E-VQA), a task requiring models to jointly output semantic answers, temporal segments, and 6 FPS dense tracked segmentation masklets, accompanied by the human-verified ST-Evidence benchmark and a 160k instruction dataset ST-Evidence-Instruct.
Background & Motivation¶
Recent Video Large Language Models (Video LLMs) have demonstrated impressive performance across Video Question Answering (Video QA) benchmarks. However, most contemporary models remain closed black-box systems: they output free-form textual answers without verifiable spatio-temporal visual grounding. Consequently, these models often rely on textual biases or produce hallucinations rather than genuinely perceiving dynamic visual scenes; when erroneous conclusions occur, debugging their underlying visual perception failures becomes virtually impossible. While recent studies explore chain-of-thought (CoT) prompting to enhance explainability, text-only reasoning chains still lack direct coupling to spatial and temporal evidence, which is unacceptable for safety-critical domains such as autonomous driving, surgical assistance, and human-robot collaboration.
Existing efforts toward grounded video understanding primarily focus on "answer grounding"—locating the specific entity that constitutes the final answer—typically relying on coarse temporal segments or sparse spatial bounding boxes sampled on keyframes. Such sparse representations are fundamentally inadequate for capturing complex, fine-grained video dynamics, including heavy occlusions, continuous state transformations, non-rigid deformations (e.g., splashing liquids or bending wires), and identity tracking across complex object interactions. Unambiguously verifying whether an AI agent has observed the legitimate causal visual evidence requires producing dense, pixel-accurate spatio-temporal segmentation masklets across time.
The central tension lies in the severe decoupling between high-level semantic reasoning and low-level pixel grounding across existing paradigms: reasoning models excel at processing multi-hop queries but lack pixel-level decoding heads, whereas grounded segmentation models can accurately track masks for simple referring expressions but fail at complex causal logic. Core idea: formalize video reasoning as an explicit triplet prediction task \((A, E_t, E_s)\) unifying the answer, temporal segments, and dense masklets, and construct a 160k aligned instruction tuning dataset via bidirectional automated synthesis to enable grounded, verifiable video perception.
Method¶
Overall Architecture¶
The E-VQA formulation extends conventional video QA by querying a model \(F\) with a video \(V\) and a question \(Q\) to jointly produce a structured triplet \((A, E_t, E_s)\). Here, \(A\) represents the natural language answer; \(E_t = \{[start_i, end_i]\}_{i=1}^N\) specifies the set of non-overlapping temporal segments supporting the logical deduction; and \(E_s\) denotes the dense spatial evidence represented as 6 FPS tracked segmentation masklets highlighting the objects causally responsible for \(A\).
To address the shortage of training data and evaluation standards for this task, the authors develop the ST-Evidence benchmark (comprising generative ST-Evidence-Gen and multiple-choice ST-Evidence-MCQ variants) and establish two automated data construction pipelines to synthesize the ST-Evidence-Instruct dataset at scale.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Input: Video V + Question Q"] --> TaskDef["Task Formulation: E-VQA Triplet<br/>Jointly predict (Answer A, Temporal Et, Masklets Es)"]
TaskDef --> Bench["Human-Verified Benchmark: ST-Evidence<br/>ST-Evidence-Gen (Generative) + ST-Evidence-MCQ (Discriminative)"]
TaskDef --> PipeG2S["Pathway 1: Grounding-to-Semantics (ViCaS source)<br/>VLM QA generation → Text-only filtering → 4 FPS temporal grounding"]
TaskDef --> PipeS2G["Pathway 2: Semantics-to-Grounding (Complex QA source)<br/>Evidence object identification → Bounding-box filtering → SAM-3 propagation"]
PipeG2S --> Dataset["ST-Evidence-Instruct Dataset<br/>160k aligned triplets (QA, Et, Es)"]
PipeS2G --> Dataset
Dataset --> JointModel["End-to-End Grounded Video LLM Baseline<br/>UniPixel architecture + Joint multi-task optimization"]
Key Designs¶
1. Dual Benchmark Formulations: Decoupling Generative Synthesis from Discriminative Perception Recognizing that standard Video LLMs lack native mask decoders, ST-Evidence establishes two evaluation paradigms to prevent unfair penalization of architecture discrepancies. ST-Evidence-Gen assesses generative reasoning where models synthesize the complete \((A, E_t, E_s)\) triplet (general models utilize a frozen UniPixel-3B segmenter as a proxy to translate predicted referring expressions into spatial masklets). ST-Evidence-MCQ evaluates discriminative capability via 4-way multiple-choice questions for all three components independently, featuring manually curated, plausible non-evidential distractors to test whether models can distinguish actual causal visual evidence from visual confounders.
2. Grounding-to-Semantics Pipeline: Upcycling Human Mask Annotations with VLM Auditing Manually annotating dense video masklets from scratch is prohibitively expensive. Pathway 1 leverages existing datasets with ground-truth masks and dense captions (such as ViCaS). Dual VLMs (Qwen3-VL and Gemini) generate candidate questions, answers, distractors, and target evidence phrases conditioned on visual content and captions. A two-stage filter discards low-confidence generations before Gemini-2.5-Pro conducts text-only causal verification to ensure the question logically depends on the specified objects. Finally, Qwen3-VL predicts temporal boundaries \(E_t\) at an elevated frame rate (4 FPS), linking directly to the pre-existing spatial masklets \(E_s\).
3. Semantics-to-Grounding Pipeline: Self-Consistency Checking and SAM-3 Volume Propagation For complex QA datasets (Perception Test, STAR, CLEVRER) lacking spatial annotations, Pathway 2 introduces a three-step decomposition: identification, localization, and propagation. Qwen3-VL-235B first identifies evidence objects and candidate timestamps. Crucially, a self-consistency check requires the VLM to answer the original question relying solely on these identified objects; if the generated answer fails to match the ground truth within five re-prompting attempts, the sample is discarded. The model then generates bounding boxes for verified entities on keyframes, which undergo geometric filtering to prune degenerate boxes. Finally, SAM-3 propagates bounding boxes across the temporal volume to produce continuous masklets, with deduplication filtering overlapping instances (\(\text{IoU} > 0.9\)).
4. Joint Perception Architecture and Multi-Task Optimization: Blending Reasoning and Dense Mask Tracking To establish competitive baselines for E-VQA, the authors fine-tune 3B and 7B variants of UniPixel, integrating Qwen2.5-VL with SAM-2.1-Base+ as the mask decoder. To prevent catastrophic forgetting and maintain general video comprehension, training blends ST-Evidence-Instruct with standard video segmentation (DAVIS, MeViS, ReVOS) and general video QA datasets. The objective combines auto-regressive next-token prediction loss \(\mathcal{L}_{\text{text}}\) for the answer and temporal segment tokens with mask decoding losses \(\mathcal{L}_{\text{mask}}\) (Focal and Dice loss) guided by projection tokens: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{text}}(A, E_t) + \lambda \mathcal{L}_{\text{mask}}(E_s)\)$
Loss & Training¶
The visual encoder and LLM backbone are updated via LoRA, while the SAM-2.1-Base+ mask decoder is fully trained. During training, the model sequentially predicts answer tokens, temporal interval brackets [[start_s, end_s]], and special segmentation tokens that are projected into prompt embeddings for decoding multi-frame masks.
Key Experimental Results¶
Main Results¶
On ST-Evidence-Gen, baseline models were evaluated across semantic accuracy (QA acc), temporal grounding (mIoU, mIoP, t-mean), and spatial grounding (\(J\), \(F\), \(J\&F\)).
| Model Category | Method | QA acc (%) | T-Evidence (t-mean) | S-Evidence (J&F) |
|---|---|---|---|---|
| Proprietary Frontier | OpenAI-o3 (via UniPixel-3B proxy) | 82.6 | 36.1 | 41.9 |
| Proprietary Frontier | Gemini-2.5-Flash (via UniPixel-3B proxy) | 81.2 | 35.2 | 42.9 |
| Proprietary Frontier | Gemini-2.5-Pro (via UniPixel-3B proxy) | 83.7 | 49.9 | 44.0 |
| Open-Source General | Video-LLaMA3-7B | 51.0 | 16.3 | 13.2 |
| Open-Source General | Qwen2.5-VL-72B | 76.2 | 25.3 | 39.5 |
| Open-Source General | Qwen3-VL-235B-A22B | 83.6 | 44.5 | 41.9 |
| Grounded Video LLM | Sa2VA-Qwen2.5-VL-7B | 76.8 | 0.0 | 26.2 |
| Grounded Video LLM | UniPixel-3B | 72.2 | 6.5 | 42.7 |
| Grounded Video LLM | UniPixel-7B | 75.0 | 1.9 | 40.3 |
| Fine-Tuned Baseline (Ours) | Ours-3B | 72.4 (+0.2) | 26.9 (+20.4) | 52.7 (+10.0) |
| Fine-Tuned Baseline (Ours) | Ours-7B | 76.0 (+1.0) | 29.1 (+27.2) | 54.1 (+13.8) |
In ST-Evidence-MCQ (Table 3), open-source general models with ~70-80% QA accuracy hovered near random guess level (25%) on spatial evidence selection (Video-LLaMA3-7B at 27.12%, LLaVA-OV-1.5-8B at 23.42%, InternVL-3.5-8B at 27.43%), exposing their lack of genuine visual discrimination. In contrast, Qwen3-VL-235B-A22B achieved 86.90% spatial accuracy due to its interleaved timestamp-frame architecture.
Ablation Study & Diagnostic Verification¶
To confirm whether annotated masklets carry true causal evidence rather than arbitrary foreground cues, the authors conducted region-corruption experiments on a 200-sample subset of ST-Evidence-Gen (Table 5):
| Model | Original Video Acc (%) | Evidence Corrupted Acc (%) | Non-Evidence Corrupted Acc (%) | Relative Drop on Evidence Corruption |
|---|---|---|---|---|
| Gemini-2.5-Flash | 81.32 | 61.12 | 76.43 | -20.20 (vs non-evidence -4.89) |
| Qwen3-VL-8B | 79.31 | 58.32 | 73.66 | -20.99 (vs non-evidence -5.65) |
Generalization to external video segmentation and reasoning benchmarks (Table 4): - On MeViS (\(J\&F\)), Ours-7B achieved 62.5 (surpassing UniPixel-7B at 61.7 and Sa2VA-4B at 52.1); on Ref-DAVIS17, it achieved 77.8 (+1.4 gain); on ReVOS, it scored 64.2 (+0.3 gain). - On MVBench, Ours-7B reached 65.6% accuracy (vs. UniPixel-7B's 64.3%), demonstrating that fine-tuning on E-VQA preserves and enhances holistic video understanding.
Key Findings¶
- Decoupling of QA and Grounding Perception: In multiple-choice evaluations, models capable of reaching ~80% textual accuracy often perform near random guess (~25%) when selecting the underlying spatial evidence, confirming heavy reliance on linguistic priors.
- Scaling Fails to Resolve Grounding Deficits: Within the Qwen2.5-VL series, scaling from 3B to 72B yielded marginal spatial gains (\(J\&F\) 31.6 to 39.5). In contrast, Qwen3-VL-4B outperformed Qwen2.5-VL-72B across all metrics due to architectural timestamp alignment, proving that data and architecture design matter far more than brute-force scaling.
- Empirical Validation of Causal Necessity: Corrupting annotated evidence regions caused QA performance to plummet by over 20%, whereas corrupting non-evidence regions degraded performance by less than 5%, verifying the causal validity of the annotations.
Highlights & Insights¶
- Rigorous Grounding Standard: Moves beyond keyframe boxes to establish 6 FPS dense masklets and temporal spans as mandatory outputs alongside the answer, creating a verifiable visual audit trail.
- Self-Consistency Data Synthesis: Enforces an answer verification loop during dataset generation—retaining evidence objects only if they independently enable the VLM to deduce the correct answer.
- Robust Multi-Stage Distillation Pipeline: Combines VLM causal reasoning, geometric box verification, and SAM-3 spatio-temporal mask propagation into a scalable automated workflow producing 160k high-quality samples.
Limitations & Future Work¶
- Dependency on Proxy Segmenter during Evaluation: General Video LLMs without native segmentation heads must be evaluated using UniPixel-3B as a proxy, conflating LLM reasoning errors with proxy segmentation errors.
- Computational Overhead for Long Complex Videos: Generating 6 FPS masklets over long videos with multiple dynamic entities via SAM-3 incurs substantial memory and latency costs.
- Future Directions: Developing natively unified Video LLM architectures that output interleaved text and continuous spatio-temporal visual tokens directly, removing the need for external segmentation decoders.
Related Work & Insights¶
- vs TVQA+ / NeXT-GQA / V-STaR: Prior grounded QA benchmarks rely on sparse bounding boxes on keyframes or restrict localization to answer entities; E-VQA demands dense 6 FPS tracking of causal evidence objects supporting the deduction.
- vs UniPixel / Sa2VA: Existing video segmentation LLMs focus on passive execution of referring expressions; this work enhances them with active multi-hop causal reasoning and temporal segment prediction via ST-Evidence-Instruct.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers the E-VQA paradigm enforcing pixel-level tracked masklet evidence for video QA]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluates 15+ models across generative and MCQ variants with rigorous region corruption sanity checks]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-articulated task definition, and transparent pipeline documentation]
- Value: ⭐⭐⭐⭐⭐ [Releases ST-Evidence benchmark and 160k ST-Evidence-Instruct dataset, providing vital resources for trustworthy video AI]