ROVA: Are Video Reasoning Models Ready to Go Outside?¶

Conference: ECCV 2026
arXiv: 2603.10652
Project Page: https://robust-video-reason.github.io/
Area: VLM Inference / Robustness
Keywords: Video Reasoning Robustness, Perturbation Robust Training, Adaptive Curriculum Learning, GRPO, Benchmark

TL;DR¶

ROVA proposes a training framework for robust video reasoning that enhances VLM performance under real-world perturbations (weather, occlusion, illumination, camera shake) through structured spatio-temporal perturbation generation, self-reflective difficulty-aware online curriculum, and dual-branch alignment based on GRPO. Concurrently, it introduces PVRBench—the first embodied video reasoning benchmark with systematically injected real-world perturbations, covering over 9K+ videos and 51K+ QA pairs.

Background & Motivation¶

Background: VLMs have progressed rapidly in video understanding and reasoning (Qwen-VL, InternVL, Embodied-R), but mainstream benchmarks (MVBench, Video-MME) primarily evaluate under clean conditions, which implicitly assumes stable illumination, unobstructed views, and smooth camera motion.

Limitations of Prior Work: In real-world deployments, VLMs frequently encounter harsh weather, dynamic occlusions, sudden lighting changes, and camera shakes. Under such challenges, the accuracy of open-source models drops by up to 35%, while closed-source models decline by 28%. Existing robustness methods (random frame masking, adversarial training, data augmentation) treat robustness as a single objective, disregarding that different perturbation types trigger distinct failure modes.

Key Challenge: The failure modes in reasoning triggered by structured, semantically meaningful visual perturbations (e.g., rain causing pavement reflection and changing object appearance) are fundamentally different from those caused by simple pixel noise—yet existing methods have never explicitly modeled these perturbation-specific failure behaviors.

Goal: (1) Design ROVA, a training framework capable of maintaining reasoning robustness under real-world perturbations; (2) Construct PVRBench, the first video reasoning benchmark with systematically injected real-world perturbations.

Key Insight: Enable the model to evaluate the training value of each perturbed sample relative to its current capability—discarding easy ones, immediately training on informative ones, and temporarily buffering difficult ones for delayed execution, thereby establishing an adaptive online curriculum.

Core Idea: Structured spatio-temporal perturbation generation of clean/corrupted video pairs \(\rightarrow\) Self-reflective difficulty-aware selective training \(\rightarrow\) Dual-branch GRPO alignment enforcing output consistency.

Method¶

Overall Architecture¶

ROVA operates in three stages: first, it applies four categories of structured spatio-temporal perturbations to video sequences to generate corrupted versions; second, the model performs a self-reflective difficulty assessment for each sample; finally, through dual-branch GRPO alignment, it enforces consistency in the reasoning outputs under clean and corrupted inputs.

flowchart TD
    A["Clean Video + Query"] --> B["Structured Spatio-Temporal Perturbation<br/>Weather/Lighting/Occlusion/Camera"]
    B --> C["Corrupted Video"]
    A --> D["Clean Branch (grad detached)"]
    C --> E["Perturbed Branch (optimized)"]
    D --> F["Self-Reflective Difficulty Assessment"]
    E --> F
    F -->|easy, high-conf| G["Discard"]
    F -->|difficult| H["Memory Buffer<br/>Periodic Re-evaluation"]
    F -->|informative| I["Dual-Branch GRPO Alignment<br/>Reasoning + Answer Consistency"]
    H -->|"re-eval→informative"| I
    H -->|">K_max times still difficult"| J["Eviction"]
    I --> K["Robust VLM"]

Key Designs¶

1. Structured Spatio-temporal Perturbations: From Pixel Noise to Semantic Distortions

For each frame \(f_t\), a style-specific mask \(P_t^{(m)} = B_t^{(m)} \odot C_t^{(m)}\) is generated (where \(B\) is a depth-aware or randomly driven binary mask, and \(C\) represents continuous intensity modulation), while frame indices are randomly permuted. Perturbations cover four categories: weather (rain, fog, snow; rendered via depth sensitivity), illumination (dusk, night, overexposure, shadows), occlusion (static/dynamic, reasonably positioned in the foreground), and camera motion (translation, zoom, rotation). The final video is represented as \(V' = \{f_{\pi(t)} \odot P_t^{(m)}\}_{t=1}^T\).

2. Self-Reflective Difficulty-Aware Online Curriculum: Empowering the Model to Select What to Learn

This is the core innovation of ROVA. At each training iteration, the model compares clean/corrupted outputs and yields a difficulty label \(d \in \{\text{easy}, \text{difficult}, \text{informative}\}\) and confidence \(c\) through a prompt template \(S_e\). Easy samples with high confidence are discarded; difficult ones are saved to a temporal memory buffer \(\mathcal{M}\) (which stores only mask metadata, not the entire video); informative and low-confidence easy samples are used for immediate training. Samples in the buffer are periodically re-evaluated, and those that remain difficult after exceeding \(K_{\max}\) evaluations are evicted. This establishes a closed-loop adaptive curriculum where the training distribution dynamically adjusts to model capability.

3. Dual-Branch GRPO Alignment: Enforcing Consistency Between Clean and Corrupted Outputs

The gradient of the clean branch is detached to serve as an anchor, while the corrupted branch is optimized for alignment. The alignment reward is decomposed into reasoning consistency \(r^{\text{align, r}}_j = \alpha_r \cdot \text{Sim}^r(o_j, \tilde{o}_j)\) and answer consistency \(r^{\text{align, a}}_j = \alpha_a \cdot \text{Sim}^a(o_j, \tilde{o}_j)\)." The total reward is \(R_j = r^F_j + r^{Acc}_j + r^A_j\) (format + accuracy + alignment), optimized via GRPO using the group-normalized advantage \(A_j\). The loss function is the standard GRPO objective: \(J(\theta) = \mathbb{E}[\frac{1}{G}\sum(\min(r_j A_j, \text{clip}(r_j, 1-\epsilon, 1+\epsilon)A_j) - \beta D_{KL}(F_\theta\|F_{ref}))]\).

Key Experimental Results¶

Main Results¶

ROVA outperforms the strongest open-source baseline, Embodied-R, by 17% at the 7B scale; larger variants (13B/72B) match or exceed the performance of GPT-4o and Gemini-1.5-Pro. Compared to the baselines, it achieves at least a 24% improvement in accuracy and a 9% gain in reasoning quality. Notably, these improvements also transfer to clean videos.

Reasoning Quality (5 Dimensions, 0-5 Scale)¶

PVRBench introduces five reasoning quality metrics: Fragility(↓), Consistency, Belief, Recovery, and Attention, evaluated by GPT-4o via a structured template. ROVA outperforms baselines across all dimensions.

Training Efficiency¶

The selective training strategy (discarding easy and delaying difficult samples) filters out low-utility samples, with the resulting gains in training efficiency offsetting the overhead of periodic re-evaluation.

Highlights & Insights¶

Self-reflective difficulty assessment is an elegant design: Allowing the model to judge if "this sample is suitable for my current capability" bypasses the hyperparameter sensitivity associated with fixed curriculums. The memory buffer's strategy of "learning the basics first before tackling hard challenges" aligns well with the concept of the "Zone of Proximal Development" in educational theory.
Clean performance also benefits: Robustness training improves performance on clean data—suggesting that robustness constraints act as a regularizer, preventing the model from overfitting to spurious correlations within the clean distribution.
PVRBench's 5-dimensional reasoning quality assessment is a core contribution of the benchmark—allowing researchers to diagnose whether "failures stem from perceptual errors or a broken chain of reasoning."
Storing only mask metadata instead of full videos is a clever engineering optimization—the memory buffer achieves the delayed training mechanism with practically negligible storage overhead.

Limitations & Future Work¶

Perturbation types are limited to 4 broad categories and 12 specific forms; the real-world perturbation space is significantly larger.
Training requires paired clean/corrupted videos, which increases the data preparation cost.
The self-reflective assessment depends heavily on the model's own capabilities, which may lead to inaccurate evaluations during the early stages of training.
The memory buffer size and re-evaluation frequency need to be adjusted based on GPU memory limits.

vs ImageNet-C: This work is the first to extend the perturbation robustness evaluation paradigm of image classification to video reasoning, maintaining spatio-temporal coherence in the perturbations.
vs Embodied-R: ROVA incorporates difficulty-aware curricula and perturbation-specific alignment rewards on top of Embodied-R's GRPO framework, serving as a robustness-enhanced version.
vs Curriculum Learning: Traditional curriculum learning schedules follow a fixed easy-to-hard progression, whereas ROVA's model-adaptive assessment is fully dynamic.

Rating¶

Novelty: ⭐⭐⭐⭐ (The combination of self-reflective curriculum and structured perturbations is novel, though GRPO alignment itself is not original)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (3 benchmarks + 5 reasoning metrics + multiple model scales + ablation)
Writing Quality: ⭐⭐⭐⭐ (Strong motivation, clear methodology)
Value: ⭐⭐⭐⭐⭐ (A critical requirement for real-world deployment, with PVRBench showing potential to become a community standard)