EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Conference: ECCV 2026
arXiv: 2511.18242
Project Page: https://people-robots.github.io/EgoVITA/
Code: None
Area: Video Understanding / Multimodal VLM
Keywords: Egocentric Video Reasoning, Plan-and-Verify Framework, Cross-view Alignment, GRPO, Dense Rewards

TL;DR

EgoVITA explicitly factorizes egocentric video reasoning into a two-stage "ego-planning + exo-verification" process. Trained via GRPO reinforcement learning guided by two dense reward signals (ACMG predictive cross-modal alignment and confidence verification preference optimization), it outperforms Qwen2.5-VL-7B by +7.7 points on EgoBlind using only 52k samples, while maintaining zero degradation in exocentric video understanding performance.

Background & Motivation

Egocentric video understanding is crucial for applications such as lifelogging, assistive technologies, and personal memory assistants. However, inherent continuous camera motion, partial observability, and frequent occlusions make procedural reasoning highly challenging. Current MLLMs in this scenario often generate plausible-sounding but visually inconsistent or weakly grounded responses; they lack explicit mechanisms to verify whether egocentric predictions conform to the actual physical constraints of the scene. For instance, a model might claim "next, put the plate into the cabinet" when no cabinet appears in the video from start to finish.

The Key Challenge of this failure lies in: egocentric planning (e.g., "reach for the cup") needs to be verified for physical feasibility from an external viewpoint (e.g., "the cup is indeed on the reachable countertop"), but paired egocentric-exocentric (ego-exo) video pairs are extremely scarce. Existing methods either use SFT to fine-tune on egocentric data, leading to a collapse in exocentric performance (catastrophic forgetting), or use large-scale sparse-reward RL (e.g., ~5M samples in EgoThinker) where exocentric performance still degenerates significantly (DocVQA -10.5), as sparse format and answer-correctness rewards fail to provide sufficient supervision for intermediate reasoning steps.

The Key Insight is: since MLLMs have been proven to possess cross-view reasoning capabilities (interpreting egocentric frames as third-person descriptions), verification can be internalized—allowing the same model to simultaneously act as both an "egocentric planner" and a "third-person auditor" without requiring external exocentric video inputs. Core Idea: Explicitly decouple egocentric video reasoning into a two-step "plan-then-verify" reasoning trajectory. Trajectory-level policy optimization is conducted within the GRPO framework using two dense reward signals that encourage alignment of predicted actions with future visual frames (ACMG) and consistency of verification reasoning (confidence reward), respectively, while leveraging lightweight exocentric regularization to prevent forgetting.

Method

Overall Architecture

EgoVITA models egocentric video reasoning as a sequential decision-making problem. Given a video \(V\) and a text prompt \(X\), the MLLM acts as a policy \(\pi_\theta\) to autoregressively generate a structured reasoning trajectory \(Y = (Y_{\text{plan}}, Y_{\text{verify}}, Y_{\text{ans}})\), which corresponds to three consecutive segments: egocentric planning (predicting egocentric action sequences), exocentric verification (auditing the spatiotemporal and logical consistency of the sequence from a third-party perspective), and the final answer. Training consists of two stages: Stage I uses SFT to initialize the "plan-then-verify" reasoning structure; Stage II uses GRPO for trajectory-level policy optimization, sampling \(k=8\) candidate trajectories for each input to update the policy based on intra-group relative rewards. The reward function comprises four components: formatting, answer correctness, ACMG cross-modal alignment, and confidence verification consistency. The architecture diagram is as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video V + Query X"] --> B["Egocentric Planning<br/>Generate egocentric action sequence"]
    B --> C["Exocentric Verification<br/>Third-party audit of logical consistency"]
    C --> D["Final Answer<br/>Integrate plan and verification results"]
    B -.->|"ACMG Reward<br/>Align predicted action with future N frames"| E["Predictive Cross-Modal Alignment"]
    C -.->|"Confidence Reward<br/>Preference optimization for verification quality"| F["Verification Consistency Optimization"]
    E --> G["GRPO Trajectory-level Policy Update"]
    F --> G

Key Designs

1. 推理分解：将第一人称推理拆为 Plan-then-Verify 双阶段

The fundamental reason behind current MLLM failures in reasoning over egocentric videos is not the lack of visual information, but the absence of an intermediate verification phase—egocentric planning requires auditing from a third-party perspective to determine its physical plausibility. EgoVITA explicitly splits the reasoning trajectory into two segments: \(Y^{\text{plan}}\) generates a sequence of temporal action clauses \(\{c_1, c_2, \dots, c_K\}\) describing the expected behavior of the camera wearer (e.g., "1. Walk to the sink, 2. Turn on the faucet, 3. Rinse the plate"); \(Y^{\text{verify}}\) reinterprets the same video in the third person, auditing year-by-year whether each action is consistent with the scene layout, object affordance, and observable context (e.g., "the sink is clearly visible in frames 5-20, the faucet is within reach, and the plan is logical"). This decoupling does not rely on paired ego-exo videos—exocentric verification operates directly on the egocentric video by simply shifting the reasoning perspective from "what am I doing" to "what is this person doing". The final answer \(Y^{\text{ans}}\) is then grounded in both egocentric procedural reasoning and third-person auditing, proving more reliable than direct single-perspective answering. The two phases have distinct roles: the planning phase specializes in egocentric procedural reasoning, while the verification phase preserves general third-person reasoning capabilities, thereby relieving the tension between specialization and generalization.

2. ACMG 密集奖励：预测性跨模态对齐

Sparse format rewards and answer correctness rewards can only evaluate final outputs and fail to provide fine-grained feedback for intermediate planning and verification steps. ACMG (Anticipatory Cross-Modal Grounding) is a dense reward signal that measures the cross-modal alignment between each action clause in the plan and subsequent visual observations. Specifically, for each action clause \(c_i\), the final hidden state \(h_i^{\text{text}}\) is extracted from the MLLM's language decoder and projected into the visual embedding space via a lightweight, trainable MLP (Anticipation Head: two linear layers + GELU + LayerNorm) to obtain the predicted future visual embedding \(\hat{v}_i = \text{MLP}(h_i^{\text{text}})\). Given the ground-truth visual embeddings \(v_{t+n}\) of the future \(N=16\) frames, the clause-level reward is computed as the maximum cosine similarity:

\[R_{\text{ACMG}}(c_i) = \max_{n \in \{1,\dots,N\}} \frac{\hat{v}_i \cdot v_{t+n}}{\|\hat{v}_i\| \|v_{t+n}\|}\]

The trajectory-level ACMG reward is the average of all clause-level rewards: \(R_{\text{ACMG}} = \frac{1}{K} \sum_{i=1}^K R_{\text{ACMG}}(c_i)\). Choosing max instead of mean is a critical design decision: a correct action (e.g., "walk to the bus stop") may only have a clear visual correspondence in a few frames (where the bus stop sign appears); averaging over all 16 frames would dilute the effective signal. Meanwhile, max is robust to temporal shifts and variations in action duration, scoring points based only on the best-aligned frame. Ablations confirm that max pooling yields a +3.9 point improvement over mean pooling on EgoBlind. The Anticipation Head is used only during training and discarded during inference, adding no deployment overhead.

3. 置信度奖励：教师引导 + 自排序的两阶段验证优化

The quality of exocentric verification directly affects the reliability of the final answer, but verification generated by the model during the initial stages of RL training is usually of low quality. EgoVITA's confidence reward adopts a two-stage design: the first 200 RL steps constitute the teacher-guided stage, which compares the model's rollout verification \(y^P\) for each trajectory with the reference verification \(y^T\) generated by a Qwen2.5-VL-72B teacher. The model is encouraged to align with the teacher through log-probability differences:

\[R_{\text{confidence}} = \gamma \left[\log \pi_\theta(y^T \mid V, X, Y^{\text{plan}}) - \log \pi_\theta(y^P \mid V, X, Y^{\text{plan}})\right]\]

where \(\gamma=0.1\) controls the strength of the preference signal. After 200 steps, it automatically switches to the self-ranking stage: each rollout is scored using the combined rewards, and the highest-scoring verification \(y^{\text{chosen}}\) and lowest-scoring verification \(y^{\text{rejected}}\) are selected for contrastive log-probability preference optimization. Ablation shows that even without the teacher warm-up stage, the model still achieves a +6.3 improvement on EgoBlind (full model +7.7), indicating that the gains primarily stem from the reward design itself rather than relying on external teachers. ACMG and confidence rewards are functionally complementary: ACMG predominantly enhances tasks with strong temporal alignment demands (EgoPlan +0.6), whereas the confidence reward strengthens verification consistency (EOC-Bench +1.3). Using them in combination yields even larger gains (EgoPlan +1.5, EOC-Bench +1.9).

4. 外中心视角正则化：防止灾难性遗忘

Training exclusively on egocentric data causes the model to lose its capability for exocentric video understanding. EgoVITA adopts a lightweight regularization: after every 200 GRPO iterations, a cross-entropy fine-tuning step is performed on MSR-VTT (a third-person video QA dataset), jointly optimized with the GRPO objective:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{GRPO}} + \lambda_{\text{exo}} \mathcal{L}_{\text{exo}}\]

where \(\lambda_{\text{exo}}=0.05\). MSR-VTT is selected because its diverse third-person everyday scenes provide complementary supervision. Ablation reveals a counter-intuitive phenomenon: a moderate amount of exocentric regularization (0.05) not only restores the exocentric performance lost during the SFT stage but, in turn, boosts egocentric performance—as the preserved general visual reasoning capability also positively affects egocentric understanding. However, an excessively large regularization factor (\(\geq 0.2\)) dilutes the dense reward signals, leading to a sudden drop in egocentric performance.

A Complete Example: Blind Navigation Assistance Scenario

An example from EgoBlind illustrates the complete reasoning process. The input video shows a visually impaired person walking along a sidewalk with a cane. The query is "What should be done next?".

1. Egocentric Planning: The model generates an action sequence: ① Walk along the sidewalk toward the bus stop area, ② Use the cane to sweep left and right to detect the curb edge, ③ Stop before the curb to wait and confirm safety, ④ Step down and cross the road after confirming no vehicles are passing.
2. ACMG Alignment: After the text embedding of clause ② "use the cane to sweep... detect the curb edge" is projected by the Anticipation Head, its cosine similarity with future frame \(t+12\) (showing the cane touching the curb stone) reaches 0.92, gaining a high ACMG reward. Conversely, if an incorrect action like "pick up the bottle on the ground" is inserted into the same context, its similarity to future frames is only 0.35 (\(\Delta=0.33\)). This demonstrates that ACMG captures action-dependent temporal changes rather than static background features.
3. Exocentric Verification: The model inspects the scenario in the third person: "The video shows a blind person walking along a sidewalk. There is a curb and a crosswalk about 3 meters ahead. The cane contacts the curb edge in frame 12. The planned step 'stop before the curb and wait' aligns with the safety logic of the scene; the plan is consistent."
4. Final Answer: "One should continue detecting the curb with the cane, stop at the curb edge to wait and confirm safety (paying attention to the sound of oncoming vehicles), and cross only after safety is confirmed." In contrast, the baseline Qwen2.5-VL directly outputs the unsafe suggestion: "Keep walking forward."

Loss & Training

Stage I: SFT. Training data are constructed using EgoProceL (5k samples with temporally aligned action annotations) and HD-EPIC (7k samples with human-annotated action-timestamp pairs). Qwen2.5-VL-72B generates planning sequences and third-person verification reasoning chains based on these annotations, which undergo quality filtering via Seed1.5-VL. Standard cross-entropy is used: \(\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(V,X,Y^*)\sim D} [\log \pi_\theta(Y^* \mid V,X)]\), with a learning rate of \(5 \times 10^{-6}\) for 1 epoch.

Stage II: GRPO. Rollouts are sampled from HD-EPIC and EgoIT (40k samples), with \(k=8\) candidate trajectories per input. Intra-group reward normalization \(\hat{A}(Y_i) = (R(Y_i) - \mu_R) / (\sigma_R + \epsilon)\) calculates relative advantages. The GRPO objective includes PPO-style clipping and a KL penalty \(\beta_{\text{KL}}=0.1\) to constrain deviations from the reference SFT policy. The combined reward is \(R = 0.1 R_{\text{format}} + 0.3 R_{\text{answer}} + 0.3 R_{\text{ACMG}} + 0.3 R_{\text{confidence}}\) (the format reward weighting is low because it saturates early). Eight NVIDIA L40S GPUs are deployed, utilizing DoRA adapters (\(r=8, \alpha=16\)) acting on the visual encoder, projector, and language decoder. During inference on a single L40S, a speed of 84.3 tokens/s and a TTFT of 0.6s are achieved, adding only 0.1s of overhead compared to the base Qwen2.5-VL.

Key Experimental Results

Main Results

EgoVITA achieves consistent egocentric improvements across three base model families without degrading exocentric performance.

Model	EgoBlind	EgoPlan	EgoThink	EOC-Bench	EgoOrient	Video-MME	MVBench
Qwen2.5-VL-7B	29.7	30.2	48.2	41.6	47.6	62.2	65.2
+ SFT	34.8	31.3	49.1	41.8	48.1	60.1	63.2
+ GRPO (Format+Answer)	35.6	31.4	49.0	42.2	48.6	62.9	66.4
EgoVITA	37.4 (+7.7)	32.9 (+2.7)	51.9 (+3.7)	44.1 (+2.5)	52.0 (+4.4)	64.6 (+2.4)	66.8 (+1.6)
InternVL-3.5-8B	47.8	34.0	58.5	37.1	36.3	65.6	71.4
EgoVITA	51.4 (+3.6)	35.9 (+1.9)	60.8 (+2.3)	39.7 (+2.6)	40.1 (+3.8)	68.3 (+2.7)	73.8 (+2.4)
Qwen3-VL-8B	48.4	33.7	62.7	46.8	60.8	71.4	68.4
EgoVITA	51.9 (+3.5)	35.0 (+1.3)	63.9 (+1.2)	48.6 (+1.8)	63.1 (+2.3)	72.2 (+0.8)	69.2 (+0.8)

Compared to the SOTA method EgoThinker (based on the same Qwen2-VL base), EgoVITA achieves a +5.8 improvement on EgoBlind using only 52k samples. In contrast, although EgoThinker was trained on approximately 5M samples, its performance plunged by -10.5 on DocVQA and -1.8 on MMMU. EgoVITA remains stable (DocVQA +0.6, MMMU +0.3), demonstrating that its cross-view stability is significantly superior to large-scale egocentric fine-tuning.

Ablation Study

Configuration	EgoPlan	EOC-Bench	Video-MME	Description
GRPO (Format+Answer only)	31.4	42.2	62.9	Sparse reward baseline
+ ACMG only	32.0 (+0.6)	43.1 (+0.9)	63.4 (+0.5)	ACMG improves temporal alignment
+ Confidence only	32.3 (+0.9)	43.5 (+1.3)	63.8 (+0.9)	Confidence improves verification consistency
EgoVITA (Full)	32.9 (+1.5)	44.1 (+1.9)	64.6 (+1.7)	Two rewards are complementary
w/o Egocentric Planning	29.6 (-3.3)	40.1 (-4.0)	63.0 (-1.6)	Removing planning hurts egocentric
w/o Exocentric Verification	31.5 (-1.4)	42.5 (-1.6)	61.2 (-3.4)	Removing verification hurts exocentric
ACMG Mean Pooling	31.5	42.6	—	Mean pooling dilutes temporal signals
N=4 window	31.0	42.4	—	Window too small; long actions lack corresponding frames
N=32 window	32.5	43.7	—	Window too large; introduces noise
Present-frame alignment	31.2	42.8	—	Present-frame alignment only captures static scenes

Key Findings

• ACMG and confidence rewards are functionally complementary: ACMG focuses on temporal alignment (EgoPlan +0.6), whereas the confidence reward focuses on verification consistency (EOC-Bench +1.3). The combined gains exceed the sum of individual gains, indicating that each contributes an independent improvement signal.
• The RL stage is more critical than the SFT stage: SFT only improves the output format, yielding limited gains in egocentric reasoning and damaging exocentric performance (MVBench -2.0, Video-MME -2.1). EgoVITA's GRPO stage not only recovers but surpasses baseline performance (MVBench +3.6 over SFT, Video-MME +4.5).
• Planning and verification assume distinct roles: Removing the planning segment severely hurts egocentric performance (EgoPlan -3.3), whereas removing the verification segment primarily hurts exocentric performance (Video-MME -3.4), validating the functional division of the two stages.
• Task-level analysis shows gains concentrate on safety-critical scenarios: The Safety Warning subcategory of EgoBlind sees the largest improvement (+5.4 over SFT); EOC-Bench's Future reasoning sees the most significant gain (+3.2); and EgoOrientBench's Choose task improves by +5.5. This indicates the framework is particularly effective for high-risk safety assistance and forward-looking reasoning tasks.
• EAGLE visualization confirms attention shifts from static backgrounds to task-critical objects: The attention of the base Qwen2.5-VL remains stuck on static backgrounds like shelves and walls and barely changes over time. EgoVITA's attention dynamically shifts among garlic, the egg bowl, the pan, and the hand region as frames progress, consistent with the design of ACMG promoting "looking into the future".

Highlights & Insights

• The "plan-then-verify" structure is a cross-view generalization paradigm without paired data: The core insight of decomposing reasoning into egocentric planning + exocentric verification is that—the same video, paired with two different prompts ("what am I doing" and "what is this person doing"), can achieve cross-view consistency without relying on paired ego-exo data. This is particularly important for resource-constrained assistive technology scenarios and can be transferred to tasks that require multi-view reasoning, such as robotic manipulation (egocentric perception + third-person workspace reasoning) and autonomous driving (driver view + bird's-eye view scene reasoning).
• The choice of max pooling for ACMG is ablation-backed and intuitively clean: Mean pooling dilutes the signal for continuous actions (e.g., "walk to the bus stop") due to a large number of mismatched frames; max is naturally robust to temporal shifts and variations in action duration. This seemingly minor pooling design choice actually contributes +3.9 points on EgoBlind (max vs. mean), indicating that in cross-modal alignment tasks, "finding the best-matching frame" is closer to human habits of verifying actions than "averaging all frames".
• The "moderate gain" from exocentric regularization challenges the intuition of Domain-Specific training: A moderate amount of exocentric regularization (0.05) not only prevents forgetting but conversely enhances egocentric performance—as the preserved general visual reasoning capability also positively affects egocentric understanding. This suggests that in multi-task or multi-domain training, mixing in a certain proportion of "out-of-domain" data might outperform "pure in-domain" data.
• Anticipation Head is discarded after training: The cross-modal projection module, comprising only a 2-layer MLP, provides dense alignment signals during training and is completely discarded during inference—achieving zero extra deployment cost. This design pattern of "training-time auxiliary modules" can be transferred to other RL tasks requiring intermediate supervision.
• 52k samples vs. 5M samples: EgoThinker runs on about 5M training samples but suffers from collapsed exocentric performance. In contrast, EgoVITA achieves stronger egocentric performance with zero exocentric degradation using less than 1% of the sample size. This proves that method design (dense rewards + perspective decomposition + exocentric regularization) matters more than simply scaling up data size.

Limitations & Future Work

• The fixed temporal window \(N=16\) fails to adapt to all action durations: Experiments indicate sensitivity to the value of N (N=4 is too short-sighted, N=32 is too noisy), whereas in reality, the duration of 'blinking' and 'cooking a pot of soup' varies wildly. The authors' proposal to use adaptive temporal windows (expanding for long actions and shrinking for rapid actions) is a reasonable and valuable direction for future expansion.
• Offline full-video reasoning assumption: Currently, EgoVITA reasons over complete videos, making it inapplicable to streaming scenarios. The incremental plan-and-verify mechanism proposed by the authors is a natural extension, but it requires addressing ACMG's alignment challenge under partial observations (where max cosine similarity cannot be computed when future frames have not yet arrived).
• Single-round verification without iterative correction: Currently, verification is executed only once. If verification uncovers errors in the plan, there is no mechanism to backtrack and correct them. An iterative plan-verify loop (planning \(\rightarrow\) verification \(\rightarrow\) plan correction \(\rightarrow\) re-verification) could further boost reliability in ambiguous scenarios, though at the expense of higher inference costs.
• Residual static scene bias: Failure case analysis shows that when the actor leaves the scene but relevant objects remain visible (e.g., the sink is still in view after the person walks away), ACMG may assign falsely high rewards based on static object co-occurrence; about 3% of rollouts fall into this failure mode (Type F). Future work could introduce causal attention masks or explicit object-human relationship modeling to alleviate this.
• Base model dependency of the Anticipation Head: ACMG assumes that the projected text embeddings via the MLP reside in the same space as the MLLM's visual encoder outputs. When switching base models, the projection layer may require recalibration, limiting the plug-and-play capability of the method.

Related Work & Insights

• vs. EgoThinker: Both use RL for egocentric video reasoning. However, EgoThinker relies on sparse formatting and answer rewards and requires ~5M samples, which severely degrades exocentric performance (DocVQA -10.5, MMMU -1.8). EgoVITA replaces sparse signals with dense ACMG and confidence rewards, achieving stronger egocentric performance with zero exocentric degradation using only 52k samples. This demonstrates that in trajectory-level RL, designing fine-grained rewards for intermediate reasoning steps is more critical than merely scaling up the dataset.
• vs. Paired Ego-Exo Methods (e.g., EgoExo-Con, Viewpoint Rosetta Stone): These methods depend on synchronously captured dual-perspective video pairs to learn viewpoint-invariant representations, incurring high data collection costs. EgoVITA demonstrates that cross-view consistency can be obtained "for free"—by simply prompting the same model to shift its reasoning perspective, without requiring external video inputs. This holds practical significance for resource-constrained scenarios like assistive technologies.
• vs. Original GRPO Usage (e.g., DeepSeek-R1): GRPO was originally designed for pure-text mathematical reasoning using rule-based sparse rewards. EgoVITA extends it to video and cross-modal scenarios by introducing dense rewards based on visual alignment, proving the feasibility and effectiveness of GRPO in multimodal trajectory-level reasoning. The relative advantages from intra-group normalization are naturally suited for ambiguous scenarios with multiple plausible reasoning paths; this characteristic is worth exploring in more perception-and-reasoning tasks.

Rating

• Novelty: 4/5 — The plan-then-verify factorization is not entirely new, but applying it to egocentric video reasoning without requiring paired ego-exo data is a unique contribution. The design of ACMG predictive cross-modal alignment as a dense RL reward is also novel.
• Experimental Thoroughness: 5/5 — Covers three base model families, nine benchmarks (ego + exo), multi-dimensional ablations (reward components, reasoning decomposition, window size, pooling strategy, teacher warm-up, visual component training, regularization strength), EAGLE visualization, t-SNE analysis, static bias testing, rollout failure classification, and clause position analysis, presenting an extremely broad coverage.
• Writing Quality: 4/5 — Clear structure, complete math formulations, and effectively supported arguments by figures/text. However, the ablation tables are scattered between the main text and supplementary materials, increasing the cognitive load, and some paragraphs are overly dense.
• Value: 4/5 — Achieving SOTA with only 52k samples while preserving exocentric performance holds high practical value for data-scarce academic research and assistive technology deployments. The plan-then-verify decomposition + dense reward training paradigm is highly transferrable to other visual tasks requiring multi-step reasoning and verification.

Related Papers

• [CVPR 2026] Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videos
• [CVPR 2026] Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning
• [CVPR 2026] Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
• [ECCV 2026] ROVA: Are Video Reasoning Models Ready to Go Outside?
• [ECCV 2026] VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement