EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Conference: ECCV 2026
arXiv: 2511.18242
Project Page: https://people-robots.github.io/EgoVITA/
Code: None
Area: Video Understanding / Multimodal VLM
Keywords: Egocentric Video Reasoning, Plan-and-Verify Framework, Cross-View Alignment, GRPO, Dense Rewards

TL;DR

EgoVITA explicitly decomposes egocentric video reasoning into a two-stage "ego-planning + exo-verification" process. Trained via GRPO reinforcement learning coupled with two dense reward signals (ACMG predictive cross-modal alignment + confidence verification preference optimization), it outperforms Qwen2.5-VL-7B by +7.7 points on EgoBlind using only 52k samples, while maintaining zero degradation in exocentric video understanding performance.

Background & Motivation

First-person (egocentric) video understanding is crucial in applications such as life logging, assistive technologies, and personal memory assistants. However, inherent continuous camera movement, partial observability, and frequent occlusions make procedural reasoning extremely difficult. Current MLLMs in this scenario often generate plausible but visually inconsistent or weakly grounded responses—they lack explicit mechanisms to verify whether egocentric predictions conform to the actual physical constraints of the scene. For example, a model might claim "next, put the plate into the cabinet," even though the cabinet never appears in the video.

The Key Challenge of this failure lies in: egocentric planning (e.g., "reaching for a cup") requires external perspective verification of its physical feasibility (e.g., "the cup is indeed on the reachable countertop"), but paired first-person/third-person video pairs (ego-exo pairs) are extremely scarce. Existing methods either fine-tune on egocentric data using SFT (which causes catastrophic forgetting of third-person capabilities) or, like EgoThinker, utilize large-scale sparse-reward RL (~5M samples), but still suffer from a significant decline in exocentric performance (DocVQA -10.5), as sparse formatting and answer correctness rewards cannot provide sufficient supervision for intermediate reasoning steps.

The Key Insight of this paper is: since MLLMs have been proven to possess cross-view reasoning capabilities (the ability to re-interpret first-person frames into third-person descriptions), verification can be internalized—allowing the same model to play the dual roles of both "first-person planner" and "third-person auditor" simultaneously, without requiring external exocentric video inputs. Core Idea: the first-person video reasoning is explicitly split into a two-step "plan-and-verify" reasoning trajectory, and trajectory-level policy optimization is performed under the GRPO framework. This is supervised by two dense reward signals that encourage alignment of predicted actions with future visual frames (ACMG) and consistency of verification reasoning (confidence reward), respectively, while incorporating lightweight exocentric regularization to prevent forgetting.

Method

Overall Architecture

EgoVITA models egocentric video reasoning as a sequential decision-making problem: given a video \(V\) and a text prompt \(X\), the MLLM acts as a policy \(\pi_\theta\) to autoregressively generate a structured reasoning trajectory \(Y = (Y^{\text{plan}}, Y^{\text{verify}}, Y^{\text{ans}})\), which corresponds to ego-planning (predicting the first-person action sequence), external-perspective verification (auditing the spatiotemporal and logical consistency of the sequence from a third-person perspective), and the final answer. Training consists of two stages: Stage I uses SFT to initialize the "plan-and-verify" reasoning structure; Stage II uses GRPO for trajectory-level policy optimization, sampling \(k=8\) candidate trajectories for each input and updating the policy based on intra-group relative rewards. The reward function comprises four components: formatting, answer correctness, ACMG cross-modal alignment, and confidence verification consistency. The framework diagram is shown below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video V + Query X"] --> B["Ego-Planning<br/>Generate First-Person Action Sequence"]
    B --> C["Exo-Verification<br/>Audit Logical Consistency from Third-Person"]
    C --> D["Final Answer<br/>Integrate Planning & Verification"]
    B -.->|"ACMG Reward<br/>Align Predicted Actions with Future N Frames"| E["Predictive Cross-Modal Alignment"]
    C -.->|"Confidence Reward<br/>Preference Optimization for Verification Quality"| F["Verification Consistency Optimization"]
    E --> G["GRPO Trajectory-Level Policy Update"]
    F --> G

Key Designs

1. Reasoning Decomposition: Splitting Egocentric Reasoning into a Two-Stage Plan-then-Verify Process

The fundamental reason for current MLLMs' failure on egocentric videos is not the lack of visual information, but the absence of an intermediate verification phase—first-person plans need to be audited by a third-person perspective to judge their physical plausibility. EgoVITA explicitly splits the reasoning trajectory into two segments: \(Y^{\text{plan}}\) generates a sequence of temporal action clauses \(\{c_1, c_2, \dots, c_K\}\), describing the camera wearer's expected behaviors (e.g., "1. walk to the sink, 2. turn on the faucet, 3. rinse the plate"); \(Y^{\text{verify}}\) re-interprets the same video in the third person, auditing whether each action is consistent with the scene layout, object affordance, and observable context (e.g., "the sink is clearly visible in frames 5-20, the faucet is within reach, the plan is reasonable"). This decoupling does not rely on paired ego-exo videos—the external perspective verification operates directly on the egocentric video by simply switching the reasoning perspective from "what am I doing" to "what is this person doing". The final answer \(Y^{\text{ans}}\) is based on both first-person procedural reasoning and third-person auditing, making it more reliable than direct answers from a single perspective. Both stages serve distinct functions: the planning stage specializes in first-person procedural reasoning, while the verification stage retains general third-person reasoning capabilities, thereby relieving the tension between specialization and generalization.

2. ACMG Dense Reward: Predictive Cross-Modal Alignment

Sparse formatting and answer correctness rewards can only evaluate the final output and cannot provide fine-grained feedback for intermediate planning and verification steps. ACMG (Anticipatory Cross-Modal Grounding) is a dense reward signal that measures the degree of cross-modal alignment between each action clause in the plan and subsequent visual observations. Specifically, for each action clause \(c_i\), its last hidden state \(h_i^{\text{text}}\) is extracted from the MLLM language decoder and projected onto the visual embedding space via a lightweight trainable MLP (Anticipation Head, comprising two linear layers + GELU + LayerNorm) to obtain the predicted future visual embedding \(\hat{v}_i = \text{MLP}(h_i^{\text{text}})\). Utilizing the ground-truth visual embeddings \(v_{t+n}\) of the future \(N=16\) frames, the clause-level reward is computed as the maximum cosine similarity:

\[R_{\text{ACMG}}(c_i) = \max_{n \in \{1,\dots,N\}} \frac{\hat{v}_i \cdot v_{t+n}}{\|\hat{v}_i\| \|v_{t+n}\|}\]

The trajectory-level ACMG reward is the average of all clause rewards: \(R_{\text{ACMG}} = \frac{1}{K} \sum_{i=1}^K R_{\text{ACMG}}(c_i)\). Taking the max instead of the mean is a key design decision: a correct action (such as "walking towards the bus stop") may have a clear visual correspondence (the bus stop sign appearing) in only a few frames; averaging over all 16 frames would dilute the valid signal. The max operator is robust to temporal shifts and variations in action duration, providing scores by identifying only the best-aligned frames. Ablation studies confirm that max pooling yields a +3.9 point improvement over mean pooling on EgoBlind. The Anticipation Head is discarded after training and is not used during inference, incurring no additional deployment cost.

3. Confidence Review: Two-Stage Verification Optimization via Teacher-Guided and Self-Ranking

The quality of the external-perspective verification directly affects the reliability of the final answer, but verification generated by the model in the early stages of RL training is often of low quality. The confidence reward in EgoVITA employs a two-stage design: the first 200 RL steps constitute the teacher-guided phase, where the model's generated verification \(y^P\) from each rollout is compared to the reference verification \(y^T\) generated by a Qwen2.5-VL-72B teacher. The model is encouraged to align with the teacher through log-probability margins:

\[R_{\text{confidence}} = \gamma \left[\log \pi_\theta(y^T \mid V, X, Y^{\text{plan}}) - \log \pi_\theta(y^P \mid V, X, Y^{\text{plan}})\right]\]

where \(\gamma=0.1\) controls the strength of the preference signal. After 200 steps, it automatically switches to the self-ranking stage: each rollout is scored using a combined reward, and pairwise log-probability preference optimization is performed using the highest-scoring verification \(y^{\text{chosen}}\) and the lowest-scoring verification \(y^{\text{rejected}}\). Ablation studies show that even without the teacher warm-up stage, the model still achieves a +6.3 improvement on EgoBlind (compared to +7.7 for the full model), indicating that the gains primarily stem from the design of the reward itself rather than reliance on external teachers. ACMG and confidence rewards are functionally complementary: ACMG mainly enhances tasks with strong temporal alignment demands (EgoPlan +0.6), while confidence rewards strengthen verification consistency (EOC-Bench +1.3), yielding greater gains when combined (EgoPlan +1.5, EOC-Bench +1.9).

4. Exocentric View Regularization: Preventing Catastrophic Forgetting

Training exclusively on egocentric data causes the model to lose its third-person video understanding capability. EgoVITA addresses this with a lightweight regularization: after every 200 GRPO iterations, cross-entropy fine-tuning is performed on MSR-VTT (a third-person perspective video QA dataset), which is joint-optimized with the GRPO objective:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{GRPO}} + \lambda_{\text{exo}} \mathcal{L}_{\text{exo}}\]

where \(\lambda_{\text{exo}}=0.05\). MSR-VTT is chosen because its diverse third-person everyday scenes provide complementary supervision. Ablation studies reveal a counter-intuitive phenomenon: a moderate amount of exo regularization (0.05) not only restores the third-person performance lost during the SFT phase but also conversely boosts egocentric performance—retaining general visual reasoning abilities positively impacts first-person understanding. However, excessively large values (\(\geq 0.2\)) dilute the dense reward signals, leading to a sharp drop in egocentric performance.

One Complete Example: Blind Navigation Assist Scenario

An example from EgoBlind illustrates the complete reasoning process. The input video shows a blind person walking along a sidewalk with a white cane, and the query is "What should be done next?".

1. Ego-planning: The model generates an action sequence—① walk along the sidewalk towards the bus stop area, ② sweep the white cane left and right to detect the curb edge, ③ stop before the curb to wait and confirm safety, ④ step down and cross the street after confirming no vehicles are passing.
2. ACMG alignment: The text embedding of clause ② "sweep the white cane left and right to detect the curb edge" is projected via the Anticipation Head, achieving a cosine similarity of 0.92 with the ground-truth future frame \(t+12\) (where the cane contacts the curb), earning a high ACMG reward. In contrast, if an incorrect action like "pick up a bottle from the ground" is inserted into the same scene, its similarity with future frames is only 0.35 (\(\Delta=0.33\)), demonstrating that ACMG captures action-dependent temporal changes rather than static background features.
3. Exo-verification: The model reviews the scene from a third-person perspective—"A blind person is walking along the sidewalk, with a curb and a crosswalk about 3 meters ahead. The white cane makes contact with the edge of the curb in frame 12. The planned 'stop before the curb and wait' step conforms to safety logic, and the plan is consistent."
4. Final Answer: "The person should continue to detect the curb with the cane, stop at the edge of the curb to confirm safety (paying attention to the sound of oncoming traffic), and then cross once safety is confirmed." In contrast, the baseline Qwen2.5-VL directly outputs the unsafe suggestion: "Keep walking straight forward."

Loss & Training

Stage I: SFT. Training data is constructed using EgoProceL (5k samples, including temporally aligned action annotations) and HD-EPIC (7k samples, including manually annotated action-timestamp pairs). Qwen2.5-VL-72B is used to generate planning sequences and third-person verification reasoning chains based on these annotations, which are then filtered for quality by Seed1.5-VL. Standard cross-entropy is used: \(\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(V,X,Y^*)\sim D} [\log \pi_\theta(Y^* \mid V,X)]\), with a learning rate of \(5 \times 10^{-6}\) for 1 epoch.

Stage II: GRPO. Rollouts are sampled from HD-EPIC and EgoIT (40k samples), with \(k=8\) candidate trajectories per input. Relative advantages are computed using intra-group standardized rewards \(\hat{A}(Y_i) = (R(Y_i) - \mu_R) / (\sigma_R + \epsilon)\), and the GRPO objective includes a PPO-style clip and a KL penalty \(\beta_{\text{KL}}=0.1\) to limit deviation from the SFT reference policy. The combined reward is \(R = 0.1 R_{\text{format}} + 0.3 R_{\text{answer}} + 0.3 R_{\text{ACMG}} + 0.3 R_{\text{confidence}}\) (formatting reward is weighted lower due to early saturation). Eight NVIDIA L40S GPUs and DoRA adapters (\(r=8, \alpha=16\)) are applied across the visual encoder, projector, and language decoder. During inference, it achieves 84.3 tokens/s and a TTFT of 0.6s on a single L40S, incurring only a 0.1s overhead compared to the base Qwen2.5-VL.

Key Experimental Results

Main Results

EgoVITA achieves consistent egocentric improvements across three base model families without degrading exocentric performance.

Model	EgoBlind	EgoPlan	EgoThink	EOC-Bench	EgoOrient	Video-MME	MVBench
Qwen2.5-VL-7B	29.7	30.2	48.2	41.6	47.6	62.2	65.2
+ SFT	34.8	31.3	49.1	41.8	48.1	60.1	63.2
+ GRPO (Format+Answer)	35.6	31.4	49.0	42.2	48.6	62.9	66.4
EgoVITA	37.4 (+7.7)	32.9 (+2.7)	51.9 (+3.7)	44.1 (+2.5)	52.0 (+4.4)	64.6 (+2.4)	66.8 (+1.6)
InternVL-3.5-8B	47.8	34.0	58.5	37.1	36.3	65.6	71.4
EgoVITA	51.4 (+3.6)	35.9 (+1.9)	60.8 (+2.3)	39.7 (+2.6)	40.1 (+3.8)	68.3 (+2.7)	73.8 (+2.4)
Qwen3-VL-8B	48.4	33.7	62.7	46.8	60.8	71.4	68.4
EgoVITA	51.9 (+3.5)	35.0 (+1.3)	63.9 (+1.2)	48.6 (+1.8)	63.1 (+2.3)	72.2 (+0.8)	69.2 (+0.8)

Compared with the SOTA method EgoThinker (based on the same Qwen2-VL base), EgoVITA achieves a +5.8 improvement on EgoBlind using only 52k samples, whereas EgoThinker, despite using ~5M training samples, drops by -10.5 on DocVQA and -1.8 on MMMU. EgoVITA remains stable (DocVQA +0.6, MMMU +0.3), proving that cross-view stability is significantly superior to large-scale egocentric fine-tuning.

Ablation Study

Configuration	EgoPlan	EOC-Bench	Video-MME	Description
GRPO (Format+Answer only)	31.4	42.2	62.9	Sparse reward baseline
+ ACMG only	32.0 (+0.6)	43.1 (+0.9)	63.4 (+0.5)	ACMG enhances temporal alignment
+ Confidence only	32.3 (+0.9)	43.5 (+1.3)	63.8 (+0.9)	Confidence enhances verification consistency
EgoVITA (Full)	32.9 (+1.5)	44.1 (+1.9)	64.6 (+1.7)	Broad complementarity of both rewards
w/o Ego-planning segment	29.6 (-3.3)	40.1 (-4.0)	63.0 (-1.6)	Removing planning harms egocentric performance
w/o Exo-verification segment	31.5 (-1.4)	42.5 (-1.6)	61.2 (-3.4)	Removing verification harms exocentric performance
ACMG Mean Pooling	31.5	42.6	—	Mean pooling dilutes temporal signals
N=4 Window	31.0	42.4	—	Window too small, no corresponding frame for long actions
N=32 Window	32.5	43.7	—	Window too large, introduces noise
Present-frame alignment	31.2	42.8	—	Current-frame alignment only captures static scenes

Key Findings

• ACMG and confidence rewards are functionally complementary: ACMG focuses on temporal alignment (EgoPlan +0.6), while confidence rewards focus on verification consistency (EOC-Bench +1.3). The combined gains exceed the sum of their individual improvements, indicating that each contributes distinct refinement signals.
• The RL stage is more critical than the SFT stage: SFT only improves the output format, yielding limited gains for egocentric reasoning and hindering exocentric performance (MVBench -2.0, Video-MME -2.1). EgoVITA's GRPO stage not only restores but surpasses the baseline (MVBench +3.6 over SFT, Video-MME +4.5).
• Planning and verification assume different roles: Removing the planning segment severely degrades egocentric performance (EgoPlan -3.3), while removing the verification segment primarily degrades exocentric performance (Video-MME -3.4), confirming the functional specialization of the two stages.
• Task-level analysis shows gains concentrated in safety-critical scenarios: The Safety Warning subcategory of EgoBlind shows the largest improvement (+5.4 over SFT), the Future reasoning of EOC-Bench sees the most significant gain (+3.2), and the Choose task of EgoOrientBench improves by +5.5, illustrating that the framework is exceptionally effective for high-risk safety-critical and prospective reasoning tasks.
• EAGLE visualization confirms attention shifts from static background to task-critical objects: The attention of the base Qwen2.5-VL remains on static backgrounds (e.g., shelves, walls) and is almost invariant over time. By contrast, EgoVITA's attention dynamically shifts among garlic, egg bowls, pans, and hands as the frames progress, consistent with the ACMG design of "paying attention to the future."

Highlights & Insights

• "Plan-and-Verify" is a cross-view generalization paradigm requiring no paired data: The core insight of decomposing reasoning into ego-planning + exo-verification is that the same video, when paired with two different prompts ("what am I doing" and "what is this person doing"), can achieve cross-view consistency without relying on paired ego-exo data. This is particularly valuable for resource-constrained assistive technology scenarios and can migrate to other tasks requiring multi-view reasoning, such as robot manipulation (first-person perception + third-person workspace reasoning) and autonomous driving (driver view + birds-eye view scene reasoning).
• The choice of max pooling in ACMG is validated by ablation and has intuitive explanations: Mean pooling dilutes signals for continuous actions (e.g., "walking to the bus stop") due to a large number of non-matching frames, while max pooling is naturally robust to temporal shifts and variations in action duration. This seemingly minor pooling design choice contributes an additional +3.9 points on EgoBlind (max vs. mean), implying that in cross-modal alignment tasks, "finding the best-matching frame" conforms more to how humans verify actions than "averaging all frames."
• The "moderate gain" of exocentric regularization challenges the intuition of Domain-Specific Training: Moderate exocentric regularization (0.05) not only prevents forgetting but conversely enhances egocentric performance—retaining general visual reasoning capabilities positively impacts first-person understanding. This suggests that in multi-task/multi-domain training, a certain proportion of "out-of-domain data mixing" may be superior to "pure in-domain data."
• Anticipation Head is discarded after training: The cross-modal projection module, comprising only a 2-layer MLP, provides dense alignment signals during training and is completely discarded during inference—resulting in zero extra deployment cost. This "auxiliary training-phase module" design pattern can be applied to other RL tasks requiring intermediate supervision.
• 52k samples vs 5M samples: EgoThinker uses ~5M samples for training yet causes exocentric performance to collapse, whereas EgoVITA achieves stronger egocentric performance with zero exocentric degradation using less than 1/100 of the sample size. This demonstrates that method design (dense rewards + perspective decomposition + exocentric regularization) is more critical than scaling data size.

Limitations & Future Work

• Fixed temporal window \(N=16\) cannot adapt to all action durations: Experiments show sensitivity to the \(N\) value (\(N=4\) is too short-sighted, \(N=32\) is too noisy). In practice, action durations differ significantly (e.g., "blinking" vs. "boiling soup"). Realizing an adaptive temporal window (expanding for long actions, shrinking for fast actions) is a reasonable and promising future extension.
• Offline full-video reasoning assumption: Currently, EgoVITA reasons over the entire video, which is not applicable to streaming scenarios. Incremental plan-and-verify is a natural extension proposed by the authors, but it requires solving the alignment issues of ACMG under partial observation (i.e., when future frames have not yet arrived and max cosine similarity cannot be computed).
• Single-turn verification without iterative correction: Currently, verification is executed only once. If the verification detects an error in the planning, there is no mechanism to backtrack and correct it. An iterative plan-verify loop (planning → verification → correcting plan → re-verification) could further improve reliability in ambiguous scenarios, albeit at the expense of higher inference costs.
• Residual static scene bias: Failure analysis shows that when a person leaves the scene but relevant objects remain visible (e.g., the sink remains in view after the person walks away), ACMG may issue false high rewards based on static object co-occurrence—accounting for about 3% of rollout failures (Type F). Future research could introduce causal attention masks or explicit relationship modeling between humans and objects to mitigate this.
• Dependence on base model for the Anticipation Head: ACMG assumes that the projected text embeddings from the MLP share the same space as the MLLM visual encoder outputs. Replacing the base model may require recalibrating the projection layer, which restricts the plug-and-play capability of the method.

Related Work & Insights

• vs EgoThinker: Both employ RL for egocentric video reasoning, but EgoThinker relies on sparse formatting and answer rewards and requires ~5M samples, resulting in heavily degraded exocentric performance (DocVQA -10.5, MMMU -1.8). EgoVITA replaces sparse signals with dense ACMG and confidence rewards, achieving stronger egocentric performance with zero exocentric degradation using 52k samples. This demonstrates that in trajectory-level RL, designing fine-grained rewards for intermediate reasoning steps is more crucial than scaling the dataset.
• vs Paired Ego-Exo Methods (EgoExo-Con, Viewpoint Rosetta Stone): These methods rely on synchronously collected dual-view video pairs to learn viewpoint-invariant representations, incurring high data acquisition costs. EgoVITA demonstrates that cross-view consistency can be obtained "for free"—by simply switching the model's reasoning perspective without external video inputs. This holds practical significance for resource-constrained assistive technology scenarios.
• vs Original GRPO Usage (DeepSeek-R1): GRPO was originally designed for pure-text mathematical reasoning using rule-based sparse rewards. EgoVITA extends it to video and cross-modal scenarios by introducing dense rewards based on visual alignment, proving the feasibility and effectiveness of GRPO in multimodal trajectory-level reasoning. The group-relative advantage normalization in GRPO is naturally suited for ambiguous scenarios with multiple reasonable reasoning paths, a characteristic worth exploring in more perception-and-reasoning tasks.

Rating

• Novelty: 4/5 —— The concept of plan-then-verify decomposition is not entirely new, but its application to egocentric video reasoning without requiring paired ego-exo data is a unique contribution. The design of ACMG predictive cross-modal alignment as a dense RL reward is also novel.
• Experimental Thoroughness: 5/5 —— Covers three base model families, nine benchmarks (ego+exo), multi-dimensional ablations (reward components, reasoning decomposition, window size, pooling strategy, teacher warm-up, vision component training, regularization severity), EAGLE visualization, t-SNE analysis, static bias tests, rollout failure classification, and clause position analysis, representing a very broad range of evaluations.
• Writing Quality: 4/5 —— Clear structure, complete formulas, and effective support of the arguments through figures and text. However, splitting the ablation tables between the main body and supplementary material increases the reading burden, and some paragraphs are somewhat dense.
• Value: 4/5 —— Achieving SOTA with only 52k samples without sacrificing exocentric performance is highly meaningful for data-scarce academic research and assistive technology deployments. The plan-verify decomposition and dense reward training paradigm are transferable to other visual tasks requiring multi-step reasoning and verification.

Related Papers

• [CVPR 2026] Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videos
• [CVPR 2026] Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning
• [CVPR 2026] Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
• [ECCV 2026] ROVA: Are Video Reasoning Models Ready to Go Outside?
• [ECCV 2026] VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement