EgoExo-Con: Exploring View-Invariant Video Temporal Understanding¶

Conference: ECCV 2026
arXiv: 2510.26113
Code: https://github.com/mjjung/EgoExo-Con
Area: Video Understanding
Keywords: Video Understanding, View Invariance, Temporal Grounding, Reinforcement Learning, Cross-View Consistency, Ego-Exo

TL;DR¶

This paper introduces the EgoExo-Con benchmark (1,148 pairs of synchronized ego-exo videos + 2,269 human-refined temporal queries) to systematically evaluate the cross-view consistency of Video-LLMs in temporal understanding for the first time, revealing that existing models' cross-view consistency barely reaches half of their single-view performance. Furthermore, it proposes the View-GRPO reinforcement learning framework, which leverages dual reasoning rewards of semantic alignment and structural consistency to significantly enhance the robustness and consistency of cross-view temporal understanding.

Background & Motivation¶

Existing Video-LLMs have made significant strides in video question answering and temporal grounding tasks, but almost all benchmarks and methods assume a fixed or minimally varying single perspective (typically the third-person "exo" perspective). However, when the same event is captured from different perspectives, the visual appearance varies drastically—for instance, a head-mounted camera (ego) and a tripod side shot (exo) recording the same cooking process look completely different, yet the underlying temporal dynamics (cutting, stirring) are identical. Humans can effortlessly track action sequences and localize temporal segments across different viewpoints, but for Video-LLMs, this cross-view temporal understanding capability remains largely unexplored.

Existing ego-exo paired datasets (e.g., CharadesEgo, LEMMA, EgoExo-4D) are either restricted to specific domains or do not evaluate cross-view temporal reasoning. More critically, the original queries in existing benchmarks are often highly templated (such as category actions in CharadesEgo) or based on atomic action-object labels (such as in LEMMA), which introduces view-induced ambiguity—key elements might be visible in one view but occluded in another. This prevents reliable evaluation of cross-view consistency.

The core Key Insight of this paper is that temporal reasoning is the most natural task to test cross-view understanding—while appearance cues vary drastically with viewpoint, the temporal structure of events remains invariant. Therefore, this paper constructs the EgoExo-Con benchmark, utilizing human-refined queries to ensure that the concepts in the queries can be reliably observed from both views, while evaluating accuracy and cross-view consistency simultaneously. Building upon this, View-GRPO is proposed to explicitly reinforce view-specific temporal reasoning via reinforcement learning while encouraging cross-view consistent understanding.

Core Idea: Pair synchronized ego-exo videos with human-refined queries to evaluate the cross-view temporal consistency of Video-LLMs, and utilize GRPO reinforcement learning with reasoning rewards to enable the model to learn view-invariant temporal reasoning.

Method¶

Overall Architecture¶

The work in this paper is divided into two parts: benchmark construction and model improvement. The EgoExo-Con benchmark collects synchronized ego-exo videos from three datasets (CharadesEgo, LEMMA, and EgoExo-4D), refines the original queries, and generates misaligned negative query samples using GPT-4o, followed by human verification to ensure that both views can reliably identify the queried actions. This yields 1,148 video pairs and 2,269 queries supporting two temporal understanding tasks: Temporal Verification (verifying whether an event occurs during a given period) and Temporal Grounding (localizing the start and end timestamps of an event).

On the model improvement side, View-GRPO is proposed, which customizes reward functions for cross-view temporal reasoning based on GRPO. The pipeline is as follows: first, GPT-5 is used to generate step-by-step reasoning chains for each view, constructing the View30K training dataset; then, during GRPO training, the model generates candidate responses for each pair of synchronized videos, optimized and updated jointly via three reward components: format reward, accuracy reward, and reasoning reward. The reasoning reward is the core innovation, composed of two sub-metrics: semantic alignment (evaluated via an LLM Judge against the reference reasoning chain) and structural consistency (measured via the Jaccard overlap of action/object sets across views).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synchronized Ego-Exo Video Pairs<br/>+ Human-Refined Queries"] --> B["GPT-5 Generates View-Specific Reasoning Chains<br/>Constructs View30K Dataset"]
    B --> C["View-GRPO Training<br/>GRPO Group Normalization + KL Regularization"]
    C --> D["Format Reward<br/>think/answer template"]
    C --> E["Accuracy Reward<br/>tIoU / Binary Accuracy"]
    C --> F["Reasoning Reward<br/>Semantic Alignment + Structural Consistency"]
    D --> G["Cross-View Consistent Temporal Predictions<br/>V-EgoExo / G-EgoExo"]
    E --> G
    F --> G

Key Designs¶

1. EgoExo-Con Benchmark Construction: Multi-Stage Query Refinement and Human Verification

Original queries from existing datasets cannot be directly used for cross-view consistency evaluation—CharadesEgo uses templated category actions while LEMMA relies on atomic action-object labels, both of which lack details and suffer from view-induced ambiguities. For example, "A person is smiling" is completely invisible from the ego perspective.

To address this issue, a three-stage refinement pipeline is designed. In the first stage, frame-level HOI labels from LEMMA (e.g., "put + cup, fridge") are aggregated into longer continuous segments, from which salient verbs and nouns are extracted and converted into natural language queries using a rule engine. In the second stage, GPT-4o is leveraged to refine the complete set of queries: given sampled frames of the target timestamp, the model verifies whether the queried action can be reliably inferred from both perspectives and templates a refined version. Concurrently, a "misaligned query" is generated for each query as a negative sample for the temporal verification task to balance the "yes/no" answer ratio. In the third stage, four human evaluators audit all samples to ensure that the refined queries are accurately grounded in both perspectives, and that the misaligned queries indeed conflict with the visual content. Ambiguous instances are further refined or discarded, with borderline cases cross-checked by the authors.

This yields 1,148 pairs of synchronized videos (426 from CharadesEgo, 558 from EgoExo-4D, and 164 from LEMMA) and 2,269 tightly aligned query-timestamp pairs. The average video duration is 86.4 seconds, with an average event segment length of 12.8 seconds, an average of 13.0 tokens for normal queries, and 15.9 tokens for misaligned queries. The three subsets present distinct developmental and temporal granularities, introducing diverse challenges to the evaluation.

2. Reasoning Reward Design: Dual-Driven by Semantic Alignment and Cross-View Structural Consistency

The core innovation of View-GRPO lies in its reasoning reward \(r_{\text{reasoning}}\), which directs the model not only to pursue correctness in final answers but also to generate interpretable, cross-view consistent reasoning paths.

The semantic alignment reward \(r_{\text{sem}}\) employs an LLM Judge (e.g., Qwen2.5-7B) to semantically compare the model-generated reasoning chain \(o\) with the reference reasoning chain \(o^*\) generated by GPT-5, outputting a score between 0 and 1:

\[r_{\text{sem}}(o) = \text{Judge}(o, o^*) \in [0, 1]\]

This encourages the model to generate reasoning processes faithful to reference explanations, preventing hallucinations caused by unconstrained free-generation.

The structural consistency reward \(r_{\text{struct}}\) approaches the problem from a cross-view perspective: for two views of the same event, it extracts the action set \(\mathcal{A}\) and object set \(\mathcal{O}\) (using spaCy) from their respective reasoning chains, and computes the cross-view Jaccard overlap:

\[r_{\text{struct}}(o) = \frac{1}{2}\big(\text{Jac}(\mathcal{A}(o), \mathcal{A}(\tilde{o})) + \text{Jac}(\mathcal{O}(o), \mathcal{O}(\tilde{o}))\big)\]

where \(\tilde{o}\) represents the reasoning chain of the alternative view. The Key Insight behind this design is that the same event under different viewpoints should share central actions and objects. A higher overlap of action/object sets extracted from the reasoning process indicates a more consistent temporal understanding of the two views.

These two components are combined into the reasoning reward: \(r_{\text{reasoning}}(o) = \lambda \cdot r_{\text{sem}}(o) + (1-\lambda) \cdot r_{\text{struct}}(o)\), with \(\lambda=0.7\). Compared to native GRPO which purely optimizes for target answer accuracy, the reasoning reward provides richer training signals: semantic alignment ensures reasoning quality, while structural consistency drives the model to internalize shared cross-view temporal abstractions rather than relying on view-specific appearance biases.

3. View-GRPO Training Framework: Three-in-One Reward-Driven Cross-View RL

View-GRPO integrates the aforementioned reasoning reward with format and accuracy rewards into a unified total reward:

\[r(o) = r_{\text{accuracy}}(o) + r_{\text{format}}(o) + r_{\text{reasoning}}(o)\]

The format reward \(r_{\text{format}}\) acts as a binary indicator, requiring responses to follow the <think>...</think><answer>...</answer> template, which facilitates structured reasoning and answer extraction. The accuracy reward \(r_{\text{accuracy}}\) uses tIoU between the predicted and ground-truth intervals for temporal grounding, and binary accuracy for temporal verification.

During training, the model is optimized based on the GRPO algorithm: for a given prompt \(p\), it generates \(G\) candidate responses, optimizes the weighted target after group-normalizing the reward values, and integrates a KL divergence regularization term to prevent drifting too far from the base model. The training freezes the visual encoder and only updates LLM parameters, leveraging eight GPU A100 cards, with a batch size of 8 and a learning rate of \(1\times 10^{-6}\).

Distinct from naively performing SFT directly on both perspectives, View-GRPO utilizes reasoning rewards to explicitly guide the model to generate faithful, step-by-step temporal explanations for each view while converging on consistent temporal conclusions. This forces the model to mitigate its reliance on view-specific biases and instead learn shared, view-invariant temporal abstractions.

Loss & Training¶

View-GRPO is based on the GRPO algorithm, optimizing the model to maximize the weighted sum of group-normalized rewards, supplemented by a KL divergence regularization term \(\beta D_{\text{KL}}(\pi_{\theta} \| \pi_{\text{ref}})\). The training dataset, View30K, is generated by GPT-5: video frames are sampled at 1 FPS to generate step-by-step reasoning chains for each view, filtering out instances where reasoning fails or the predicted tIoU is below 0.7 in either view, resulting in 3.3k videos and 30k reasoning instances. Training uses the AdamW optimizer with the visual encoder frozen, a maximum image resolution of 2.8M pixels, and an intermediate reasoning length of 256 tokens (experiments show that excessively short outputs lack context, whereas excessively long outputs lead to over-exploration and hallucinations). The SFT baseline is fine-tuned using LoRA for 3 epochs on four A100 GPUs.

Key Experimental Results¶

Main Results¶

Existing models' performance on EgoExo-Con: Humans achieve 92.1%/91.3% (Exo/Ego) and 89.4% consistency on temporal verification, and 72.4%/73.0% with 67.3% consistency on temporal grounding. Among closed-source models, Gemini-2.5 Flash is the strongest (V around 70%, G around 42-46%), yet its cross-view consistency is only 52.3% (V) and 20.8% (G), significantly lagging behind single-view metrics. Among open-source models, TimeChat-VT achieves the best temporal verification consistency (42.1%), while TimeSuite achieves the best temporal grounding consistency (18.7%). However, all open-source models' consistency scores barely reach half of their single-view performance. Models with training data containing ego-views do not exhibit higher consistency than those trained solely on exo-views, indicating that simply mixing ego-exo data does not naturally nurture view invariance.

Method	V-Exo	V-Ego	V-ExoEgo	G-Exo	G-Ego	G-ExoEgo
Human	92.1	91.3	89.4	72.4	73.0	67.3
GPT-5	60.5	61.3	52.5	34.5	32.8	20.1
Gemini-2.5 Flash	70.4	70.1	52.3	42.0	45.9	20.8
TimeChat-VT (best open)	62.1	61.4	42.1	27.8	26.2	16.3
Video-LLaMA3	56.7	54.6	36.6	27.7	28.0	16.2

View-GRPO consistently and significantly outperforms both SFT and naive GRPO baselines across three backbones. Taking Qwen2.5-VL-7B as an example, View-GRPO increases temporal verification consistency from 33.0% to 45.1% (+12.1%), and temporal grounding consistency from 6.9% to 18.7% (+11.8%). The gain is even more pronounced on InternVL3.5-8B: V-ExoEgo rises from 50.7% to 62.4% (+11.7%), and G-ExoEgo rises from 3.0% to 10.6% (+7.6%). The most remarkable gains are consistently reflected in the cross-view consistency metrics, demonstrating the effectiveness of the View-GRPO design.

Method	V-Exo	V-Ego	V-ExoEgo	G-Exo	G-Ego	G-ExoEgo
Qwen2.5-VL-7B (zero-shot)	54.3	56.3	33.0	14.2	11.4	6.9
+ SFT	57.6	58.0	41.4	18.3	17.8	14.9
+ GRPO	55.2	57.6	39.8	18.6	16.1	14.3
+ View-GRPO	58.6	58.2	45.1	22.0	21.6	18.7
InternVL3.5-8B (zero-shot)	64.4	64.7	50.7	12.8	6.7	3.0
+ View-GRPO	73.1	74.4	62.4	20.5	16.8	10.6

View-GRPO also exhibits strong generalization capability on general benchmarks like Video-MME and TVGBench: Video-MME rises from 61.1% to 69.7%, and TVGBench [email protected] improves from 19.5% to 25.0%, suggesting that reinforced temporal reasoning capabilities can transfer back to standard benchmarks.

Ablation Study¶

Both sub-components of the reasoning reward contribute significantly. Integrating only the semantic reward \(r_{\text{sem}}\) improves V-ExoEgo from 39.8% to 44.7%, and G-ExoEgo from 14.3% to 18.3%. Adding the structural consistency reward \(r_{\text{struct}}\) further climbs to 45.1% and 18.7%. This indicates that while semantic alignment establishes the baseline reasoning quality, structural consistency provides the crowning touch for cross-view alignment.

Configuration	V-Exo	V-Ego	V-ExoEgo	G-Exo	G-Ego	G-ExoEgo
GRPO (w/o reasoning reward)	55.2	57.6	39.8	18.6	16.1	14.3
+ \(r_{\text{sem}}\)	58.3	58.1	44.7	21.5	21.0	18.3
+ \(r_{\text{sem}}\) + \(r_{\text{struct}}\)	58.6	58.2	45.1	22.0	21.6	18.7

Furthermore, the scale of the LLM Judge significantly impacts the quality of reasoning rewards: using Qwen2.5-0.5B as a Judge awards excessively high reasoning rewards from the very first step, leading to a decay of approximately 3% in G-EgoExo. This indicates that the calibration and reliability of the Judge are vital for RL optimization. Ablation on output length reveals that a medium length of 256 tokens remains the most stable; 128 tokens yield poor context leading to low accuracy and unstable optimization, while 512 tokens, despite scoring high reasoning rewards, suffer an accuracy drop (suggesting the Judge may favor verbose style over factual correctness).

Key Findings¶

Reasoning Reward is Key to View-GRPO: Naive GRPO without reasoning rewards performs even worse than SFT (V-ExoEgo 39.8% vs 41.4%), while adding the reasoning reward facilitates a significant comeback. This proves that RL signals optimizing solely for correct answers are insufficient to resolve cross-view consistency, and the reasoning process must be explicitly guided.
Naive Multi-View SFT is Unreliable: On CharadesEgo, TimeSuite concurrently trained on both perspectives exhibits an 8.1% drop in consistency compared to training purely on the exo-view. This demonstrates that without explicit alignment constraints, mixing views potentially introduces conflicting priors, destroying temporal signals rather than reinforcing consistency.
Temporal Reasoning Capability Outweighs Frame Count: Video-LLaMA2 (8 frames) consistently outperforms VideoChat2 (16 frames), and TimeChat-VT (96 frames) surpasses multiple models utilizing higher or lower frame counts, showing that reasoning and temporal modeling ability are more pivotal than merely expanding dense input frame rates.
Unfreezing the Vision Encoder Brings No Benefits: Unfreezing the vision encoder during SFT yields no improvement and often causes performance degradation, likely because the current data scale is insufficient to support end-to-end fine-tuning of large visual backbones. This further accentuates the fundamental challenge of view-invariant understanding.

Highlights & Insights¶

"Semantic + Structural" Dual-Driven Reasoning Reward Design: Instead of forcing the two views to yield identical outputs, the mechanism implicitly induces consistency by aligning key concepts (actions and objects) within the reasoning process. This is more flexible than directly constraining raw outputs, encouraging models to assimilate view-invariant temporal abstractions rather than memorizing superficial patterns.
Misaligned Query as Automated Negative Sample Generation: Using GPT-4o to automatically generate semantically relevant but temporally mismatched negative samples for each refined query effectively balances the "yes/no" target distribution for the temporal verification task without incurring human labeling costs. This design paradigm can be easily ported to other VQA benchmark constructions requiring negative samples.
Finding on LLM Judge Scale Affecting RL Stability: Insufficiently sized Judge models yield poorly calibrated, overly generous reasoning rewards, which directly degrades performance consistency. This finding serves as a cautionary tale for any training pipeline utilizing LLM-as-a-Judge for RL, emphasizing the need to carefully select judge scale and monitor reward distributions.
Quantitative Revelation of Cross-View Consistency Gap: All open-source models obtain consistency scores barely reaching half of their single-view counterparts (e.g., V-ExoEgo 23-42% vs. V-Exo 46-62%). This surprisingly wide gap suggests that current Video-LLMs' "strong performance" heavily relies on view-specific appearance shortcut cues rather than genuine temporal reasoning—a reality the research community must confront.

Limitations & Future Work¶

Limited Data Scale: View30K comprises only 3.3k videos. Although augmented with GPT-5 reasoning chains, its capacity falls heavily short of what large-scale pre-training requires. The authors note that collecting physical large-scale ego-exo paired data is expensive and impractical; hence, exploring synthetic ego-view generation from exo-videos (e.g., via cropping/re-projecting hand regions) stands out as a viable augmentation alternative.
Architectural Constraints: Existing Video-LLM training pipelines process individual videos independently, leaving the model incapable of simultaneously perceiving and contrasting both views. Architectures natively supporting multi-video joint processing or explicit cross-view alignment layers could further enhance consistency.
LLM Judge Biases in Reasoning Reward: Judges might favor verbosity over factual accuracy (as seen in the lower accuracy of 512-token generations) and struggle with calibration at smaller scales. Future research should explore more robust reward formulations, such as structured evaluations based on ground-truth temporal footprints.
Temporal Grounding in Long Videos Remains Highly Challenging: Grounding accuracy on the EgoExo-4D and LEMMA subsets lags severely behind CharadesEgo, highlighting that current models heavily struggle to localize brief events in long video contexts—a massive bottleneck for practical validation.
Limited Temporal Task Coverage: Temporal verification and grounding serve as vital probe tasks, yet cross-view understanding should naturally encompasses richer dimensions like action recognition and procedural reasoning. The EgoExo-Con framework can be expanded to these frontiers.

vs. EgoExoBench (He et al.): EgoExoBench is a contemporaneous work that also explores cross-view temporal reasoning of Video-LLMs, but focuses primarily on multiple-choice action sorting. It neither explicitly handles prediction consistency nor proposes direct model improvements. EgoExo-Con is far more comprehensive in its task design (verification + grounding vs. sorting), consistency metrics, and algorithmic optimization.
vs. Traditional Ego-Exo Representation Learning (CharadesEgo, EgoExo-4D): These works primarily utilize contrastive learning or cross-view distillation to learn view-invariant representations, yet they seldom evaluate the temporal reasoning consistency of Video-LLMs. This paper advances view-invariance from the representation layer to the reasoning layer, exposing a fundamental concern: even with aligned representations, the reasoning pipeline itself may still latch onto view-specific shortcut patterns.
vs. Video-LLM Temporal Reasoning Works (TimeChat, VTimeLLM, TimeSuite): These works push the frontiers of fine-grained temporal comprehension in Video-LLMs, yet they are all designed under single-view assumptions. The finding here—that over half of these models' observed "strong capabilities" stem from view-specific biases—offers crucial methodological insights for the field: cross-view consistency should be incorporated as a standard benchmark dimension moving forward.
vs. GRPO / RL for Video Reasoning (Video-R1, Time-R1): These contributions utilize RL to elevate video reasoning, but sidestep cross-view environments. The dual-driven (semantic + structural) reasoning reward appended to GRPO in View-GRPO models a highly generic design pattern that could extend to other RL pipelines demanding multi-modal consistency (such as multilingualism or multi-modal alignment).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ It systematically defines and evaluates the cross-view temporal consistency of Video-LLMs for the first time. EgoExo-Con fills a crucial benchmark gap, and the dual-driven reasoning reward in View-GRPO is highly original.
Experimental Thoroughness: ⭐⭐⭐⭐ The work covers 8 open-source models + 2 closed-source models + a human baseline, testing View-GRPO across 3 backbone architectures. Ablation, Judge analysis, and reasoning length curves are fully complete, though it lacks verification on ultra-large-scale models (e.g., >70B).
Writing Quality: ⭐⭐⭐⭐⭐ The problem definition is clear, the motivation is fully-linked, and the logical progression from empirical analysis to the proposed methodology is coherent. The appendix is thorough.
Value: ⭐⭐⭐⭐⭐ It reveals the underlying viewpoint bias driving current Video-LLM success, offering a fresh evaluation paradigm and optimization path for the community. Both the EgoExo-Con benchmark and the View-GRPO methodology carry significant independent value.