Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

Conference: ECCV2026
Authors: Dominick Reilly, Manish Kumar Govind, Le Xue, Srijan Das
Paper: https://eccv.ecva.net/virtual/2026/poster/3825
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/3277.pdf
Code: https://github.com/dominickrei/EgoExo4ADL
Area: Model Compression / Video Understanding
Keywords: Knowledge distillation, privileged supervision, cross-view transfer, adaptive visual tokens, hand-object interaction

The official page uses the title above; the PDF additionally prefixes it with “From My View to Yours.” The code link comes from the paper, which promises to release code, models, and data. This does not establish that all resources are already available.

TL;DR

Ego2ExoVLM teaches an exocentric-only student to reproduce an egocentric teacher's answers and uses adaptive visual tokens to extract interaction cues that would otherwise be overwhelmed by the background. It improves VideoLLaMA3's average accuracy on Ego-in-Exo Perception from 71.2% to 74.2%, without requiring an egocentric camera at test time.

Background & Motivation

Fixed cameras are convenient for recording everyday activities, but they may not clearly capture the details that determine an action's meaning: which object someone picks up, which hand they use, or what they are manipulating. Exocentric footage often emphasizes room layouts and whole-body motion, while hands and objects occupy only small regions. An egocentric camera naturally sits closer to the interaction. A general-purpose vision-language model seeing both views during training does not automatically learn to locate evidence relevant to egocentric tasks in an exocentric input. Training questions and answers generated from exocentric footage alone may further reinforce scene descriptions rather than subtle interaction details.

Adding egocentric video at test time could help, but asking people to wear cameras continuously is impractical for everyday activity monitoring. The paper therefore adopts privileged supervision: temporally synchronized egocentric and exocentric videos are available during training, whereas deployment retains only exocentric input. The goal is neither to transfer the other view's pixels nor to shrink the student into a smaller network. It is to transfer semantics that the teacher obtains from a clearer observation. Because the two views look substantially different, forcing their visual features to match may not effectively teach a generative model to answer questions.

Core idea: use language answers as the bridge for cross-view knowledge distillation, while giving the student a dedicated visual pathway for fine-grained interaction evidence. This grounds egocentric supervision in cues that remain visible, but inconspicuous, in exocentric footage.

Method

Overall Architecture

Each training sample contains synchronized egocentric and exocentric videos and the same natural-language question. A frozen teacher watches the egocentric video and generates an answer. The student sees only the exocentric video, supplements its original visual representation with Ego Adaptive Visual Tokens, and learns the teacher's answer through Ego2Exo Sequence Distillation. Inference removes the teacher and distillation supervision: only the exocentric student runs, producing action-related answers or video descriptions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Exocentric video and question"] --> B["Frozen visual encoder"]
    B --> C["Ego Adaptive<br/>Visual Tokens"]
    B -->|Original visual representation| D["Student language model"]
    C -->|Dedicated connector| D
    E["Synchronized egocentric video<br/>and frozen teacher"] --> F["Ego2Exo Sequence Distillation"]
    D -->|Training predictions| F
    D -->|Inference| G["Exocentric-only answer"]

Both teacher and student are initialized from VideoLLaMA3, consisting of a SigLIP visual encoder, a vision-language connector, and a Qwen2.5 language model. The teacher is not a larger specialized network; its advantage comes from its training-time viewpoint. Nor does the student generate egocentric video. The original visual pathway preserves scene and action context, while the new token pathway supplies local interaction information. Both feed the language model rather than replacing the full scene with local evidence.

Key Designs

1. Ego Adaptive Visual Tokens: a dedicated pathway for reading inconspicuous interaction regions

Simply providing teacher answers makes the supervision more detailed, but the student's visual representation may still primarily encode the background and coarse body shape. The paper introduces learnable visual probes called Ego Adaptive Visual Tokens. Acting as queries, these tokens use cross-attention to read spatiotemporal features from intermediate visual-encoder layers, accessing both lower-level detail and higher-level semantics. They are not pre-cropped hand regions and require no additional object boxes or hand-location annotations.

Details readily identifiable in one view may leave only a few useful patches in the other. Dedicated queries aggregate those cues into additional tokens without requiring the main visual encoder to fundamentally change its representations. Following the paper's cross-attention expression, the computation at one layer can be written as:

\[ \widetilde E^{\ell}=\operatorname{softmax}\!\left(\frac{(E^{\ell}W_q^{\ell})(X^{\ell}W_k^{\ell})^{\top}}{\sqrt{d_v}}\right)(X^{\ell}W_v^{\ell}). \]

Here, \(E^{\ell}\) denotes the learnable probes, \(X^{\ell}\) the video features at that layer, \(W_q^{\ell},W_k^{\ell},W_v^{\ell}\) the layer-specific projections, and \(d_v\) the visual feature dimension. Attention reads from the collection of spatiotemporal features. Its output is eventually mapped into the language-model space through an independent connector and concatenated with the original exocentric visual tokens. The independent connector allows the new pathway to adapt to interaction details without sharing the original connector's projection.

“Ego Adaptive” describes the role these tokens acquire through supervision, not an egocentric image input at test time. During inference, they still read only exocentric video. Attention visualizations show a stronger focus on interaction regions, but this is qualitative evidence consistent with the mechanism, not proof that the model has acquired a reliable hand-object detector.

2. Ego2Exo Sequence Distillation: align answer semantics rather than force cross-view appearance alignment

The frozen teacher generates a complete answer conditioned on the egocentric video and question, supplying a pseudo-target for the exocentric student. Training fits this answer autoregressively, token by token. Each prediction is conditioned on the preceding teacher-answer tokens, the question, the original exocentric visual representation, and the adaptive tokens. The intended lesson is “this exocentric observation should support these interaction semantics,” not “both cameras must produce identical visual vectors.” This makes the design particularly suited to language-generating VLMs rather than directly importing mean-squared-error distillation for visual encoders.

The following restates the paper's Eq. (3) in standard conditional-probability notation because text extraction damages its typesetting:

\[ \mathcal L_{\mathrm{seq}}=-\sum_{m=1}^{M}\log P_S\!\left(a_m^{\mathrm{ego}}\mid A_{<m}^{\mathrm{ego}},Q,Z^{\mathrm{exo}},\phi_{\mathrm{ego}}(\widetilde E)\right). \]

Here, \(M\) is the number of tokens in the teacher's answer; \(a_m^{\mathrm{ego}}\) and \(A_{<m}^{\mathrm{ego}}\) denote the current target token and target prefix. \(Q\) is the question, \(Z^{\mathrm{exo}}\) the student's original visual embeddings, \(\phi_{\mathrm{ego}}(\widetilde E)\) the new token pathway's output, and \(P_S\) the student's conditional distribution. This is negative log likelihood over generated answers, not KL distillation that matches the teacher's full-vocabulary logits. The KL divergence reported later in the paper is an analysis metric, not an additional training term here.

The two designs interact in a concrete way: teacher answers identify which details matter, while visual probes give the student a route to retrieve the corresponding evidence from the external view. Answer supervision can help discover information present in the image, but cannot guarantee factual recovery of completely occluded objects. Temporal synchronization ensures that the videos depict the same activity instance, reducing mismatches between the teacher's description and the student's scene. The paper also explores semantic-retrieval pseudo-pairs, without treating them as equivalent to strict synchronization.

A Worked Example

Consider the vegetable-preparation scene in the paper's illustration. During training, two cameras record the same action, and a question asks which vegetable is being prepared. The egocentric teacher clearly observes the interaction target and generates an answer mentioning a tomato and details such as the holding hand. The exocentric student may mainly see a person at a counter, with the object occupying a small area. This is the paper's explanatory illustration, not an additional quantitative test example.

The student's original visual tokens supply the counter, person, and action context. The default 16 adaptive tokens aggregate interaction details from intermediate layers. The training objective then asks the student to reproduce the teacher's answer, updating probe-related projections, connectors, and the trainable parts of the language model. At test time, the same kind of question is paired only with exocentric video. The probes remain active, but neither the teacher answer nor egocentric footage is an input. If an object is entirely invisible, a plausible guess should not be mistaken for reliable recognition.

Loss & Training

The main experiments use approximately 9k ego-exo video samples from the EgoExo4D keystep recognition training split. For each video pair, one question is sampled from each of three predefined categories—description, temporal, and spatial—and answered by the frozen teacher. The paper reports approximately 28k question-answer pairs overall. Both counts are approximate; multiplying them should not be used to infer an exact sample count.

The entire teacher and the student's visual encoder are frozen. Trainable components comprise vision-language connectors, adaptive-token parameters and cross-attention projections, and LoRA modules in the language model. The LoRA rank is 64, with a learning rate of \(1\times10^{-5}\); other trainable parameters use \(2\times10^{-6}\). Training runs for 3 epochs on 4 NVIDIA H200 GPUs. The visual and language embedding dimensions are 1152 and 3584, respectively, and the default number of adaptive tokens is 16.

Cross-view knowledge distillation is the central mechanism; the paper does not report a smaller student or a parameter-compression ratio. Its placement under “Model Compression” follows the project's convention of collecting knowledge-distillation papers in that category, not a claim that model size or inference acceleration is the main experimental objective.

Key Experimental Results

Main Experiments

Ego-in-Exo Perception is built from cooking videos in the EgoExo4D keystep recognition test subset. Initial filtering covers 2,319 videos and approximately 10k atomic action annotations. Questions are generated using only egocentric frames, action annotations, and parsed scene objects. Four participants then verify them, removing erroneous questions, trivial distractors, and questions that cannot be answered from exocentric footage because of occlusion or framing. The final benchmark contains 3,881 multiple-choice questions across action understanding, task-relevant regions, hand-object interaction, and hand identification. All four report accuracy, with higher values better.

The following results come from the paper's Table 2. Avg. retains the reported values. The egocentric teacher receives more favorable input and is only an upper-bound reference, not a competing baseline under identical input conditions.

Method and input	Action ↑	Task Regions ↑	Hand-Object Interaction ↑	Hand ↑	Avg. ↑
Teacher, egocentric input	81.8	83.9	78.9	67.1	77.9
Qwen2.5-VL, exocentric input	68.2	75.9	70.8	64.8	69.9
VideoLLaMA3, exocentric input	66.5	79.6	74.5	64.1	71.2
EE4D-VideoLLaMA3, exocentric input	59.5	66.6	71.6	65.0	65.7
Ego2ExoVLM, exocentric input	73.4	82.0	76.2	65.3	74.2

Average accuracy improves by 3.0 percentage points over the initial VideoLLaMA3 backbone and by 8.5 percentage points over EE4D-VideoLLaMA3, which is fine-tuned only on this dataset's exocentric videos. The student still trails the egocentric teacher by 3.7 percentage points, so the viewpoint gap has not disappeared.

ADL-X adds 4 multiple-choice tasks and 2 video-description tasks. In the paper's Table 4, Ego2ExoVLM and VideoLLaMA3 achieve ADL-MCQ average accuracies of 78.8 and 76.9, respectively. Their Charades description averages are 75.5 and 70.6, and Toyota Smarthome description averages are 88.7 and 87.2. Description tasks use Video-ChatGPT scores for correctness, detail, context, temporal understanding, and consistency; these scores must not be treated as multiple-choice accuracy.

Ablation Studies

The following results come from the paper's Table 5, with all metrics being Ego-in-Exo Perception accuracy. The Ego+Exo row uses both views for training, not simultaneous dual-view input at test time.

Training configuration	Sequence distillation	Adaptive tokens	Action ↑	Task Regions ↑	Hand-Object Interaction ↑	Hand ↑	Avg. ↑
Exo	No	No	59.5	66.6	71.6	65.0	65.7
Ego+Exo	No	No	59.2	68.5	71.1	65.0	66.0
Exo	Yes	No	67.2	67.9	73.8	65.3	68.5
Exo	No	Yes	70.0	79.4	74.3	65.1	72.2
Exo, full model	Yes	Yes	73.4	82.0	76.2	65.3	74.2

Simply adding training data from both views improves the average by only 0.3 percentage points. Removing adaptive tokens from the full model reduces the average by 5.7 percentage points; removing sequence distillation reduces it by 2.0 percentage points. These interventions do not provide independent, additive gains: their improvements over the baseline cannot simply be summed to predict the full model's result.

Key Findings

• Adaptive tokens particularly improve task-relevant regions: adding tokens alone raises that score from 66.6 to 79.4 over the Exo baseline. Hand-object interaction rises from 71.6 to 74.3 in the same comparison. The paper's prose emphasizes interaction, but the largest task-level increase in the table is actually for task-relevant regions.
• Visual-feature distillation averages 67.8 in the paper's Table 7, compared with 74.2 for the full method. This supports language-level transfer in this setting, not a universal rejection of visual distillation.
• With pseudo-paired Ego4D + HowTo100M data, the average rises from 68.1 to 70.4, but hand-object interaction and hand identification fall from 75.1 and 65.3 to 71.7 and 62.8. The appropriate conclusion is that unsynchronized supervision still helps on average, not that every task benefits.

Highlights & Insights

• The teacher's advantage comes from observation conditions rather than model scale. The same architecture can provide more informative answer supervision through a privileged viewpoint, separating “knowing more” from “having more parameters.”
• Language supervision and visual evidence retrieval are complementary. The former specifies the semantics to learn, while the latter supplies a pathway to information in inconspicuous regions. Ablations also show that merely mixing both views is insufficient.
• The benchmark explicitly separates the view used to generate questions from the view available at test time. Human filtering of questions unanswerable from exocentric video respects observability more carefully than simply asking the model to guess egocentric details.

Limitations & Future Work

• Observability remains a hard constraint. The benchmark actively filters questions with severe occlusion or poor framing, so its results do not establish reliable recovery of invisible interactions. Real fixed-camera deployment may be harder.
• The benchmark mainly covers cooking, and teacher-generated pseudo-answers can contain errors. The results do not establish equal effectiveness across all everyday activities, camera layouts, or long-tail objects.
• In the bias analysis, text-only input still averages 55.7, a single center frame reaches 69.8, and video input scores 73.4. These are GPT-4.1-Mini results. They show that video helps, but do not establish that language priors or static shortcuts have been eliminated.
• Training details and module ablations are substantial, but the main text does not provide latency, throughput, or compression ratios sufficient to quantify deployment cost. The actual overhead of extra tokens and the effect of teacher errors on the student require more direct evaluation.

Related Work & Insights

• vs VideoLLaMA3 / EE4D-VideoLLaMA3: The method inherits initialization from the former but adds privileged-view distillation and visual probes. The latter shows that in-domain exocentric fine-tuning alone does not guarantee retaining or improving understanding of egocentric attributes.
• vs Quattrocchi et al.'s view transfer: Earlier work transfers pooled visual features with mean squared error, whereas this method carries knowledge through answer sequences. The difference concerns both the objective and the goal of transferring from egocentric to exocentric views in a generative VLM.
• vs Exo2EgoVLM: This model addresses the reverse transfer direction, from exocentric to egocentric. In the paper's Table 7, its average is 70.3, but hand identification reaches 81.1, above this method's 65.3. Leading on average does not imply leading in every fine-grained capability.
• vs EgoExoBench: Both involve paired views and VLM evaluation. Here, question generation deliberately relies only on egocentric evidence, aiming to avoid questions that merely reflect scene information already salient in exocentric footage.

Rating

• Novelty: 4/5 — A clear combination of privileged-view sequence distillation and visual probes, although the components are not fundamentally new mechanisms.
• Experimental Thoroughness: 4/5 — Covers a new benchmark, ADL-X, module and transfer-strategy ablations, and pseudo-pairs, but lacks deployment-cost measurements and broader activity coverage.
• Writing Quality: 4/5 — Clearly distinguishes task boundaries and training versus inference, although some ablation interpretations require checking against the tables.
• Value: 4/5 — Offers useful guidance for fine-grained activity understanding with fixed cameras, but does not replace real deployment validation.