From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation¶

Conference: ECCV2026
Paper: ECCV 2026 Poster
Code: https://github.com/insait-institute/Syn2Seq
Area: Video Generation
Keywords: Egocentric video generation / Cross-view generation / Frame interpolation / Diffusion forcing / Sequence modeling

TL;DR¶

This paper reformulates exocentric-to-egocentric (Exo2Ego) video generation from a condition-output task into the generation of a single continuous sequence: a frozen frame interpolator synthesizes a transition video between the last exo frame and the first ego frame, camera poses are bridged by Slerp plus linear interpolation, and the exo, transition, and ego segments are concatenated into \(S=(X,I,G)\) for a Diffusion Forcing Transformer to model with per-frame independent noise, surpassing TrajectoryCrafter, Wan-FCtrl, Wan VACE, and Exo2EgoSyn in PSNR, SSIM, and LPIPS on the Health, Bike, and Cooking categories of Ego-Exo4D.

Background & Motivation¶

Exocentric-to-egocentric (Exo2Ego) video generation synthesizes a first-person observation of a scene from a third-person video, which matters for robotics and AR/VR, where a system must reconstruct an interactive first-person experience from a viewpoint that cannot see its own hands. Supervision is not the bottleneck: datasets such as Ego-Exo4D and Ego-ExoLearn provide temporally synchronized exo and ego videos together with per-frame camera poses, so prior work has largely treated the problem as conditional generation and differed only in how conditioning is done. PMYS uses a two-stage pipeline that predicts hand trajectories before generating egocentric video; Exo2Ego-V leverages four exocentric views with a PixelNeRF-style 3D prior; EgoExo-Gen conditions on action text plus the first ground-truth ego frame; and Exo2EgoSyn guides a large pretrained generator with a single predicted ego frame under per-frame camera control (see Tab. 1 of the paper for a structured comparison).

This paper argues, however, that synchronization gives paired supervision and, at the same time, creates two discontinuities that make Exo2Ego fundamentally different from ordinary video generation. Visually, the viewpoint gap between the end of the source video, \(x_T\), and the beginning of the target video, \(g_1\), is often large, producing an abrupt spatio-temporal jump. Geometrically, the camera poses are just as discontinuous: the configuration of \(p^x_T\) at the end of the source segment can differ sharply from \(p^g_1\) at the start of the target segment, so the pose trajectory itself is broken. Camera-controllable generators such as TrajectoryCrafter and ReCamMaster all assume a camera moving along a continuous trajectory with smoothly evolving content, and therefore break on such jumps. The difficulty is amplified because the exocentric camera is often static (\(p^x_1=\cdots=p^x_T\)), leaving almost no multi-view geometric cue: the model must infer missing scene structure from highly under-constrained evidence, which is why prior work generally resorts to explicit 3D priors or additional hand-object interaction labels.

The angle taken here is straightforward: since the discontinuity is created by synchronization itself, one should stop solving for it inside the synchronous formulation and instead interpolate, splitting one large jump into several smooth steps, treating interpolation as a lightweight proxy for depth reasoning — rather than estimating geometry explicitly, the model is guided toward the target viewpoint through intermediate frames, i.e. "if the depth is unknown, reach the target via interpolation." Core idea: reframe Exo2Ego from condition-output modeling into sequential signal modeling, where an interpolated transition segment stitches exo and ego into one continuous signal that a diffusion-forcing sequence model with flexible history conditioning generates end to end. The reformulation improves performance and, because the formulation carries no notion of direction, the same model also supports the reverse Ego2Exo setting.

Method¶

Overall Architecture¶

Syn2Seq-Forcing addresses the following problem: given an exocentric video sequence \(X=\{x_1,\dots,x_T\}\), per-frame exocentric poses \(P_x\), and per-frame egocentric poses \(P_g\), synthesize an egocentric video \(G=\{g_1,\dots,g_T\}\) that stays temporally aligned with X. Instead of conditioning on X and diffusing G, the paper treats the task as the generation of one continuous signal. A frozen frame interpolator synthesizes a transition segment \(I\) between the last exocentric frame \(x_T\) and the first egocentric frame \(g_1\); the exocentric, transition, and egocentric segments are then concatenated into a single temporal stream \(S=(X,I,G)\), while the pose stream is completed into \(P=(P_x,P_i,P_g)\), where \(P_i\) is an interpolated transition pose trajectory.

Modeling is delegated to a Diffusion Forcing Transformer (DFoT), which applies independent noise levels per frame: heavily noised frames act as masked context whereas lightly noised frames retain usable history, so the model accepts histories of arbitrary length and structure. At sampling time, CFG-style history guidance trades off fidelity, temporal consistency, and diversity (the paper builds on the HG-v / HG-t / HG-f / HG-tf variants of history guidance). Training is an offline data-construction plus online random-sampling pipeline: a frozen WFLF expands each pair \((X,G)\) into a cached triplet \((X,I,G)\), and each training step samples one adjacent transition from this three-segment sequence. At inference, the model is conditioned on clean exocentric frames and appends a noise-only suffix of length \(2T\), so a single denoising pass emits both the transition segment \(\hat I\) and the egocentric segment \(\hat G\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["input: exo video X<br/>poses P_x and P_g"] --> B["Unified sequence signal<br/>S = (X, I, G)"]
    B --> C["Video interpolation: frozen WFLF<br/>x_T to g_1 yields segment I"]
    B --> D["Interpolated pose trajectory<br/>Slerp plus linear, Plucker embedding"]
    C --> E["Two-stage training<br/>random sub-task decomposition"]
    D --> E
    E --> F["DFoT per-frame independent noise<br/>one denoising pass over 2T frames"]
    F --> G["output: transition I and ego segment G"]

Key Designs¶

1. Unified sequence signal: recasting Exo2Ego from condition-output into one continuous sequence

The condition-output paradigm treats X as the condition and G as the output, so the model sees "one video that suddenly jumps to another viewpoint," which directly conflicts with the smooth-motion assumption baked into video diffusion models and produces artifacts exactly at the jump. This paper instead builds a unified temporal stream \(S=(X,I,G)\) with a matching pose stream \(P=(P_x,P_i,P_g)\), letting the source, transition, and target segments condition one another inside one sequence — past frames condition future frames. Choosing DFoT as the backbone is not incidental: its per-frame independent noise levels act as noising-as-masking, so history can have arbitrary length and structure; defining the unconditional branch as fully noising (masking) the history and the conditional branch as preserving selected history frames yields a classifier-free guidance acting over the history dimension, which tunes how much the result leans toward fidelity versus consistency. Two consequences follow directly. First, the transition segment and the target egocentric segment are synthesized jointly in a single denoising process and are therefore temporally coherent by construction. Second, the formulation carries no directional semantics, so swapping the roles of \((X,P_x)\) and \((G,P_g)\) yields Ego2Exo without any architectural change.

2. Video interpolation: a frozen WFLF supplies pseudo-ground-truth transitions

Real data contain no transition video that walks from the exocentric viewpoint to the egocentric one, so the model has no intermediate state to learn from — and that gap is precisely the source of the jump. The paper delegates this to a dedicated frame interpolator: taking \(x_T\) as the start and \(g_1\) as the end, WFLF (WAN2.2 First/Last Frame) generates the interior transition frames, and the two real boundary frames are explicitly attached back, yielding a transition segment of the same length as X and G:

\[I'=\mathrm{WFLF}(x_T,\ g_1,\ \text{prompt}),\quad |I'|=T-2,\qquad I=[x_T,\ I',\ g_1],\quad |I|=T\]

The prompt is a dataset-specific text prompt that steers the interpolation. WFLF itself is built on the 8-step LoRA of Wan2.2-Lightning and used in the FLF2V setting (conditioning on first and last frames), which makes it a cheap interpolator: no CFG is required, very few steps suffice, and global structure and temporal consistency are preserved. Crucially, WFLF stays frozen throughout training; its output only serves as pseudo ground truth, so no real transition video is needed. The benefit is twofold: one intractable jump is outsourced to a model that is good at local interpolation, and egocentric generation additionally receives visual cues pointing toward the target viewpoint. The paper's ablation shows that interpolating video frames alone, with no pose interpolation at all, already brings a large gain (PSNR on Health rises from 13.54 to 16.10), confirming that the dominant difficulty comes from the spatio-temporal discontinuity itself rather than from the quality of the conditioning signal.

3. Interpolated pose trajectory and Plücker embedding: making the camera geometry continuous too

Connecting the visuals does not connect the geometry: concatenating poses with the true \(p^x_T\) and \(p^g_1\) leaves a teleport at the seam, and a model conditioned on per-frame poses is especially sensitive to it. Each pose is decomposed into intrinsics \(k\) and a rigid transform \([R\,|\,\mathbf t]\), and the two endpoints are interpolated over normalized time \(\tau_j=(j-1)/(T-1)\):

\[R^i_j=\mathrm{Slerp}\!\left(R^x_T,\ R^g_1;\ \tau_j\right),\qquad \mathbf t^i_j=(1-\tau_j)\,\mathbf t^x_T+\tau_j\,\mathbf t^g_1,\qquad p^i_j=\big(k^i_j,\ [R^i_j\,|\,\mathbf t^i_j]\big)\]

Rotation uses Slerp along the shortest geodesic on \(SO(3)\) rather than element-wise linear interpolation, translation is interpolated linearly, intrinsics are fixed to the exocentric ones throughout, and boundary consistency \(p^i_1=p^x_T\), \(p^i_T=p^g_1\) is enforced, so \(P=(P_x,P_i,P_g)\) becomes a genuinely continuous pose trajectory in the geometric sense. How poses enter the network is also ablated: a 16-dimensional global pose vector per frame, a 180-dimensional per-pixel ray encoding, and a 6-dimensional per-pixel Plücker embedding; Plücker wins (PSNR 15.69 for the transition segment on Bike, 0.61 above the global pose and 0.09 above ray encoding). The plausible explanation is that a per-pixel ray representation explicitly encodes "which direction this pixel looks along," which fits a framework that approaches the target viewpoint pixel by pixel through interpolated frames, whereas a 16-dimensional global vector discards the spatial layout and only describes the camera itself.

4. Two-stage training with random sub-task decomposition: one model learns both transitions

The unified sequence creates a new problem: it hides two transitions of different nature — Exo→Interp and Interp→Ego — and learning the whole sequence end to end forces the model to absorb cross-view alignment and interpolation-trajectory continuation at once. The paper therefore adopts a pretraining-then-finetuning strategy and, during finetuning, decomposes the sequence into two sub-tasks sampled at random at each step:

\[D=\big((X,I),\ (P_x,P_i)\big)\quad\text{or}\quad D=\big((I,G),\ (P_i,P_g)\big)\]

Pretraining runs on 356k direct Exo→Ego clips (no interpolation) to obtain a strong initialization; finetuning uses 40k videos per category with interpolation, teaching both transitions inside one framework. This stochastic decomposition meshes with DFoT's history conditioning: what the model learns is not a mapping from A to B but how to continue an arbitrary piece of history, so at inference it suffices to feed clean exocentric frames and append \(2T\) noise slots — the first \(T\) are denoised into the transition segment \(\hat I\) and the remaining \(T\) into the egocentric segment \(\hat G\), both produced in a single denoising process. The authors note that this property means reversing the segment order yields Ego→Exo; in this work that direction is only briefly explored.

Loss & Training¶

The training objective is the standard denoising loss of a sequence model: each frame of the unified sequence is noised at a different level to give \(S^n\), and the model predicts the added noise (equivalently the velocity field) conditioned on the pose stream \(P\):

\[\mathcal{L}=\mathbb{E}_{S,\,\epsilon,\,n}\left\|\epsilon-\epsilon_\phi\!\left(S^n,\ P\right)\right\|_2^2 \quad \text{⚠️ this equation is corrupted in the cached text; it is stated here in the standard diffusion-forcing form, refer to the original paper for the exact symbols}\]

Both stages share the same objective. Pretraining runs 20 epochs on 356k direct Exo→Ego clips without interpolation (about 4 days on 8 NVIDIA H200 GPUs); finetuning runs 150 epochs on 40k videos per category (again about 4 days on 8 H200s), producing one model per category. For preprocessing, because WFLF requires the frame count to satisfy \(4n+1\), only 9 frames are sampled per video with a skip-frame step of 4 to retain sufficient motion, and all videos are resized to \(256\times256\); the official Ego-Exo4D test splits are used, and only one of the four available exocentric videos is used at random. Implementation details, hyper-parameters, and the Ego2Exo setup are deferred to the supplementary material.

Key Experimental Results¶

Main Results¶

Experiments cover three Ego-Exo4D categories — Bike (363 videos), Cooking (678), and Health (397) — with PSNR, SSIM, and LPIPS (AlexNet features), following the evaluation protocol of Exo2Ego-V. For fairness, metrics are computed on the generated egocentric segment only, excluding the transition segment; otherwise baselines without transitional capability would not be comparable. Numbers are taken from Tab. 2 of the paper, rounded to two decimals.

Method	Health (PSNR↑/SSIM↑/LPIPS↓)	Bike (PSNR↑/SSIM↑/LPIPS↓)	Cooking (PSNR↑/SSIM↑/LPIPS↓)
TrajectoryCrafter [51]	14.31 / 0.418 / 0.562	14.01 / 0.341 / 0.600	13.76 / 0.373 / 0.600
Wan-FCtrl (Wan Fun Control) [1]	13.90 / 0.432 / 0.573	13.57 / 0.331 / 0.620	13.42 / 0.360 / 0.607
Wan VACE [19]	14.19 / 0.430 / 0.597	13.21 / 0.333 / 0.616	13.37 / 0.369 / 0.613
Exo2EgoSyn [31]	15.62 / 0.482 / 0.499	15.13 / 0.390 / 0.505	13.90 / 0.404 / 0.613
Ours (Syn2Seq-Forcing)	16.71 / 0.573 / 0.483	15.63 / 0.472 / 0.501	14.39 / 0.453 / 0.585

The method leads on all three metrics in all three categories. Against the strongest prior method, Exo2EgoSyn, it gains 1.09 PSNR, 0.09 SSIM, and 0.016 LPIPS on Health, and 0.50 PSNR with 0.08 SSIM on Bike. Notably, the three purely camera-controllable baselines (TrajectoryCrafter, Wan-FCtrl, Wan VACE) all cluster in the 13.2–14.3 PSNR range, indicating that transforming the viewpoint of an existing video is not the same capability as synthesizing a new view along a continuous trajectory. Exo2Ego-V requires four exocentric views and EgoExo-Gen has no public implementation, so neither is included in the comparison.

Ablation Study¶

Effect of interpolation (Tab. 3 of the paper). Exo2Ego-Direct predicts the ego view directly from the exo view without interpolation; Exo2Ego-FI interpolates frames only, using the egocentric poses and filling the remaining poses with zero vectors; Exo2Ego-FPI interpolates both frames and poses and is the full model.

Config	Health (PSNR/SSIM/LPIPS)	Bike (PSNR/SSIM/LPIPS)	Cooking (PSNR/SSIM/LPIPS)
Exo2Ego-Direct	13.54 / 0.418 / 0.582	14.01 / 0.355 / 0.587	13.16 / 0.383 / 0.618
Exo2Ego-FI (frames only)	16.10 / 0.543 / 0.490	15.12 / 0.441 / 0.539	14.21 / 0.440 / 0.590
Exo2Ego-FPI (frames + poses, full)	16.71 / 0.573 / 0.483	15.63 / 0.472 / 0.501	14.39 / 0.453 / 0.585

Which interpolator, and how poses are embedded (Tab. 4 / Tab. 5 of the paper, both on Bike; INT denotes the first T frames, i.e. the interpolated segment, and EGO the next T frames).

Config	Bike-INT (PSNR/SSIM/LPIPS)	Bike-EGO (PSNR/SSIM/LPIPS)
WFLF pseudo ground truth (ours)	15.69 / 0.475 / 0.502	15.63 / 0.472 / 0.501
DFoT native inference-time interpolation	13.11 / 0.320 / 0.633	13.90 / 0.334 / 0.619
Pose embedding Global (16-dim per frame)	15.08 / 0.459 / 0.512	15.02 / 0.457 / 0.518
Pose embedding Ray Encoding (180-dim per pixel)	15.60 / 0.474 / 0.510	15.57 / 0.473 / 0.505
Pose embedding Plücker (6-dim per pixel, ours)	15.69 / 0.475 / 0.502	15.63 / 0.472 / 0.501

Key Findings¶

Interpolation itself accounts for most of the gain. On Health, moving from Direct to FI jumps PSNR by 2.56 (13.54→16.10), while FI to FPI adds only 0.61 more. This directly supports the paper's central claim: the main bottleneck of Exo2Ego is the synchronization-induced spatio-temporal discontinuity, not a weak conditioning signal.
Pose interpolation mainly buys structural and perceptual consistency. Adding it raises SSIM from 0.543 to 0.573 and lowers LPIPS from 0.490 to 0.483, a clearer relative improvement than PSNR, suggesting that once the geometric trajectory is continuous the frames look structurally "smoother" rather than merely being more accurate per pixel.
Transition quality propagates to the final egocentric segment. Replacing WFLF with DFoT's own inference-time interpolation drops the transition segment by 2.58 PSNR (15.69→13.11) and drags the ego segment down to 13.90 (15.63 for the full model). Which component produces the intermediate state is therefore not a mere engineering detail.
Pose representation has a clear ordering. Plücker (6-dim per pixel) > Ray Encoding (180-dim per pixel) > Global (16-dim per frame). Global lags by 0.61 PSNR on the transition segment, showing that compressing the camera state into a spatially agnostic vector is costly when the task requires inferring scene structure pixel by pixel.
Category difficulty is consistent: Cooking is hardest for every method (Ours 14.39), Health is easiest (16.71), and Bike sits in between; the relative ordering of methods is unchanged across categories, so the conclusion does not hinge on one category.
Evaluation caveat: all metrics are computed on the egocentric segment only, with the transition segment excluded so that baselines lacking transitional capability remain comparable. The numbers in the tables therefore cannot be read as the quality of the full sequence including the transition.

Highlights & Insights¶

Re-identifying synchronization as the source of the problem rather than a convenience is the most valuable step in the paper. Everyone uses synchronized data, but few point out that synchronization itself creates the jump; once seen that way, the fix (interpolating between the two endpoints) is almost self-evident.
Interpolation as a proxy for depth reasoning is a neat trade: no explicit geometry estimation, no 3D prior, no hand-object interaction labels, and a single exocentric view, yet the model is walked to the target viewpoint through intermediate frames, degrading one under-constrained cross-view generation problem into a chain of smooth sub-problems.
Pseudo ground truth requires no real transition video: a frozen off-the-shelf interpolator supplies the supervision and is discarded after training, which keeps the engineering cost low.
One formulation covers both directions: since the unified sequence carries no directional semantics, Ego→Exo only requires swapping the roles of the segments, with no architectural change and no second model.
Transferable idea: for any conditional generation task connecting two discontinuous domains (cross-sensor, cross-modality, cross-view, or even temporal jumps in video editing), one can insert an intermediate state produced by an existing model to split a large jump into small steps; using it as pseudo ground truth rather than as an inference-time component avoids inheriting the latency of the auxiliary model.

Limitations & Future Work¶

The scale is small and conclusions extrapolate cautiously: only 9 frames per video, \(256\times256\) resolution, a single dataset (Ego-Exo4D) and three categories, so long-sequence capability is effectively untested; the authors also admit Ego2Exo is only briefly explored, with no quantitative results in the main text.
The performance ceiling is locked by the interpolator: transition frames are WFLF pseudo ground truth, and their error propagates through the sequence conditioning into the ego segment (the 1.7 PSNR drop in Tab. 4 when swapping the interpolator is direct evidence), while WFLF's quality itself depends on dataset-specific prompts. How stronger or weaker interpolators change the picture is not studied systematically.
The transition length is hard-wired to T, equal to the exo and ego segments; it is neither adaptive nor is the obviously relevant question of "how many interpolated frames are enough" discussed.
The metric suite is narrow: only per-frame fidelity metrics such as PSNR/SSIM/LPIPS, with no distribution-level or temporal-consistency metric such as FVD and no user study — yet high per-frame scores do not imply temporal stability, which is precisely what the paper claims to improve.
Cost is not low: each category needs its own finetuning run of 150 epochs on 40k videos, about 4 days on 8 H200 GPUs, and cross-category or cross-dataset generalization is not reported at all.
Possible improvements: make the number or length of transition frames adapt to the magnitude of the pose jump; weight the losses of the transition and ego segments separately to damp the propagation of pseudo-ground-truth error; replace WFLF with a stronger interpolation or generation model and refresh the pseudo ground truth periodically in a self-training manner; add FVD and long-horizon (tens to hundreds of frames) experiments to substantiate the sequential-modeling claim.

vs Exo2EgoSyn [31]: both target Exo2Ego, but from opposite directions. Exo2EgoSyn repurposes a large pretrained video generator, guiding generation with a predicted egocentric frame and applying per-frame camera control (which does not match the model's temporally coupled attention); this paper trains a DFoT better suited to per-frame pose conditioning, consumes the full exocentric video, and is the only method in the paper's Tab. 1 that can generate exo-to-ego transitions. The cost is that it still requires poses for both cameras, whereas Exo2EgoSyn used four exocentric views.
vs Exo2Ego-V [26]: they rely on four exocentric views plus a PixelNeRF-style 3D prior to compensate for geometry; this paper uses a single exocentric view and no explicit 3D representation, substituting interpolation for geometric reasoning — fewer inputs, but it also gives up the genuine geometric constraints of multiple views, trading accuracy for smoothness when the exocentric camera is static.
vs EgoExo-Gen (X-Gen) [48]: that method needs action text plus the ground-truth first egocentric frame and has no public implementation, so it could not be compared directly (the paper states this explicitly); readers should not read the reported gains as a win over EgoExo-Gen.
vs camera-controllable generators (TrajectoryCrafter [51], Wan-FCtrl [1], Wan VACE [19], ReCamMaster [2], etc.): they follow a continuous camera trajectory during generation and excel at novel-trajectory rendering, but do not address transforming the viewpoint of an existing video; their PSNR being more than 2 points lower in Tab. 2 reflects a task difference rather than a pure capability difference.
vs DFoT / History Guidance [41]: the mechanism of DFoT is left untouched; instead, its arbitrary-history conditioning is used to carry a sequence lengthened by interpolation, showing that the backbone applies more broadly than the task demonstrated in its original paper.

Rating¶

Novelty: ⭐⭐⭐⭐ Identifying synchronization as the root cause and reframing condition-output as sequential modeling is a clear, self-consistent angle; however, "bridging with frame interpolation" is a simple operation, so the contribution is more conceptual than technical.
Experimental Thoroughness: ⭐⭐⭐ Three categories, four comparisons, and two ablations that target the central claim; but only 9 frames at 256 resolution, a single dataset, no FVD or user study, and an Ego2Exo direction asserted without numbers in the main text.
Writing Quality: ⭐⭐⭐⭐ The motivation chain is clear, Fig. 1 makes the "jump causes failure" story intuitive, and Tab. 1 is informative; the downside is that implementation details and Ego2Exo results are deferred to the supplementary material and some equations are corrupted in the cached text.
Value: ⭐⭐⭐⭐ A general, reusable reformulation (interpolation bridge + sequence modeling + pseudo ground truth) that is inspiring for cross-view video generation; practical adoption is still limited by per-category finetuning cost and the 256-resolution scale.