Infinite-Homography as Robust Conditioning for Camera-Controlled Video Generation

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: camera control, infinite homography, residual parallax, latent-space warping, trajectory augmentation

TL;DR

InfCam supplies depth-independent rotational conditioning through infinite homography, combines frozen Wan2.1 with trainable control layers to learn translation-induced residual parallax, and removes training-pair biases through trajectory and focal-length augmentation, reducing WebVid rotation error from TrajectoryCrafter's 5.007 to 3.162.

Background & Motivation

Repositioning the camera in an existing dynamic video requires preserving subject motion and scene content while changing the camera position, orientation, and field of view. This is not simply cropping or shifting an image: camera movement displaces foreground and background differently and can expose regions never visible in the source. Methods such as GEN3C and TrajectoryCrafter estimate depth, reproject the source into the target camera, and use a generative model to complete missing content. Accurate depth provides strong geometric conditioning, but inaccurate depth first places objects at incorrect image locations. When the external depth network is not trained jointly, these misplaced conditions persist, leaving the generator to infer both which regions are reliable and which require correction.

Another approach learns the relationship between camera trajectories and videos directly, but training-pair construction also determines its behavior. Although MultiCamVideo contains diverse camera motions, cameras within a scene start at the same position, so source and target first frames are always identical during training. The model can learn to copy the input first frame instead of respecting the requested initial pose; a fixed focal length within each scene can likewise encourage it to ignore the target focal length. Having multiple trajectories and focal lengths in a dataset therefore does not mean that its training pairs cover changes in both. Adding a camera encoder alone cannot automatically remove this mismatch between control inputs and supervision.

The paper selects a component of projection geometry that is determined without depth: given intrinsics and relative rotation, infinite homography specifies rotational correspondences. The remaining displacement caused by finite depth and camera translation is handled by the video generation prior and trainable control layers. This choice does not reconstruct a complete 3D scene; it avoids treating unreliable explicit depth as a fixed condition while augmented pairs teach the model to respond to different starting poses and fields of view. Core Idea: first narrow novel-view prediction with reliable analytical rotational conditioning, then learn residual parallax while explicitly breaking the unchanged-first-frame and unchanged-focal-length shortcuts in training.

Method

Overall Architecture

Inputs comprise a source video, source and target intrinsics, and a target trajectory expressed in the coordinate system of the source video's first camera pose; the output shows the same dynamic scene along that trajectory. The target trajectory specifies rotation and translation for each frame, while the model does not require the true extrinsics of every source frame. LLaVA extracts a source-video description for Wan2.1's text conditioning, rather than providing additional geometric supervision. Before training, Trajectory and Intrinsic Augmentation constructs synchronized source-target pairs; inside the network, Infinite Homography Warping is followed by Homography-Guided Attention. The former produces rotation-aligned reference features, and the latter lets each target frame access both contemporaneous source content and the geometric reference. A 3D VAE encodes the source; during training, the target video is also encoded and noised, whereas inference supplies no target ground truth and produces target latents through denoising. The 3D VAE finally decodes the video, and the augmented target video supplies supervision only during training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    DATA["Synchronized multiview videos"] --> AUG["Trajectory and<br/>Intrinsic Augmentation"]
    AUG -->|Training input| INPUT["Source video latents<br/>Intrinsics and target trajectory"]
    INPUT --> WARP["Infinite Homography<br/>Warping"]
    INPUT --> ATTN["Homography-Guided<br/>Attention"]
    WARP --> ATTN
    NOISE["Noisy target latents<br/>Text condition"] --> ATTN
    ATTN --> OUT["Wan2.1 denoising and decoding<br/>Target video"]
    AUG -.->|Training only: target ground truth| SUP["Training supervision"]
    OUT -.-> SUP

Key Designs

1. Trajectory and Intrinsic Augmentation: make training pairs change their starting viewpoint and field of view

MultiCamVideo provides 10 synchronized trajectories per scene, with 81 frames per video and motions including translation, rotation, and arcs. The authors exploit the shared starting point by reversing one video and appending another forward-playing video from the same scene, removing the duplicated first frame at the junction. This produces 161 frames, or 81 + 80; the junction corresponds to the same scene timestamp and camera starting point, rather than arbitrarily joining unrelated actions. They then extract 81-frame windows with matching starting timestamps across augmented sequences from the same scene, preserving dynamic synchronization across viewpoints. A window need not begin at the original shared camera origin, allowing the source and target first frames to show different viewpoints. This changes the input-output relationships encountered during training, so copying the first frame no longer consistently satisfies supervision. It also introduces reversed actions: this is an augmentation for reusing existing data, not new physical capture or simulation.

Focal-length augmentation addresses a second invariant in the pairs: the original data contain 18, 24, 35, and 50 mm focal lengths, but cameras within a scene share their focal length. The authors select a longer available focal length, resize according to the focal-length ratio, and center-crop to simulate a narrower field of view. This operation uses only content already present in the original image and does not fabricate wider-view ground truth. When constructing a pair, source and target are selected from synchronized cameras in the same scene, and each video receives focal-length augmentation with probability 0.5. Although an individual augmentation only increases focal length, placing an augmented video on either side of a pair covers both relative increases and decreases. For example, a cropped narrow-view source paired with the original wider-view target supplies supervision for expanding the field of view. The paper reports 3,400 original scenes per focal length, totaling 13,600, but separately lists 47,432 scenes in the AugMCV training split without fully explaining how these counts correspond.

2. Infinite Homography Warping: explicitly fix only the geometric component that needs no depth

A general plane-induced homography depends on camera motion, plane orientation, and distance; as the plane recedes to infinity, its translation-dependent term disappears. With source intrinsics \(K_s\), target intrinsics \(K_t\), and relative rotation \(R\), the standard form is:

\[ H_{\infty}=K_t R K_s^{-1}. \]

This is the standard form corresponding to the textual definition in Section 3.2; equations (1) and (2) have damaged formatting and operators in the extraction, so the broken reprojection equation is not reconstructed as an exact author formula. Transformed homogeneous image coordinates require normalization, yielding a reference position under rotation and intrinsic changes rather than the final pixel position under arbitrary translation. For pure rotation, this mapping does not depend on object depth; translation adds parallax that varies with depth. The paper explains the residual's directional constraint through epipolar geometry and uses the relationship between the infinity-plane reference and the epipole to motivate the reduced search space. The network does not explicitly output per-pixel depth or claim to hard-project every generated pixel onto a line segment. It is therefore geometrically meaningful conditioning, not a strict guarantee of multiview consistency in generated results.

The implementation warps the source video's initial latent features rather than raw RGB frames to reduce computation. The initial frame is used because the target trajectory is defined in the initial source-camera coordinate system; a corresponding infinite homography is computed for each target pose. The warped result passes through a zero-initialized convolution and is added residually to the original initial-frame latent, followed by camera embeddings carrying target translation and intrinsic information. This lets the new pathway begin from a restrained residual state instead of immediately disrupting pretrained representations. The analytically determined rotation is presented to the model in advance, while end-to-end training absorbs parallax arising from translation and implicit scene depth. The authors call the rotational condition "noise-free," assuming reliable camera parameters; this does not mean intrinsic estimation, interpolation, or latent warping is error-free. Comparisons against per-frame source warping and pixel-space warping are deferred to supplementary material that is not included in the available full text.

3. Homography-Guided Attention: fuse content and geometric references at the same timestamp

Each temporal position has three feature groups: its source latent, current target latent, and reference latent warped according to the target camera. The camera encoder is a linear layer with 16 input dimensions: 9 for a flattened rotation matrix, 3 for translation, and \(f_x,f_y,c_x,c_y\). Its output is broadcast spatially and added to the relevant latent features; the same encoder is shared across DiT blocks. The target branch uses the specified target pose and intrinsics, while the source branch uses source intrinsics and the identity pose \([I|0]\) because its per-frame poses are unknown. This identity pose is a source-conditioning convention, not a statement that the actual input camera must be stationary. For the reference branch, explicit homography already handles rotation, and subsequent camera embeddings supply translation and intrinsic information.

Attention concatenates the three feature groups spatially and treats each temporal position as a separate batch item. For batch size \(b\), latent temporal length \(f\), spatial dimensions \(h,w\), and channel dimension \(d\), the attention input has shape \(bf\times 3hw\times d\). This lets the target frame access source motion at the same time rather than arbitrarily mixing different timestamps in the added control layer. Although reference features originate from the initial frame, the synchronized source branch retains per-frame content, so the model is not reconstructing all motion from the first frame alone. After fusion, features are split into three branches again, the reference branch is discarded, and only source and target proceed into subsequent Wan2.1 layers. The pretrained layers treat source and target as a unified batch, while the base video model continues to supply spatiotemporal generation capabilities. The architecture assigns cross-view conditioning exchange to the new layers and preserves the large-scale pretrained generative prior in the frozen backbone.

A Worked Example

Consider a video of a moving person where the requested camera starts off to one side, follows an arc, and uses a shorter focal length than the source; this illustrates the mechanism, not an additional experiment. During training, trajectory augmentation provides synchronized pairs with different initial viewpoints, and focal-length augmentation exposes the model to narrow-view inputs paired with wider-view targets. Inference does not run this training-data augmentation or supply a target video; it receives source content, intrinsics, and the requested trajectory. At each target timestamp, rotation and target intrinsics first determine how to warp the initial-frame latent into a geometric reference. That reference alone cannot implement translation: the model must also use translation conditioning to infer distinct residual parallax for the person and background. Homography-Guided Attention reads the corresponding source action, so changing the camera does not require freezing the person in their first-frame pose. Newly exposed regions from the shorter focal length remain unobserved and must be completed by the generative prior, not interpreted as recovery of uniquely correct hidden geometry.

Loss & Training

Wan2.1's pretrained weights are frozen; the text explicitly trains the new camera encoder and Homography-Guided Attention layers, initializing the latter from corresponding pretrained Transformer weights. The warping module also uses the zero-initialized convolutional residual described above, and noised target-video latents provide generative training supervision. The main text does not specify a complete loss equation, noise schedule, or geometric auxiliary loss, so no unverified exact objective is added here. Main experiments use 81 frames at \(480\times 832\), 15k training iterations, 4 H100 GPUs, and batch size 8, taking approximately one week. Ablations use 41 frames at \(320\times 544\), 20k iterations, 4 H100 GPUs, and batch size 32, taking about four days. For both settings, the text reports Adam with learning rate \(10^{-5}\) and weight decay 0.01; it is not silently changed to AdamW. "Depth-free" specifically means not reprojecting external depth maps for conditioning; when WebVid source intrinsics are unavailable, inference still estimates them with UniDepth.

Key Experimental Results

Main Results

The AugMCV test split contains 168 scenes, each using a static-camera source to generate 10 target trajectories, totaling 1,680 videos. Source and target share intrinsics in 96 scenes and differ in 72; PSNR and SSIM are higher-is-better, and LPIPS is lower-is-better, measured against ground-truth target-view videos. The following reproduces the selected results from Table 1, page 12, under the main 81-frame, \(480\times 832\) setting.

Method	Shared intrinsics PSNR ↑	SSIM ↑	LPIPS ↓	Different intrinsics PSNR ↑	SSIM ↑	LPIPS ↓
GEN3C	16.891	0.479	0.548	17.449	0.525	0.467
ReCamMaster	21.130	0.617	0.420	19.665	0.558	0.510
TrajectoryCrafter	21.228	0.660	0.296	19.557	0.586	0.390
InfCam	22.677	0.718	0.246	22.261	0.699	0.270

WebVid evaluation randomly selects 100 real videos and generates 20 target trajectories per video, totaling 2,000 outputs; each source has 10 trajectories sharing its initial pose and 10 starting at different poses. There is no target-view ground truth: FID/FVD evaluate frame-level and video-level distributional quality, while ViPE estimates generated trajectories for rotation and translation error against the requested paths. S.C. denotes VBench Subject Consistency and M.S. Motion Smoothness, both higher-is-better; neither substitutes for camera-trajectory correctness. The following corresponds to Table 2, page 12; the nearby text does not specify pose-error units or normalization details, so values are reported without inventing degree or meter units.

Method	RotErr ↓	TransErr ↓	FID ↓	FVD ↓	S.C. ↑	M.S. ↑
GEN3C	9.588	3.012	43.790	331.768	0.908	0.990
ReCamMaster	8.375	1.027	39.930	302.025	0.944	0.994
ReCamMaster w/ interp.	9.016	1.590	35.720	303.012	0.946	0.993
TrajectoryCrafter	5.007	0.735	30.877	289.879	0.940	0.991
InfCam	3.162	0.438	29.702	286.952	0.949	0.993

Ablation Study

The following corresponds to Table 3, page 13; T.A. is trajectory augmentation, I.A. intrinsic augmentation, and W.M. the warping module. It uses the 41-frame, \(320\times 544\) ablation setting, so its higher PSNR cannot be compared directly with Table 1 as an additional model improvement.

Config	Shared intrinsics PSNR ↑	SSIM ↑	LPIPS ↓	Different intrinsics PSNR ↑	SSIM ↑	LPIPS ↓
No augmentation or warping	19.228	0.562	0.427	18.480	0.507	0.525
T.A.	20.820	0.615	0.353	18.865	0.523	0.499
T.A. + I.A.	22.807	0.680	0.250	21.866	0.649	0.293
T.A. + I.A. + W.M.	24.412	0.733	0.198	24.311	0.720	0.203

Key Findings

• With both augmentations enabled, adding warping raises different-intrinsics PSNR from 21.866 to 24.311, a gain of 2.445; shared-intrinsics PSNR rises from 22.807 to 24.412, a gain of 1.605.
• With trajectory augmentation alone, different-intrinsics PSNR is 18.865, increasing to 21.866 after intrinsic augmentation; better trajectory control does not replace training coverage of focal-length changes.
• InfCam improves WebVid rotation and translation errors over TrajectoryCrafter, but its M.S. of 0.993 is below ReCamMaster's 0.994, so it does not lead every metric.
• Table 4, page 14, separately compares data strategies: translation errors are 0.839 for AugMCV and 1.441 for mixed MCV+SCV training, supporting targeted augmentation without merging these values into the different setting of Table 2.

Highlights & Insights

• More complete geometric conditioning is not necessarily better: fixing reliable rotation and learning uncertain parallax is easier to interpret than passing an entire erroneous depth reprojection to the generator. The transferable principle is to divide analytical and learned components according to reliability.
• Augmentation targets responsiveness to conditions rather than appearance diversity alone. Hidden first-frame and focal-length invariants encourage ignoring controls, and augmentation directly changes those relationships.
• Initial-frame geometry and per-frame source content play distinct roles. The former establishes a common coordinate reference, while the latter preserves dynamic synchronization rather than treating a static geometric cue as the content source for the entire video.
• Evaluation must consider camera errors alongside visual quality. Smooth videos may simply underreact to requested motion, so M.S. alone cannot establish successful control.

Limitations & Future Work

• The authors acknowledge limitations for extreme novel viewpoints and unseen regions in the conclusion, deferring detailed failures to the supplement; that material is unavailable here, so no specific failure thresholds are supplied.
• Avoiding depth maps does not remove calibration dependence. Real-video intrinsics are estimated through UniDepth, and errors can still contaminate the supposedly "noise-free" rotational condition, limiting the assumption.
• WebVid pose errors depend on external ViPE estimates, and target-view ground truth is absent. This supports practical controllability but does not directly measure hidden-geometry accuracy.
• The original 13,600 scenes and augmented training split of 47,432 scenes are not fully reconciled in the main text; reproduction needs further augmentation-expansion and split details.
• From a reader's perspective, useful further tests would cover moving source cameras, intrinsic perturbations, and large translations, with separate quality reporting for observed and newly exposed regions; these are proposals, not completed experiments.

Related Work & Insights

• vs GEN3C / TrajectoryCrafter: both build reprojection conditions from estimated depth, whereas InfCam uses an infinite-homography reference and learns residual parallax; it removes propagation of fixed depth errors but still generates unseen content.
• vs ReCamMaster: both leverage video generation priors for camera control, but InfCam explicitly handles rotation and changes MultiCamVideo pairing. The interpolation variant in Table 2 also fails to resolve pose bias, suggesting that transitional frames at inference cannot fully replace training-distribution changes.
• vs SynCamMaster / SynCamVideo: static multiview data provide different initial viewpoints, while InfCam reuses existing data through reversal and concatenation of dynamic trajectories. Table 4 indicates that mixing the two dataset types is not necessarily equivalent to targeted conditional augmentation.
• Geometric background: reference [4] is the planes-and-homographies chapter of Hartley and Zisserman's Multiple View Geometry in Computer Vision, which explains the plane at infinity, epipoles, and parallax.

Rating

• Novelty: 4/5. The integration of classical projective geometry with video diffusion conditioning is clear; the contribution is representation choice and complementary training, not a new geometric theorem.
• Experimental Thoroughness: 4/5. Synthetic and real videos, shared and different intrinsics, and component ablations are covered, but supplementary results and calibration sensitivity cannot be verified here.
• Writing Quality: 4/5. Motivation, architecture, and ablations connect clearly; dataset counts, training objectives, and error units need fuller specification, while damaged equation extraction is a separate reading-material limitation.
• Value: 4/5. A reusable approach to controlled camera editing of existing videos, with remaining risks of hallucinated content under extreme viewpoints.