InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision / Video Generation
Keywords: novel view synthesis, continuous latent masks, inverse problems, video inpainting, camera control

TL;DR

InverseCrafter formulates monocular video recapture along a specified camera trajectory as latent-space inpainting, predicts continuous multichannel masks with a lightweight encoder, and enforces measurement consistency through back-propagation-free DDS, achieving a runtime of 71 seconds and measurement PSNR of 29.35 dB without fine-tuning the video diffusion model.

Background & Motivation

Video recapture requires more than generating a similar clip from text: the same scene and motion must be presented along a new camera trajectory specified by the user. Pretrained video diffusion models can generate natural imagery, but text and first-frame conditioning generally cannot specify camera motion precisely or guarantee preservation of source details. A direct approach estimates video depth, projects visible pixels into the target view, and lets a generative model complete newly exposed regions. Geometric reprojection then handles content determined by the input, while the generative prior handles content that was never observed. Methods such as TrajectoryCrafter further fine-tune the video model for this inpainting task, introducing training data requirements, model updates, and dependence on particular geometry reconstruction pipelines.

Keeping the pretrained model suggests enforcing agreement with known pixels at every denoising step, but latent compression creates another obstacle. Pixel-space comparisons require repeated traversal of the nonlinear VAE decoder, increasing computation and potentially destabilizing optimization. Simply resizing the pixel mask is cheaper, but assumes that the VAE preserves visibility at corresponding positions and that all latent channels can share one mask. Modern video VAEs compress both space and time, making both assumptions unreliable. When an object is visible in only some frames of a temporal group, logical-AND downsampling can mark the entire group as unknown and discard motion information that could constrain generation.

The paper therefore shifts attention from retraining the generator to expressing observations appropriately in its latent space. The objective is to avoid per-step decoding while retaining an observation operator simple enough for established linear inverse solvers. Training-free here means no fine-tuning of the video diffusion model; the camera-control version still learns a small mask encoder, so the entire system is not free of training. Core Idea: learn the continuous effect of pixel occlusion across latent channels, approximate measurement constraints through an elementwise linear mask, and solve the resulting latent inverse problem under a frozen video generative prior.

Method

Overall Architecture

The inputs are a source video, a target camera trajectory, and generation conditioning; the output shows the same dynamic scene along the target trajectory. DepthCrafter estimates per-frame depth, camera intrinsics back-project pixels into point clouds, and relative camera transformations render each target view. The resulting warped video contains visible content and holes: it is the observation for inpainting, not complete target-view ground truth. The pixel visibility mask is denoted by \(m\), with 1 at visible positions; the ideal pixel observation model is \(y=m\odot x\), where \(x\) is the target video to recover. Training constructs the Continuous Latent Mask Target and learns the Lightweight Mask Encoder; inference combines the predicted mask and observation latents in Latent-space DDS. The video diffusion model and VAE remain frozen, and iterations modify generated latents rather than updating generator weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    Source["Source video + target trajectory"] --> Warp["Depth and reprojection<br/>Observed video"]
    subgraph Training["Offline training"]
        Target["Continuous Latent<br/>Mask Target"] -->|Supervision| Encoder["Lightweight Mask<br/>Encoder"]
    end
    Warp -->|Inference input| Encoder
    Warp -->|Frozen VAE encodes observation| Solver["Latent-space DDS"]
    Encoder -->|Continuous multichannel mask| Solver
    Prior["Frozen video diffusion model"] -->|Per-step denoising estimate| Solver
    Solver --> Output["Final VAE decoding<br/>Novel-view video"]

Key Designs

1. Continuous Latent Mask Target: represent how occlusion affects latent features

An occluded pixel does not imply that a corresponding compressed latent position becomes entirely unusable. Nonlinear VAE encoding mixes neighboring spatial, temporal, and channel information, so the relevant question is how the latent representation changes after masking. The authors encode a complete video and its masked counterpart, normalize their latent difference, and construct a continuous mask with the same shape as the latent representation. The target neither interpolates the pixel mask nor copies one channel across all latent channels; each channel and spatiotemporal position can have a different retention strength. Following Eq. (5), the normalization function is retained in its abstract form:

\[ h=1-f\!\left(\mathcal E(x)-\mathcal E(m\odot x)\right), \qquad h\in[0,1]^{C\times M}. \]

Here \(\mathcal E\) is the frozen VAE encoder, \(C\) is the number of latent channels, \(M\) is the number of compressed spatiotemporal positions, and \(f\) maps the difference into \([0,1]\). The available main text describes activation-based normalization but does not provide its complete reproducible specification; substituting a particular sigmoid or absolute-value formula would be unsupported. This construction uses latent changes to provide soft constraint strengths, but does not mathematically guarantee exact multiplicative operator equivalence for arbitrary nonlinear VAEs. Operator equivalence is therefore best read as the observation relation the method seeks to approximate, rather than an identity proven for every input.

Training for camera recapture lacks actual target-camera videos, so complete and masked target-view pairs cannot be used directly to calculate this difference. The authors use double reprojection: an existing complete video undergoes a viewpoint transformation and is projected back to produce a corresponding version with geometric occlusions. The original video serves as the complete reference, and the synthesized masked video provides the degraded observation required for latent-mask supervision. This avoids paired real dynamic multiview 4D data, although training occlusions still depend on depth estimates and the synthetic trajectory distribution. An apparently more direct alternative divides masked latents by complete latents, but the authors note numerical instability when denominators approach zero. The continuous difference target avoids that division, while its normalization remains an approximation open to improvement.

2. Lightweight Mask Encoder: predict multichannel constraints directly from observations

Complete videos are available during training, but actual recapture provides only a target-view observation with holes, preventing recomputation of a target mask that requires complete ground truth. The mask encoder therefore learns to predict \(h\) directly from the observed video \(y\), converting offline supervision into an inference-time constraint. Importantly, Section 3.2 defines the network input as the pixel-space observed video, not the binary pixel mask alone. The name mask encoder describes its output role and should not be interpreted as a network that receives only geometric masks. Its architecture is a reduced Wan2.1 VAE, with channel width decreased from 96 to 16 and 1.5M parameters in total. Additional learning consequently focuses on representing the observation operator rather than teaching a large video generator scene content or camera control again.

At inference time, the observed video needs one observation-latent encoding and one continuous-mask prediction, which subsequent denoising steps reuse. Once the encoder output is fixed, the mask acts on the unknown latent representation through elementwise multiplication and is therefore linear. A nonlinear network producing a mask is compatible with a linear observation operator conditioned on that mask; the solver exploits the latter property. The paper also states that masked regions are filled before encoding during training and inference to reduce input distribution mismatch, but the available main text does not fully specify the fill-value rule. This preliminary filling is an encoding procedure, not a claim that unknown regions have become genuine observations. Its usefulness must still be assessed through preservation of visible content in the generated result, not merely through fitting the mask target.

3. Latent-space DDS: alternate denoising priors with measurement correction

With observation latents \(w=\mathcal E(y)\) and a predicted mask, the method approximates the relationship between target latents and observations as \(w\approx h\odot z\). DDS, the Decomposed Diffusion Sampler, first obtains a clean-latent estimate \(\hat z_{0|t}\) from the frozen flow model and then performs a separate measurement-consistency correction. The correction should make masked latents agree with observations without moving excessively far from the estimate already supplied by the generative prior. Writing the positive weight in the paper's Eq. (8) consistently as \(\lambda\), the proximal subproblem is:

\[ z^{(y)}=\arg\min_z \frac{\lambda}{2}\lVert w-h\odot z\rVert_2^2 +\frac{1}{2}\lVert z-\hat z_{0|t}\rVert_2^2. \]

The first term imposes data constraints, while the second keeps the correction near the denoised estimate; together they determine the update. Because a fixed mask defines a diagonal linear operator, the paper solves this quadratic problem with conjugate gradient without back-propagating through the VAE or video diffusion model. The corrected clean-latent estimate and the flow model's noise-end estimate are then passed to an ODE solver to advance to the next timestep. Only the final latent representation is decoded into a video, so eliminating decoding means eliminating repeated decoding for per-step measurement optimization. It does not turn video generation into a single-step algorithm: the pretrained model still uses multistep sampling.

Measurement constraints are not applied indiscriminately at every timestep; an update-window hyperparameter controls when consistency correction begins. Earlier correction more strongly preserves source content, whereas later correction gives the generative prior more opportunity to form visually natural completions. Figure 9(c) compares 0.6, 0.8, and 0.9, and the text reports 0.8 as a useful compromise between semantic alignment and structural preservation. This timing control differs from the proximal weight in Eq. (8) and should not be confused with the number of conjugate-gradient iterations. Algorithm and timestep-set notation is partially corrupted in the supplied extraction, so this note retains the readable mechanism and parameter comparison without reconstructing damaged pseudocode.

A Worked Example

Consider zooming out or translating upward, as illustrated in Figure 6: depth estimates first turn the source video into per-frame point clouds. Changing the target viewpoint moves previously visible content into new positions, while newly exposed background has no corresponding source pixels. The warped result is now a video with holes, and the mask encoder estimates which latent features require strong constraints and which should allow completion. Each denoising step proposes a possible latent novel-view video, DDS pulls observation-inconsistent content back toward the measurements, and sampling continues. The final result must follow the target camera motion and retain source dynamics; unseen background remains a generative prediction rather than verified reconstruction.

Text-guided object replacement differs because the complete source video and object mask are both known, allowing the latent mask to be computed directly from the encoding difference. The editing experiments therefore do not need the learning process required to predict masks, whereas the main camera-recapture experiments use the learned version described above. An off-the-shelf image inpainting model supplies first-frame conditioning, and subsequent video inpainting uses the target text; this does not assume access to unobserved target-view ground truth during recapture.

Loss & Training

The mask encoder is trained with L1 error between predicted and target masks plus a weighted \(1-\operatorname{SSIM}\) term, combining elementwise agreement with structural similarity. Training uses 7,750 VidSTG videos, with 6 double-reprojection samples per video, yielding 46,500 training samples. The training resolution is \(240\times416\), the AdamW learning rate is \(1\times10^{-4}\), and weight decay is \(3\times10^{-2}\). The total batch size is 16 across 4 GPUs, with 4 samples per GPU, and the authors report 1 day of training. Main inference experiments use Wan2.1-Fun-V1.1-1.3B-InP at \(480\times832\) on NVIDIA RTX A6000 hardware with 48GB VRAM. The savings therefore concern task-specific video diffusion fine-tuning and per-step measurement optimization, not the elimination of small-network training, depth estimation, or sampling computation.

Key Experimental Results

Main Results

Camera control is evaluated on 1,000 UltraVideo clips with captions, sampling one of 6 target trajectories per video; the following selection comes from Table 1 on page 10. PSNR, LPIPS, and SSIM measure framewise agreement with warped observations, while DINO distance and FVD reference source videos; these are not reconstruction metrics against complete target-view ground truth. Runtime is in seconds; higher PSNR, SSIM, and VBench are better, while lower LPIPS, DINO, FVD, and runtime are better.

Method	Runtime (s)	PSNR (dB)	LPIPS	SSIM	DINO	FVD	VBench
GCD	44	Not reported	Not reported	Not reported	0.0691	250.85	0.8865
TrajCrafter	134	28.37	0.0573	0.8942	0.0376	120.03	0.8954
NVS-Solver	696	26.68	0.0816	0.8314	0.0393	96.90	0.8955
CogNVS	164	15.41	0.4339	0.4016	0.1108	2175.93	0.8193
InverseCrafter	71	29.35	0.0485	0.8827	0.0359	99.73	0.8977

Relative to TrajCrafter, PSNR improves by 0.98 dB and runtime falls from 134 seconds to 71 seconds, but SSIM decreases from 0.8942 to 0.8827. NVS-Solver still has lower FVD at 96.90 versus 99.73, and GCD is faster at 44 seconds; the supported conclusion is a favorable overall trade-off rather than dominance on every metric.

Ablation Study

The following table is Table 3 on page 13, evaluating measurement consistency of VAE reconstructions under different mask strategies on a DAVIS subset, not an end-to-end camera-recapture ablation. The target mask directly uses the latent difference between complete and masked videos, while the predicted mask comes from the learned encoder.

Mask strategy	PSNR (dB)	LPIPS	SSIM
Binary mask resizing	27.76	0.0559	0.8794
Elementwise division	26.59	0.0688	0.8458
Ours, target mask	28.10	0.0526	0.8764
Ours, predicted mask	27.97	0.0515	0.8772

Compared with binary resizing, the predicted mask reduces LPIPS from 0.0559 to 0.0515 and increases PSNR from 27.76 to 27.97, but does not exceed the binary baseline on SSIM. The predicted mask also has lower PSNR than the target mask, so results from the supervision target should not be attributed directly to network predictions.

Key Findings

• The method combines more detailed observation modeling with inexpensive solving rather than retraining a larger generator.
• Text editing is additionally evaluated on 20 DAVIS videos with 5 target prompts each; Table 2 on page 12 reports FVD of 1422.72 and VLM alignment of 0.6510 for the proposed method.
• Its editing CLIP Score is 24.98, below Zero4D's 25.83; the authors attribute the latter's image-metric advantage to nearly static outputs, a qualitative explanation rather than an isolated causal ablation.
• Near-zero additional overhead is the paper's claim relative to base diffusion sampling; Table 1 does not provide a bare-model timing under identical settings that independently quantifies that overhead.

Highlights & Insights

• The reusable design is to learn parameters of an easily solvable operator instead of an arbitrary nonlinear measurement mapping. Linearity after fixing the mask allows expressive encoding to coexist with inexpensive numerical solving.
• Continuous multichannel masks replace binary known-versus-unknown decisions with latent-feature retention strengths, specifically addressing information mixing under temporal compression. This explains why spatial resizing alone is insufficient for video VAEs.
• Whether mask prediction is needed depends on whether the task provides a complete reference video. Camera recapture and object editing share a solver without requiring identical mask acquisition procedures.

Limitations & Future Work

• The authors acknowledge that initial reprojection relies on monocular depth, so depth errors become erroneous observations; the solver may faithfully preserve those errors rather than correct geometry automatically.
• Larger viewpoint changes introduce more occlusion and less valid visual context, potentially weakening latent-difference mask targets; generated content also inherits pretrained-model biases.
• The mask encoder is tied to a specific VAE architecture and is not guaranteed to transfer directly to another latent space; multistep diffusion sampling still limits speed.
• From a reader's perspective, soft-mask operator equivalence is mainly supported empirically, without an error bound in the main text; more principled target construction and cross-VAE adaptation remain useful directions.
• The supplied full text ends at the references and does not include cited Appendices B.2, D, or E; the detailed VLM scoring protocol, additional training-free camera-control variant, and appendix analyses were not verified.
• Some equations and algorithms are damaged in the extraction; this note does not guess normalization implementations, timestep sets, or unreported solver hyperparameters.

Related Work & Insights

• vs TrajectoryCrafter (reference 66): both use geometric reprojection and inpainting, but TrajectoryCrafter adapts the video model; InverseCrafter reuses its double-reprojection idea while learning only a small mask encoder.
• vs NVS-Solver / Zero4D (references 65 / 39): these methods establish the potential of frozen video priors, while InverseCrafter further addresses latent-mask mismatch, particularly spatiotemporal compression and channel differences.
• vs SILO (reference 43): SILO learns a latent forward operator that remains nonlinear in the unknown latents; InverseCrafter preserves a linear action after learning the mask and can use inexpensive DDS consistency steps.
• Insight from DDS (reference 14): separating prior denoising from measurement optimization shifts the adaptation challenge toward constructing a suitable forward operator for the new modality, without necessarily redesigning the generator.

Rating

• Novelty: 4/5. The combination of continuous latent masks and linear solving is clear, although operator equivalence remains approximate modeling.
• Experimental Thoroughness: 4/5. Camera recapture, text editing, and mask comparisons are covered, but reconstruction ablations do not isolate end-to-end gains.
• Writing Quality: 4/5. The main argument is clear, but training-free requires qualification, and formula extraction in the supplied text limits access to reproduction details.
• Value: 4/5. Useful for efficient geometric control under frozen video priors, with practical performance still dependent on depth quality and the particular VAE.