ConfCtrl: Enabling Precise Camera Control in Video Diffusion via Confidence-Aware Interpolation

Conference: ECCV2026
Paper: Official Paper Page · PDF
Area: Video Generation / Novel View Synthesis
Keywords: Endpoint interpolation, camera control, point-cloud projection, confidence weighting, predict-update conditioning

TL;DR

ConfCtrl uses endpoint-conditioned video interpolation as a generative prior, changes the starting distribution of rectified flow with confidence-weighted projected point-cloud latents, and refines camera conditioning through predict-update modules, achieving 15.54 dB PSNR, 0.143 m translation error, and 0.103 rad rotation error on CO3D-Hydrant.

Background & Motivation

Given two images of a scene, synthesizing intermediate views along a specified camera trajectory is more demanding than making a smooth visual transition. As the camera moves around an object, the model must complete surfaces and backgrounds absent from the inputs while applying the correct perspective changes to visible structures. Feedforward reconstruction methods such as PixelSplat and MVSplat can render according to camera poses, but geometric errors and unobserved regions become rendering artifacts under sparse, wide-baseline inputs. Video diffusion models have stronger completion priors, yet plausible-looking frames need not follow the requested trajectory.

Camera parameters and projected point clouds provide different information. The former specify where to look from; the latter indicate which content already has geometric support in that view. CameraCtrl primarily uses pose conditioning, while methods such as ViewCrafter introduce point-cloud projections. However, incorrect depth, scale, or occlusion handling can feed distorted observations into the generator. Even a paired-image geometry foundation model such as VGGT does not make projections noise-free. Simply concatenating these representations does not explicitly distinguish trustworthy regions from those requiring generative completion.

The paper also chooses an endpoint-conditioned interpolation backbone instead of generating forward from a single image: observations at both ends constrain intermediate frames and help limit accumulating drift. Geometry then influences both the initial distribution and the internal feature updates, where it is treated as a fallible observation. Core idea: initialize generation with reliable geometric projections, then combine camera-driven prediction with projection-residual updates so that uncertain point clouds guide the model without becoming unquestionable ground truth.

Method

Overall Architecture

The inputs are two endpoint images and a target camera trajectory; the output is a sequence of synthesized intermediate views along that trajectory. ConfCtrl builds on the 1.3B Wan-2.1-InP model. VGGT estimates a point cloud and point-wise confidence from the inputs, after which geometry and confidence are projected into the target camera views. The model operates on latents of these projections rather than directly regressing a final set of independently renderable Gaussian parameters.

Three mechanisms have distinct roles: the endpoint interpolation prior constrains appearance at both ends, confidence-weighted initialization determines where rectified flow starts, and predict-update camera conditioning refines features during generation. The architecture uniformly inserts 10 Kalman DiT modules. Each module contains three DiT blocks, so these 10 modules should not be mistaken for only 10 additional ordinary DiT blocks.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Two input images<br/>and target camera trajectory"] --> B["Endpoint Interpolation Prior"]
    A --> C["VGGT geometry and confidence<br/>projected into target views"]
    B --> D["Confidence-Weighted Initialization"]
    C --> D
    D --> E["Predict-Update Camera Conditioning"]
    C --> E
    E --> F["Novel-view sequence<br/>along the requested trajectory"]

Key Designs

1. Endpoint Interpolation Prior: constrain intermediate views with observations at both ends

A video model starting from one frame can carry early deviations into later frames, whereas novel view synthesis already provides a second real observation. ConfCtrl inherits the ability of a pretrained endpoint interpolation model to generate the content between these observations. Generation must not merely depart from the first image; it must also arrive at the last. This supplies appearance and spatiotemporal continuity priors, but does not make the intermediate trajectory correct automatically. Different camera paths can connect the same endpoints, which is why explicit camera conditioning and geometric projections remain necessary.

This choice also helps explain adaptation with limited training data. Most generative capacity comes from existing video pretraining, while the current training adapts that capacity to wide-baseline image pairs instead of learning the full appearance distribution from a small collection of sequences. Importantly, the ablation without interpolation replaces the backbone with a pretrained camera-guided video diffusion model. It does not simply switch off an ordinary interpolation operator, so the underlying backbone change must remain part of the interpretation.

2. Confidence-Weighted Initialization: place reliable projections in the flow's starting distribution

Conventional conditioning begins with pure Gaussian noise and introduces projected geometry through an auxiliary control branch. ConfCtrl instead obtains projected point-cloud latents and the corresponding target-view confidence weights, multiplies them element-wise, and adds the result to noise. Reliable projections exert more influence on the initial state, while uncertain regions receive weaker geometric guidance and leave more room for generative completion. This is neither a direct copy of projected pixels into the final image nor a mask-based composition of final outputs.

Some symbols in cached Eq. (3) are corrupted. The following expression reconstructs the weighted sum explicitly described by the adjacent prose; it is a semantic transcription, not a verbatim recovery:

\[ z_0=\lambda_1\bigl(w\odot\hat z_{pc}^{0}\bigr)+\lambda_2\epsilon, \qquad \epsilon\sim\mathcal N(0,I). \]

Here, \(\hat z_{pc}^{0}\) is the projected point-cloud latent, \(w\) contains corresponding confidence weights, and \(\odot\) denotes element-wise multiplication. The experiments set \(\lambda_1=1\) and \(\lambda_2=1\), so this is not a convex interpolation whose weights must sum to one. The projected latent is also linearly interpolated to match temporal channels and spatial resolution. That resizing operation is distinct from endpoint-conditioned video interpolation. The main text does not fully specify confidence normalization, occlusion-conflict handling, or values for missing projections; these implementation details cannot be inferred from the equation alone.

3. Predict-Update Camera Conditioning: turn geometric disagreement into a learned correction signal

Initialization sets a starting point but cannot ensure that every subsequent layer interprets the target pose correctly. Each Kalman DiT module therefore produces a prediction and then updates it using the discrepancy from projected geometry. The prediction branch encodes target camera Plücker representations, forms a control feature through cross-attention between camera and projected point-cloud latents, and adds that feature to the backbone through a zero-initialized linear layer. Although the introductory description calls prediction conditioned solely on the target pose, the implementation explicitly includes projection features in this cross-attention. The implementation description should guide interpretation; prediction is not completely isolated from the point cloud.

The update branch treats the projected latent, already processed by earlier modules, as a noisy measurement. It computes the discrepancy between prediction and measurement, processes that discrepancy with additional DiT blocks, and injects the learned correction through another zero-initialized linear layer. The following combines Eqs. (4)-(6), preserving the prediction-minus-observation sign:

\[ \begin{aligned} z_{\mathrm{pred}}^n &= z^n+\operatorname{zero\text{-}linear}(u^k),\\ z_{\mathrm{update}}^n &= z_{\mathrm{pred}}^n+ \operatorname{zero\text{-}linear}\!\left( \operatorname{Diff}\!\left(z_{\mathrm{pred}}^n-\hat z_{pc}^{k-1}\right)\right). \end{aligned} \]

The index \(n\) identifies a backbone layer, \(k\) identifies the corresponding Kalman DiT module, and \(u^k\) is the control feature produced by the cross-attention above. \(\operatorname{Diff}\) is the network learning the discrepancy correction, not an independent diffusion sampling process. The two zero-initialized linear layers are separate branch interfaces: they initially avoid strongly disturbing pretrained features and learn to inject control during training. Compared with direct concatenation, the network explicitly receives where prediction and observation disagree, allowing it to learn to attenuate erroneous projections.

The connection to Kalman filtering is architectural only. The authors explicitly rule out recursive Bayesian state estimation across diffusion steps and do not provide state-covariance propagation or an analytical Kalman gain. Confidence maps explicitly weight initialization, whereas reliability-dependent refinement in the update branch is learned through residual corrections. These should not be conflated into an unsupported claim that every layer explicitly multiplies by the same confidence map.

A Worked Example

Consider two images captured around a hydrant, with the objective of producing a 33-frame view sequence including the endpoints. VGGT supplies the visible-surface point cloud and confidence information. At an intermediate target viewpoint, a reliable projection of the hydrant's front contributes more strongly to the initial latent, while uncertain side or rear regions receive weaker geometric guidance. This is an illustrative walkthrough of the mechanism, not a claim that the paper reports a particular pixel's confidence value.

The prediction branch then combines the target camera and projection features to steer the current backbone representation toward the desired view. If a projected boundary disagrees with the prediction, the update branch learns a correction from that discrepancy instead of being forced to copy the warped boundary. Endpoint images continue to constrain appearance, and the generative prior completes unobserved content. The result is a view sequence, not a three-dimensional model with guaranteed ground-truth unseen surfaces.

Loss & Training

Training uses rectified flow to evolve the confidence-aware initialization \(z_0\) toward the ground-truth video latent \(z_1\). Following the prose accompanying Eq. (7), the linear path can be written as:

\[ z_t=(1-t)z_0+t z_1,\qquad z_{\mathrm{target}}=z_1-z_0. \]

The velocity network learns this target direction. Changing initialization necessarily changes the starting point used in velocity supervision; substituting pure noise for \(z_0\) would describe a different objective. The authors additionally align horizontal and vertical spatial gradients of predicted and target velocities in latent space to preserve local structure and reduce noise or flicker during abrupt viewpoint changes. The total objective combines the rectified flow term with a gradient regularizer weighted by \(\lambda_{\mathrm{grad}}=0.05\). Norm and aggregation symbols in cached Eqs. (8)-(10) are incomplete, so an unverifiable exact expansion of these losses is deliberately omitted.

The three datasets are trained and evaluated separately. CO3D-Hydrant, CO3D-Teddybear, and DL3DV use 250, 450, and 200 training sequences, respectively, with the remaining sequences used for testing. Training uses \(256\times256\) resolution, 33 frames, batch size 1, learning rate \(10^{-5}\), and 20k steps. It requires approximately 40 GB of GPU memory and 24 hours on a single GPU. Sampling with 50 steps takes about 10 seconds per scene, but the main text does not specify the GPU model or separate geometric preprocessing from generation in the timing breakdown.

Key Experimental Results

Main Results

The following values are selected from Table 1. Higher PSNR is better; lower LPIPS, translation error \(E_t\), and rotation error \(E_r\) are better. \(E_t\) is the mean Euclidean distance between predicted and ground-truth camera positions in m, and \(E_r\) is the mean angular difference between their rotation matrices in rad. VGGT estimates the trajectory from generated video relative to the first input image; evaluation does not simply read the generator's internal camera parameters.

Dataset	Method	LPIPS	PSNR (dB)	SSIM	\(E_t\) (m)	\(E_r\) (rad)
CO3D-Hydrant	Gen3R	0.341	15.32	0.321	0.189	0.144
CO3D-Hydrant	Uni3C	0.350	15.41	0.322	0.219	0.167
CO3D-Hydrant	ConfCtrl	0.339	15.54	0.328	0.143	0.103
CO3D-Teddybear	Gen3R	0.368	16.92	0.452	0.234	0.159
CO3D-Teddybear	Uni3C	0.382	16.45	0.438	0.271	0.198
CO3D-Teddybear	ConfCtrl	0.354	17.27	0.480	0.210	0.155
DL3DV	Gen3R	0.296	16.34	0.391	0.225	0.160
DL3DV	Uni3C	0.306	15.99	0.395	0.213	0.159
DL3DV	ConfCtrl	0.287	16.49	0.427	0.195	0.149

CameraCtrl, ViewCrafter, and Uni3C are adapted to the same endpoint-conditioned video diffusion backbone and trained with the same data setting. ViewCrafter and Uni3C also receive the same VGGT projections. SeVA and Gen3R are additionally evaluated as large-scale pretrained models, so equal pretraining budgets cannot be claimed for every method. In the cross-domain experiment from Table 3, training on DL3DV and testing zero-shot on GraspNet gives ConfCtrl a PSNR of 15.23, \(E_t\) of 0.099, and \(E_r\) of 0.071; Uni3C obtains 15.05, 0.128, and 0.091, respectively.

Ablation Study

These rows are selected from Table 4. All results are on CO3D-Hydrant, avoiding comparisons across different datasets.

Configuration	LPIPS	PSNR (dB)	SSIM	\(E_t\) (m)	\(E_r\) (rad)
Full ConfCtrl	0.339	15.54	0.328	0.143	0.103
(a) Pure-noise initialization	0.347	15.44	0.326	0.174	0.128
(b) Uniform confidence of 1.0	0.366	14.75	0.296	0.164	0.117
(c) Without the update branch	0.386	14.65	0.305	0.194	0.139
(d) Without update and confidence-aware initialization	0.362	14.85	0.304	0.213	0.155
(e) Without gradient regularization	0.339	15.49	0.322	0.152	0.110
(f) Pose-only prediction	0.370	14.90	0.312	0.192	0.141
(g) Direct addition of pose and projection	0.363	15.06	0.313	0.174	0.124
(h) Non-interpolation backbone	0.491	13.45	0.193	0.457	0.362
(i) More parameters for the no-update variant	0.382	14.56	0.306	0.202	0.145
(l) Replace VGGT with Pi3	0.333	15.56	0.326	0.146	0.107

Key Findings

• Changing the interpolation backbone causes the largest degradation: PSNR falls from 15.54 to 13.45, while translation error rises from 0.143 to 0.457. This supports the importance of the pretrained starting point, but it is not a single-module removal with everything else held fixed.
• Uniform confidence of 1.0 produces worse image quality than pure-noise initialization despite slightly better pose errors. Incorrect geometry can simultaneously constrain the camera more strongly and damage appearance. Removing both components is not worse than either individual removal on every metric; complementarity is most evident in camera control.
• Gradient regularization brings a modest gain: LPIPS remains 0.339 without it. Pi3 improves LPIPS to 0.333 and PSNR to 15.56 relative to default VGGT, but slightly worsens SSIM and pose errors, revealing a trade-off in geometry-backbone selection.
• DL3DV FVD in Table 2 provides a clear exception to universal superiority: ConfCtrl obtains 113.46 versus Gen3R's 105.52, where lower is better. The across-metric lead in Table 1 should not be extended to every generative-quality metric.

Highlights & Insights

• Confidence can shape the initial generative distribution rather than only reweighting a loss. Reliable geometry then influences generation from the outset instead of depending entirely on later control branches to correct drift.
• Prediction-observation disagreement is itself useful conditioning. Learning how to correct that disagreement is better suited to wide baselines and occlusions than assuming every geometric feature is trustworthy.
• Endpoint constraints and explicit trajectory constraints are complementary: the former provide appearance boundary conditions, while the latter determine the intermediate camera path. This division may inform other sparse-observation generation tasks, although transfer benefits require separate experiments.

Limitations & Future Work

• The authors identify the video VAE as a bottleneck: it favors smooth changes between adjacent frames and is poorly suited to abrupt camera motion or large positional jumps. They propose VAE architectures tailored to novel view synthesis or VAE-free generation as possible directions.
• Evaluation is limited to \(256\times256\), 33 frames, and relatively small training sets. It does not establish high-resolution, long-sequence, or real-time performance; approximately 10 seconds per scene is not real-time interaction.
• VGGT supplies geometric conditioning and also estimates evaluation trajectories, creating a possible shared-model bias. This is an evaluation concern raised by this note, not demonstrated misconduct or failure. Independent pose estimators and calibrated geometric evaluation would provide useful cross-checks.
• The available cache lacks supplementary details on full training splits and confidence normalization, and it does not report result variance or confidence intervals. Small metric differences require caution, and plausible unseen content is not equivalent to true three-dimensional recovery.

Related Work & Insights

• Versus PixelSplat, MVSplat, and AnySplat: feedforward methods construct explicit scene representations and render them at requested poses, whereas ConfCtrl directly uses video priors to complete views. The latter can handle unknown appearance more flexibly but does not automatically yield a unified, arbitrarily rerenderable three-dimensional model.
• Versus CameraCtrl, ViewCrafter, and Uni3C: the contribution is not the first use of camera parameters or point clouds. It changes where geometry enters the model and how it is fused. Shared VGGT inputs and same-backbone retraining help isolate the contribution of this conditioning design.
• Versus classical Kalman filtering: the method borrows a predict-update architectural prior without analytical uncertainty-estimation guarantees. Explicit confidence calibration could be evaluated as an extension to residual updates, but that untested possibility should not be presented as an existing capability.

Rating

• Novelty: 4/5. Confidence-weighted flow initialization and predict-update conditioning form a targeted combination, while relying on established interpolation, geometry foundation models, and residual control.
• Experimental Thoroughness: 4/5. Three datasets, cross-domain evaluation, and diverse ablations support the method, but resolution, training scale, and independent pose validation remain limited.
• Writing Quality: 4/5. The pipeline maps clearly onto the ablations, although the prediction-branch overview and cross-attention implementation use inconsistent wording.
• Value: 4/5. Useful for sparse-view generation and camera-control research, especially as an approach to handling unreliable geometric conditioning.