World Reconstruction From Inconsistent Views

Conference: ECCV2026
Authors: Lukas Höllein, Matthias Nießner
Affiliation: Technical University of Munich
Paper: ECCV paper
Project: Video to World
Area: 3D Vision
Keywords: non-rigid registration, generative drift, point cloud alignment, inverse-deformation rendering, world reconstruction

TL;DR

The paper treats geometric drift in generated videos as observation deformation to remove, combining non-rigid point cloud alignment with deformation-aware 2D Gaussian Splatting (2DGS) optimization to reconstruct a single static world, achieving 79.29 in 3D consistency and 86.59 in photometric consistency across video models.

Background & Motivation

Video diffusion models can generate detailed indoor and outdoor environments, but individually plausible frames do not necessarily describe the same three-dimensional world. When a camera moves around a house, for example, window frames may shift or walls may stretch; even camera-conditioned generators do not always follow the supplied trajectory precisely. Directly optimizing a Gaussian scene against these images can split one surface into several layers, producing blurry training views and floating artifacts from novel viewpoints.

Prior methods mainly strengthen camera control, maintain a 3D cache, or fine-tune consistency objectives on the generation side, yet drift remains and training is generator-specific. This paper instead exploits a property of geometric foundation models: they may not correct generative drift, but they expose it as misalignment between per-frame point clouds. The problem becomes aligning existing observations in a common space rather than requiring a flawless generator; however, aligning geometry alone is insufficient because the supervision images remain inconsistent.

Core Idea: first recover consistent geometry in canonical space through non-rigid registration, then deform it back into each observation space during rendering-based training, allowing photometric optimization to improve appearance without reintroducing drift into the static world.

Method

Overall Architecture

The input is one or more generated videos depicting a static scene; the output is a unified 2DGS scene that supports real-time novel-view rendering, not a time-varying dynamic scene. DepthAnything-3 (DA3) predicts depth, confidence, and cameras to create per-frame colored point clouds; reliable-point filtering, non-rigid frame-to-model registration, and global refinement then resolve surface misalignment. Finally, an inverse deformation from canonical space to each frame space is learned and used to calculate the rendering loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Generated videos + DA3 predictions"] --> FILTER["Reliable-point filtering"]
    FILTER --> ICP["Non-rigid frame-to-model<br/>registration"]
    ICP --> GLOBAL["Global geometry refinement"]
    GLOBAL --> RENDER["Inverse-deformation-aware<br/>rendering"]
    RENDER --> OUTPUT["Unified static 2DGS world"]

Key Designs

1. Reliable-point filtering: retain dense observations that support surface alignment

DA3 predicts per-pixel depth and confidence together with pinhole camera intrinsics and extrinsics; when the video generator supplies a camera trajectory, that trajectory conditions depth and confidence prediction. Depth is unprojected using the camera intrinsics and transformed into a common coordinate system to form a colored point cloud. Instead of applying one global confidence cutoff, the method assigns points to voxels, calculates a local confidence percentile within each voxel, and computes a percentile threshold over voxel point counts. A point survives only if its confidence exceeds the local threshold and its voxel has sufficient occupancy, combining local reliability with support from repeated observations.

This initialization aims to retain dense, alignable surfaces rather than removing problematic regions and asking a generator to hallucinate replacements. It can still contain duplicate walls or window frames: confidence concerns geometric prediction reliability, not perfect consistency of generated content across frames; resolving that mismatch is the next stage's responsibility.

2. Non-rigid frame-to-model registration: progressively merge displaced surfaces into a common model

The first frame defines the canonical reference, and subsequent frames align to the accumulated model rather than only to the immediately preceding frame. For each frame, the method jointly optimizes camera extrinsics and a hash-grid MLP that predicts a six-dimensional twist at each 3D position, mapped to a per-point SE(3) transform through the exponential map. Each point receives a rigid transform, but the transform varies spatially, allowing the surface as a whole to deform non-rigidly; local deformation occurs in the current camera space before the camera transform maps it into canonical space.

Registration combines complementary evidence: point-to-plane ICP constrains surface distance, colored ICP aligns local tangent-plane color gradients, and RoMa matches provide confidence-weighted correspondences across frames. Nearest-neighbor ICP only accepts points within a distance threshold, so a window displaced beyond that threshold receives no attraction toward its counterpart; image correspondences can connect these distant but visually corresponding locations. The weighted correspondence term in original Eq. (6) can be written as follows, where both endpoints are 3D points associated with matched pixels and transformed into the same coordinate system, and \(w_m\) is matching confidence:

\[ \mathcal{L}_{\mathrm{corr}}= \frac{\sum_m w_m\|\mathbf{x}_m-\mathbf{y}_m\|_2^2}{\sum_m w_m}. \]

The method also compares each point's predicted twist with those of its six axis-aligned neighbors one voxel away, discouraging locally inconsistent deformation and arbitrary surface tearing. Optimization proceeds coarse to fine, activating the non-rigid network only at the finest scale: stable rigid updates first address overall pose, followed by local deformation. During fusion, an adaptive rejection rule uses the median absolute deviation (MAD) of historical geometric and color ICP losses; a point enters the model only when both losses pass their thresholds.

3. Global geometry refinement: update both ends of correspondences to reduce accumulated error

Sequential fusion builds a complete scene but updates only the current frame at each step, allowing earlier errors to accumulate into surfaces that overlap approximately yet remain thick. Global refinement jointly optimizes all cameras and per-frame deformation networks, finding 5 nearest neighbors from other frames for each point to construct geometric and color objectives. Crucially, gradients now reach both correspondence endpoints instead of treating the accumulated model as immutable ground truth, allowing surfaces from multiple frames to tighten together.

To prevent joint collapse or unrestricted camera drift, sampled per-frame anchor points are encouraged to retain their pre-refinement twists, and the camera twist parameters receive a similar constraint. This stage refines an established alignment rather than solving registration again from scratch. The fence example in original Figure 5 illustrates the intended effect: thinner surfaces also produce sharper texture boundaries that were previously overlaid.

4. Inverse-deformation-aware rendering: do not force aligned geometry to directly explain contradictory images

Passing an aligned point cloud into ordinary Gaussian optimization would still let inconsistent training images pull Gaussians back toward duplicated surfaces. The method therefore analytically inverts the rigid camera transforms and trains a shared inverse-deformation hash-grid MLP conditioned on a learnable view embedding. Supervision comes from the existing forward registration: sample original camera-space points, map them into canonical space, and require inverse deformation to recover the originals, using 3D mean-squared error and local smoothness regularization.

The canonical point cloud initializes 2D Gaussian disks: points supply positions, normals determine rotations, the mean Euclidean distance to 10 nearest neighbors sets scales, opacity starts at 0.1, and colors become degree-zero spherical-harmonic coefficients. Before rendering a training view, the corresponding inverse deformation transforms Gaussian positions and rotations, scales are handled with the rigid inverse transform as described in the paper, and 2DGS rasterization with alpha blending produces the image. The following is explanatory notation for the computational path in Eqs. (10)-(13), not an additional loss definition from the paper; \(\mathcal{G}\) denotes the canonical Gaussian scene, \(D_i^{-1}\) the inverse transform for that frame, and \(\mathcal{R}\) rendering:

\[ \hat I_i=\mathcal{R}\!\left(D_i^{-1}(\mathcal{G})\right). \]

Image L1 and LPIPS losses, together with 2DGS depth and normal regularizers, back-propagate through inverse deformation to optimize the canonical Gaussians and cameras. Training can consequently explain an observation in which a window is displaced, while novel-view exploration renders the canonical scene without that per-frame drift. Unlike dynamic reconstruction, the deformation is an auxiliary explanation of inconsistent observations, not real motion that the final world should replay.

A Worked Example

Consider a generated video moving around a house in which the same window changes position between frames. This illustrates the mechanism and is not an additional experimental result. DA3 exposes the discrepancy as two window-shaped point layers; reliable-point filtering removes low-confidence and sparsely supported points while retaining both credible observations. When displacement exceeds the ICP threshold, RoMa correspondences first provide attraction between the layers, local non-rigid deformation merges them, and global refinement tightens the surface. During Gaussian training, each frame's inverse deformation temporarily moves the unified window back to its observed position before image comparison; exploration renders the unified window instead of an averaged, blurry boundary.

Loss & Training

The frame-to-model objective combines geometric ICP, colored ICP, sparse correspondences, and deformation smoothness; this combination is explicitly given in original Section 3.2:

\[ E=\mathcal{L}_{\mathrm{data}}+0.05\mathcal{L}_{\mathrm{color}} +1.0\mathcal{L}_{\mathrm{corr}}+10.0\mathcal{L}_{\mathrm{tv}}. \]

Section 4.1 uses uniform settings throughout: voxel sizes of 4 cm and 2 cm, nearest-neighbor distance limits of 5 cm and 3 cm, 50 and 150 iterations respectively, and an Adam learning rate of \(10^{-3}\). The local confidence percentile is 15 and the voxel occupancy percentile is 50; the RoMa setup uses up to 20 image pairs and up to 5,000 correspondences. Global refinement runs for another 100 iterations with anchor weight 50.0; for 50 frames, the two geometry stages average 25 minutes and 20 GB of GPU memory on one A6000. The point cloud is subsampled to approximately 1.5M Gaussians; 5,000 optimization iterations without densification take approximately 10 minutes and 8 GB of GPU memory. Specular scenes can receive another 10,000 iterations of higher-degree spherical-harmonic optimization, freezing cameras and positions to avoid encoding geometric drift as view-dependent appearance.

Key Experimental Results

Main Results

Original Table 1 samples 50 frames per video and averages across source models: Wan-2.2, ViewCrafter, Gen3C, SEVA, Voyager, Genie3, and HY-WorldPlay. All entries below are scores with higher values preferred, not percentage accuracies or geometric errors in centimeters. Consistency and fidelity follow WorldScore; the cached main paper does not expand its underlying metric formulas, so these scores should not be interpreted as absolute geometric accuracy.

Method (original Table 1)	3D Consistency	Photometric Consistency	CLIP-IQA+	CLIP Aesthetic
DA3	69.53	71.62	31.64	36.64
3DGS-MCMC	67.34	69.58	35.56	35.06
VGGT-X	65.73	65.58	38.21	37.24
VGGT-X†	69.66	67.05	41.44	37.53
Ours	79.29	86.59	46.56	37.61

The DA3 baseline is not a raw depth-prediction score: it uses DA3 points to initialize 3DGS with a DA3 depth constraint; VGGT-X† substitutes DA3 for its original geometry predictions. Relative to VGGT-X†, the first three metrics improve by 9.63, 19.54, and 5.12 points; the aesthetic gain is only 0.08 points, so not every appearance metric improves substantially. Input-video CLIP-IQA+ and CLIP Aesthetic scores are 47.39 and 39.04, still above the reconstructed results of 46.56 and 37.61.

Ablation Study

Original Table 2 averages over four SEVA scenes, a different evaluation set from the cross-model average in Table 1.

Config (original Table 2)	3D Consistency	Photometric Consistency	CLIP-IQA+	CLIP Aesthetic
only rigid: rigid alignment only	76.98	64.93	11.43	49.12
no inv: no inverse-deformation rendering	78.28	55.16	15.81	46.39
no filt: no point filtering	72.58	64.88	55.17	51.05
no corr: no sparse correspondences	65.62	74.76	44.81	46.27
no global: no global refinement	63.90	77.85	49.69	51.61
Full model	79.79	86.88	55.30	52.40

Key Findings

• Removing inverse-deformation rendering reduces photometric consistency from 86.88 to 55.16, a 31.72-point drop; geometric pre-alignment cannot replace an appropriate image-supervision path.
• Removing global refinement reduces 3D consistency from 79.79 to 63.90, a 15.89-point drop; successful sequential alignment does not guarantee globally thin surfaces.
• Rigid-only alignment obtains 11.43 CLIP-IQA+, versus 55.30 for the full model; camera-extrinsic correction alone cannot explain local object deformation.
• Large-scale experiments use up to 32 SEVA video sequences and compare with WorldExplorer and VGGT-X; Figure 7 shows more stable novel views, but the main paper provides no corresponding quantitative table.

Highlights & Insights

• Model canonical geometry separately from the way each observation deviates from it. Preserving deformation mappings after alignment prevents subsequent photometric training from undoing geometric repairs.
• RoMa correspondences and ICP are complementary rather than redundant. The former supplies attraction beyond the nearest-neighbor threshold under large drift, while the latter provides precise local surface alignment.
• Dense initialization enables 2DGS training without densification. Computation shifts toward preliminary geometric registration, so the final 10 minutes should not be reported as the total cost.

Limitations & Future Work

• The authors distinguish geometric drift from content hallucination: displacement and deformation can be registered, but objects appearing or disappearing and textures changing on revisits may have no valid common correspondence.
• Per-scene optimization remains necessary; the reported geometry and basic Gaussian stages sum to approximately 35 minutes, excluding video generation and optional spherical-harmonic training, so reconstruction is not end-to-end real time.
• The target is a static scene, with non-rigid transforms compensating errors. The results do not establish physically correct dynamic interaction or recovery of real motion.
• As an evaluation limitation, the main paper reports ablations on only four SEVA scenes and does not detail the WorldScore implementation; large-scene evidence is mainly qualitative, without confidence intervals.
• The authors suggest robust mechanisms to identify hallucinated frames or using alignments as fine-tuning signals for generators; further work should also test failure detection under low overlap and severe content replacement.

Related Work & Insights

• vs VGGT-X: rigid correspondence alignment improves cameras and reconstruction; this method additionally models spatially varying deformation to address generated-video errors that no single camera transform can explain.
• vs WorldExplorer: its focus is autoregressive expansion of navigable scenes; this paper uses its progressive expansion strategy for multiple videos but contributes primarily to reconstruction and supervision.
• vs DynamicFusion / dynamic NeRF: both use canonical space and deformation fields, but here deformation is an observation disturbance to remove from a static result rather than genuine dynamics.
• vs generation-side consistency fine-tuning: this method reuses multiple video sources without retraining their diffusion models; the trade-off is additional optimization for every scene, not cost-free universal post-processing.

Rating

• Novelty: 4/5. Non-rigid registration is established, but inverse deformation in Gaussian supervision directly addresses inconsistent generated views.
• Experimental Thoroughness: 4/5. Multiple generators and key-component ablations provide useful evidence, while large-scene quantification and error statistics remain limited.
• Writing Quality: 4/5. The relationship between canonical space, observation space, and ablations is clear; more metric and inverse-transform implementation detail would improve reproducibility.
• Value: 4/5. A practical reconstruction route for existing video models, though content hallucination and per-scene optimization cost remain unresolved.