Raymap-Guided Coupling for Drift-Robust Unposed Feed-Forward 3D Reconstruction¶
Conference: ECCV 2026
Paper: ECCV Official Link
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Feed-Forward 3D Reconstruction, Pose Drift, Raymap-Guided Coupling, Dual-Frequency Scheduling
TL;DR¶
To resolve the critical problem of cumulative camera pose drift that causes geometric collapse and blurry rendering in long-sequence unposed feed-forward 3D Gaussian Splatting, NoDrift3R explicitly anchors 3D Gaussian centers to dense raymap-and-depth geometry, establishes a bidirectional synergistic "Rendering-to-Geometry Gain" loop, and integrates a dual-frequency viewpoint scheduler to achieve drift-robust, globally consistent 3D reconstruction across extended trajectories.
Background & Motivation¶
Feed-forward 3D Gaussian Splatting (3DGS) has rapidly emerged as a dominant paradigm for real-time 3D scene reconstruction and novel view synthesis by predicting explicit scene representations directly via a single neural network forward pass, bypassing time-consuming per-scene test-time optimization. As research transitions from calibrated multi-view settings with known camera parameters toward practical pose-free scenarios, modern models aim to simultaneously predict camera poses and 3D scene representations directly from uncalibrated input images. However, when extending to longer image trajectories or wider baselines, existing pose-free feed-forward models suffer severe performance degradation. This vulnerability stems from an acute, bidirectional coupling between predicted camera poses and scene representation: minor inaccuracies in camera rotation and translation immediately distort 3D spatial geometry, while defective scene structures feed back into subsequent pose regressions, creating cumulative pose drift that compounds over time and culminates in ghosting artifacts, severe motion blur, and structural dislocation in long-sequence reconstructions.
A deeper examination of this training instability reveals two fundamental bottlenecks in existing pose-free paradigms. First, Structure-from-Motion (SfM) pseudo ground-truth camera parameters inherently carry matching noise and sensor imperfections, whereas completely unconstrained rendering-based self-supervision (e.g., NoPoSplat, Uni3R) leads to volatile optimization and sub-optimal local minima due to the under-constrained joint search over both camera poses and ungrounded 3D geometry. Furthermore, prior conditioning strategies (e.g., YoNoSplat) that concatenate low-dimensional extrinsic vectors into the Gaussian attribute head create a severe geometric information bottleneck, failing to provide the dense per-pixel spatial cues needed to anchor Gaussians accurately. Second, existing curriculum-based training schedules that progressively enlarge view baselines based on visual overlap (such as E-RayZer) suffer from catastrophic forgetting: as the view interval widens linearly during training, the network loses its grasp on small-interval, high-overlap frame pairs, destroying local geometric consistency and destabilizing training across wide temporal ranges.
To escape this vicious cycle of pose drift and optimization collapse, the primary angle of attack is to tightly intertwine appearance rendering supervision with dense ray-based geometric guidance. Core idea: anchor 3D Gaussian centers deterministically to dense per-pixel raymaps and depth predictions, establishing a bidirectional "Rendering-to-Geometry Gain" optimization loop where appearance reconstruction refines camera geometry and raymap constraints regularize Gaussian distributions, complemented by a dual-frequency viewpoint scheduling strategy that balances progressive wide-interval expansion with stochastic small-interval replay.
Method¶
Overall Architecture¶
NoDrift3R adopts an end-to-end feed-forward pipeline built upon a standard Vision Transformer backbone (vanilla DINOv2 initialized with Depth Anything v3 weights) without modifying the internal transformer blocks. Given an unposed multi-view image sequence, the model first applies lightweight patch embeddings to produce per-view tokens, which pass through the multi-view transformer to facilitate cross-view geometric and contextual reasoning. Subsequently, a Dual-DPT head decodes two dense geometric signals for each view: a 6-channel raymap (specifying per-pixel ray directions and ray origins) and a scalar depth map, which together deterministically compute 3D Gaussian center coordinates. Concurrently, a Gaussian head predicts per-primitive opacity, scale, rotation quaternion, and spherical harmonics/color, while a camera head estimates camera extrinsics. During training, a unified objective simultaneously enforces RGB rendering reconstruction, camera parameter regularization, and raymap consistency, enabling bidirectional gradient flow between appearance and geometry.
The overall architecture and stage progression are illustrated in the flowchart below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Unposed Multi-View Image Sequence"] --> B["Dual-Frequency Viewpoint Scheduling<br/>Bounded overlap curriculum and small-interval replay"]
B --> C["Multi-View Feature Interaction Encoding<br/>Vanilla DINOv2 ViT backbone with cross-view attention"]
C --> D["Raymap-Guided Coupling Module<br/>Dual-DPT heads predict raymaps and depth to anchor Gaussian centers"]
D --> E["Synergistic Joint Optimization Objective<br/>Bidirectional feedback coupling appearance and raymap geometry"]
E --> F["Global 3D Gaussians and Refined Camera Poses"]
Key Designs¶
1. Dual-Frequency Viewpoint Scheduling: stabilizing geometric optimization via bounded interval expansion and stochastic replay To overcome the severe catastrophic forgetting observed in linear curriculum learning—where training on wide baselines degrades performance on dense, small-interval view pairs—this design establishes a dual-frequency sampling regimen. First, a visual overlap curriculum computes pairwise semantic similarity using pre-trained DINOv2 feature embeddings:
As training iterations \(t\) advance, the required overlap threshold anneals progressively from 1.0 down to 0.75, determining the candidate frame interval \(g_{\text{curr}}(t)\). Crucially, the maximum allowed interval is strictly capped at \(g_{\text{max}} = 15\) (\(g_{\text{overlap}}(t) = \min(g_{\text{curr}}(t), g_{\text{max}})\)) to avoid unmanageably sparse supervision that could destabilize early representations. Second, a stochastic replay mechanism intervenes during the second half of the training phase: with probability \(p_{\text{small}} = 0.5\), the scheduler randomly selects a small-interval view pair drawn from a uniform discrete distribution \(g_{\text{small}} \sim \mathcal{U}\{6, \dots, 10\}\). By interleaving wide-baseline exploratory pairs with dense, high-overlap revisits, the model maintains sharp local geometric consistency while scaling gracefully across extended trajectories.
2. Raymap-Guided Coupling Module: anchoring Gaussian centers to dense raymap and depth geometry To resolve the information bottleneck inherent in conditioning Gaussian prediction on low-dimensional camera extrinsic vectors, this module establishes a dense, per-pixel geometric foundation for 3D Gaussian primitives. For each view, the Dual-DPT head decodes a 6-channel raymap \(R \in \mathbb{R}^{H \times W \times 6}\) and a scalar depth map \(D \in \mathbb{R}^{H \times W}\). In the raymap, the first three channels represent the normalized ray direction \(r \in \mathbb{R}^{H \times W \times 3}\) and the remaining three channels denote the ray origin \(o \in \mathbb{R}^{H \times W \times 3}\). The spatial center position \(p_j \in \mathbb{R}^3\) of each pixel-aligned Gaussian primitive \(j\) is then deterministically computed via unprojection along the predicted ray:
Setting the Gaussian center \(\mu_j = p_j\) eliminates unconstrained 3D point drift and provides an explicit geometric anchor. The auxiliary Gaussian head is thus dedicated solely to predicting localized deformation and appearance parameters: opacity \(\alpha_j = \sigma(f_j^{\alpha})\), anisotropic scales \(s_j = \exp(f_j^s)\), unit rotation quaternions \(q_j = \text{normalize}(f_j^q)\), and color vectors \(c_j\). This rigid grounding binds Gaussian primitives directly to the dense visual geometry field.
3. Synergistic Joint Optimization Objective: establishing a bidirectional Rendering-to-Geometry Gain loop Instead of treating pose estimation, geometric regression, and radiance field synthesis as loosely connected sub-tasks, this design unifies them into a synergistic, closed-loop objective. Because Gaussian positions are directly derived from the depth map and raymap, gradients from the RGB rendering loss \(\mathcal{L}_{\text{rgb}}\) backpropagate through the differential 3D Gaussian splatting rasterizer directly into the underlying ray directions \(r_j\) and origins \(o_j\). This allows multi-view photometric and textural consistency to act as an implicit geometric regularizer that refines ray orientations and camera extrinsics. Concurrently, the explicit raymap loss \(\mathcal{L}_{\text{ray}}\) penalizes directional errors, imposing structural rigidity that prevents Gaussian primitives from collapsing into trivial, floating-artifact local optima. This bidirectional coupling—termed "Rendering-to-Geometry Gain"—ensures that geometry stabilizes appearance synthesis while photometric supervision simultaneously sharpens geometry and camera trajectory estimates.
A Worked Example¶
Consider reconstructing an unposed 24-frame continuous indoor trajectory: 1. Sampling & Scheduling: The input sequence is ingested by the dual-frequency scheduler. If the training iteration is in the latter half and triggers the \(p_{\text{small}} = 0.5\) branch, it randomly alternates between dense view intervals (e.g., 6 frames apart) and maximum-span frames (capped at 15 frames apart), maintaining both high visual overlap and long-baseline coverage. 2. Feature Encoding & Dual Geometry Prediction: Multi-view tokens interact across all views in the ViT backbone. The Dual-DPT head outputs a \(224 \times 224 \times 6\) raymap \(R\) (origins \(o\) and directions \(r\)) and a \(224 \times 224\) depth map \(D\) for each view. 3. Deterministic Lifting & Rasterization: For each of the \(224 \times 224 = 50,176\) pixels, its Gaussian center is anchored at \(p_j = o_j + D_j \cdot r_j\). The Gaussian head outputs scale, rotation, and opacity attributes, allowing the splatting rasterizer to render synthesized target views \(\hat{I}\). 4. Bidirectional Gradient Flow: The rendered views \(\hat{I}\) are compared against target frames \(I\) via MSE and LPIPS losses. The photometric gradients flow directly back into the raymaps and depth, refining pixel-level rays. Concurrently, predicted camera parameters \(\hat{\theta}\) and raymaps \(\hat{R}\) are penalized against pseudo-ground truth via Huber and absolute error losses, achieving rapid, drift-free convergence across all 24 frames.
Loss & Training¶
The overall training objective is formulated as a multi-task weighted combination:
The individual loss components are structured as follows: 1. RGB Reconstruction Loss: Combines pixel-level Mean Squared Error (MSE) and perceptual LPIPS loss across all \(N\) frames: $\(\mathcal{L}_{\text{rgb}} = \frac{1}{N} \sum_{i=1}^N \left( \lambda_{\text{mse}} \|I_i - \hat{I}_i\|_2^2 + \lambda_{\text{lpips}} \text{LPIPS}(I_i, \hat{I}_i) \right)\)$ 2. Camera Regularization Loss: Applies a Huber loss between predicted camera parameters \(\hat{\theta}_i\) and reference camera parameters \(\theta_i\): $\(\mathcal{L}_{\text{cam}} = \frac{1}{N} \sum_{i=1}^N \text{Huber}(\hat{\theta}_i, \theta_i)\)$ 3. Raymap Consistency Loss: Measures the pixel-wise mean absolute error between predicted raymaps \(\hat{R}_i\) and ground-truth raymaps \(R_i\): $\(\mathcal{L}_{\text{ray}} = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_{\text{abs}}(\hat{R}_i, R_i)\)$
Training Details and Compute Hardware: - Giant Model (used for primary SOTA benchmarking): Employs a ViT backbone with hidden dimension 1536 and 40 transformer blocks, initialized from Depth Anything v3 weights. Trained on 8 \(\times\) NVIDIA H100 GPUs for 150k steps with a per-GPU batch size of 1. - Large Model (used for all ablation experiments): Employs a ViT backbone with hidden dimension 1024 and 24 transformer blocks, initialized from Depth Anything v3 weights. Trained on 8 \(\times\) NVIDIA RTX 5090 GPUs for 150k steps with a per-GPU batch size of 1. All inputs are resized to \(224 \times 224\) resolution.
Key Experimental Results¶
Main Results¶
The model was evaluated on the large-scale DL3DV benchmark across varying input view counts (6, 12, and 24 views) for both novel view synthesis and camera pose estimation.
Table 1: Novel view synthesis comparison across varying input view settings on the DL3DV benchmark (\(p\): ground-truth poses; \(k\): ground-truth intrinsics)
| Method | Venue | Pose \(p\) | Intrinsics \(k\) | 6v PSNR↑ | 6v SSIM↑ | 6v LPIPS↓ | 12v PSNR↑ | 12v SSIM↑ | 12v LPIPS↓ | 24v PSNR↑ | 24v SSIM↑ | 24v LPIPS↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MVSplat | ECCV 2024 | ✓ | ✓ | 22.659 | 0.760 | 0.173 | 21.289 | 0.709 | 0.224 | 19.975 | 0.662 | 0.269 |
| DepthSplat | CVPR 2025 | ✓ | ✓ | 23.418 | 0.797 | 0.136 | 21.911 | 0.753 | 0.179 | 20.088 | 0.690 | 0.240 |
| NoPoSplat | ICLR 2025 | ✗ | ✓ | 22.766 | 0.743 | 0.179 | 19.380 | 0.563 | 0.318 | 17.860 | 0.495 | 0.397 |
| AnySplat | SA 2025 (TOG) | ✗ | ✗ | 19.027 | 0.554 | 0.235 | 18.940 | 0.549 | 0.262 | 19.703 | 0.596 | 0.249 |
| YoNoSplat | ICLR 2026 | ✗ | ✗ | 24.531 | 0.804 | 0.142 | 22.933 | 0.746 | 0.187 | 22.174 | 0.720 | 0.209 |
| E-RayZer | CVPR 2026 | ✗ | ✗ | 24.814 | 0.791 | 0.184 | 20.454 | 0.639 | 0.317 | 18.750 | 0.571 | 0.406 |
| NoDrift3R (Ours) | ECCV 2026 | ✗ | ✗ | 24.922 | 0.826 | 0.127 | 24.250 | 0.797 | 0.141 | 24.242 | 0.794 | 0.142 |
Table 2: Camera pose estimation comparison on the DL3DV dataset (\(224 \times 224\) resolution)
| Model | 6v AUC@5°↑ | 6v AUC@10°↑ | 6v AUC@20°↑ | 12v AUC@5°↑ | 12v AUC@10°↑ | 12v AUC@20°↑ | 24v AUC@5°↑ | 24v AUC@10°↑ | 24v AUC@20°↑ |
|---|---|---|---|---|---|---|---|---|---|
| VGGT (518×280) | 0.700 | 0.848 | 0.924 | - | - | - | - | - | - |
| \(\pi^3\) (518×280) | 0.795 | 0.897 | 0.949 | - | - | - | - | - | - |
| NoPoSplat (256×256) | 0.538 | 0.735 | 0.853 | - | - | - | - | - | - |
| AnySplat (448×448) | 0.596 | 0.776 | 0.884 | 0.517 | 0.732 | 0.864 | 0.476 | 0.708 | 0.851 |
| YoNoSplat (224×224) | 0.833 | 0.917 | 0.958 | 0.804 | 0.902 | 0.951 | 0.778 | 0.885 | 0.942 |
| E-RayZer (256×256) | 0.846 | 0.972 | 0.983 | 0.421 | 0.699 | 0.767 | 0.389 | 0.654 | 0.707 |
| NoDrift3R (Ours) | 0.967 | 0.983 | 0.992 | 0.961 | 0.979 | 0.990 | 0.949 | 0.972 | 0.985 |
Ablation Study¶
To dissect the contribution of each component within the Raymap-Guided Coupling (RGC) module and the Dual-Frequency scheduling framework, ablations were conducted on the Large model variant.
Table 3: Loss ablation across different input view counts on the Large model
| Configuration | 6v PSNR↑ | 6v SSIM↑ | 6v AUC@5°↑ | 6v AUC@10°↑ | 12v PSNR↑ | 12v SSIM↑ | 12v AUC@5°↑ | 12v AUC@10°↑ | 24v PSNR↑ | 24v SSIM↑ | 24v AUC@5°↑ | 24v AUC@10°↑ | Description |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| w/o raymap-guided | 23.062 | 0.752 | 0.821 | 0.907 | 22.375 | 0.717 | 0.753 | 0.867 | 21.379 | 0.599 | 0.690 | 0.821 | Predict geometry purely from camera head poses |
| w/o raymap loss | 21.869 | 0.708 | 0.869 | 0.927 | 20.933 | 0.655 | 0.814 | 0.902 | 20.609 | 0.635 | 0.782 | 0.875 | Remove explicit raymap loss constraint |
| w/o rgb loss | 12.608 | 0.321 | 0.851 | 0.922 | 12.375 | 0.325 | 0.801 | 0.891 | 12.378 | 0.328 | 0.754 | 0.851 | Remove photometric appearance supervision |
| Ours (Large) | 23.302 | 0.766 | 0.874 | 0.935 | 21.909 | 0.696 | 0.829 | 0.907 | 21.626 | 0.675 | 0.787 | 0.883 | Complete bidirectional synergistic framework |
Table 4: Viewpoint sampling strategy, upper-bound clipping \(g_{\text{max}}\), and replay ablation (6-view input)
| Scheduler Config | Range 0–50 PSNR↑ | Range 0–50 SSIM↑ | Range 0–50 AUC@5°↑ | Range 0–50 AUC@10°↑ | Range 0–150 PSNR↑ | Range 0–150 SSIM↑ | Range 0–150 AUC@5°↑ | Range 0–150 AUC@10°↑ | Analysis |
|---|---|---|---|---|---|---|---|---|---|
| original sampler | 22.696 | 0.747 | 0.782 | 0.884 | 18.815 | 0.571 | 0.591 | 0.744 | Standard random interval sampler |
| \(g_{\text{max}}=10\), w/o replay | 23.302 | 0.766 | 0.874 | 0.935 | 19.542 | 0.579 | 0.687 | 0.803 | Restricted to short range, weak long-range expansion |
| \(g_{\text{max}}=20\), w/o replay | 22.892 | 0.752 | 0.872 | 0.935 | 19.690 | 0.592 | 0.709 | 0.826 | Wider interval boosts long range but drops short range |
| \(g_{\text{max}}=\infty\), w/o replay | 22.744 | 0.742 | 0.862 | 0.930 | 19.712 | 0.588 | 0.722 | 0.832 | Unbounded annealing leads to short-range degradation |
| \(g_{\text{max}}=\infty\), replay 50% | 23.006 | 0.745 | 0.864 | 0.929 | 19.745 | 0.592 | 0.721 | 0.832 | Small-interval replay recovers short-range fidelity |
Key Findings¶
- Drift Suppression in Long Sequences: As input sequence length scales from 6 to 24 views, competing pose-free methods degrade drastically: YoNoSplat drops by 2.357 dB in PSNR, while E-RayZer suffers a catastrophic drop of 6.064 dB and an AUC@5° plunge to 0.389. In contrast, NoDrift3R maintains an almost flat PSNR curve (from 24.922 dB down to 24.242 dB, a minor change of 0.68 dB), while preserving a camera pose AUC@5° of 0.949 at 24 views, outperforming YoNoSplat by 2.068 dB PSNR.
- Empirical Proof of Bidirectional Synergy: The loss ablation in Table 3 demonstrates the mutual dependence between geometry and appearance. Removing the raymap loss (w/o raymap loss) leads to a substantial decrease in novel view synthesis quality (24v PSNR drops from 21.626 dB to 20.609 dB), proving that explicit ray constraints regularize Gaussian distributions. Conversely, eliminating RGB rendering loss (w/o rgb loss) causes camera pose estimation accuracy to plummet (24v AUC@5° drops from 0.787 to 0.754), conclusively showing that photometric appearance consistency serves as a vital geometric prior.
- Superior Cross-Dataset Generalization: When trained on DL3DV and tested zero-shot on ScanNet++ up to 128 views, NoDrift3R achieves 19.714 dB PSNR compared to YoNoSplat's 17.641 dB (a +2.073 dB margin). In zero-shot pose estimation on RE10K, NoDrift3R attains an AUC@5° of 0.755, outperforming geometry foundation models such as MASt3R (0.609) and VGGT (0.566).
Highlights & Insights¶
- Dense Raymaps as a Geometric-Appearance Highway: Rather than relying on low-dimensional camera pose vectors that choke spatial information flow, predicting 6-channel raymaps directly anchors each Gaussian center at \(o_j + D_j \cdot r_j\), allowing photometric gradients from 3DGS rasterization to propagate directly into the underlying camera ray field.
- Breaking the Pose-Geometry Ambiguity in Pose-Free Learning: The synergistic "Rendering-to-Geometry Gain" formulation demonstrates that joint optimization does not require convoluted auxiliary objectives; a streamlined combination of RGB reconstruction, raymap consistency, and camera regularization achieves mutual stabilization.
- Dual-Frequency Scheduling Resolves Curriculum Forgetting: Combining progressive visual-overlap annealing with a 50% stochastic replay of short-interval view pairs effectively prevents catastrophic forgetting of dense local geometry during wide-baseline training.
Limitations & Future Work¶
- Supervisory Dependence during Training: While the model is completely pose-free during inference, training still relies on precomputed SfM pseudo-ground-truth camera poses and dense raymap targets. Extending the paradigm to self-supervised or unposed training data remains an open challenge.
- Static Scheduling Hyperparameters: The visual overlap threshold annealing schedule (1.0 to 0.75) and interval clipping parameter (\(g_{\text{max}} = 15\)) are fixed across diverse datasets. Incorporating adaptive, scene-dependent scheduling based on trajectory speed and visual complexity would be a promising direction.
Related Work & Insights¶
- vs YoNoSplat (ICLR 2026): YoNoSplat relies on teacher-forcing mixed training and low-dimensional camera pose conditioning, which leads to cumulative pose drift on long trajectories (AUC@5° drops to 0.778 on 24v); NoDrift3R anchors Gaussians directly to dense raymaps, sustaining an AUC@5° of 0.949 and a 2.068 dB PSNR lead.
- vs E-RayZer (CVPR 2026): E-RayZer introduces curriculum learning based on linear visual overlap extension but degrades on short-interval evaluation and collapses under long sequences; NoDrift3R introduces dual-frequency replay and bounded clipping, ensuring robustness across arbitrary intervals.
- vs NoPoSplat (ICLR 2025) & AnySplat (SA 2025 TOG): These unconstrained pipelines suffer from severe pose ambiguity and local minima due to the lack of rigid geometric grounding; NoDrift3R demonstrates that explicit raymap-guided coupling is essential for stable, drift-robust feed-forward reconstruction.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the "Rendering-to-Geometry Gain" loop by explicitly anchoring Gaussians to dense raymaps and designs a dual-frequency replay scheduler to suppress cumulative pose drift.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations from 6 to 128 views across DL3DV, RE10K, and ScanNet++, backed by clear quantitative ablations on loss synergy and sampling strategies.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative that clearly identifies the long-sequence drift bottleneck and articulates the theoretical motivation behind the proposed solutions.
- Value: ⭐⭐⭐⭐⭐ Provides an impactful, drift-robust blueprint for scaling unposed feed-forward 3D Gaussian Splatting to long-sequence, large-scale scene reconstruction.