Déjà View: Looping Transformers for Multi-View 3D Reconstruction

Conference: NeurIPS 2026
arXiv: 2605.30215
OpenReview: https://openreview.net/forum?id=Eidz3t6qmk
Code: https://research.nvidia.com/labs/dvl/projects/dvlt/
Area: 3D Reconstruction & Rendering Keywords: Multi-View 3D Reconstruction, Looping Transformer, Weight Sharing, Inference-Time Compute Knob, Pose and Depth Estimation

TL;DR

DéjàView introduces an explicit looping transformer architecture for multi-view 3D reconstruction that iteratively refines per-view representations using a single shared block; with only 117M parameters, it matches or surpasses billion-parameter feed-forward baselines while offering an inference-time compute knob.

Background & Motivation

Recovering 3D scene geometry and camera poses from multiple uncalibrated images has traditionally relied on Structure-from-Motion (SfM) pipelines. By alternating between feature extraction, matching, pose estimation, triangulation, and global bundle adjustment, classical SfM is intrinsically iterative; however, it remains brittle in textureless regions, under wide baselines, or across complex in-the-wild environments. Recent feed-forward reconstruction architectures, notably DUSt3R, VGGT, Pi3, and Depth Anything 3, replace this pipeline with an end-to-end transformer that directly regresses 3D structure in a single forward pass. Their reconstruction gains have followed the general trend of computer vision: scaling model capacity through wider and deeper backbones, with parameter counts surging into the 1B+ regime.

Yet this iteration has not truly disappeared—it has merely been implicitly absorbed into network depth. Probing studies and mechanistic analyses (such as RAPTOR in vision transformers and layer-by-layer probing in DUSt3R) reveal that contiguous transformer layers execute remarkably similar transformations, while decoder depth progressively refines predicted pointmaps into sharper geometry. Relying on independently parameterized layers to achieve progressive refinement introduces severe parameter redundancy and permanently fixes the inference compute budget at deployment.

This paper's angle of attack is that if the primary benefit of depth in geometric transformers stems from iteration, that iteration should be made explicit in the architecture rather than simulated inefficiently with billions of unique parameters. Core idea: recurrently apply a single shared transformer block conditioned on continuous time to DINOv2-initialized multi-view features, training with variable step counts to expose an inference-time compute knob and outperforming 1B-scale feed-forward baselines with only 117M parameters.

Method

Overall Architecture

DéjàView adopts a recurrent refinement pipeline comprising three stages: single-view patch encoding, shared-block iterative refinement, and dual-branch geometric decoding. Given \(V\) input RGB images \(\{\mathbf{I}_i\}_{i=1}^V\), the model first extracts patch tokens using a frozen/pretrained DINOv2 ViT-B backbone, prepending learnable register tokens and camera tokens to form the initial multi-view state \(\mathbf{z}_0\). Next, a single looped transformer block is recurrently applied for \(K\) steps under continuous-time interval conditioning, driving the residual stream through directional geometric refinement. Finally, parallel decoder heads map the refined state \(\mathbf{z}_K\) into dense depth maps, uncertainty maps, and pixel-wise ray maps, analytically reconstructing consistent global 3D point clouds and camera poses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view image input<br/>V RGB images"] --> B["DINOv2 patch encoding<br/>prepend register and camera tokens z0"]
    B --> C["Shared looped block with continuous-time modulation<br/>alternating frame and global attention"]
    C --> D["Variable-K training & inference compute knob<br/>train with K ∈ [8,16], inference budget on demand"]
    D --> E["Directional refinement in residual stream<br/>cosine alignment while norm growth is absorbed by LN"]
    E --> F["Dual-branch geometric decoding<br/>ray branch + convolutional depth branch"]
    F --> G["Output: global 3D point clouds & camera poses<br/>analytical points X = R_origin + D · R_dir"]

Key Designs

1. Shared looped block with continuous-time modulation: eliminating parameter redundancy and enabling arbitrary step interpolation Prior multi-view reconstruction models dedicate unique attention and feed-forward weights to every layer, scaling parameters linearly with depth. DéjàView constrains its iterative backbone to a single shared transformer block that alternates between single-view frame attention (equipped with 2D RoPE) and multi-view global cross-attention. To enable this single block to discern whether the current step is handling coarse global camera alignment or fine-grained local surface adjustment, the architecture incorporates continuous-time interval conditioning. The refinement process is parameterized over a partition of the unit interval \(0=t_0<t_1<\dots<t_K=1\), updating the state via \(\mathbf{z}_{k+1} = f_\theta(\mathbf{z}_k, t_k, t_{k+1})\). The block applies three channel-wise scale vectors \((\mathbf{s}_{\text{attn}}, \mathbf{s}_{\text{mlp}}, \mathbf{s}_{\text{out}})\) across residual pathways: $$ \mathbf{z}' = \mathbf{z}k + \mathbf{s}}} \odot \text{LS}_1(\text{Attn}(\text{LN}_1(\mathbf{zk))), \quad \mathbf{z}'' = \mathbf{z}' + \mathbf{s}}} \odot \text{LS2(\text{MLP}(\text{LN}2(\mathbf{z}'))), \quad \mathbf{z}} = \mathbf{s'' $$ where scale vectors are generated by a zero-initialized MLP from concatenated sinusoidal embeddings of }} \odot \mathbf{z\(t_k\) and \(t_{k+1}\): \(\mathbf{s} = \mathbf{1} + \text{MLP}(\gamma(t_k, t_{k+1}))\). Unlike discrete step indices, continuous-time conditioning decouples weights from any fixed step count \(K\) and allows smooth interpolation across arbitrary step budgets at test time.

2. Variable-K training and inference compute knob: single checkpoint supporting multiple compute budgets Standard feed-forward networks have rigid computational costs, while training separate models for different step budgets is computationally prohibitive. DéjàView introduces a variable-\(K\) training recipe: during training, step count \(K\) is sampled per batch from a Beta distribution \(K \sim \text{Beta}(2, 1)\) scaled and rounded to \([K_{\text{min}}, K_{\text{max}}] = [8, 16]\), running uniform intervals \(t_k = k/K\) and supervising only at the final step \(\mathbf{z}_K\). This eliminates the intermediate decoding overhead typical of RAFT-style sequence losses (saving \((K-1)\) decoder forward and backward passes per training iteration). At inference time, the single trained checkpoint exposes \(K_{\text{inf}}\) as an adjustable compute knob: users can pick \(K_{\text{inf}}=12\) for low-latency edge deployment or \(K_{\text{inf}}=16\) for maximal reconstruction fidelity without any retraining.

3. Directional refinement dynamics and dual-branch geometric decoding: representation alignment and artifact elimination Analyzing the residual stream dynamics reveals that the recurrence does not contract to a fixed point in feature space as in deep equilibrium networks (DEQs); instead, the state norm \(\|\mathbf{z}_k\|_2\) grows monotonically. However, the dynamics exhibit clear directional refinement: cosine similarity \(\cos(\mathbf{z}_k, \mathbf{z}_K)\) monotonically approaches 1.0, and the relative update norm \(\|\Delta \mathbf{z}_k\| / \|\mathbf{z}_k\|\) decays from \(\sim 0.5\) at step 1 to \(\sim 0.1\) at step 16. The pre-norm LayerNorm at the entrance of each decoder branch absorbs the parallel norm growth component, allowing the decoded representation to converge stably in direction. In the decoding stage, the final tokens \(\mathbf{z}_K\) feed into two parallel branches: a ray decoder outputting pixel-wise ray origins and directions \(\mathbf{R}_\theta \in \mathbb{R}^{H \times W \times 6}\) via shallow attention and pixel shuffle, and a convolutional depth decoder outputting depth \(\mathbf{D}_\theta\) and confidence \(c_\mathbf{D}\). The convolutional depth head avoids patch-boundary blocking artifacts, and points are derived analytically via \(\mathbf{X}_\theta = \mathbf{R}_\theta^o + \mathbf{D}_\theta \cdot \mathbf{R}_\theta^d\).

Loss & Training

Optimization follows a two-stage schedule. The first stage trains end-to-end for 200K iterations across all five geometric loss terms: $$ \mathcal{L} = |\hat{s}\mathbf{D}\theta - \bar{s}\mathbf{D}|2 + \mathcal{L}}}(\hat{s}\mathbf{D\theta, \bar{s}\mathbf{D}) + |\hat{s}\mathbf{R}\theta - \bar{s}\mathbf{R}|1 + |\hat{s}\mathbf{X}\theta - \bar{s}\mathbf{X}|2 + \mathcal{L}}}(\mathbf{c\theta, \mathbf{c}) $$ where \(\hat{s}\) and \(\bar{s}\) are scale factors computed from inverse mean distance to normalize predicted and ground-truth coordinates; \(\mathcal{L}_{\text{grad}}\) enforces multi-scale depth gradients on synthetic data; \(\mathcal{L}_{\text{cam}}\) applies \(\ell_1\) losses with weights \((1, 1, 0.5)\) across translation, rotation, and field of view. The second stage attaches the convolutional depth head and finetunes the depth decoder for 40K steps with frozen backbone and ray heads, supervising with uncertainty-weighted regression \(\mathcal{L}_{\text{unc}}(c, \mathbf{a}, \mathbf{b}) = c\|\mathbf{a}-\mathbf{b}\|_2 - \lambda_c \log c\) (\(\lambda_c=0.2\)).

Key Experimental Results

Main Results

The model was evaluated across five diverse benchmarks covering object-centric scenes (DTU), outdoor wide-baseline sequences (ETH3D), indoor environments (7-Scenes, ScanNet++), and autonomous driving (nuScenes).

Dataset	Metric	Ours (DéjàView)	Pi3 (2025)	VGGT-Ω-1B (2026)	DA3-G (2025)	Note
DTU	Rel. L2 ↓ IR (%) ↑ AUC@3° ↑ AUC@30° ↑	0.009 97.1 83.2 98.8	0.009 97.3 70.2 97.3	0.009 97.2 77.4 98.1	0.010 97.1 74.9 97.9	Tied for lowest Rel. L2; leads in wide-angle pose AUC@30 at 98.8%
ETH3D	Rel. L2 ↓ IR (%) ↑ AUC@3° ↑ AUC@30° ↑	0.026 78.3 66.0 95.4	0.034 66.8 43.2 84.9	0.024 78.6 64.1 95.4	0.129 64.7 55.4 82.4	Outperforms all baselines in tight pose AUC@3 (66.0%) in outdoor environments
7-Scenes	Rel. L2 ↓ IR (%) ↑ AUC@3° ↑ AUC@30° ↑	0.035 74.2 13.9 81.7	0.032 77.8 11.3 81.3	0.039 65.3 21.3 87.0	0.037 71.1 12.0 80.9	High inlier ratio (74.2%), surpassing the 1B VGGT-Ω baseline by 8.9% IR
ScanNet++	Rel. L2 ↓ IR (%) ↑ AUC@3° ↑ AUC@30° ↑	0.015 93.3 79.4 98.0	0.014 94.3 76.5 97.3	0.032 70.6 29.9 87.3	0.041 57.8 26.4 80.2	AUC@3 reaches 79.4%, outperforming Pi3 and beating VGGT-Ω by 49.5 points
nuScenes	Rel. L2 ↓ IR (%) ↑ AUC@3° ↑ AUC@30° ↑	0.067 58.5 43.4 85.3	0.078 51.0 24.8 82.4	0.055 62.3 42.4 85.5	0.080 42.0 37.7 82.0	Best pose accuracy under surround-view driving cameras (AUC@3 = 43.4%)
5-Benchmark Overall Efficiency	Params (M) ↓ Compute (TFLOPs) ↓ Peak Mem (GiB) ↓ Mean IR (%) ↑ Mean AUC@30 ↑	117 75.9 4.9 80.3 91.8	959 153.8 6.6 77.4 88.6	1144 99.8 7.9 74.8 90.7	1201 178.7 13.0 66.5 84.7	At 117M parameters (~1/10th of baselines), achieves highest average IR (80.3%) and AUC@30 (91.8%)

Ablation Study

1. Looped block design components (averaged across 5 benchmarks):

Config	Weight sharing	Residual gates (s_attn, s_mlp)	State gate (s_out)	Rel. L2 ↓	IR (%) ↑	AUC@3° (%) ↑	AUC@30° (%) ↑	Note
Decoupled	✗	✗	✗	0.056	61.1	23.0	82.0	16 independent unshared transformer blocks
Shared	✓	✗	✗	0.045	66.4	30.2	84.8	Weight sharing alone drops error by 20% with 16× fewer parameters
Shared + residual gates	✓	✓	✗	0.042	67.0	31.5	85.9	Attention and MLP continuous-time modulation
Shared + state gate (Full model)	✓	✓	✓	0.040	69.2	33.3	86.9	Complete time-conditioned architecture achieves optimal metrics across the board

2. Step-count training strategies and inference compute knob:

Scheme	Training step sampler	Inference \(K_{\text{inf}}\)	Rel. L2 ↓	IR (%) ↑	AUC@3° (%) ↑	AUC@30° (%) ↑	Observation
Fixed \(K=12\)	fixed 12	12	0.044	67.6	30.4	85.5	Model trained exclusively for 12 steps
Ours (Variable)	Beta on [8, 16]	12	0.043	66.7	29.6	85.4	Variable checkpoint evaluated at 12 steps matches dedicated model within <1%
Fixed \(K=16\)	fixed 16	16	0.041	69.6	33.8	86.8	Model trained exclusively for 16 steps
Ours (Variable)	Beta on [8, 16]	16	0.040	69.2	33.3	86.9	Variable checkpoint at 16 steps matches or edges fixed model at zero extra cost

Key Findings

• Shared weights provide a stronger inductive bias than capacity scaling: Under identical training data and compute budgets, a single weight-shared looping block (16× fewer parameters) outperforms 16 independent unshared layers, reducing Rel. L2 error from 0.056 to 0.045 and boosting AUC@3 from 23.0% to 30.2%. Explicit iteration is inherently better aligned with multi-view geometry estimation than unshared depth scaling.
• Monotonic improvement across refinement steps: Per-iteration probing demonstrates that geometry and pose metrics improve monotonically from step 1 to 16. Residual updates slow down rapidly (\(\|\Delta \mathbf{z}_k\|/\|\mathbf{z}_k\|\) falls from \(\sim 0.5\) to \(\sim 0.1\)), indicating smooth convergence in direction.
• Elastic step counts enable flexible compute allocation: The variable-\(K\) checkpoint allows dropping inference steps from 16 to 12 with minimal metric impact (<1% AUC), immediately saving 25% of recurrent transformer compute without needing distinct checkpoints.

Highlights & Insights

• Looping blocks as an alternative to parameter scaling: Rather than chasing billion-parameter models, DéjàView shows that recurrently executing a single shared block across a residual stream achieves top-tier reconstruction quality with only 117M parameters and <5 GiB peak memory.
• Continuous-time gating for seamless interpolation: Modulating attention and MLP branches via sinusoidal interval embeddings \((\mathbf{s}_{\text{attn}}, \mathbf{s}_{\text{mlp}}, \mathbf{s}_{\text{out}})\) enables flexible step counts without learning rate decay or retraining.
• Decoder-free intermediate training: Unlike RAFT or recurrent flow models that compute intermediate losses at every recurrence step, DéjàView trains purely on the final step \(\mathbf{z}_K\), saving substantial backward-pass computation.

Limitations & Future Work

• Divergence beyond the trained step range: When \(K_{\text{inf}}\) exceeds \(K_{\text{max}}=16\), representation norm growth causes individual channels to diverge, limiting unbounded test-time compute scaling.
• Absence of dynamic object modeling: The architecture assumes rigid multi-view static scenes; moving objects or dynamic elements in driving datasets can cause transient geometry artifacts.
• Slight gap compared to fixed-step specialists: Variable-\(K\) training trades a marginal amount of peak accuracy (~0.5% AUC) at a specific budget for compute flexibility across the full \([8, 16]\) range.

Related Work & Insights

• vs VGGT / Pi3 / Depth Anything 3: Modern SOTA baselines stack tens of independent transformer layers (900M–1.2B parameters) with fixed inference cost. DéjàView achieves superior average geometry with only 117M parameters while exposing an inference-time compute knob.
• vs RAFT / Iterative MVS: Classical iterative methods run lightweight recurrent units over precomputed correlation volumes and decode at each iteration. DéjàView interleaves cross-view attention and geometric refinement directly within the same repeated transformer block.
• vs RAPTOR / ALBERT: While ALBERT uses weight sharing in NLP and RAPTOR distills pretrained ViTs into looping blocks post-hoc, DéjàView trains a single looping block end-to-end directly on 3D reconstruction task loss without a teacher network.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Successfully applies continuous-time conditioned looping transformers to multi-view 3D reconstruction, challenging the feed-forward depth-scaling paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across five standard benchmarks, extensive ablations on recurrence components and step counts, and thorough residual stream probing.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, insightful mathematical and empirical analysis of directional refinement versus fixed-point convergence.
• Value: ⭐⭐⭐⭐⭐ Offers a highly practical, parameter-efficient paradigm for edge and mobile 3D reconstruction with adaptive test-time compute.

Related Papers

• [CVPR 2026] Scaling View Synthesis Transformers (SVSM)
• [NeurIPS 2026] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction
• [NeurIPS 2026] MvFFN: Multi-view Floor-Plan Feed-Forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction
• [NeurIPS 2026] PhyMo: Learning Physical Dynamics with Accurate and Continuous Motion from Multi-View Videos
• [NeurIPS 2026] No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos