LVSPM: Long Sequence View Synthesis and Pose Estimation Model¶
Conference: ECCV 2026
Paper: ECCV Official
Code: Project Page
Area: 3D Vision
Keywords: Novel View Synthesis / Camera Pose Estimation / Test-Time Training / Long Sequence / Pose-Free
TL;DR¶
LVSPM introduces an end-to-end forward sequence prediction framework based on Large-Chunk Test-Time Training (LaCT) and learnable camera tokens, achieving accurate pose estimation and real-time novel view synthesis across uncalibrated sequences up to 256 frames without requiring dense 3D ground truth.
Background & Motivation¶
Novel View Synthesis (NVS) stands as a foundational problem in 3D computer vision and neural rendering. From scene-specific optimization methods (such as NeRF and 3D Gaussian Splatting) to feed-forward reconstruction architectures (like pixelSplat and LVSM), conventional techniques almost universally require precisely calibrated camera poses. These poses typically rely on offline Structure-from-Motion (SfM, e.g., COLMAP) pipelines. However, traditional SfM is computationally intensive and prone to registration failure in low-texture, repetitive, or illumination-varying environments, severely impeding real-world applicability in in-the-wild scenarios.
To bypass pose dependence, recent research has explored pose-free novel view synthesis (e.g., AnySplat, NoPoSplat) and geometric pose estimation models (e.g., DUSt3R, VGGT, DepthAnything3). Yet both families face severe limitations: pose-free NVS methods are typically constrained to sparse sets of fewer than 10–32 views, struggling to scale to long, expansive trajectories, and often rely on pre-trained geometric models. Conversely, geometric estimators require costly, dense 3D point cloud or depth ground truth for training and lack native image rendering capabilities.
The central challenge lies in developing a unified feed-forward model capable of scaling to hundreds of frames for globally consistent pose estimation while simultaneously delivering high-fidelity image synthesis without dense 3D supervision. Embracing the minimal-inductive-bias philosophy of LVSM, this paper formulates the task as a joint sequence-to-sequence token prediction problem. By utilizing the dynamic hidden states of Test-Time Training (TTT) as compact scene representations, it demonstrates that novel view synthesis supervision inherently provides strong multi-view geometric constraints. Core idea: Formulate an end-to-end architecture using a Large-Chunk Test-Time Training (LaCT) backbone with learnable camera tokens and asymmetric query routing, trained solely on RGB rendering and sparse camera pose losses, leveraging view synthesis supervision to reinforce geometric pose accuracy across uncalibrated sequences up to 256 frames with 58.8 FPS parallel rendering.
Method¶
Overall Architecture¶
LVSPM adopts a sequence-to-sequence transformation paradigm with minimal explicit 3D inductive biases. The input comprises \(N\) uncalibrated multi-view RGB images \(\mathcal{I} = \{I_i\}_{i=1}^N\). Each image is divided into non-overlapping patches and linearly projected into patch tokens, augmented with a learnable camera token. The first frame serves as the canonical reference coordinate system with a fixed reference token \(c_r\), while subsequent input frames share a generic learnable camera token \(c_x\). For novel target views, Plücker rays corresponding to the target camera parameters are linearly mapped and concatenated with trigger tokens \(c_t\) to form target query tokens.
All input and target tokens are processed by a 24-layer backbone of stacked Large-Chunk Test-Time Training (LaCT) blocks. Each LaCT block incorporates intra-view window self-attention, large-chunk TTT dynamic parameter update layers, and feed-forward MLP mixing layers. At the output stage, lightweight two-layer MLP heads decode: (1) absolute camera poses (quaternions, translation vectors) and normalized focal lengths from the input camera tokens; and (2) RGB pixels for target novel views directly from the target query tokens.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Uncalibrated Multi-View Images<br/>N Frames RGB Input Sequence"] --> B["Image Patchification & Linear Projection<br/>Append Camera Tokens & Reference Anchoring"]
C["Target View Plücker Rays<br/>Ray Encoding & Linear Mapping"] --> D["Stacked LaCT Transformer Blocks (24 Layers)<br/>Window Self-Attention + Large-Chunk TTT + FF-MLP"]
B --> D
D --> E["Asymmetric Query Routing<br/>Inputs Update Weights, Targets Read-Only"]
E --> F["Lightweight Two-Layer MLP Heads<br/>Decode Pose/Intrinsics & Target View RGB"]
F --> G["Joint Multi-Task Output<br/>Accurate Poses + 58.8 FPS Parallel Rendering"]
Key Designs¶
1. Learnable Camera Tokens and Canonical Reference Anchoring: Unbiased Input Representation Traditional forward reconstruction pipelines often force unknown camera parameters into spatial positional encodings, causing geometric entanglement across uncalibrated long sequences. LVSPM explicitly decouples camera parameters by appending a learnable camera token \(c_i\) to each patchified image sequence. To establish a self-consistent global coordinate reference without absolute world coordinates, the first view is designated as the canonical reference frame (fixing its translation to zero and rotation to unit quaternion) with dedicated reference token \(c_r\). Subsequent frames share a generic token \(c_x\). Features are normalized by the maximum scene extent defined by the farthest view, bounding the sequence within a unit scale. This allows self-attention to freely exchange contextual textures and camera cues across views without handcrafted epipolar constraints.
2. Large-Chunk Test-Time Training and Asymmetric Query Routing: Long-Sequence Scalability Standard quadratic attention suffers from memory explosion on long sequences, whereas recurrent or small-chunk state updates induce catastrophic forgetting over extended trajectories. LVSPM integrates Large-Chunk Test-Time Training (LaCT) with SwiGLU-MLP parameter layers. Given an input sequence, gradients are computed across large chunks aggregating all input image and camera tokens (defaulting to 8,224 tokens) to perform a single weight update: $\(G_l = \nabla_{\text{TTT-MLP}_l} \left( 1 - \text{TTT-MLP}_l(k_l) \cdot v_l^T \right)\)$ Crucially, only input view keys and values propagate gradients to update dynamic weights. Target ray tokens act purely as read-only queries passing through the updated TTT-MLP. This asymmetric routing prevents local geometric overfitting while enabling completely independent, unblocked parallel rendering of arbitrary numbers of novel views at 58.8 FPS.
3. Implicit Geometric Self-Supervision via Novel View Synthesis: Bypassing Dense 3D Ground Truth While methods like DUSt3R and VGGT rely on dense 3D point clouds or depth labels, LVSPM demonstrates that high-fidelity novel view rendering provides sufficient physical constraints for multi-view geometric alignment. In novel view synthesis, slight misalignment in estimated camera poses or scene representations produces severe photometric and perceptual reconstruction penalties when projected along target Plücker rays. By jointly optimizing RGB reconstruction loss alongside camera pose regression, novel view synthesis provides a supervisory signal matching explicit 3D point cloud supervision without requiring dense geometric annotations.
4. Lightweight Decoupled Pose Decoder Heads: Minimalist Parameter Regression In contrast to architectures using heavy inter-frame cross-attention or dual-branch networks, LVSPM deploys a minimal two-layer MLP head atop the 24-layer LaCT backbone. The pose decoder head directly predicts a 9-dimensional vector: 4D unit rotation quaternion \(r_i\), 3D translation vector \(t_i\), and 2D normalized effective focal lengths \((\hat{f}^x_i, \hat{f}^y_i)\). Because the LaCT blocks have already blended multi-view spatiotemporal features, the lightweight decoder reduces parameter overhead (312M total parameters) while outperforming complex geometric baselines.
Loss & Training¶
The overall training objective combines photometric reconstruction loss \(\mathcal{L}_{\text{rgb}}\), camera pose loss \(\mathcal{L}_{\text{pose}}\), and camera intrinsics loss \(\mathcal{L}_{\text{int}}\): $\(\mathcal{L}_{\text{total}} = \lambda_{\text{rgb}} \mathcal{L}_{\text{rgb}} + \lambda_{\text{pose}} \mathcal{L}_{\text{pose}} + \lambda_{\text{int}} \mathcal{L}_{\text{int}}\)$ where \(\mathcal{L}_{\text{rgb}} = \|I_{\text{pred}} - I_{\text{gt}}\|_2^2 + \lambda_{\text{LPIPS}} \cdot \text{LPIPS}(I_{\text{pred}}, I_{\text{gt}})\), \(\mathcal{L}_{\text{pose}} = \|t_{\text{pred}} - t_{\text{gt}}\|_2^2 + \|r_{\text{pred}} - r_{\text{gt}}\|_2^2\), and \(\mathcal{L}_{\text{int}} = \|\hat{f}^x_{\text{pred}} - \hat{f}^x_{\text{gt}}\|_2^2 + \|\hat{f}^y_{\text{pred}} - \hat{f}^y_{\text{gt}}\|_2^2\).
Training proceeds across three stages on 64 NVIDIA H100 GPUs: Stage 1 pre-trains on synthetic ASE at \(128 \times 128\) for 90k iterations; Stage 2 trains on mixed synthetic and real scenes (DL3DV-10K, ScanNet++, Hypersim, Co3Dv2) for 60k iterations; Stage 3 scales resolution to \(512 \times 288\) with 128 input and 64 target views. The model generalizes zero-shot to 256 input views during inference.
Key Experimental Results¶
Main Results¶
Using the equi-temporal evaluation protocol on DL3DV-10K across 16 to 256 input views, evaluating camera pose AUC (\(3^\circ, 5^\circ, 30^\circ\)) and novel view synthesis metrics:
| Method | Dense 3D Supervision | 16 views AUC3 | 16 views AUC5 | 16 views AUC30 | 64 views AUC3 | 64 views AUC5 | 128 views AUC3 | 128 views AUC5 | 256 views AUC3 | 256 views AUC5 | 256 views AUC30 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Cut3R | Yes | 48.8 | 60.5 | 88.2 | 52.2 | 64.1 | 34.1 | 48.2 | 19.2 | 31.4 | 73.0 |
| Fast3R | Yes | 22.7 | 34.0 | 75.2 | 26.9 | 39.6 | 14.8 | 24.7 | 8.0 | 15.3 | 52.4 |
| Flare | No | 60.6 | 69.9 | 90.4 | 76.2 | 82.5 | 74.0 | 80.8 | 71.1 | 79.1 | 92.8 |
| VGGT | Yes | 74.7 | 81.9 | 94.7 | 85.5 | 89.4 | 84.1 | 88.2 | 83.6 | 87.9 | 95.2 |
| TTT3R | No | 51.4 | 63.3 | 89.3 | 58.7 | 68.8 | 46.5 | 58.8 | 36.5 | 50.7 | 83.3 |
| DA3 | Yes | 84.0 | 88.7 | 96.2 | 86.1 | 90.0 | 84.5 | 89.1 | 83.8 | 88.5 | 95.3 |
| LVSPM (Ours) | No | 88.4 | 91.6 | 96.9 | 92.0 | 93.5 | 90.0 | 92.0 | 90.1 | 92.1 | 95.7 |
Novel view synthesis on DL3DV-10K across 16 to 256 views:
| Method | Pose Input | 16 views PSNR | 16 views SSIM | 16 views LPIPS | 64 views PSNR | 64 views SSIM | 128 views PSNR | 128 views SSIM | 256 views PSNR | 256 views SSIM | 256 views LPIPS |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DepthSplat | GT Poses Required | 24.80 | 0.849 | 0.134 | 18.39 | 0.617 | – (OOM / Fails) | – | – | – | – |
| AnySplat | Pose-Free | 23.15 | 0.757 | 0.158 | 20.52 | 0.643 | 19.45 | 0.593 | 18.55 | 0.543 | 0.339 |
| LVSPM (Ours) | Pose-Free | 25.77 | 0.811 | 0.152 | 24.33 | 0.752 | 23.31 | 0.710 | 22.09 | 0.654 | 0.250 |
On RealEstate10k at 128 views, LVSPM achieves AUC3 of 61.3 (VGGT: 20.6, Flare: 35.9). On Co3Dv2 at 128 views, LVSPM reaches AUC3 of 68.8 and AUC5 of 76.6, surpassing DepthAnything3 (65.7 and 75.3).
Ablation Study¶
Ablations on DL3DV-10K and synthetic ASE under fixed 32 input and 32 target views:
| Dataset | Variant | AUC30 ↑ | AUC5 ↑ | AUC3 ↑ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Key Takeaway |
|---|---|---|---|---|---|---|---|---|
| DL3DV | w/o Syn (No Synthetic Pretrain) | 92.7 | 84.3 | 79.9 | 20.08 | 0.589 | 0.267 | Synthetic pre-training bolsters geometric priors |
| DL3DV | Full Model | 94.5 | 85.6 | 80.2 | 21.12 | 0.638 | 0.222 | Optimal balance across real-world scenes |
| ASE | w/o RGB (No NVS Rendering Loss) | 87.9 | 62.6 | 50.9 | – | – | – | AUC3 drops by 21.0 points: NVS loss is critical |
| ASE | Half Chunk Size (4112 tokens, 2 updates) | 88.3 | 74.3 | 67.9 | 24.98 | 0.702 | 0.263 | Small-chunk updates cause local drift |
| ASE | 12 TTT Layers (Shallower Backbone) | 90.1 | 69.2 | 59.3 | 24.18 | 0.673 | 0.288 | Reduced capacity degrades long-sequence tracking |
| ASE | Ours Full | 93.7 | 79.5 | 71.9 | 25.01 | 0.701 | 0.257 | Full 24-layer LaCT with large chunk achieves best metrics |
Key Findings¶
- NVS Rendering Loss Drives Camera Pose Precision: Removing RGB rendering supervision (
w/o RGB) causes AUC3 to drop from 71.9 to 50.9 on ASE, demonstrating that photometric rendering loss enforces precise multi-view geometric alignment without explicit 3D point cloud labels. - Large-Chunk Single Updates Outperform Multi-Step Updates: Halving chunk size with two updates reduces AUC3 from 71.9 to 67.9, indicating that frequent test-time parameter modifications lead to localized overfitting and catastrophic forgetting.
- Robust Long-Sequence Scaling: As input views increase from 16 to 256 frames, baselines like Fast3R and Cut3R degrade to AUC3 of 8.0 and 19.2, whereas LVSPM maintains 90.1 AUC3 with modest PSNR drop (from 25.77 to 22.09 dB), outperforming AnySplat by +3.54 dB at 256 views.
- Inference Efficiency: On 256 frames, LVSPM completes forward pose inference in 1.78 seconds on a single H100 GPU (versus 13.6 seconds for VGGT) and renders novel views at 58.8 FPS.
Highlights & Insights¶
- Photometric Supervision as Geometric Anchor: Validates that multi-view RGB rendering loss serves as a powerful geometric regularizer for pose estimation, bypassing expensive 3D point cloud annotations.
- Asymmetric Key/Value Query Routing: Decouples scene representation updates (input tokens) from novel view synthesis queries (target tokens), facilitating 58.8 FPS parallel rendering regardless of query view count.
- LaCT for Implicit 3D Memory: Extends test-time training to multi-view 3D vision, replacing explicit point cloud storage with dynamic transformer weights to achieve linear memory scaling over long sequences.
Limitations & Future Work¶
- Training Complexity: The current framework relies on a multi-stage progressive curriculum across synthetic and real datasets.
- Textureless and Drastic Lighting Variations: Performance degrades in expansive textureless environments or under extreme lighting shifts due to weak feature correspondences.
- Metric Scale Alignment: Predicted poses operate in normalized coordinate frames rather than absolute metric scale.
Related Work & Insights¶
- vs. VGGT & DepthAnything3: VGGT and DA3 require dense 3D point cloud labels and heavy frame-attention layers, lacking native novel view rendering. LVSPM matches or exceeds their pose accuracy while simultaneously providing real-time NVS without dense 3D ground truth.
- vs. AnySplat & NoPoSplat: 3DGS-based pose-free methods are limited to sparse sets (<32 views) and encounter memory bottlenecks on long sequences. LVSPM scales gracefully to 256 frames with stable rendering quality.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant integration of LaCT and learnable camera tokens for pose-free NVS without dense 3D ground truth.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous equi-temporal benchmarking across 16–256 frames on DL3DV, RE10k, and Co3Dv2 with extensive ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear theoretical exposition, well-structured methodology, and insightful empirical analysis.
- Value: ⭐⭐⭐⭐⭐ Sets a new benchmark for scalable, pose-free 3D reconstruction and real-time novel view synthesis.