RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://compvis.github.io/rayder
Area: 3D Vision
Keywords: novel view synthesis / self-supervised learning / video geometry learning / transformers / dynamic modeling

TL;DR

RayDer unifies camera pose estimation, scene reconstruction, and novel view rendering into a single feed-forward transformer backbone while absorbing video dynamics as a nuisance factor via state dropout, breaking the static-scene data barrier and unlocking predictable power-law scaling on unconstrained real-world video.

Background & Motivation

Novel view synthesis (NVS) holds the theoretical promise of becoming a highly scalable self-supervised learning paradigm: by observing different viewpoints across video sequences and jointly predicting camera geometry and visual representations, one can directly turn the vast, uncurated corpus of web video into rich geometric supervision without manual annotations. However, existing self-supervised NVS pipelines (such as RayZer, Pensieve, and Less3Depend) remain severely bottle-necked by a foundational assumption: they require training on curated, strictly static-scene datasets. When exposed to unconstrained real-world videos containing moving humans, vehicles, or shifting objects, minimizing photometric reconstruction error forces these models to erroneously absorb time-varying spatial changes into their camera pose representations, triggering catastrophic representation drift, gradient spikes, and training divergence.

Beyond data curation bottlenecks, prior self-supervised NVS frameworks suffer from structural flaws in architectural scalability. Established architectures like RayZer rely on multi-network pipelines composed of three distinct networks: a camera pose estimator, an intermediate scene representation encoder, and a separate rendering decoder. While manageable at modest parameter counts, scaling such heterogeneous, interacting networks to hundreds of millions of parameters becomes prohibitively unstable and empirical tuning intractable. Distributing capacity across discrete modules disrupts smooth gradient propagation and prevents self-supervised geometric systems from exhibiting the clean, predictable power-law scaling laws seen in large language models.

This work addresses this scalability bottleneck by re-evaluating the role of dynamics and architectural modularity: the ultimate objective remains static-scene NVS rather than costly 4D dynamic scene modeling, meaning dynamic content should simply be absorbed as a nuisance factor during training rather than explicitly reconstructed; simultaneously, the fragmented geometry-rendering pipeline is consolidated into a single unified transformer backbone. Core idea: consolidate camera estimation, scene reconstruction, and view rendering into a single feed-forward transformer backbone while introducing a minimal per-view dynamic state variable treated as a nuisance factor with state dropout, turning self-supervised NVS from unconstrained real-world video into a well-behaved single-model scaling problem governed by clean power-law scaling laws.

Method

Overall Architecture

RayDer replaces the conventional multi-network setup of self-supervised NVS with a single feed-forward transformer backbone \(\mathcal{M}\). Given an uncalibrated, unposed set of video frames \(\{I_i\}\), the model adaptively performs two cooperative tasks within the same self-attention latent space: first, estimating per-frame camera pose tokens \(p_i \in SE(3)\) (parameterized via Plücker rays) along with auxiliary dynamic state tokens \(s_i \in \mathbb{R}^{d_{\text{state}}}\); second, rendering novel target views \(\hat{I}_{\text{target}}\) conditioned on query target camera rays \(p_{\text{target}}\).

To ensure stable optimization across real-world dynamic video, prevent temporal pose shortcuts, and achieve high-resolution detail with minimal overhead, RayDer integrates temporal dynamic state absorption, parallel-target attention factorization, random-order autoregressive training, and shallow local high-resolution attention layers into its end-to-end architecture:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Uncalibrated Dynamic Video Sequence {I_i}"] --> B["Dynamic State Prediction & State Dropout<br/>Jointly predict poses and dynamic state (p_i, s_i)"]
    B --> C["Single-Network Consolidation & Parallel-Target Attention<br/>Unified transformer representation with factorized attention"]
    C --> D["Random-Order Autoregression over Views<br/>Sever temporal shortcuts and enforce geometric conditioning"]
    D --> E["Local High-Resolution Layers<br/>Intra-frame neighborhood attention for high-frequency details"]
    E --> F["Synthesized Target Novel View I_target"]

Key Designs

1. Dynamic State Prediction & State Dropout: Absorbing Scene Dynamics as a Nuisance Factor

In unconstrained real-world video, differences between an input view \(i\) and target view \(j\) arise from two intertwined factors: camera pose displacement \(p_i \to p_j\) and scene dynamic deformation \(s_i \to s_j\). When models expose only camera pose conditioning, the photometric loss forces dynamic scene changes into the pose representations, causing severe training divergence. RayDer bypasses the need for full 4D scene modeling (which requires scarce multi-view dynamic captures) by predicting a compact per-view dynamic state embedding \(s_i \in \mathbb{R}^{d_{\text{state}}}\) alongside the camera pose:

\[E_{\text{cam,state}}: \{I_i\} \mapsto \{(p_i, s_i)\}, \quad D_{\text{render,state}}: z, (p_{\text{target}}, s_{\text{target}}) \mapsto \hat{I}_{\text{target}}\]

The dynamic state token acts purely as a nuisance factor, absorbing non-rigid motions and transient lighting changes. To eliminate test-time dependence on ground-truth target states \(s_{\text{target}}\), RayDer adopts state dropout during training: randomly replacing \(s_{\text{target}}\) with a zero vector \(\mathbf{0}\). This encourages the unified network to synthesize high-fidelity static scene content regardless of whether dynamic state conditioning is available. At test time, static background geometry remains geometrically consistent while unconditioned moving elements smoothly fade into a coherent low-frequency average, entirely stabilizing training on open-world dynamic video.

2. Single-Network Consolidation & Parallel-Target Attention: Eliminating Multi-Network Scaling Bottlenecks

Traditional multi-network pipelines isolate pose estimation from neural rendering, preventing mutual representation sharing and causing unpredictable capacity allocation during scaling. RayDer integrates camera/state estimation, scene reconstruction, and rendering into a single shared transformer backbone \(\mathcal{M}\). By modulating token roles via adaptive normalization, the same network alternates between processing input views to establish 3D geometry and rendering target views:

\[\mathcal{M}: \{(I_i, p_i, s_i)\} \cup (\cdot, p_j, s_j) \mapsto \hat{I}_j\]

Treating this unified model as a standard autoregressive decoder would recompute attention over all input views for every target view, multiplying computational costs. RayDer resolves this via a factorized parallel-target attention mask: input view tokens perform full bidirectional self-attention amongst themselves to construct an invariant 3D scene representation; meanwhile, target query tokens attend to themselves and all input tokens, but remain causally masked from other target tokens. This allows all input key-value representations (KV Cache) to be computed once and reused across queries, enabling parallel target view synthesis during training and reducing per-target rendering compute by approximately \(7\times\).

3. Random-Order Autoregression over Views: Breaking Temporal Shortcuts for Robust Geometry

Training on contiguous video clips allows models to exploit an "implicit temporal shortcut": because camera motion is continuous over time, the network can predict camera transformations merely by indexing frame order along the temporal axis rather than inferring actual spatial geometry. Poses learned this way fail to transfer to sparse or single-view NVS benchmarks.

To force true geometric pose reasoning, RayDer trains the unified backbone autoregressively across views. Instead of following the chronological capture order, it implements random-order autoregression: the sequence of input frames is randomly permuted, dynamically varying context size \(|I_{\text{input}}| \in \{1, 2, \dots, |I_{\text{total}}| - 1\}\) with causal masking along the permuted order. The model is forced to predict camera poses and novel views under extreme view sparsity (even from a single image) using pure perspective and visual cues, progressively refining geometry as more context views are introduced. This mechanism dramatically improves camera rotation accuracy (\(R@10^\circ\) jumps from 70.1% to 84.4%).

4. Local High-Resolution Layers: High-Frequency Recovery at Negligible Compute Cost

Standard Vision Transformers rely on coarse patchification (e.g., \(16 \times 16\)), which discards fine geometric edges and textures; conversely, reducing patch size inflates computational complexity at an \(O(p^4)\) rate. Following modern hourglass vision architectures, RayDer wraps its central unified transformer backbone with shallow local high-resolution layers.

These layers operate strictly intra-frame via Neighborhood Attention (NAT). High-resolution feature patches are initially aggregated locally within confined spatial windows before downsampling into the global cross-view transformer backbone. Once long-range epipolar and geometric correspondence reasoning is completed, local decoding layers restore high-frequency textures using localized neighborhood attention assisted by skip connections. This design delivers sharp edges and fine spatial details without incurring prohibitive global attention overhead.

Loss & Training

RayDer optimizes strictly through photometric reconstruction error on held-out target views without any 3D supervisory signals, depth priors, optical flow supervision, or pretrained geometric foundation models. The objective combines mean squared error (MSE) and perceptual loss (LPIPS):

\[\mathcal{L} = \mathcal{L}_{\text{MSE}}(\hat{I}_{\text{target}}, I_{\text{target}}) + \lambda_{\text{LPIPS}} \mathcal{L}_{\text{LPIPS}}(\hat{I}_{\text{target}}, I_{\text{target}})\]

The model is trained using AdamW on SpatialVid (~2.7M videos) with a global batch size of 256 at \(256 \times 256\) resolution. Training sequences sample 8 views per video clip spaced by roughly 0.5 seconds. Four parameter scales are evaluated: XS (59M), S (145M), B (422M), and L (743M).

Key Experimental Results

Main Results

RayDer demonstrates outstanding zero-shot generalization across open-set benchmarks. Operating entirely without external camera pose annotations at test time (relying exclusively on its own predicted camera poses), RayDer-L matches or outperforms large-scale video diffusion models and 3D-supervised baselines.

Table 1: Open-Set Novel View Synthesis Benchmark Comparison (PSNR ↑)
(Evaluated across both small-viewpoint and large-viewpoint splits across 1, 3, and 6 input views. RayDer uses self-predicted poses)

Model	Params	Self-sup.	Views	LLFF (3-view)	DTU (3-view)	CO3D (3-view)	WRGBD (3-view)	Mip360 (3-view)	T&T (3-view)
MVSplat	12M	✗	3	12.50	15.52	13.52	12.54	13.56	13.22
DepthSplat	354M	✗	3	12.62	16.24	13.77	14.23	14.01	14.35
ViewCrafter	1.4B	✗	3	13.52	16.40	14.72	12.66	14.59	18.07
SEVA	1.3B	✗	3	19.48	20.82	19.25	18.91	16.70	15.16
Kaleido	3.1B	✗	3	20.71	–	–	–	18.03	–
E-RayZer	246M	✓	3	18.01	16.97	17.76	16.18	15.86	10.36
RayDer-L (Ours)	743M	✓	3	21.38	17.92	19.09	17.23	16.25	18.74

To investigate whether self-supervision is preferable to leveraging off-the-shelf pose estimators on dynamic videos, Table 2 evaluates RayDer against LVSM trained with MegaSaM pseudo-ground-truth camera poses:

Table 2: Self-Supervised vs. Pseudo-Pose Supervised Training on Dynamic Video (Zero-Shot RE10K)

Model	Training Dataset	Camera Supervision	Pose Generation Cost	PSNR ↑	LPIPS ↓	SSIM ↑
LVSM	SpatialVid (~2.7M)	MegaSaM Pseudo-GT	~69,000 GPU-h	25.44	0.184	0.729
RayDer-B (Ours)	SpatialVid (~2.7M)	Fully Self-Supervised	0 GPU-h (None)	28.35	0.151	0.879

Note: RayDer-B requires only ~1,200 GPU-h of training while outperforming the pseudo-pose supervised LVSM by +2.91 dB PSNR, demonstrating that self-supervised representation learning avoids pose fitting noise.

Ablation Study

Table 3 details the incremental impact of each technical component on training stability, zero-shot view synthesis (RE10K), and camera pose estimation (DL3DV-10K):

Table 3: Progressive Architectural Ablation Study on Dynamic Video Datasets

Config	Variant Description	Training Stable	SA-V: PSNR (w/o state) ↑	SA-V: PSNR (w/ state) ↑	SA-V: Camera R@10° ↑	SA-V: Camera [email protected] ↑	SV-HQ: PSNR (w/o state) ↑	SV-HQ: Camera R@10° ↑
A	RayZer-like Baseline (~140M)	Diverges (Unstable)	22.53*	–	59.8*	6.5*	22.69*	66.0*
B	+ Dynamic State Prediction	Fully Stable ✓	13.42†	24.01	56.1	7.0	13.48†	54.4
C	+ State Dropout	Fully Stable ✓	23.01	23.76	62.4	8.1	23.02	69.2
D	+ Single-Network Consolidation	Fully Stable ✓	24.93	25.33	68.8	16.3	26.98	74.1
E	+ Parallel-Target Attention	Fully Stable ✓	24.04	25.12	70.1	15.6	25.91	70.9
F	+ Ordered Autoregression	Fully Stable ✓	23.08‡	24.49‡	73.6	24.9	23.53‡	76.5
G	+ Random-Order Autoregression	Fully Stable ✓	25.45	26.28	84.4	37.2	27.27	86.0
H	+ Local High-Res Layers	Fully Stable ✓	25.61	26.87	85.0	40.2	27.78	88.7

*Selected from non-diverging runs; †without dropout, test inference depends on unobservable target states; ‡ordered autoregression exhibits distribution shift on standard NVS benchmarks.

Key Findings

• Dynamic State & Dropout Guarantee 100% Training Stability: While the baseline consistently crashes when trained on open-world video (SA-V), predicting dynamic state tokens (Config B) completely prevents gradient spikes. Incorporating state dropout (Config C) restores zero-shot unconditioned synthesis quality from 13.42 dB to 23.01 dB.
• Single-Network Consolidation Amplifies Both Representation and Speed: Merging three disjoint models into one shared backbone (C \(\to\) D) boosts PSNR by almost 4.0 dB on SV-HQ (23.02 \(\to\) 26.98 dB) and doubles translation accuracy \([email protected]\) (8.2 \(\to\) 19.7). Factorizing target attention (D \(\to\) E) yields a \(7\times\) speedup per target query with negligible quality degradation.
• Random-Order Autoregression Neutralizes Temporal Shortcuts: Transitioning from temporal order to randomized autoregressive view conditioning (F \(\to\) G) dramatically enhances camera rotation accuracy \(R@10^\circ\) (73.6% to 84.4%) and translation accuracy \([email protected]\) (24.9 to 37.2), validating that viewpoint randomization enforces genuine 3D perspective reasoning.
• Strict Joint Compute-Data Power-Law Scaling: RayDer's compute-optimal frontier closely obeys the parametric law \(L(C, D) = L_\infty + A C^{-\alpha} + B D^{-\beta}\) with \(R^2 > 0.99\). Scaling data and compute simultaneously is mandatory: on small data budgets (1%, ~27k videos), larger models overfit and fall behind smaller models. Furthermore, training on general dynamic video at scale (SpatialVid 2.7M, achieving 29.38 dB) decisively outperforms aggregating all existing static-scene datasets combined (~250k videos, achieving 28.68 dB).

Highlights & Insights

• Formulating Dynamics as a Nuisance Factor Unlocks Internet-Scale Data: Rather than getting trapped in the scarce-data regime of 4D reconstruction, RayDer insightfully focuses on static-scene NVS as the target task and treats non-rigid motion as an auxiliary nuisance variable to be absorbed and dropped, clearing the path for scaling across uncurated video.
• Consolidation Dividend in 3D Foundation Models: Paralleling the transition from modular vision pipelines to unified transformers in 2D perception, RayDer demonstrates that camera estimation and neural rendering benefit from a unified representation space, producing mutual gradient regularization and superior feature scaling.
• Superior Economics of Self-Supervision over Pseudo-Pose Pipelines: Offline SfM tools like MegaSaM demand massive compute (over \(50\times\) the training cost of RayDer-B) and introduce systematic tracking artifacts that cap model performance. Self-supervised joint geometry-rendering optimization is both vastly cheaper and structurally more accurate.

Limitations & Future Work

• Blurry Mean Reconstructions in Unobserved Regions: Due to the MSE regression objective, regions that are completely occluded or absent from context views collapse into a low-frequency average color rather than generating plausible hallucinated textures. Integrating generative diffusion decoders into the unified backbone represents a promising future avenue.
• Dynamic Content Blurring in Dynamic Scenes: Because the model does not build an explicit 4D spatiotemporal representation, dynamic entities in the scene degrade into semi-transparent blurred averages during view synthesis, even while static background geometry renders crisply.
• Domain Gap on Textureless Laboratory Captures: On synthetic or sterile lab datasets such as DTU (featuring pure white backgrounds and artificial lighting), RayDer's camera pose estimation exhibits occasional drift, reflecting the domain discrepancy from natural in-the-wild video.

Related Work & Insights

• vs RayZer [28]: RayZer pioneered self-supervised NVS from monocular video but remains constrained to curated static scenes and relies on three separate ViTs. RayDer solves the dynamic video divergence problem and scales smoothly through its unified single-transformer backbone.
• vs LVSM [29] / VGGT [70]: LVSM requires ground-truth or pseudo-ground-truth camera poses. VGGT unifies multiple geometry tasks within a single transformer but operates under full 3D supervision. RayDer extends the unified backbone philosophy to a completely self-supervised, pose-free paradigm.
• vs Video Diffusion Models (SEVA [91] / ViewCrafter [84] / Kaleido [41]): Video diffusion models leverage massive web-scale video generation pretraining to hallucinate views but require external pose inputs and immense compute budgets. RayDer trains from scratch in a single stage with zero pose supervision, achieving competitive fidelity at a fraction of the parameter and compute budget.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Elegant formulation of video dynamics as a nuisance factor to stabilize self-supervised NVS, alongside single-backbone consolidation and scaling law validation)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive scaling across 4 model sizes, 3 orders of data magnitude, dozens of zero-shot benchmarks, and rigorous train-test leakage checks)
• Writing Quality: ⭐⭐⭐⭐⭐ (Crystal-clear motivation, rigorous progression from instability analysis to architectural consolidation, and strong empirical figures)
• Value: ⭐⭐⭐⭐⭐ (Overcomes the fundamental static-data barrier for self-supervised 3D vision, establishing a scalable roadmap toward real-world 3D foundation models)