ViewSplat: View-Adaptive 3D Gaussian Splatting for Feed-Forward Synthesis¶

Conference: ECCV 2026
arXiv: 2603.25265
Code: https://cvlab-uos.github.io/ViewSplat (Project Page)
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Feed-Forward Reconstruction, Novel View Synthesis, View Adaptation, Hypernetworks

TL;DR¶

ViewSplat introduces a "view-adaptation" step to pose-free feed-forward 3DGS. It first predicts a set of base Gaussians and scene-conditioned View MLPs. During rendering, these MLPs take target view coordinates to dynamically calculate residual offsets for all Gaussian attributes (position, scale, rotation, opacity, color) on the fly. While maintaining real-time rendering (the largest backbone still achieves 90 FPS), this approach reconstructs high-frequency view-dependent effects such as specular highlights and sharp reflections that are typically lost in static representations, achieving a new state-of-the-art in pose-free NVS fidelity.

Background & Motivation¶

Feed-forward 3D Gaussian Splatting (feed-forward 3DGS) has pushed the speed of "directly reconstructing 3D scenes from a few images" to the limit over the past two years: models like pixelSplat, MVSplat, and GS-LRM no longer require thousands of optimization iterations per scene, but rather output a full set of Gaussian primitives in a single feed-forward pass to achieve instant novel view synthesis. To eliminate the dependence on SfM-precomputed camera poses, a recent batch of work (Splatt3R, NoPoSplat, SelfSplat, SPFSplat/SPFSplatV2) further leverages 3D foundation models such as DUSt3R, MASt3R, and VGGT to estimate camera parameters and geometry jointly in an end-to-end manner, achieving true pose-free reconstruction of uncalibrated in-the-wild images. Even so, a significant fidelity gap remains between feed-forward methods and per-scene optimization-based methods.

The authors attribute this gap to a structural bottleneck: static primitive regression. Existing feed-forward networks pin down all primitive attributes (3D position, scale, rotation quaternion, opacity, spherical harmonics SH coefficients) at once using a single-step regression. However, this set of attributes must simultaneously satisfy all possible viewing directions. The problem is that regressing a representation that is optimal for every viewpoint is fundamentally an ill-posed task for a single feed-forward pass, especially in the presence of complex geometries or non-Lambertian surfaces. While optimization-based methods can slowly resolve sharp specular highlights and view-dependent SH coefficients over thousands of iterations, feed-forward models have no opportunity to correct their initial predictions during inference. Even worse, the authors' experiments reveal that simply increasing the SH degree of the baseline from 4 to 8 yields almost no change or even a slight drop in PSNR. This indicates that the standard SH representation has hit its expressive capacity ceiling under the pose-free setting, and merely scaling up SH degrees cannot resolve view-dependent effects.

The key insight of the authors is to completely pivot the network's objective: rather than forcing the network to guess a "one-size-fits-all" static 3D Gaussian representation, it is better to focus on generating updates "specifically for a target viewpoint"—since "rendering a high-fidelity 2D image from a target viewpoint is relatively easy as long as the underlying 3D primitives can be tailored on-the-fly for that view." Learning the latter is far more feasible than learning the former. Core Idea: Transition the paradigm from static primitive regression to view-adaptive splatting. The network first generates a set of base Gaussians and scene-conditioned View MLPs. During rendering, these MLPs take the target view's coordinates as input to predict view-dependent residual offsets for all Gaussian attributes on the fly. This enables each primitive to correct its initial estimation errors and reconstruct high-frequency view-dependent details that are lost in static representations.

Method¶

Overall Architecture¶

ViewSplat is built upon the pose-free feed-forward framework of SPFSplat / SPFSplatV2. Given \(N\) pose-free images \(\{I^v\}_{v=1}^{N}\), a shared geometric Transformer backbone (MASt3R or VGGT) performs multi-view feature aggregation, which is then fed into three parallel heads: a pose head that estimates camera extrinsic parameters \(P^{v\to 1}\) relative to a reference view (defined as the canonical coordinate system on \(I_1\)), and two DPT Gaussian heads (center head + parameter head) that regress a set of static base Gaussians \(\{\mathcal{G}^{v\to 1}\}\) per-pixel in the canonical space. These capabilities constitute the baseline.

The key addition of ViewSplat is the fourth head—the view-dependent head. It does not directly resolve offsets; instead, it adopts a hypernetwork structure. By taking the feature tokens from the geometric Transformer, it generates the weights of a small, secondary network (referred to as the View MLP) per-pixel. When rendering a target viewpoint, the target pose \(P^{t\to 1}\) is first reparameterized into a compact 4D representation (3D unit viewing direction + 1D log distance). This representation is passed pixel-by-pixel into the corresponding View MLP to compute the residual offsets \(\Delta\mathcal{G}\) for all Gaussian attributes. The offsets are element-wise added back to the canonical Gaussians to obtain the refined Gaussians \(\hat{\mathcal{G}}\), which are then passed to the rasterizer to render the target image from \(P^{t\to 1}\). The entire pipeline is as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pose-free input images"] --> B["Geometric Transformer Backbone<br/>(MASt3R / VGGT)"]
    B --> C["Pose Head + Gaussian Heads<br/>canonical static Gaussians + camera poses"]
    B --> D["View-dependent Head: Hypernetwork<br/>generates View MLP weights per-pixel"]
    C --> E["Target pose reparameterization<br/>16-dim extrinsics → 4D (direction + log distance)"]
    D --> F["View MLP computes residual offsets on the fly<br/>Δ position/scale/rotation/opacity/color"]
    E --> F
    F -->|Residuals added to canonical Gaussians| G["Rasterization to render target view"]

Key Designs¶

1. View-Adaptive Splatting: Replacing Static Primitive Regression with On-the-Fly Refinement

This is the core proposition of the paper. As discussed, feed-forward networks determine a single set of Gaussians to fit all viewing angles through single-step regression, which inevitably struggles with specular/non-Lambertian surfaces, whereas optimization methods iteratively correct attributes. ViewSplat brings this corrective action into the feed-forward framework. Specifically, the canonical Gaussians are no longer expected to be directly usable; instead, they serve as an "initial guess," on top of which view-dependent residual corrections are applied when rendering each target viewpoint. This allows each Gaussian primitive to be locally adjusted in terms of both geometry and photometry for the currently observed viewpoint: diffuse regions remain largely unchanged, while specular/high-frequency boundary regions are heavily corrected. This shift is crucial because it changes what the network needs to learn: from "an ill-posed, one-size-fits-all static solution" to "a far more feasible, viewpoint-conditioned increment." The latter is naturally much easier to learn and enables the feed-forward model to represent view-dependent reflections and highlights for the first time. The authors' error map analysis validates this "spatial selectivity": ViewSplat reduces errors primarily at specular reflections and high-frequency geometric boundaries, while keeping diffuse regions stable.

2. Scene-Conditioned View MLP: Letting the Scene Decide "How to Refine" via Hypernetworks

If a fixed-parameter network were used to regress offsets directly from "context features + target pose," two issues would arise: first, a fixed-parameter MLP uses the same response function for all pixels and materials, limiting expressiveness; second (see Design 4), the target pose would couple with the heavy DPT decoder, requiring a rerun of the dense network for every viewpoint change, which ruins rendering efficiency. ViewSplat sidesteps this using a hypernetwork: the view-dependent head consumes the geometric Transformer's feature tokens (which carry rich geometric and material context) to generate weights for a dedicated View MLP per-pixel. In other words, the network first lets the scene context decide "what kind of response function this pixel should use"—specular surfaces can receive more complex mappings to capture moving reflections, while diffuse regions only need a simple function. This decoupling of "context-driven weight generation" and "pose-driven offset computation" liberates the model from a predefined static parameter space, enabling it to depict fine-grained, high-frequency appearance changes. Empirically, this View MLP requires only a single hidden layer of size 16 (or even 8 without much performance decay), keeping the overhead minimal.

3. Target Pose Reparameterization: Compressing 16D Extrinsics to 4D "Direction + Log Distance"

Feeding the \(4\times 4\) world-to-camera extrinsic matrix directly into the View MLP is highly problematic, as mathematical constraints like rotation matrix orthogonality are extremely difficult for an MLP to optimize. The authors reformulate the target pose into a compact and geometrically meaningful representation: for each canonical Gaussian center \(\mu_j^{v\to 1}\), the target camera center in the world frame is first recovered as \(C^{t\to 1}=-(R^{t\to 1})^{\top}t^{t\to 1}\). Then, the displacement vector from the Gaussian center to the camera is computed as \(\mathbf{d}_j^v=C^{t\to 1}-\mu_j^{v\to 1}\), which is finally decomposed into a 3D unit viewing direction and a 1D log distance:

\[\mathbf{u}_j^v=\frac{\mathbf{d}_j^v}{\|\mathbf{d}_j^v\|_2},\qquad l_j^v=\log(\|\mathbf{d}_j^v\|_2+\epsilon)\]

This reduces the input dimension from 16 to 4, retaining key view-dependent cues while making it significantly easier to learn. The log distance is not a random choice: under perspective projection, camera movement has a far greater visual impact on near objects than far ones; \(\log(\cdot)\) compresses the dynamic range of distances and amplifies near-field sensitivity, leading the View MLP to prioritize refining Gaussians where motion causes the most rapid visual changes. Ablation studies show that the log distance consistently outperforms linear distance on RE10K/ACID.

4. View-Dependent Gaussian Refinement: Comprehensive Residual Addition, Position and Visibility Must Be Synced

With the 4D pose representation and the per-pixel View MLP, a full set of attribute offsets \(\{(\Delta\mu,\Delta\alpha,\Delta r,\Delta s,\Delta c)\}\) can be predicted and element-wise added (e.g., \(\hat{\mu}=\mu+\Delta\mu\)) to yield refined Gaussians. Crucially, "residuals + all attributes + holistic refinement" must be combined to work: in the ablated experiments (Table 6), refining only a subset of attributes yields modest improvements, but when only spatial displacement \(\Delta\mu\) is applied without updating opacity \(\Delta\alpha\) simultaneously, the PSNR catastrophically drops from 25.5 to 15.7, causing severe artifacts, blurriness, and structural collapse. This is because spatial shifts must sync with visibility updates; otherwise, the occlusion relationships of the moved Gaussians become misaligned. This insight supports the "all-attribute joint refinement" design—the full model (all 8 items active) performs best across all metrics. Another easily overlooked detail: the geometric reprojection loss only supervises the canonical center \(\mu\), not \(\mu+\Delta\mu\). Treating \(\mu\) as a stable 3D geometric anchor while letting \(\Delta\mu\) dynamically model view-dependent appearance is shown to perform better than constraining the refined centers.

Loss & Training¶

Following the pose-free end-to-end training of SPFSplat, the total loss consists of the rendering loss and the geometric reprojection loss: \(\mathcal{L}_{total}=\mathcal{L}_{render}+\lambda_{reproj}\mathcal{L}_{reproj}\), with \(\lambda_{reproj}=0.001\). The rendering loss is a combination of MSE photometric loss and LPIPS perceptual loss (\(\lambda_{LPIPS}=0.05\)). The reprojection loss projects canonical Gaussian centers back onto context views using estimated relative poses and intrinsics, penalizing deviation from the 2D pixel coordinates, which stabilizes the geometry of the canonical space and prevents collapse under pose-free settings. The training employs a frozen backbone strategy: the geometric Transformer, the two Gaussian heads, and the pose head are all initialized with pretrained weights from the corresponding SPFSplat variant and frozen; only the new view-dependent head is trained. Furthermore, this head is zero-initialized, ensuring that the network retains the exact static baseline reconstruction in early stages before gradually learning the view-adaptive refinements. Adam optimizer is used with a learning rate of \(1\times10^{-4}\) and a batch size of 12 (one scene per batch), with curriculum learning in which the frame interval between input views increases as training progresses. Training resolution is \(256\times256\) (and \(224\times224\) for the V2-L variant as default by its backbone).

Key Experimental Results¶

Main Results¶

Novel view synthesis is evaluated on RE10K and ACID. During evaluation, rendering is performed using estimated target poses (rather than ground truth poses) to jointly assess reconstruction quality and pose-Gaussian alignment. ViewSplat, as a plug-and-play module, consistently improves performance across three SPFSplat backbones. The V2-L variant establishes a new state-of-the-art across all metrics:

Dataset	Backbone	PSNR↑	SSIM↑	LPIPS↓
RE10K	SPFSplat	25.484	0.847	0.153
RE10K	+ ViewSplat	26.317	0.857	0.144
RE10K	SPFSplatV2-L	25.668	0.855	0.137
RE10K	+ ViewSplat	26.798	0.870	0.124
ACID	SPFSplatV2-L	26.674	0.806	0.162
ACID	+ ViewSplat	27.509	0.820	0.149

As a comparison, pose-required MVSplat only achieves 24.012 PSNR on RE10K; ViewSplat (pose-free) outperforms it by roughly 2.8 dB. In terms of efficiency (on an RTX 4090), the ViewSplat module adds only 11–13 ms to the forward pass; however, since residuals must be re-evaluated for each new viewpoint, the rendering framerate drops from 386 FPS (baseline) to 154 FPS (base backbone). The largest V2-L backbone still delivers 90 FPS, which is well above the 30 FPS threshold for real-time performance.

Cross-dataset zero-shot generalization (trained on RE10K, tested on ACID/DTU/DL3DV) also shows consistent improvements across all three backbones. On DL3DV, V2-L + ViewSplat achieves 20.563 PSNR compared to the baseline's 19.743, verifying that the view-adaptive refinement learns transferrable representations.

Ablation Study¶

Configuration	PSNR↑	Description
(1) Baseline SPFSplat	25.484	Static baseline
(4) w/o \(\Delta\alpha\) (with \(\Delta\mu\), no opacity)	15.684	Decoupling location and visibility leads to structural collapse
(8) Full ViewSplat (all attributes)	26.317	Full model performs best
View MLP (Ours)	26.317 / 154 FPS	Hypernetwork scheme
Direct Regression	26.396 / 72 FPS	Direct regression gains only +0.08 dB but halving the framerate
SH degree 8 baseline	25.418	Scaling up SH degree is ineffective, proving the necessity of view refinement

Key Findings¶

The most critical yet fragile coupling is between \(\mu\) and \(\alpha\): adding positional offset without opacity updates tanks the PSNR by nearly 10 dB (from 25.5 to 15.7), proving that spatial movement must sync with visibility; all-attribute joint refinement is non-negotiable.
Hypernetwork vs Direct Regression is a trade-off of efficiency, not accuracy: Direct regression couples the target pose with the DPT decoder, forcing the heavy dense network to rerun for every view, which balloons the rendering overhead by +436% compared to the baseline (dropping from 154 FPS to 72 FPS) while yielding only +0.08 dB in accuracy. The hypernetwork scheme computes weights only once and runs lightweight View MLPs during rendering, preserving real-time performance.
Standard SH has hit its expressive ceiling: Increasing baseline SH degree from 4 to 8 yields negligible or even negative gains, whereas ViewSplat with SH=4 outperforms all baseline variants, directly proving the bottleneck is the lack of view-dependent refinement rather than SH frequency bands.
Refinement is true "3D refinement", not 2D overfitting: The pose AUC remains identical to the baseline (as the pose head is frozen), and depth consistency \(\delta < 1.25\) evaluated with DepthAnything3 rises from 0.947 to 0.952, showing that the performance gain comes from physical geometric improvement rather than compensating for incorrect poses using 2D artifacts.

Highlights & Insights¶

Pivoting the objective from "learning a static solution" to "learning viewpoint increments" is highly clever: It transforms an ill-posed all-viewpoint regression problem into a much more manageable viewpoint-conditioned residual prediction task. This is a key breakthrough for feed-forward fidelity, and the idea of "predicting query-specific corrections instead of directly predicting the target" can be easily adapted to any scenario where a single forward pass must satisfy diverse queries.
The hypernetwork localizes "how to refine" down to the pixel level: generating per-pixel View MLP weights from context features and then driving offsets via target poses decouples the two processes. This design enhances expressiveness while avoiding the efficiency disaster of rerunning the dense network per viewpoint, presenting an elegant compromise between expressive power and real-time performance.
Log-distance reparameterization is a small but solid trick: Compressing extrinsics into 4D and using log-distance to align with the near-field sensitivity of perspective projections yields stable performance increments at almost zero computational cost, which is highly recommendable for any module using poses as MLP inputs.
Plug-and-play: It brings consistent improvements to MASt3R-based, VGGT-based, and even independent architectures like YoNoSplat, demonstrating its generality as a paradigm rather than being tuned for a specific backbone.

Limitations & Future Work¶

Inability to hallucinate unobserved regions: Inheriting from the reconstructive nature of baselines without generative priors, unseen areas under sparse views still turn out blurry or empty. The authors note that integrating generative modeling is the way forward for full-scene completion.
View-conditioning sacrifices rendering throughput: Because attributes are refined on the fly for each target viewpoint, a single static 3D representation cannot be cached. Consequently, the rendering framerate drops noticeably compared to the static baseline (from 386 FPS to 154 FPS); though still real-time, this poses a hard constraint for ultra-high-framerate applications.
Multi-view consistency becomes "a family of view-conditioned approximations" rather than a single rigid geometry: The authors acknowledge this is theoretically an open question—the same scene has slightly different refined versions across different viewpoints, and ensuring strict consistency warrants further exploration.
A natural direction for future work is to share View MLP weights across spatial/material clusters to further reduce the cost of per-view recalculation, or to explore cacheable intermediate representations to mitigate framerate drops.

vs SPFSplat / SPFSplatV2 (Baseline): These resolve pose-free feed-forward reconstruction but output static Gaussians. This work overlays a view-dependent refinement head, pivoting "static primitives" into "on-the-fly refinable primitives," offering full-scale fidelity gains as a plug-and-play module at the expense of rendering speed.
vs NoPoSplat / SelfSplat / PF3plat: These follow the same pose-free feed-forward roadmap but are constrained by single-step static regression. This paper identifies and breaks through this collective bottleneck of "static primitive regression."
vs Scaffold-GS: Also discusses "view-adaptive rendering," but Scaffold-GS relies on anchor structures within a per-scene optimization framework. In contrast, this model performs view modulation via hypernetworks in a feed-forward manner without requiring per-scene training.
vs GaussianShader / Spec-Gaussian / Ref-NeRF (View-dependent rendering): These achieve high fidelity but are tied to expensive, non-generalizable per-scene optimizations. This work achieves similar high-frequency/non-Lambertian expressiveness in a feed-forward manner, generalizing instantly to unseen scenes.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The paradigm shift of 'learning viewpoint increments instead of static solutions' coupled with the per-pixel View MLP hypernetwork is clear and compelling, though components are clever assemblies of existing techniques.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Consistent gains across three backbones, generalization across four datasets, pose/depth consistency evaluations, and extensive ablations (including the catastrophic decoupling case of \(\mu/\alpha\)) construct a complete chain of evidence.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation and bottlenecks are argued with high clarity, and the appendix details pose reparameterization and loss derivations thoroughly.
Value: ⭐⭐⭐⭐☆ Closes the fidelity gap with optimization-based methods by bringing view-dependent refinement to feed-forward 3DGS. The plug-and-play nature makes it highly adoptable for future work, though framerate drop and unobserved regions remain open challenges.