StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/J-C-Zhao/StructSplat
Area: 3D Vision
Keywords: Gaussian Splatting, uncalibrated sparse views, structured representation, pixel-aligned feature injection, camera alignment

TL;DR

StructSplat assigns geometry, semantics, and texture to separate encoding paths and aligns cameras without leaking target-image features into Gaussian prediction, achieving 28.045 dB PSNR on DL3DV with two source views, no camera-parameter inputs, and no post-optimization.

Background & Motivation

Turning a few photographs into a renderable scene requires both locating surfaces and preserving their colors and fine textures. Traditional NeRF or 3DGS usually fits each scene separately; generalizable feed-forward models avoid this repeated optimization, but many still depend on calibrated cameras. Even with geometry foundation models such as VGGT or MASt3R, predicting Gaussians from uncalibrated inputs does not guarantee faithful appearance. Figure 3 tests this directly: attaching a color head to VGGT features produces color and detail errors even when reconstructing the input images, showing that geometry features are not lossless appearance carriers.

A separate issue concerns the data path used during training and evaluation. To estimate source and target cameras in one coordinate system, a model may encode both image sets together with cross-view attention. Selecting only source tokens afterward does not remove the target information already incorporated into those tokens, so rendered-view supervision is no longer a clean test of an unseen viewpoint. Deleting target tokens cannot undo earlier attention exchanges.

StructSplat does not ask a single backbone to handle geometry, semantics, texture, and leakage-free evaluation on its own. Core Idea: let geometry determine spatial structure, semantics supply object and scene priors, and pixel-aligned texture preserve detail, while target images are used only in a separate camera-estimation path whose target cameras are aligned to the source-only reconstruction frame.

Method

Overall Architecture

The inputs are uncalibrated RGB source images of one scene, with no user-provided intrinsics or extrinsics; the output is a global set of 3D Gaussians that can be rendered with Gaussian Splatting. Structured feature roles first extract geometry, semantics, and texture, and pixel-aligned Gaussian decoding predicts local Gaussian attributes for source pixels. Dual-stream camera alignment provides consistent target-camera coordinates for source-only reconstruction, while predicted source cameras transform local Gaussians into the world frame.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Uncalibrated source images"] --> Features["Structured feature roles"]
    Features --> Decoder["Pixel-aligned<br/>Gaussian decoding"]
    Features --> Camera["Dual-stream<br/>camera alignment"]
    Target["Target images<br/>Camera-estimation path only"] --> Camera
    Decoder --> World["Local Gaussians to world frame"]
    Camera -->|Source cameras| World
    World --> Render["Gaussian Splatting<br/>and target views"]
    Camera -->|Aligned target cameras| Render

Here, requiring no camera parameters means that known calibration is not an input; it does not mean rendering needs no target camera. During the paper's training and evaluation, target images still participate in mixed-stream camera estimation, but never enter the feature stream used to predict Gaussians. This distinction matters: the method isolates target appearance-feature leakage, rather than establishing a camera-localization protocol that never accesses target images.

Key Designs

1. Structured feature roles: do not force the geometry backbone to carry all appearance information

The VGGT geometry encoder aggregates scene relationships through cross-view attention and produces geometric tokens for depth and camera estimation. A separate semantic encoder extracts high-level context from each image, adding priors about object organization, scene layout, and correspondences. The appendix specifies a separate frozen DINOv3: features from DINOv2 fine-tuned for geometry inside VGGT can lose appearance information, whereas an independent semantic path avoids that task-specific bias. There is no additional semantic segmentation output; semantics condition Gaussian prediction.

The texture encoder uses lightweight convolutions to retain high-resolution local appearance aligned with individual pixels. Instead of compressing colors and textures into viewpoint-invariant concepts, it preserves the actual pixel color and nearby edge variations. The three paths are complementary: geometry constrains placement, semantics reduce structural ambiguity under sparse views, and texture supplies high-frequency information that the other two paths can discard. The appendix also describes using the texture encoder to reduce interference among depth, opacity, color, scale, and rotation attributes, but does not provide a complete reproducible convolutional-layer specification.

2. Pixel-aligned Gaussian decoding: fuse structure first, then restore detail near the output

The decoder modifies DPT's multi-scale reassembling and fusion architecture rather than simply concatenating three feature vectors at the final output. Reassembling blocks project, spatially resize, and concatenate geometric and semantic features, then project them back to the original DPT channel dimensions before cascaded reassembling and fusion. High-resolution texture features are injected immediately before the final prediction layer; Figure 4 depicts this as a late addition path. Texture therefore does not need to survive all low-resolution structural processing before being recovered, shortening its route to Gaussian attributes.

Separate heads predict depth, opacity, color, scale, and rotation. Position is parameterized by depth along the camera z-axis, and rotation is represented by a quaternion in the local camera frame; predicted intrinsics and extrinsics must then transform these attributes into world coordinates. Local Gaussians from different source cameras cannot simply be concatenated without this transformation. Table 7 gives output channel counts of 1, 1, 3, 3, and 4, respectively; depth and scale use Softplus-based activations, opacity uses a shifted Sigmoid, and rotation is normalized. The color head multiplies its raw output by 0.25 without a Sigmoid to avoid gradient saturation, although a linear output does not mathematically guarantee a valid color range.

3. Dual-stream camera alignment: share camera anchors, not target appearance features

The geometry encoder separately processes mixed stream A, containing source and target images, and source-only stream B. Only B supplies Gaussian-prediction features; A supplies target cameras. The two estimates can use different reference frames, so target cameras from A cannot directly render the Gaussians reconstructed in B. Source cameras, present in both streams, serve as matching anchors from which the alignment is estimated and transferred to target-camera parameters.

Rotation alignment finds a common correction directly in unit-quaternion space; the main text uses a Lagrange multiplier formulation, and the appendix gives a closed-form normalized sum of relative quaternions. Translation alignment goes beyond scale matching: it jointly fits a scale and a three-dimensional offset by minimizing squared residuals between corresponding source-camera translations. Using the paper's camera-parameter notation, the target-camera transfer is:

\[ q^B_{t,j}=\Delta q^*q^A_{t,j},\qquad \tau^B_{t,j}=\lambda^*\tau^A_{t,j}+\Delta\tau^*. \]

The quaternion correction, scale, and offset are estimated from corresponding source cameras, rather than obtained by iteratively fitting a scene representation to target-pixel errors. This equation transfers the camera parameters as defined by the paper and should not silently be substituted into another camera-center or extrinsic convention. Compared with adjusting only translation scale at evaluation, it accounts for rotation, scale, and translation offset while keeping Gaussian features source-only during training.

Loss & Training

Training renders the predicted Gaussians into both source and target views and combines pixel, structural, and perceptual supervision. The objective in Equation (3) can be expressed with the following compact notation:

\[ \mathcal{L}=\sum_{i=1}^{N_{\mathrm{src}}+N_{\mathrm{tar}}} \left[w_1\operatorname{MSE}(I_i,\hat I_i) +w_2\left(1-\operatorname{SSIM}(I_i,\hat I_i)\right) +w_3\operatorname{LPIPS}(I_i,\hat I_i)\right]. \]

Training uses BF16, DeepSpeed, and a Warmup-Stable-Decay learning-rate schedule; the reconstruction training does not use ground-truth depth or an explicit depth-consistency loss. This does not imply the geometry encoder has no pretrained priors or that training uses no target RGB supervision. The cache does not report numerical loss weights, a complete learning-rate configuration, or batch size, so standard defaults should not be assumed. The appendix reports approximately 2.5 days on one H100 for the two-view setting, 73 GB peak memory, 1.535 B parameters, and 164.657 ms per inference pass; this timing should not be extrapolated to arbitrary view counts or resolutions.

Key Experimental Results

Main Results

Table 1 uses the official DL3DV benchmark test split and evaluates the DL3DV-trained model on ACID and RealEstate10K for cross-dataset transfer. All outputs and ground-truth images are resized and center-cropped to 256×256; evaluation metrics use target views only, with no post-optimization. The DL3DV two-source-view results match Table 2. The following selection comes from Table 1; PSNR is measured in dB, while SSIM and LPIPS are dimensionless.

Dataset	Method	Camera input condition	PSNR ↑	SSIM ↑	LPIPS ↓
DL3DV	AnySplat	No camera parameters	22.377	0.716	0.150
DL3DV	StructSplat	No camera parameters	28.045	0.888	0.091
DL3DV	NoPoSplat	Known intrinsics	25.592	0.838	0.104
ACID	AnySplat	No camera parameters	22.433	0.651	0.237
ACID	StructSplat	No camera parameters	24.372	0.712	0.219
ACID	NoPoSplat	Known intrinsics	25.521	0.750	0.185
RealEstate10K	AnySplat	No camera parameters	20.521	0.686	0.212
RealEstate10K	StructSplat	No camera parameters	22.240	0.729	0.201
RealEstate10K	NoPoSplat	Known intrinsics	24.724	0.818	0.145

The PSNR gains over AnySplat are 5.668, 1.939, and 1.719 dB on the three datasets, respectively. The best-performance claim is restricted to camera-parameter-free methods in Table 1; known-intrinsics NoPoSplat remains stronger in cross-dataset evaluation, so the input conditions should not be merged into an unconditional superiority claim.

Ablation Study

The following results are from Tables 3 and 6 on DL3DV, with PSNR in dB. Table 3 adds features cumulatively; Table 6 separately tests camera alignment and is not another step in the same feature-addition sequence.

Original table	Config	PSNR ↑	SSIM ↑	LPIPS ↓
Table 3	Geometry only	20.610	0.622	0.343
Table 3	Geometry + semantics	26.236	0.848	0.151
Table 3	Geometry + semantics + texture	28.045	0.888	0.091
Table 6	Without camera alignment	27.338	0.879	0.097
Table 6	Full model	28.045	0.888	0.091

Adding semantics improves PSNR by 5.626 dB, adding texture afterward adds 1.809 dB, and removing camera alignment costs 0.707 dB. These results support feature specialization and clean training paths, but the cumulative ablation is not an additive decomposition of independent module contributions.

Key Findings

• In Table 2, 2, 4, and 6 source views yield PSNR values of 28.045, 30.415, and 31.137 dB, indicating that additional observations improve reconstruction.
• Target-extrinsic post-optimization in Table 4 raises the result to 29.287 dB; this setting must be reported separately from the main-table result of 28.045 dB.
• In Table 5, rotation ATE is 0.0866 for AnySplat and 0.0016 for StructSplat; translation ATE is 0.1003 and 0.0093. Units and normalization details are not specified there, so these values should not be labeled as degrees or meters.
• Table 8 uses lower-rate sampling to cover larger scenes, a different protocol: 4 source views yield only 20.171 dB, which must not be conflated with the 30.415 dB in Table 2.

Highlights & Insights

• A geometry foundation model's strength does not imply pixel fidelity. Diagnosing input-view color reconstruction before adding a texture path is a more targeted design rationale than simply enlarging the backbone.
• Late texture injection does not abandon 3D consistency; it restores local information after structural fusion. This is relevant to feed-forward reconstruction systems with plausible geometry but blurry rendering.
• Leakage checks must trace features back to the attention operation. Keeping only source tokens at the output does not make them clean if they previously interacted with target tokens.

Limitations & Future Work

• The authors acknowledge difficulties with extremely sparse views, severe occlusions, complex view-dependent effects, and lighting; dynamic scenes and large-scale environments remain future directions rather than solved capabilities.
• Two geometry-estimation streams and a separate semantic encoder impose real costs. Single-GPU training is not synonymous with a lightweight model: 1.535 B parameters and 73 GB peak memory constrain deployment.
• Evaluation still estimates target cameras from target images. Isolating target appearance features does not establish equivalent performance when target images are entirely unavailable and arbitrary user-specified cameras are used.
• Table 3 lacks a full factorial feature ablation and error bars; whether semantic and texture gains vary with training scale or scene category remains untested.
• Alignment depends on source-camera anchor quality. The cache does not adequately specify robustness to degenerate trajectories, tiny baselines, or failed pose predictions, nor the precise inclusion of both streams and alignment in inference timing.

Related Work & Insights

• vs AnySplat: both perform feed-forward Gaussian reconstruction without intrinsic or extrinsic inputs; StructSplat emphasizes independent semantic and texture paths and alignment of rotation, scale, and offset.
• vs Splatt3R / Depth Anything 3: all exploit geometry foundation models, but StructSplat argues that a geometry backbone plus Gaussian heads still lacks appearance detail and instead modifies feature fusion and late texture injection.
• vs NoPoSplat / FLARE: these methods use known intrinsics in Table 1 and should not be treated as sharing the fully camera-parameter-free condition; their cross-dataset advantages also show the continuing value of calibration information.
• A useful follow-up would fix the geometry backbone and training budget, compare semantic priors and texture-injection locations, and separately report source-only reconstruction cost and target-camera localization cost.

Rating

• Novelty: 4/5. Structured appearance modeling and leakage-free camera alignment form a clear combination, largely built on existing geometry and vision foundation models.
• Experimental Thoroughness: 4/5. Cross-dataset tests, multiple view counts, ablations, and resolution extensions are included, but training details and statistical uncertainty remain incomplete.
• Writing Quality: 4/5. The diagnostic in Figure 3 and leakage illustration in Figure 5 are intuitive, while camera conventions, ATE units, and runtime accounting need clarification.
• Value: 4/5. Useful for both uncalibrated feed-forward reconstruction and evaluation-path design, subject to large-model costs and reliance on target-camera estimation.