GlobalSplat: Efficient Feed-Forward 3D Gaussian Splatting via Global Scene Tokens

Conference: ECCV2026
Paper: ECCV paper page
Area: 3D Vision
Keywords: Gaussian Splatting, novel view synthesis, global scene tokens, dual-branch attention, capacity curriculum

TL;DR

GlobalSplat aligns multiple views into a fixed set of global scene tokens before decoding compact 3D Gaussians, achieving 28.53 dB PSNR, 77.88 ms per forward pass, and a 3.8 MB scene asset with 16K Gaussians on 24-view RealEstate10K evaluation, while retaining less perceptual detail than the heavier ZPressor.

Background & Motivation

Feed-forward 3D Gaussian Splatting aims to turn several images into an explicit, repeatedly renderable scene in one pass, avoiding per-scene iterative optimization. Methods such as pixelSplat and MVSplat typically predict on image pixels, depth maps, or feature grids before lifting the results into 3D. This structure benefits from established dense prediction techniques, but also carries input-view redundancy into the output: a wall appearing in multiple frames may produce several view-anchored groups of Gaussians instead of becoming shared scene structure from the outset.

Adding images therefore increases not only coverage but potentially also the local geometry to reconcile, memory consumption, and asset size. Post-processing compression can shrink an existing Gaussian set without necessarily removing redundant computation during its construction. LVSM takes another route by compressing views into a latent representation and directly decoding target images, but still requires a neural decoder for each new viewpoint. The missing combination is early fusion with the reusable standard 3DGS asset interface.

GlobalSplat imposes its constraint before explicit geometry is generated: a fixed set of scene tokens repeatedly reads all views, forms a shared interpretation, and only then determines Gaussian placement. Here, alignment means multiview fusion in latent space, not a separate explicit point-matching procedure or automatic estimation of unknown camera poses. Core Idea: absorb cross-view redundancy in a global latent bottleneck before using geometry/appearance-separated decoding and a coarse-to-fine capacity curriculum to generate explicit Gaussians whose count does not grow with the number of input views.

Method

Overall Architecture

The inputs are multiple RGB images and their camera intrinsics and extrinsics; the output is a standard 3D Gaussian set with positions, scales, rotations, opacities, and colors. Cameras are first transformed into a canonical coordinate system, while image patches, rays, and camera metadata form the input context. Fixed scene tokens read this context through geometry and appearance branches and are decoded into an asset usable by a 3DGS renderer.

The default configuration uses 2,048 scene tokens of dimension 512, with 8 output Gaussians per token, yielding 16,384 Gaussians, abbreviated as 16K in the paper. The number of input frames can change without changing this output budget; this does not make input feature extraction or cross-attention computation completely constant.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multiview RGB<br/>Camera intrinsics and extrinsics"] --> B["Camera-Augmented Context"]
    B --> C["Dual-Branch Global Fusion"]
    C --> D["Capacity-Curriculum Decoding"]
    D --> E["Explicit Gaussian Asset<br/>Standard 3DGS Rendering"]
    E --> F["Novel-View Image"]
    E -. "Training: depth and opacity" .-> H["Cross-Subset Geometry Consistency"]
    T["Another Input Subset<br/>Independent Pass with Stop-Gradient"] -. "Training reference" .-> H
    H -. "Training update" .-> C

Solid arrows show reconstruction and rendering, while dashed arrows indicate training-only geometry constraints. RGB rendering loss and regularization also supervise training, as described below. At deployment, producing a scene does not require a second input subset or loss optimization.

Key Designs

1. Camera-Augmented Context: give patches comparable spatial information

Different videos have inconsistent origins, orientations, and scales. Feeding them directly to the network would require it to learn scene structure while also handling arbitrary coordinate changes. The authors place the origin at the average camera center, re-orthonormalize the average viewing directions to construct canonical axes, and normalize translations by the maximum distance between camera centers. Putting camera layouts into comparable scales and orientations lets the network devote more capacity to internal scene structure instead of locating the entire video in a coordinate system.

Images are converted into RGB patch tokens, while dense Plucker-ray features are also patchified and linearly embedded. Rather than relying on rays alone, the method combines Fourier positional encoding of the camera center with an MLP embedding of normalized intrinsics. This per-view camera code is broadcast and added to the ray-patch features, which are then concatenated with RGB features. Rays describe local projection geometry, while the additional metadata explicitly conditions on camera location and focal information. The ablation supports this augmentation, but does not establish that Plucker representations contain no location-related information at all.

2. Dual-Branch Global Fusion: share the scene interpretation before separating geometry and appearance prediction

The 2,048 learnable scene tokens are not 2,048 input pixels and are not preassigned to fixed cells in any view. They query relevant evidence from all input patches and exchange scene context through self-attention. Additional learnable register tokens capture global information and discourage local-feature overfitting. Because every view contributes to the same latent slots, repeated observations of a surface can be reconciled before Gaussian generation, instead of producing separate copies that must be merged afterward.

The network stacks 4 encoder blocks, each projecting the tokens into geometry and appearance streams. Within each branch, scene features query the image context through cross-attention, and the reported 2 self-attention layers process global relationships. A two-layer Mixer MLP then fuses the streams for the next block. Separation does not mean isolation: the streams process structure and texture independently and then communicate, reducing opportunities to hide inaccurate geometry with color fitting. The output heads retain this division: the geometry head predicts positions, scales, quaternions, and opacity, while the appearance head predicts color attributes. The standard 3DGS representation includes view-dependent spherical-harmonic color coefficients.

3. Capacity-Curriculum Decoding: learn large-scale structure before releasing local Gaussian capacity

Each latent slot predicts 16 Gaussian candidates, but early training does not let all candidates independently explain the images. Instead, they are merged into one representative Gaussian. Effective per-slot capacity then increases through 1, 2, 4, and 8, with the default model stopping at 8. When global geometry is still unstable, the model cannot as easily compensate for local image errors with numerous small primitives. More Gaussians are released once structure has stabilized to capture local detail. This is a training curriculum, not iterative per-scene optimization repeated during inference.

Candidate reduction uses temperature-scaled softmax importance weights and weighted aggregation tailored to Gaussian attributes. The main text delegates the parameter-aware aggregation details to supplementary material, which is absent from the supplied cache. This note therefore does not assume that rotations or other attributes are simply averaged elementwise. Candidate count, scene-token count, and final Gaussian count must be distinguished: the default 16K means 2,048 tokens times 8 outputs, not one Gaussian per token or all 16 candidates being emitted by default.

4. Cross-Subset Geometry Consistency: make different observation sets support compatible scene structure

With target RGB supervision alone, color and opacity can conceal structural errors. The authors therefore divide input views into two subsets, reconstruct a scene independently from each, and compare the resulting rendered depth and opacity maps. A stop-gradient operation is applied to the reference branch, and the other branch is encouraged to agree with it. No external ground-truth depth labels are required: the constraint asks different subsets of observations to produce compatible geometry, matching the purpose of global scene tokens.

This objective neither matches Gaussian indices one by one nor requires identical color pixels between reconstructions. It measures differences in rendered geometric quantities. The main text identifies depth, opacity, and stop-gradient, but the cached equation has missing characters; unprovided details such as viewpoint sampling and occlusion handling are not reconstructed here. The objective is used only during training and does not enlarge the final asset's Gaussian budget.

A Worked Example

Consider 24 context images from RealEstate10K in which the same wall appears repeatedly. Each frame first produces camera-conditioned patch features, but these features do not directly become 24 separate sets of wall Gaussians. The same 2,048 scene tokens read them through 4 dual-branch blocks and progressively integrate evidence about the wall's position and appearance.

Early in training, each slot has one effective Gaussian, giving a budget of 2,048. Capacity gradually increases to 8 per slot, producing 16,384 explicit Gaussians. This arithmetic explains the paper's configuration rather than introducing another experiment. Once training is complete, a new scene requires one forward pass to produce the final budget, followed by rendering with a target camera. Increasing context from 24 to 36 frames preserves that budget but does not guarantee improvement in every image-quality metric.

Loss & Training

The paper separates its objective into rendering, cross-subset consistency, and regularization terms. The total objective is clearly stated in the main text:

\[ \mathcal{L}=\lambda_{\mathrm{ren}}\mathcal{L}_{\mathrm{ren}}+\lambda_{\mathrm{con}}\mathcal{L}_{\mathrm{con}}+\lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}. \]

The rendering term combines squared target-image error with a perceptual loss. Consistency constrains depth and opacity, while regularization includes feature soft thresholding and a frustum constraint. Several component equations in the text cache have missing operators, so they are not reconstructed as exact author formulas. Loss weights, optimizer settings, training steps, and capacity-stage transition times are likewise not invented.

GlobalSplat is trained from scratch without a pretrained pixel-prediction backbone or latent features reused from dense baselines. The main model has approximately 83.4M parameters; the single-stream ablation is widened to 90M to reduce parameter-count confounding. RealEstate10K is used for the main training and evaluation, and ACID for zero-shot cross-dataset testing. DL3DV-10K results use a model trained from scratch on that dataset, not the same zero-shot transfer experiment.

Key Experimental Results

Main Results

Evaluation follows C3G's protocol based on the NoPoSplat split: the two anchor context frames and held-out targets remain fixed, while extra frames between the anchors expand context to 12, 24, or 36 images. RealEstate10K and ACID are evaluated at 256×256 resolution. PSNR measures pixel reconstruction error and SSIM measures structural similarity, both higher-is-better; LPIPS measures perceptual distance and is lower-is-better.

The table combines the 24-view results from original Tables 1 and 3. Gaussian count measures scene primitives, and disk size refers to the output scene asset rather than network weights. Latency measures one forward pass, not rendering frame rate. The paper reports a single "A100 with 64GB VRAM" for efficiency testing. That memory specification needs clarification from the authors; this note preserves the reported condition without treating it as independently verified hardware information.

Method	PSNR / dB	SSIM	LPIPS	Gaussians / K	Peak Memory / GB	Forward / ms	Scene / MB
GlobalSplat	28.53	0.883	0.140	16	1.79	77.88	3.8
ZPressor	28.51	0.911	0.097	393	3.70	194.20	134
DepthSplat	19.66	0.743	0.239	1572	29.84	669.50	534
C3G	23.80	0.747	0.198	2	6.04	387.14	0.1
GGN	17.20	0.634	0.343	512	25.08	1800.64	174

GlobalSplat exceeds ZPressor by only 0.02 dB PSNR, which is not an across-the-board image-quality win. It achieves fewer primitives and shorter forward latency while accepting worse SSIM and LPIPS. C3G has fewer Gaussians and a smaller disk footprint still, so GlobalSplat's advantage is the combined quality-resource trade-off, not first place on every compactness metric.

For zero-shot ACID transfer with 24 views, original Table 2 reports 28.03 / 0.813 / 0.208 for GlobalSplat and 28.53 / 0.860 / 0.138 for ZPressor, ordered as PSNR, SSIM, and LPIPS. Original Table 4 uses 32K Gaussians on DL3DV-10K at 256×448 with 24 views, obtaining 24.90 PSNR versus ZPressor's 688K and 24.26. ZPressor still has better SSIM and LPIPS. C3G entries in Table 4 use 224×224 and cannot establish a same-protocol victory from their raw numbers.

Ablation Study

The following results come from original Table 6 on RealEstate10K. Neither that table nor its accompanying discussion explicitly labels the context-view count, so these results are not presented as 24-view evaluation. PSNR drops are calculated relative to that table's full model.

Config	PSNR / dB	SSIM	LPIPS	PSNR Drop / dB
Full model	28.57	0.885	0.139	0.00
Plucker rays only	28.30	0.880	0.140	0.27
Without consistency loss	28.15	0.876	0.143	0.42
Single stream, 90M parameters	28.02	0.873	0.151	0.55
Direct full capacity from the start	27.69	0.867	0.150	0.88

The 12-view model in original Table 1 and the default model in Table 5 report LPIPS 0.138, whereas Table 6 reports 0.139 for the full model. Values are preserved from their respective tables rather than silently harmonized. Table 5 also separates latent capacity from output Gaussian count; its context-view count is likewise not explicitly labeled in the table:

Total Gaussians	Scene Tokens	Gaussians per Token	PSNR / dB	SSIM	LPIPS
2,048	256	8	25.25	0.785	0.250
2,048	2,048	1	26.83	0.838	0.198
16,384	2,048	8	28.57	0.885	0.138
32,768	2,048	16	28.58	0.884	0.135
32,768	4,096	8	29.54	0.903	0.121

Key Findings

• The capacity curriculum has the largest individual effect in Table 6: predicting full capacity immediately reduces PSNR by 0.88 dB, supporting structural stabilization before releasing detail capacity.
• At the same 32,768-Gaussian budget, increasing tokens from 2,048 to 4,096 while reducing outputs per token from 16 to 8 raises PSNR from 28.58 to 29.54. Stronger scene representation helps more than merely decoding additional primitives, although this is not a compute-matched comparison.
• RealEstate10K PSNR is 28.57, 28.53, and 28.45 for 12, 24, and 36 views. The fixed budget is relatively stable as inputs increase, rather than demonstrating that more views necessarily improve quality.

Highlights & Insights

• Moving fusion before explicit primitive generation changes where redundancy arises. The transferable principle is to establish a shared scene representation before allocating geometry capacity, rather than merely replacing a compressor.
• Geometry and appearance branches process information separately while communicating through the Mixer. The weaker result of the larger single-stream model provides evidence for the division of labor itself.
• Token count and Gaussians per token are distinct capacity controls. At a fixed asset budget, strengthening latent scene representation may be more valuable than adding local candidates.

Limitations & Future Work

• The authors acknowledge that a fixed budget may be insufficient for complex city-scale scenes. The current model also assumes static environments, and extremely sparse input with 2 to 3 images remains difficult. Their proposed directions include hierarchical tokens, spatiotemporal modeling, and monocular priors.
• This note's assessment: competitive PSNR does not replace perceptual detail quality. ZPressor's SSIM and LPIPS advantages remain relevant to deployment choices, and the use of 32K on DL3DV shows that 16K is not a sufficient budget for arbitrary scenes.
• This note's assessment: a fixed output budget does not guarantee constant encoding cost. Without dedicated tests of unknown or noisy camera poses, the results should not be generalized to uncalibrated reconstruction. The capacity table also lacks a comparison at matched training compute.
• Material available here: the cache contains the main paper and references but not the cited supplementary material. Parameter-aware reduction, regularization details, and complete training settings therefore cannot be described at implementation-level reproducibility. The hardware specification and final-digit LPIPS discrepancy also remain verification boundaries.

Related Work & Insights

• vs pixelSplat / MVSplat: typical dense view-aligned pipelines generate local geometry first. GlobalSplat first fuses global latent information, reducing the direct transfer of repeated observations into redundant Gaussian assets.
• vs LVSM: both use a fixed latent scene representation, but LVSM directly decodes target images while GlobalSplat produces a reusable explicit 3DGS asset. Their deployment interfaces differ.
• vs C3G: C3G is also a compact query-driven Gaussian method and is important concurrent work. GlobalSplat adds iterative dual branches and a capacity curriculum while training from scratch. Comparisons must account for Gaussian budgets, backbone sizes, and input resolutions.
• vs ZPressor: ZPressor compresses view-centric intermediate features and preserves sharper detail with a heavier asset. GlobalSplat shows a different resource trade-off through latent fusion and explicit rendering, not the elimination of all quality loss.

Rating

• Novelty: 4/5. Global latent representations are not new, but the combination of dual branches, capacity curriculum, and compact explicit decoding is well targeted.
• Experimental Thoroughness: 4/5. Multiview settings, cross-dataset transfer, resource measurements, and key ablations are covered, although some protocol and hardware details need clarification.
• Writing Quality: 4/5. The main argument is clear, but claims of no quality sacrifice need to be read alongside LPIPS and SSIM trade-offs.
• Value: 4/5. Useful for rapid scene creation and lightweight assets; extremely sparse views and dynamic scenes are not yet strengths.