ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Conference: ECCV2026
Paper: ECCV paper
Project: ZeroSplat
Authors: Jiayu Ding, Meilu Song, Xiaoyi Zhang, Hongbo Jin, Yichen Jin, Xiangtian Si
Area: 3D Vision
Keywords: Gaussian Splatting, generalized referring segmentation, vision-language model, multi-view geometry, spatial label diffusion

Identity check: manifest ID 3373, the stub, and the conference PDF have the same title; no subtitle change was found. This note follows Sections 3-4 and Tables 1-8 of the conference PDF. The cached text does not contain the annotation-protocol appendix mentioned in the paper.

TL;DR

ZeroSplat converts category and localization priors from a vision-language model into Gaussian-level binary labels through multi-view masks and geometric constraints, requiring no per-scene semantic training or additional semantic feature storage and achieving 50.8 and 41.2 mIoU on GR-LERF and GR-ScanNet.

Background & Motivation

3D Gaussian Splatting supports real-time scene rendering, but visual fidelity does not imply understanding instructions such as selecting all red chairs next to a table. LangSplat-style approaches distill 2D model features into a scene and render feature maps for text matching; OpenGaussian-style methods organize semantics at Gaussian points. The former emphasize image-plane retrieval, while the latter usually address category names rather than expressions combining attributes, quantities, and spatial relations.

ReferSplat introduces complex referring expressions into 3DGS but assumes that every expression identifies one object. Selecting exactly one instance misses targets in plural queries and produces false positives when the requested object is absent. The proposed GR3DGS task instead assigns foreground or background to every Gaussian, allowing 0, 1, or multiple instances while avoiding a large semantic feature field that must be optimized for each new scene.

Core Idea: let the vision-language model establish what is being requested and where it is, then let multi-view geometry determine which Gaussians belong to it, turning semantic querying into filtering and label diffusion over the original Gaussian set rather than learning another scene representation.

Method

Overall Architecture

The inputs are an already reconstructed 3DGS scene, multi-view images with camera information, and a free-form referring expression. The output is a binary label for each Gaussian. The pipeline comprises geometry-guided semantic parsing, mask generation and 3D lifting, and multi-view verification and spatial refinement, following Figure 2 and Sections 3.2-3.4.

The first stage exposes only a compact keyframe set to the vision-language model (VLM), obtaining category labels and 2D boxes for the referred targets. The second runs SAM3 over a broader set of views and traces rendered mask evidence back to individual Gaussians. Target boxes then reject same-category distractors, and 3D neighborhoods fill internal gaps, separating category recall, referring localization, and structural completion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Existing Gaussian scene<br/>Multi-view images and expression"] --> B["Geometry-guided<br/>semantic parsing"]
    B -->|Category labels| C["Mask generation<br/>and 3D lifting"]
    B -->|Keyframe target boxes| D["Multi-view verification<br/>and spatial refinement"]
    C --> D
    D --> E["Gaussian-level binary labels<br/>Zero, one, or multiple targets"]

Key Designs

1. Geometry-guided semantic parsing: preserve geometric evidence before separating category and referring localization

Passing every video frame to a VLM wastes computation on repeated background, whereas uniform sampling can miss informative edges and corners. ZeroSplat downsamples Gaussian centers, estimates local surface variation from neighborhood covariance eigenvalues, and rescales that variation into saliency weights. It favors geometrically informative regions, but its objective is coverage of previously unseen high-value voxels rather than selecting individually complex frames repeatedly.

Local surface variation is \(\sigma_p=\lambda_3/(\lambda_1+\lambda_2+\lambda_3+\epsilon)\), where \(\lambda_1\geq\lambda_2\geq\lambda_3\). Each voxel receives the maximum weight of its points. Depth and camera parameters identify the voxels observed by each candidate view. A greedy procedure repeatedly selects the view contributing the largest weight over uncovered voxels and updates the covered set until the keyframe budget is exhausted, preserving more scene-wide evidence than repeated views of one complex region.

The keyframes support two VLM interactions. The first combines images and the full expression to extract concise category labels. The second supplies those labels, the original expression, and the keyframes to predict normalized target boxes. Labels drive SAM3; boxes recover which instances within the category are intended. The simplified label is therefore an intermediate representation, not the final referring answer.

For example, an expression about chairs next to a table can first yield chair to improve 2D mask recall, while boxes conditioned on the original expression exclude distant chairs. Multiple instances can produce multiple boxes whose spatial union is used later. All Gaussians may also remain background; the output is not constrained by an argmax that must select one instance.

2. Mask generation and 3D lifting: aggregate reliable views using rendering contributions

SAM3 takes each category label and produces candidate masks with confidence scores in each view. The method does not use every view indiscriminately. It selects the high-confidence set when that set is sufficiently large; otherwise it takes the highest-confidence views from a lower base-threshold set, falling back to a still lower safe-threshold set when necessary. The VLM keyframes and the mask views have different purposes and are not the same set.

Within a category, the highest-confidence mask is retained, and the runner-up is merged only when it passes a strict confidence threshold. Masks are then united across categories. This can repair some over-segmentation, but it is not an unconditional union of every candidate instance and does not by itself guarantee full recall for arbitrary plural queries.

Lifting does not simply label every Gaussian projecting into a mask as foreground. A Gaussian's contribution to a pixel is its accumulated transmittance multiplied by effective opacity. These contributions are accumulated separately over foreground and background pixels, and the two evidence totals are compared. The following restores the damaged typesetting of Equations 7-8 according to their accompanying definitions:

\[ W_k(g)=\sum_{I_i\in\mathcal I_{\mathrm{sel}}}\sum_{r\in R_i(g)}\mathbf 1[M_i(r)=k]T_i(r,g)\alpha_i(r,g),\qquad g\in\mathcal G_{\mathrm{fg}}\iff W_1(g)>W_0(g). \]

Here, \(R_i(g)\) denotes the Gaussian's projected pixel set in a view, with \(k=1\) for foreground and \(k=0\) for background. Transmittance suppresses contributions hidden behind earlier content along the ray. This is more informative than a projection hit alone because it asks whether the Gaussian materially contributes to the target image.

Cross-view background cropping then counts valid in-frame projections and projections landing on mask background. A candidate is relabeled as background when the conflict fraction exceeds a threshold, preventing mask spillover in one view from permanently absorbing background geometry. The count uses in-frame projections and should not be interpreted as a complete occlusion-aware visibility test.

3. Multi-view verification and spatial refinement: reject distractors with referring boxes, then repair gaps with neighborhoods

Category masks may contain both the referred objects and irrelevant instances of the same category. The method therefore returns to the target boxes predicted during parsing. It projects foreground Gaussians into keyframes and counts in-frame observations \(N_{\mathrm{vis}}\) and out-of-box observations \(N_{\mathrm{out}}\). Removal requires both enough observations and an excessive out-of-box fraction. Restating the mechanism from the text of Section 3.4:

\[ R_{\mathrm{out}}(g)=\frac{N_{\mathrm{out}}(g)}{N_{\mathrm{vis}}(g)+\epsilon},\qquad \operatorname{remove}(g)\iff N_{\mathrm{vis}}(g)\geq\tau_{\mathrm{views}}\ \land\ R_{\mathrm{out}}(g)>\tau_{\mathrm{box}}. \]

The observation minimum and fractional tolerance avoid deleting genuine geometry because one box is inaccurate. This complements background cropping: cropping tests whether a point is outside the category mask, whereas box verification tests whether it belongs to the instances specified by the original expression. Section 4.2 interchanges the names of these two steps; this note assigns parameters according to the mechanisms in Sections 3.3-3.4.

Finally, a KD-Tree retrieves the 3D nearest neighbors of an unlabeled Gaussian. If the foreground fraction among its \(k\) neighbors is at least \(\tau_{\mathrm{knn}}\), the point becomes foreground, repairing gaps caused by occlusion and mask boundaries. A fixed neighbor count adapts better to Gaussian-density variation than a fixed radius. It does not create new Gaussians or recover geometry entirely absent from the input reconstruction.

A Worked Example

Consider an instruction to find all red chairs beside a table; this is an illustrative walkthrough, not a reported test case. The method selects 30 geometric keyframes, extracts the chair category, and predicts target boxes using the color and spatial constraints. SAM3 provides chair masks from reliable views, and rendering contributions assign their evidence to the original Gaussians.

Gaussians with excessive background conflicts are removed first, followed by same-category distractors outside the keyframe boxes. With the reported \(k=40\) and \(\tau_{\mathrm{knn}}=0.8\), an unlabeled Gaussian requires at least 32 foreground neighbors to be filled. Multiple targets need not compete for one winning instance. Reliable empty outputs still depend on upstream masks and geometric filtering: the main text reports neither a separate rejection classifier nor no-target accuracy.

Loss & Training

The method adds no learnable semantic parameters, semantic loss, or per-scene semantic optimization. It uses a pretrained VLM and SAM3 together with an already reconstructed 3DGS scene. Training-free does not mean the foundation models were never trained or that scene reconstruction is unnecessary.

The implementation uses PyTorch and reports a single NVIDIA RTX 4090D. The VLM is qwen3-vl-30b-a3b-instruct. Geometry settings are base voxel size \(s=0.1\), downsampling resolution \(0.05\), curvature search radius \(0.25\), saliency range \([1,10]\), and \(K=30\) keyframes.

Mask settings are \(\tau_{\mathrm{high}}=0.6\), \(\tau_{\mathrm{base}}=0.3\), \(N_{\mathrm{target}}=30\), \(N_{\mathrm{safe}}=6\), and \(\tau_{\mathrm{merge}}=0.8\). The fallback threshold is \(0.15\) for referring tasks and \(0\) for open-vocabulary tasks. Background conflict uses \(\tau_{\mathrm{conf}}=0.8\); box verification uses \(\tau_{\mathrm{views}}=8\), \(\tau_{\mathrm{box}}=0.8\), and \(\epsilon=10^{-6}\). KNN uses \(k=40\) and \(\tau_{\mathrm{knn}}=0.8\).

Key Experimental Results

Main Results

GR-LERF evaluates rendered-view pixels in Ramen, Teatime, Figurines, and Waldo. GR-ScanNet evaluates 3D points in the 10 ScanNet indoor scenes used by OpenGaussian. Both cover generalized referring expressions. Values below follow the paper's percentage-scale mIoU, and gains are absolute points; mIoU values from different tasks should not be directly conflated.

Dataset / setting	ZeroSplat	Comparator	Comparator value	Gain	Source
GR-LERF, generalized referring, four-scene mean	50.8	GOI	31.6	+19.2	Table 2
GR-ScanNet, generalized referring, point-level	41.2	InstanceGaussian	24.5	+16.7	Table 2
Ref-LERF, single-target referring	32.7	ReferSplat	29.2	+3.5	Table 3
LERF, open vocabulary, point-based comparison	52.4	LUDVIG	50.4	+2.0	Table 5
LERF, open vocabulary, pixel-based comparator	52.4	3DVLGS	62.0	-9.6	Table 5

The GR-LERF scene scores are 46.5, 56.3, 52.1, and 48.4. Teatime remains below GOI's 60.0, so the method does not lead on every scene. Section 4.6 claims a new open-vocabulary LERF state of the art, but Table 5 reports 62.0 for 3DVLGS and 61.3 for Occam's LGS. The 52.4 result leads only the point-based methods in that table; this note does not repeat the unrestricted claim.

Ablation Study

All results below use GR-ScanNet, with differences calculated against the full system at 41.2. Component toggles in Table 6 and internal-mechanism removals in Table 7 are different ablation protocols; their low-scoring configurations should not be treated as the same baseline.

Config	mIoU	Difference from full system	Source
Neither VLM nor KNN	24.7	-16.5	Table 6
VLM only, no KNN	38.6	-2.6	Table 6
KNN only, no VLM	26.2	-15.0	Table 6
Full system	41.2	0.0	Tables 6-8
Without semantic label extraction	24.3	-16.9	Table 7
Without bounding-box filtering	40.1	-1.1	Table 7
Random keyframe sampling	39.6	-1.6	Table 7
Fixed-radius spatial search	39.8	-1.4	Table 8

Key Findings

• Removing semantic label extraction causes the largest loss. Complex instructions are not interchangeable with category prompts for SAM3, and geometric refinement cannot repair an incorrect semantic target.
• In Table 6, VLM integration adds +13.9 over the baseline, and KNN adds another +2.6 when the VLM is present. Localization and completion are complementary rather than gains arising from smoothing alone.
• Table 1 reports 0 additional semantic feature storage and approximately 10 GB peak VRAM, versus approximately 3 GB and 28 GB for ReferSplat. These are not total scene storage or established full VLM-serving costs, and per-query latency is not reported.

Highlights & Insights

• Separating category segmentation from referring localization gives SAM3 short labels suited to its interface while retaining complex relations through VLM boxes. The label-extraction ablation directly supports this interface choice.
• Geometry decides which views the VLM sees, not just where semantic evidence is projected afterward. Marginal coverage of previously unseen voxels could also guide budget-limited multi-view scene question answering.
• Zero-feature means no persistent semantic vector is attached to every Gaussian, not that visual features are absent throughout the system. It reduces scene-specific storage while shifting some work to query-time foundation-model calls.

Limitations & Future Work

• The main text has no dedicated limitations section; the following are reading-based assessments of its mechanisms and reporting scope. The missing appendix prevents verification of instruction counts, 0/1/N proportions, annotation agreement, and the full evaluation implementation.
• Aggregate mIoU does not establish reliable no-target rejection, and the main text does not specify scoring when prediction and ground truth are both empty. Future evaluation should separate no-target false positives from single- and multi-target recall.
• Performance depends on the existing 3DGS, camera and depth quality, VLM boxes, and SAM3 predictions. KNN may propagate labels across nearby objects and cannot reconstruct geometry absent from the original scene.
• The accounting boundary of approximately 10 GB peak VRAM, VLM deployment precision, and serving setup are unclear. End-to-end query latency and scene-scaling curves are also missing, so resource-constrained deployment requires further validation.

Related Work & Insights

• vs ReferSplat: ReferSplat learns a referring field and localizes single targets in rendered pixels. ZeroSplat selects Gaussians directly and permits generalized target counts; a higher Ref-LERF mean does not imply superiority on every scene.
• vs OpenGaussian / InstanceGaussian: These methods emphasize learned point-level instance or semantic representations. ZeroSplat instead uses frozen-model priors and geometric filtering to avoid per-scene semantic optimization while preserving the original representation.
• vs LUDVIG / Dr.Splat: Learning-free or direct feature lifting can still require stored features. ZeroSplat avoids persistent per-Gaussian semantic vectors, illustrating why training-free, zero additional semantic features, and zero computation are different claims.

Rating

• Novelty: 4/5. Generalized 0/1/N referring and a zero-feature point-level pipeline form a clear contribution, although the components mainly use existing foundation models and geometry tools.
• Experimental Thoroughness: 3/5. Generalized, single-target, and open-vocabulary tasks plus component ablations are covered, but no-target breakdowns, scaling, and latency remain underreported.
• Writing Quality: 3/5. The pipeline is understandable, but resource-accounting boundaries and the Table 5 state-of-the-art claim require clarification.
• Value: 4/5. The method offers a practical route to language-based selection in existing Gaussian scenes, with deployment benefits still dependent on actual query workloads.