Enhancing Embodied Reasoning and Grounding by Novel View Synthesis¶

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/4036
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/4217.pdf
Project: https://splatreasoner.github.io/
Area: Multimodal Reasoning
Keywords: novel view synthesis, 3D Gaussian Splatting, embodied question answering, visual grounding, view selection

TL;DR¶

SplatReasoner turns a Gaussian scene into visual memory that can be reframed, combining visibility-based retrieval, local novel view synthesis, and text-based judging with verification to give a VLM more useful evidence, raising GPT-4o LLM-Match on OpenEQA from 3D-Mem's 54.6 to 57.8.

Background & Motivation¶

Embodied question answering requires more than recognizing a current image: it must answer questions about locations, attributes, and relationships using past observations. 3D-Mem compresses historical RGB-D observations into object-centric memory snapshots, preserving visual detail while reducing redundancy. Its candidate images, however, still come from previously recorded camera positions. If an object is occluded, truncated at an image boundary, or too small, stronger language reasoning may still lack the necessary evidence.

Language-enhanced 3D Gaussian Splatting (3DGS) offers a different kind of memory: it stores geometry and appearance while supporting rendering from unrecorded viewpoints. Yet attaching CLIP features to Gaussians and searching by text similarity usually retrieves object categories rather than resolving instance-level relationships such as which object is near a window. Instead of training a new model that directly understands 3D point clouds, the paper puts existing 2D vision-language models (VLMs) to work over renderable 3D memory.

The key is not simply to provide more images, but to change how evidence images are obtained. Geometry and semantics first narrow the search, local views then reframe relevant objects, and a final comparison checks whether the new images supply information needed by the question. Core Idea: use Gaussian novel view synthesis as an inference-time evidence acquisition tool, rather than limiting reasoning to historical camera positions.

Method¶

Overall Architecture¶

The inputs are historical scene images with camera poses and a user question, with the scene reconstructed in advance as semantically enriched 3D Gaussians. Semantic Visibility Retrieval finds a useful recorded view for each candidate object cluster; Local Novel View Synthesis adds nearby views; Text-Based Judging and Verification selects useful evidence from them. The output can be a natural-language answer or a grounded 3D region constrained by the selected views.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Historical images and poses<br/>Question and semantic Gaussian memory"] --> B["Semantic Visibility Retrieval"]
    B --> C["Local Novel View Synthesis"]
    C --> D["Text-Based Judging and Verification"]
    D --> E["Question answering output<br/>or grounded 3D region"]

A novel view here is rendered from an existing 3D reconstruction, not captured after a robot physically moves. It can reorganize evidence already recorded in scene memory, but cannot guarantee recovery of genuinely unobserved content. The pipeline remains passive episodic-memory question answering rather than an online exploration policy.

Key Designs¶

1. Semantic Visibility Retrieval: find candidate entities, then identify recorded views that actually see them

The Gaussians must first become searchable by the question. Question-answering experiments follow 3D-Mem's closed-set object-category protocol: an object detector supplies categories, and multiview semantic observations are directly registered onto Gaussians. Grounding instead uses Dr. Splat's open-set pipeline: SAM segments image regions, CLIP extracts region features, and those features are registered onto Gaussians. These serve different comparison protocols; the question-answering experiments should not be described as fully open-vocabulary retrieval.

An LLM then extracts categories likely to supply evidence, rather than passing the entire question directly to CLIP. A question about where to nap, for example, retrieves evidence objects such as pillows and cushions. The later VLM handles the answering intent; this stage only locates relevant regions. Closed-set memory activates Gaussians by category, whereas open-set grounding uses cosine similarity to evidence text with a threshold of 0.5. The cached multicategory similarity equation is corrupted, and the prose does not clearly specify aggregation, so no summation or maximum rule is reconstructed here.

Activated Gaussians are local primitives, not complete objects. HDBSCAN groups their 3D centers into candidate instance clusters; when category labels are available, clustering occurs within each category to avoid mixing object types. Perfect instance segmentation is unnecessary because this stage mainly reduces candidate viewpoints, while language-based relationship reasoning happens later.

Selecting a recorded view for each cluster requires more than checking whether its projection falls inside the image or counting image pixels. The method renders a cluster activation map, then checks which Gaussian in the entire scene contributes the largest rendering weight at each activated pixel. A dominant Gaussian counts as visible target evidence only if it also belongs to the target cluster. Foreground occluders therefore do not count as visible target Gaussians merely because they overlap its projection. The recorded view with the highest following score is selected:

\[ S^{\mathrm{vis}}_{l,i}=\frac{|\mathcal{G}^{\mathrm{vis}}_{l,i}|}{|\mathcal{G}^{\mathrm{cluster}}_l|}. \]

The denominator counts Gaussians in cluster \(l\), and the numerator counts that cluster's Gaussians actually visible from camera view \(i\); a Gaussian is not counted repeatedly for covering multiple pixels. This definition follows the recoverable meaning of the paper's Eq. (8). It acts more like a proxy for object surface coverage than screen area or a Gaussian-volume-weighted sum, making it less susceptible to oversized outlier Gaussians.

2. Local Novel View Synthesis: add views around relevant regions instead of revisiting the entire memory

Given \(L\) candidate clusters and their initial views, the method perturbs each view left, right, forward, and backward, rendering \(V=4\) novel views with 3DGS. It retains the initial view in the candidate pool, yielding \(1+V\) candidates per cluster and \(L+L\times V\) candidates overall. This is local augmentation, not unconstrained optimization over arbitrary camera positions throughout the scene.

Local synthesis addresses cases where the relevant object has been retrieved but the composition remains poor. An initial view might reveal only part of a target, while a nearby rendering may place both the target and a reference object in view, exposing color, shape, or relative-position evidence. Semantic retrieval concentrates the synthesis budget on question-relevant regions, and visibility scoring supplies a relatively reliable starting point. Novel view generation therefore does not begin by guessing where to look in an empty search space.

The design does not train a generative image editor or introduce a question-answering-specific 3D encoder. It reuses the renderability of an existing reconstruction, so its benefits depend on reconstruction quality. The main paper specifies the four perturbation directions but does not adequately describe translation magnitudes or all camera-adjustment details. The current cache alone does not support a fully reproducible pose-sampling implementation.

3. Text-Based Judging and Verification: compare what each view can answer, not just geometric visibility

Seeing more of an object does not necessarily make a view more useful for the question. A larger image of a chair might still exclude the window needed to identify the chair beside it. Candidate views therefore undergo question-conditioned VLM evaluation. Each image and the question are processed independently to obtain an answer supported by that view, or a statement that evidence is insufficient. An LLM then selects among these text candidates alone. This avoids presenting a long image sequence to the VLM at once and reduces cross-image confusion.

Text aggregation here does not mean precompressing the entire scene into generic captions. It retains each view's judgment about the current question: one may be inconclusive, while another clearly reveals the target attribute. Selecting the better-supported candidate answer implicitly selects its associated view. Text-only judging outperforms a judge that reads both images and text in the reported ablation, but this is an empirical result for this setup, not a general reason to discard visual inputs in every multi-image task.

Verification then compares the selected view with its corresponding initial view to determine whether previously missing information has become available. The original image remains a reference rather than assuming every rendering is better. This is not an independent authenticity check: incorrect renderings or descriptions can still mislead the judge. The main paper provides the comparison principle but does not fully specify reproducible verification prompts and decision rules.

The number of output views depends on the task. Embodied question answering retains one final view per initial candidate, giving \(L'=L\), and feeds these RGB images with the question to the answering model. Single-target localization and 3D referring segmentation use \(L'=1\), whereas the number for multitarget localization varies with the query. For grounding, activated Gaussians are first restricted to the union of final-view frustums, then the VLM's question-conditioned reasoning selects the target region. Frustums supply spatial constraints and language reasoning distinguishes same-category instances; everything in view is not automatically part of the answer.

A Worked Example¶

Consider a relational question asking which bed is closest to the window. The LLM first extracts beds as the evidence category, semantic retrieval activates beds in the scene, and spatial clustering separates candidate regions. Visibility-based retrieval picks a relatively clear recorded image for each cluster. At this point, the system has candidate beds but has not decided which satisfies the relationship.

Each candidate view then gains four local novel views. An initial image may truncate the window, while a new view may show the bed and window together. Per-image VLM evaluation expresses these relational cues as text judgments, which Text-Based Judging and Verification compares. The referring-segmentation task retains a single final view and uses its spatial constraints and reasoning output to select target Gaussians. This walkthrough illustrates the mechanism; it is not an additional quantitative experiment reported by the paper.

Loss & Training¶

The new contribution is primarily an inference-time retrieval, rendering, and selection pipeline. It introduces no new loss for training the VLM and requires no additional fine-tuning of a 2D model for 3D reasoning. The system still needs a 3DGS reconstruction and semantic features registered from existing visual models. No further training refers to the reasoning model, not an absence of reconstruction costs across the system.

The main question-answering experiments use GPT-4o and Qwen3-VL-8B, respectively, under the corresponding closed-set memory protocol for comparison with 3D-Mem. Grounding uses open-set semantic registration. The cached main paper does not fully list model versions, prompts, and invocation costs for every stage, so the description should not be expanded into an unsupported training configuration.

Key Experimental Results¶

Main Results¶

OpenEQA uses the passive EM-EQA protocol, with 1,623 questions in all reproduced experiments. LLM-Match uses a language model to judge semantic agreement between predicted and reference answers; higher is better, and it is not exact-string-match accuracy. The following results come from Table 1. Frame counts cover only final views supplied to the answering stage, excluding candidate images additionally rendered and evaluated during selection.

Method	GPT-4o LLM-Match ↑	Qwen3-VL-8B LLM-Match ↑	Average Final Frames ↓
Multi-Frame	49.1	48.2	3.0
3D-Mem	54.6	50.8	2.7
SplatReasoner	57.8	51.6	2.6

The grounding table uses the paper's Gaussian-space evaluation rather than directly reproducing a standard point-cloud or bounding-box leaderboard. ScanRefer contributes 141 scenes and 9,508 queries; Multi3DRefer uses its multitarget validation subset with 133 scenes and 2,757 queries. The referring-segmentation evaluation additionally constructs 49 spatial questions and target annotations. The following subset of Table 2 reports 3D mIoU, which measures volumetric overlap between activated Gaussians and ground-truth regions, depends on Gaussian scales, opacity, and the paper's volume normalization, and is higher-is-better.

Method	ScanRefer 3D mIoU ↑	Multi3DRefer 3D mIoU ↑	3D Referring Segmentation 3D mIoU ↑
Dr. Splat (category queries)	8.73	4.69	10.03
Dr. Splat (sentence queries)	9.44	4.14	10.56
ReferSplat	3.14	1.25	2.34
SplatReasoner	11.12	6.32	12.46

These results support improved relational grounding under the paper's direct Gaussian-search protocol, but not a universal lead on every metric. Referring-segmentation Acc@8 is 45.14, slightly below sentence-query Dr. Splat's 45.21. Acc@8 is the proportion of predictions whose IoU with the ground truth exceeds 8%, not localization accuracy at a more conventional high IoU threshold.

Ablation Study¶

The following combines Tables 3 and 5, both evaluated on 184 question-answering examples. The first group compares initial-view scoring, while the second compares judge input formats during novel-view selection. They should not be treated as one controlled module-removal experiment.

Comparison Group	Config	LLM-Match ↑	Average Final Frames
Initial view	3D-Mem	45.4	3.1
Initial view	Volume score	47.8	2.7
Initial view	Visibility score	48.2	2.7
Judge input	Image	47.1	2.7
Judge input	Image and text	46.9	2.7
Judge input	Text only	50.5	2.7

Key Findings¶

Relative to 3D-Mem, main-experiment gains are 3.2 LLM-Match score points with GPT-4o and 0.8 with Qwen3-VL-8B. The gain depends on the model setup and should not be presented as uniformly large.
Visibility scoring exceeds volume scoring by 0.4 points for initial retrieval. The authors attribute the latter's weakness to oversized outlier Gaussians disproportionately affecting scores, whereas count-based visibility is more stable.
Text-only judging exceeds image-and-text judging by 3.6 points. Identical final frame counts do not imply identical total computation: local rendering and per-image evaluation still incur additional costs.

Highlights & Insights¶

Treat the viewpoint as an adjustable source of evidence. Rather than asking a VLM to infer answers from missing cues, the system changes the observation before reasoning. This is the central distinction from fixed-snapshot memory.
Assign different roles to geometry and semantics. Visibility addresses occlusion and coverage, whereas question-conditioned judging assesses relevance. Combining them avoids confusing greater visibility with better answerability.
Preserve visual detail while compressing the selection interface. Final answering can still read images, but selection aggregates per-view textual judgments. The experiments explicitly compare this interface choice.

Limitations & Future Work¶

The authors explicitly restrict the setting to passive episodic memory, without demonstrating active robot navigation or long-horizon exploration. Offline synthesis is not equivalent to gathering new information through physical action.
Reconstruction, semantic registration, clustering, and judging form a chain of dependencies. An incorrect reconstruction or omitted retrieval category can remove the correct candidate entirely, and the paper does not establish reliable self-correction of that chain.
The 3D evaluation converts data into Gaussians, processes volumes, and includes a small manually constructed question set. ReferSplat's supervision is also constructed by the authors using foundation models. Conclusions should remain specific to this protocol rather than imply dominance on standard grounding leaderboards.
The main paper does not adequately report end-to-end latency, invocation costs, or complete pose-sampling details. Fewer final images save capacity at an interface; they do not establish that the entire system is cheaper.

vs 3D-Mem: Both reduce redundant historical observations. 3D-Mem selects fixed memory snapshots, whereas this method places semantic memory in a renderable Gaussian scene and expands local view candidates, at the cost of reconstruction and additional evaluation.
vs Dr. Splat: The method reuses direct semantic registration and 3D search, but does not make text similarity the final arbiter of relationships. The VLM and view selection distinguish target instances among same-category candidates.
vs ReferSplat: ReferSplat optimizes Gaussian features using fine-grained language supervision, whereas this method relies more on existing 2D models reading images at inference time. Compatibility differences between direct Gaussian search and rendering-based optimization are important context for the comparison.

Rating¶

Novelty: 4/5. Question-conditioned evidence selection incorporates novel view synthesis in a focused, understandable way, although most underlying components come from earlier methods.
Experimental Thoroughness: 3/5. Two question-answering model settings, 3D grounding, and judge ablations provide useful coverage, but custom evaluation and incomplete cost reporting limit generalization of the conclusions.
Writing Quality: 3/5. The overall argument is clear, while grounding, verification, and camera-perturbation details require supplementary material; the local cache also contains corrupted equation and table formatting.
Value: 4/5. The work offers a concrete path for existing 2D VLMs to use renderable 3D memory, particularly when fixed viewpoints leave insufficient evidence.