ActiveStructure: Plane Scene Graph-Guided Active 3D Gaussian Splatting

Conference: ECCV 2026
Paper: Official page · PDF
Area: 3D Vision
Keywords: Active reconstruction, 3D Gaussian splatting, plane scene graph, next-best-view, structural completeness

TL;DR

ActiveStructure augments active 3DGS mapping with a persistent plane scene graph, uses relational anomalies and localized 3D gaps to select views, and turns established walls from exploration attractors into suppressors, achieving 97.47% coverage and 32.13 dB PSNR on Replica within a fixed 2000-frame budget.

Background & Motivation

Active reconstruction must decide not only how to fit observed images, but also where to look next. Methods such as ActiveGAMER select views using rendering coverage or pixel/Gaussian uncertainty, which helps discover unobserved regions. However, two walls rendering convincingly on their own does not establish that their relationship is reliable. Whether a corner joins correctly, whether the floor and ceiling are parallel, and whether distant point-cloud fragments belong to the same wall are questions involving multiple surfaces. A factorized pixel score has no explicit representation of those relationships.

Neither plane priors nor scene graphs are new. PGSR and related work use planes to improve geometry from existing observations, while scene graphs often organize a reconstruction for later semantic use. This paper changes their role: unverified relationships generate new observation requests instead of merely constraining the reconstruction afterward. Large planes also have different values at different stages. Early observations of walls, floors, and ceilings establish the scene scaffold, whereas repeatedly observing those surfaces after they are complete can consume the budget needed for smaller furniture surfaces.

The method therefore does not replace the Gaussian representation or introduce a new renderer. It adds relational assessment and exploration control to the mapping system. Core idea: use a plane scene graph to measure structural deficits beyond pixel coverage, then reverse the exploration role of large planes as the scaffold matures so that acquisition and optimization focus on surfaces still lacking supervision.

Method

Overall Architecture

The input is a stream of posed RGB-D frames collected by an indoor agent; the output is an incrementally updated 3DGS map. Three representations coexist: the Gaussian map supplies color/depth rendering and pixel coverage, the plane scene graph supplies surface relationships and deficiency assessment, and a signed distance field, or SDF, supports traversability and collision avoidance.

Plane segments from keyframes are lifted into 3D and merged into the graph. Per-plane quality and relational anomalies then turn reconstruction deficits into structural gain for candidate viewpoints. Candidates combine that gain with pixel gain, the current exploration phase, and distance-related preferences. The selected view is executed through SDF-based path planning, and the resulting observation updates both maps. The four designs below follow this closed loop.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Posed RGB-D<br/>Gaussian map and SDF"] --> B["Plane Scene Graph"]
    B --> C["Quality and Relational Anomalies"]
    C --> D["Deficiency Targeting<br/>and Dual-Channel Gain"]
    A -->|Pixel coverage| D
    D --> E["Informational Polarity Reversal"]
    E --> F["View selection and safe paths<br/>Acquisition or refinement"]
    F -->|Update with observations| A

Key Designs

1. Plane Scene Graph: turn local plane observations into a persistent relational scaffold

At periodic keyframes, plane segments detected in RGB-D are back-projected into 3D and fitted using SVD. Each node stores its normal and offset, point set, area, observation count, centroid, and normal history. The difficult step is not fitting a single frame, but associating new fragments with existing surfaces. Strict normal alignment and point-to-plane proximity permit a relaxed centroid threshold, allowing distant fragments of the same wall to merge. Other candidates face a stricter centroid gate, and a depth-discontinuity test rejects geometrically similar but physically separated surfaces. Matched nodes undergo exponential moving-average updates; unmatched segments create nodes.

Edges combine normal alignment and spatial proximity, with smaller weights denoting more reliable relationships. A minimum spanning tree retains strong relationships as a backbone. This affinity should not be misread as a direct reward for every orthogonal relationship. A node is classified as structural if it has high tree betweenness centrality, high tree degree, or jointly exceeds area and point-count thresholds; the remaining nodes represent object surfaces. The first two criteria identify relational hubs, while the last protects a large but not yet well-connected wall from being treated as a small object. This geometric classification evolves with the graph rather than relying on semantic wall or chair labels.

2. Quality and Relational Anomalies: distinguish an incomplete surface from an inadequately anchored one

Intrinsic quality spans eight signals: spatial coverage, point density, fitting stability, observation count, boundary completeness, normal consistency, temporal stability, and interior completeness. Structural planes emphasize coverage, interior completeness, and normal consistency; object surfaces emphasize density, boundaries, and observation count. A sliding history of normals prevents a single good fit from being mistaken for stable convergence. Type-specific weighted aggregation produces a quality score whose higher values indicate a better reconstructed surface.

Relational assessment adds three anomalies. Low tree degree suggests inadequate anchoring; nearby nodes without graph edges indicate missing neighborhood relationships; and inter-plane angles departing from typical indoor parallel or orthogonal configurations suggest unstable normals or incomplete observations. All three scores retain a nonzero floor so that one dimension does not completely zero out the others. The central fusion rule, from Eq. (10) and its preceding definition, is:

\[ \Phi_i=\phi_i^{\mathrm{c}}\phi_i^{\mathrm{r}}\phi_i^{\mathrm{g}},\qquad \gamma_i=(1-q_i)\Phi_i. \]

Here \(q_i\) is intrinsic quality; the three \(\phi\) terms are connectivity, neighborhood, and geometric anomalies; and \(\gamma_i\) is the structural information deficit used for planning. Multiplication acts as a soft conjunction: priority becomes large when intrinsic inadequacy and several relational anomalies coincide, instead of triggering extensive revisitation for one suspicious angle. The trade-off is that high intrinsic quality or one small anomaly score can suppress the overall signal. Representing relational problems therefore does not imply responding strongly to every such problem. The paper also blends the mean and maximum anomalies over visible nodes to balance broad weakness against isolated severe defects; the cached text does not fully specify how that blended quantity is substituted into the final candidate score.

3. Deficiency Targeting and Dual-Channel Gain: turn abstract deficits into observable 3D targets

A low quality score alone does not tell the planner where to look. The method projects a plane's points into the two-dimensional tangent frame obtained by SVD, rasterizes them into a grid, and uses morphological closing to estimate an interior mask. It then identifies cells inside that mask that lack observations. The paper defines the interior deficiency ratio as:

\[ g_i=\frac{|I_i\setminus O_i|}{|I_i|+\epsilon}. \]

\(I_i\) is the interior mask and \(O_i\) contains occupied cells. Empty-cell centers are projected back into 3D as explicit targets. This differs from merely counting sparse points or extending an outer boundary. However, the interior mask is an estimate, not ground-truth surface extent: distinguishing a real opening from a missing observation still depends on geometric assumptions and observation quality.

Candidate positions come from free voxels, with directions sampled using a Fibonacci lattice on the sphere. Pixel gain counts pixels without confident Gaussian coverage; a candidate that mostly sees already tracked, low-deficit planes receives a discount. Structural gain sums deficits over visible planes and rewards views of the localized 3D gaps. Visibility uses frustum culling, distance gating, and view-angle filtering; candidates observing tracked planes also retain a nonzero visibility floor. Eqs. (13) and (14) can be clearly written as:

\[ \mathrm{IG}_{\mathrm{str}}(T_c)=\sum_{i\in V_c^{\mathrm{vis}}}\alpha_i^{(\psi)}\gamma_i+\omega_d\mathrm{IG}_{\mathrm{def}}(T_c), \]

\[ \mathrm{IG}(T_c)=(1-\rho_c)\mathrm{IG}_{\mathrm{px}}(T_c)+\omega^{(\psi)}\mathrm{IG}_{\mathrm{str}}(T_c). \]

\(\psi\) denotes the current phase, \(\alpha\) weights plane types, \(\omega_d\) weights localized gaps, \(\rho_c\) discounts pixel gain, and \(\omega\) controls the structural channel. This information gain is a computable planning proxy, not a calibrated reduction in probabilistic entropy; the channels are not required to be statistically independent. Candidate log-gains undergo softmax normalization before multiplication by a factor favoring nearby, reachable targets. RRT on the SDF produces a path with minimum-clearance constraints, followed by smoothed interpolation of translation and rotation. Gaussian optimization remains unchanged: the planner changes where supervision comes from.

4. Informational Polarity Reversal: stop complete large planes from continually attracting exploration

The first phase emphasizes structural-plane deficits and conservatively prunes candidates, prioritizing the wall, floor, and ceiling scaffold until the candidate pool is exhausted. The second shifts attention to object surfaces and raises the discount cap. Views dominated by well-reconstructed structural planes receive additional discounts, turning those planes from early attractors into later suppressors. This reversal neither negates Gaussian color or geometry losses nor requires making every plane's gain negative. It changes type weights, discounts, and termination signals.

The third phase revisits keyframes with the lowest rendering fidelity and increases mapping iterations until the target keyframes exceed a quality threshold. Phase boundaries are thus driven by reconstruction state rather than predetermined frame allocations. That does not make the evaluation budget-free: all phases remain within the 2000-frame sequence limit. Final refinement can improve images while reducing geometric completeness, so the multi-metric ablation matters more than assuming every stage monotonically improves every metric.

A Worked Example

Consider an office whose walls have mostly been observed, but one wall contains an uncovered interior region and its relationship to the floor remains unstable. This is an illustrative mechanism walkthrough, not an additional experiment. The graph first merges fragments belonging to the wall. Its quality signals detect missing interior coverage, while relational scores identify weak anchoring, jointly raising the structural deficit. Empty grid cells become 3D targets, so a candidate facing the gap can remain valuable despite offering few new pixels.

After additional views complete the wall, its deficit falls. In the second phase, candidates repeatedly viewing that wall are discounted and furniture surfaces become more valuable. Finally, the agent returns to low-fidelity keyframes for optimization instead of seeking new regions. The Replica Off2 ablation illustrates the trade-off: the deficiency-targeting configuration achieves 97.21% coverage versus 96.58% for the full model, while PSNR rises from 29.03 to 30.94 dB.

Loss & Training

Incremental mapping updates Gaussians through color and depth supervision. Keyframes initialize new Gaussians in uncovered regions, and the map undergoes splitting, cloning, and pruning. The cached extraction of Eq. (3) is truncated at the depth residual, so this note does not reconstruct the full loss or invent loss weights, iteration counts, plane thresholds, the number of candidate directions, or phase thresholds.

Experiments use posed RGB-D observations in Habitat with a \(60^\circ\times90^\circ\) field of view, a single RTX 4090, and 2000 frames per sequence including all three phases. The paper states that active GS comparison frameworks use SplaTAM as the mapping backbone without changing Gaussian attributes. Passive systems receive ground-truth poses with tracking disabled. The results therefore primarily test exploration strategies, not real-robot SLAM under localization error.

Key Experimental Results

Main Results

Replica contains 8 indoor scenes and Matterport3D, abbreviated MP3D, contains 5 larger environments. Acc is geometric accuracy error and Com. is completion error, both in cm and lower-is-better; C.R. is coverage at a 5 cm threshold and higher-is-better. PSNR is measured in dB; higher SSIM and lower LPIPS are better. The following averages are selected from Tables 1 and 2.

Dataset	Method	Acc ↓	Com. ↓	C.R. ↑	PSNR ↑	SSIM ↑	LPIPS ↓
Replica	ActiveSplat	1.43	1.58	93.64%	24.72	0.854	0.170
Replica	ActiveGAMER	1.28	1.74	96.59%	30.99	0.960	0.137
Replica	ActiveStructure	1.16	1.49	97.47%	32.13	0.974	0.116
MP3D	ActiveSplat	4.05	6.66	84.81%	21.79	0.737	0.253
MP3D	ActiveGAMER	1.63	2.54	94.63%	25.43	0.912	0.285
MP3D	ActiveStructure	1.57	1.91	96.23%	26.08	0.920	0.256

Relative to ActiveGAMER, Replica PSNR improves by 1.14 dB and coverage rises by 0.88 percentage points. On MP3D, completion error falls from 2.54 to 1.91 cm, approximately a 24.8% relative reduction, while coverage increases by 1.60 percentage points. MP3D LPIPS remains slightly worse than ActiveSplat. On Replica, MonoGS obtains a lower LPIPS of 0.085 but only 73.20% coverage, illustrating why one appearance metric cannot establish overall reconstruction quality.

Ablation Study

Table 3 cumulatively adds components on Replica Off2 rather than independently removing them. Depth denotes depth error in cm.

Configuration	Acc ↓	Com. ↓	C.R. ↑	PSNR ↑	SSIM ↑	LPIPS ↓	Depth ↓
A: Pixel counts only	1.51	4.65	91.24%	28.41	0.962	0.163	1.47
B: Add PSG gain	1.42	3.45	92.07%	28.65	0.963	0.157	1.39
C: Add anomaly detection	1.33	1.68	96.98%	29.12	0.964	0.135	1.22
D: Add deficiency targeting	1.17	1.56	97.21%	29.03	0.963	0.131	1.07
E: Full model	1.22	1.65	96.58%	30.94	0.976	0.115	0.91

Key Findings

• B to C reduces completion error by 1.77 cm and increases coverage by 4.91 percentage points, the largest single structural improvement. Building a graph alone is insufficient; converting its anomalies into planning weights matters.
• D to E increases PSNR by 1.91 dB but worsens Acc by 0.05 cm and reduces coverage by 0.63 percentage points. This change includes both phase control and final optimization, so it does not isolate the causal contribution of polarity reversal alone.
• Table 4 reports 20.5 minutes on Room0 versus 114.5 minutes for ActiveGAMER, approximately a 5.6-fold speedup; Room1 takes 17.6 versus 99.1 minutes. These are total times on two scenes, not a dataset-wide average frame rate.

Highlights & Insights

• A structural representation can request observations, not just regularize optimization. Separating apparent coverage from stable relationships is a more transferable idea than changing the renderer.
• Back-projecting deficiencies turns diagnosis into a 3D target. Other active mapping systems could reuse this pattern, provided they check whether the mask confuses real openings with missing observations.
• The same surface type has different information value at different stages. State-driven changes in type weights and discounts better reflect indoor reconstruction needs than continually rewarding large surfaces.

Limitations & Future Work

• Author-acknowledged trade-off: final appearance optimization sacrifices some geometric accuracy and completeness; the full model is not best on every metric.
• Domain assumptions: plane and canonical-angle priors suit indoor buildings, while curved surfaces, slanted structures, and atypical layouts may cause false alarms. The cache provides no targeted evaluation of these cases.
• Evaluation boundary: posed simulator observations do not establish robustness to real depth noise, pose drift, or dynamic occlusions. The efficiency table covers only two Replica scenes.
• Interpretation and reproducibility: cumulative ablations cannot exclude component interactions. The cache reports no error bars or multiple-seed statistics, some equations are damaged in extraction, and several thresholds are unspecified. Independent component toggles, noise tests, and parameter sensitivity analyses would strengthen the evidence.
• Relational signals remain proxies: a leaf in a minimum spanning tree need not indicate insufficient observation, and multiplicative fusion can suppress an isolated severe anomaly. Confidence-aware graphs and more selective anomaly aggregation are possible follow-ups, not validated contributions of this paper.

Related Work & Insights

• Versus ActiveGAMER / ActiveSplat: the former relies on rendering coverage and the latter combines voxel and Gaussian information. ActiveStructure adds cross-surface relationships without discarding pixel coverage as the foundation for exploration.
• Versus ActivePolicy: its graph nodes are candidate viewpoints, and spectral properties guide selection. Here nodes are persistent scene surfaces, shifting the question from which viewpoint matters to which structural relationship remains unreliable. The cache contains no direct main-table comparison with ActivePolicy, so the conceptual distinction does not establish an empirical win.
• Versus PGSR / 2DGS / PlanarRecon: these improve planar representations, surface reconstruction, or plane detection. This paper instead turns plane deficiencies into acquisition signals. Graph-driven exploration could in principle combine with improved surface representations, but that combination is not evaluated here.

Rating

• Novelty: 4/5. Persistent surface relationships guide active planning with phase-dependent suppression, although many underlying components have precedents.
• Experimental Thoroughness: 4/5. Two datasets, multiple baselines, and cumulative ablations support effectiveness; real noise, isolated component effects, and statistical stability remain open.
• Writing Quality: 4/5. The distinction between pixel coverage and relational completeness is clear, but structural proxies are not guarantees, and damaged cached equations constrain verification.
• Value: 4/5. The method offers practical ideas for plane-dominated indoor active reconstruction and useful evidence about allocating a budget between geometry and appearance.