---

title: >- [Paper Note] Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow description: >- [ECCV 2026][3D Vision][3D Scene Completion] Seen2Scene presents the first visibility-guided flow matching model trained directly on incomplete real-world 3D scans for realistic 3D scene completion and generation. tags: - ECCV 2026 - 3D Vision - 3D Scene Completion - Real-world Scene Generation - Flow Matching - Sparse Transformer date: 2026-09-19 content_hash: b238d57bd80c8de9

Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow

Conference: ECCV 2026
Paper: ECCV 2026
Project Page: https://quan-meng.github.io/projects/seen2scene/
Area: 3D Vision
Keywords: 3D scene completion, real-world scene generation, flow matching, visibility guidance, sparse transformer

TL;DR

Seen2Scene is the first flow matching approach trained directly on incomplete real-world 3D scans for scene completion and generation, introducing visibility-guided flow matching and masked sparse VAE compression that mask out unknown regions to predict high-fidelity, coherent scene geometry.

Background & Motivation

Modern 3D content creation is rapidly shifting from synthetic, isolated objects toward realistic, immersive, and interactive world models. A significant challenge in this transition is generating complete 3D scenes from partial observations—a capability increasingly central to visual simulation, robotics planning, and virtual reality content pipelines. Recent progress has been made in 3D scene synthesis leveraging deep generative models, particularly diffusion models and flow matching frameworks. However, training existing models heavily relies on synthetic 3D datasets such as 3D-FRONT that provide complete, clean ground-truth scene geometry and axis-aligned object arrangements. This fundamentally limits their ability to generalize to complex, unconstrained real-world environments.

Unlike synthetic data, real-world scans collected by RGB-D sensors (e.g., ScanNet++, ARKitScenes) are inherently incomplete and noisy due to occlusions, limited sensor coverage, and reflective surfaces. When standard generative models are trained directly on real scans, an acute tension emerges: because unobserved regions lack valid geometry labels, the models naively treat sensor-occluded spaces (such as areas beneath tables, behind furniture, or in front of windows) as empty space. Consequently, during completion and synthesis, they learn the wrong distribution and generate severe hole artifacts replicating sensor shadows. Conversely, models trained solely on synthetic complete scenes suffer from a large domain gap, failing to capture the rich variability, clutter, and off-axis object alignments of real rooms.

This paper's key insight stems from the observation that truncated signed distance field (TSDF) volumes inherently encode sensor visibility: voxels that were never reached by the camera line-of-sight are distinctly flagged with a sentinel value. Instead of fabricating pseudo ground truth, one can explicitly decouple valid observations from unobserved space across the entire generative pipeline. Core idea: introduce visibility-guided flow matching over TSDF volumes, where unknown regions are explicitly masked out during sparse VAE compression and transformer velocity field learning, paired with a ControlNet branch for partial scan injection to enable realistic 3D scene completion directly trained on partial real scans.

Method

Overall Architecture

Seen2Scene aims to reconstruct geometrically complete, structurally coherent, and realistic 3D indoor scenes from incomplete, noisy real-world scans. The scene is represented as TSDF voxel patches of resolution \(256^3\) with a voxel size of 1.1 cm, reconstructed from raw depth sequences via volumetric fusion. The pipeline consists of four cooperating components: first, a visibility-aware masked sparse VAE compresses the TSDF patch into a compact geometry latent grid while masking out unobserved regions; second, a 3D semantic layout conditioning module encodes object bounding boxes and open-vocabulary CLIP labels into spatially aligned tokens; third, a 28-block sparse transformer backbone (SparseDiT) learns the generative velocity field via visibility-aware masked flow matching, supervised exclusively on known regions; finally, for 3D scan completion, a ControlNet branch injects partial scan conditions into the frozen base model to guide generation while strictly preserving observed surface details.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Incomplete Real 3D Scan<br/>TSDF Patch (256³)"] --> B["Masked Sparse VAE<br/>Keep surface/empty, mask unknown"]
    B --> C["Compact Geometry Latents (z)"]
    D["3D Semantic Layout Boxes B<br/>CLIP Label Encoding + Painting"] --> E["Layout Conditioning Tokens"]
    C & E --> F["Visibility-Guided Flow Matching (Gψ)<br/>SparseDiT velocity field (exclude unknown)"]
    G["Partial Input Scan vp"] --> H["ControlNet Guidance Branch (Cϕ)<br/>Frozen backbone, multi-scale injection"]
    F & H --> I["High-Fidelity Complete 3D Scene<br/>Seamless observed & completed geometry"]

Key Designs

1. Visibility-Aware Structured Scene Representation and Masked Sparse VAE: Neutralizing Unobserved Regions in Latent Space
Volumetric TSDF grids naturally differentiate observed surface, free space, and unobserved regions: voxels with signed distance values within the truncation band encode known geometry, whereas voxels holding the sentinel value of \(-3 \times \text{voxel\_size}\) mark unknown space never swept by the camera frustum. To compress the high-resolution volume without letting unknown space distort the latent distribution, the authors design a masked sparse VAE. Built with residual sparse convolutional blocks, the encoder \(E_\tau\) downsamples the sparse voxel features by \(8\times\) to produce 8-channel latent representations \(z\). The decoder \(D_\tau\) features two specialized prediction heads: a category head classifying voxels as surface (distance \(< 3 \times \text{voxel\_size}\), target 1) or empty (distance \(= 3 \times \text{voxel\_size}\), target 0), and a TSDF regression head predicting continuous signed distance only for surface voxels. Critically, unknown voxels are masked out:

\[L_{\text{vae}} = L_{\text{tsdf}} + L_{\text{cat}} + \lambda_{\text{KL}} D_{\text{KL}}(q(z|v) \parallel p(z))\]

Here \(L_{\text{tsdf}} = \frac{1}{|m_s|} \sum_{j \in m_s} |\hat{v}_j - v_j|\) is evaluated strictly over surface voxels \(m_s\), and the binary cross-entropy \(L_{\text{cat}}\) is computed over all known voxels \(m = m_s \cup m_e\). By excluding unobserved voxels from training losses, the VAE learns clean, compact latent representations of observed scene geometry without hallucinating erroneous empty voids.

2. Open-Vocabulary 3D Semantic Layout Conditioning: Bridging Heterogeneous Datasets and Real Object Diversity
To guide synthesis in expansive unobserved regions, the model conditions on a 3D semantic layout represented as a set of axis-aligned 3D bounding boxes \(B = \{b_k = (c_k, s_k, l_k)\}_{k=1}^K\), where \(c_k\) is the center, \(s_k\) is the size, and \(l_k\) is the semantic label. Across heterogeneous datasets (ScanNet++, ARKitScenes, 3D-FRONT), discrete category indices suffer from incompatible taxonomies (e.g., overlapping terms like "TV", "monitor", and "screen"). To resolve this, labels are embedded into continuous feature vectors using a frozen CLIP ViT-B/32 text encoder, augmented via an LLM (GPT-4) with 20 lexical variations per category (synonyms, abbreviations, plurals, typos, and hyphenations). The semantic vectors are painted into a 3D layout volume matching the geometry latent grid, then embedded into tokens. Classifier-free guidance (CFG) randomly drops \(B\) during training, making layout conditioning fully optional at inference time.

3. Visibility-Aware Masked Flow Matching: Learning True Geometry Distributions from Partial Real Scans
Rather than relying on curvature-heavy diffusion trajectories, the generative backbone \(G_\psi\) is formulated via flow matching with a 28-layer rotary position embedding (RoPE) sparse transformer (SparseDiT). The model learns a vector field pushing Gaussian noise \(z_0 \sim p_0\) to target geometry latents \(z_1 \sim p_{\text{geo}}\) along the linear interpolation \(z_t = (1 - t)z_0 + t z_1\) with target velocity \(z_1 - z_0\). Crucially, geometry tokens belonging to unobserved space (derived from the sparse VAE visibility mask) are excluded from the loss calculation:

\[L_{\text{gen}} = \mathbb{E}_{z_0 \sim p_0, z_1 \sim p_{\text{geo}}, t \sim U(0,1)} \left[ \| G_\psi(z_t, t, B) - (z_1 - z_0) \|_2^2 \right]\]

Because different scanning passes across thousands of real rooms capture objects from complementary viewing angles, the transformer discovers the underlying true geometric distribution of furniture and room structures across aggregated observations, learning to infer missing geometry without being penalized for unobserved voxels during training.

4. ControlNet-Guided Scan Completion and MultiDiffusion Large-Scale Tile Assembly
For the completion task, the input is an arbitrary partial scan \(v_p\). To preserve known geometric details while leveraging the pretrained generative prior, a ControlNet branch \(C_\phi\) is initialized from the weights of \(G_\psi\). Training operates self-supervised: a more degraded partial scan \(v_p\) is simulated by dropping depth frames from an existing scan \(v\). The frozen VAE encodes \(v_p\) into latent condition \(z_p = E_\tau(v_p)\), and \(C_\phi\) injects multi-scale control signals into the frozen \(G_\psi\), optimizing only \(\phi\) with flow matching loss \(L_{\text{sc}}\). For large multi-room environments, a MultiDiffusion tiled generation strategy processes overlapping \(256^3\) chunks with an overlap ratio of 0.2 simultaneously, performing average-based blending at each Euler step to ensure seamless global 3D geometry fusion.

Loss & Training

The framework is optimized in two stages using the AdamW optimizer with an initial learning rate of \(10^{-4}\), cosine annealing, 1,000 warmup steps, and a batch size of 64 on 4 NVIDIA H100 GPUs using BF16 mixed precision. Stage 1 trains the masked sparse VAE, and Stage 2 trains the masked flow matching generative backbone and the ControlNet completion branch. At inference time, 50-step Euler ODE integration is employed with a classifier-free guidance scale of 3.0.

Key Experimental Results

Main Results

Quantitative evaluations for 3D scan completion were conducted over 4,000 samples across ScanNet++, ARKitScenes, and 3D-FRONT against deterministic baseline methods SG-NN and NKSR. Metrics include Chamfer Distance (CD, on 3D-FRONT with complete mesh ground truth), \(L_2\) distance and Total Mutual Difference (TMD, measuring completion diversity via pairwise CD) in observed regions, and Uni3D Fréchet Point Cloud Distance (U3D-FPD):

Method	CD \(\times 10^{-2}\) ↓	\(L_2 \times 10^{-4}\) ↓	TMD \(\times 10^{-2}\) ↑	U3D-FPD \(\times 10^{-2}\) ↓
SG-NN (CVPR 2020)	10.77	1.90	0.00	15.41
NKSR (CVPR 2023)	5.22	2.55	0.00	21.78
Seen2Scene (w/o bbox)	1.92	0.53	1.69	9.57
Seen2Scene (Full)	2.05	0.51	3.33	7.79

For 3D scene generation, Seen2Scene was compared on the 3D-FRONT test split against BlockFusion, LT3SD, and WorldGrow, reporting DINOv2-FID (rendered over 30 views per patch), U3D-FPD, and a comprehensive VLM perceptual quality score assessed by Qwen3-VL-8B-Instruct (0–10 scale):

Method	DINOv2-FID \(\times 10^2\) ↓	U3D-FPD \(\times 10^2\) ↓	VLM Quality Score ↑
BlockFusion (ToG 2024)	12.92	28.30	3.51
LT3SD (CVPR 2025)	3.51	28.55	5.20
WorldGrow (2025)	4.44	50.83	4.69
Seen2Scene (Synthetic only)	2.73	11.55	5.61
Seen2Scene (Full)	1.90	14.56	5.70

Ablation Study

The authors thoroughly evaluated the impact of masked training, training on real scans, and semantic label conditioning choices:

Table 1: Masked Training Ablation (Reconstruction on VAE, Generation on Flow Matching) | Configuration | Recon \(L_1 \times 10^{-4}\) ↓ | Recon \(L_2 \times 10^{-6}\) ↓ | Recon CD \(\times 10^{-3}\) ↓ | Gen U3D-FPD \(\times 10^2\) ↓ | Gen VLM Score ↑ | |---|---|---|---|---|---| | w/o Masked Training | 3.24 | 7.1 | 10.64 | 20.28 | 5.66 | | Seen2Scene (Full) | 2.01 | 2.0 | 9.04 | 14.56 | 5.70 |

Table 2: Ablation Study on Generative Design Choices (Evaluated Across Datasets) | Configuration | DINOv2-FID \(\times 10^2\) ↓ | U3D-FPD \(\times 10^2\) ↓ | VLM Score ↑ | Note | |---|---|---|---|---| | Synthetic only | 2.90 | 21.49 | 5.61 | Fails on complex real-world layouts and clutter | | Discrete Category | 3.17 | 21.46 | 5.64 | Incompatible across dataset label spaces | | Without Label Synonym | 2.14 | 18.77 | 5.62 | Degraded robustness to open-vocabulary inputs | | Seen2Scene (Full) | 1.90 | 14.56 | 5.70 | Optimal fidelity and distribution alignment |

Key Findings

• Masked Training is Essential for Real Scans: Without visibility masking, reconstruction \(L_2\) error jumps from \(2.0 \times 10^{-6}\) to \(7.1 \times 10^{-6}\), and generative U3D-FPD degrades from 14.56 to 20.28. Unmasked models erroneously treat unscanned under-table voids as empty air, reproducing gaping holes in completed scenes.
• Bounding Box Guidance Balances Detail vs Large Completion: The model without bounding boxes achieves a marginally better CD (\(1.92 \times 10^{-2}\) vs \(2.05 \times 10^{-2}\)), indicating that coarse boxes can be slightly restrictive for small surface patches. However, 3D boxes are vital for recovering large missing entities (e.g., entirely occluded chairs) and boost generation diversity (TMD increases from 1.69 to 3.33).
• Training on Real Scans Breaks the Synthetic Domain Barrier: Compared to training exclusively on synthetic 3D-FRONT data, incorporating real scans reduces DINOv2-FID from 2.90 to 1.90 and U3D-FPD from 21.49 to 14.56, allowing the model to faithfully synthesize complex non-axis-aligned furniture and cluttered objects (e.g., clothes on an office chair).

Highlights & Insights

• Formulating Visibility as Loss Masking: Instead of relying on unreliable synthetic inpainting labels, Seen2Scene leverages the natural \(-3 \times \text{voxel\_size}\) sentinel value of TSDF volumes to mask unobserved regions, allowing pristine generative training on noisy, raw physical scans.
• ControlNet Adaptation to Sparse 3D Flow Matching: By adapting the 2D ControlNet paradigm to 3D sparse transformers, the model cleanly separates generative world prior modeling from partial scan conditioning, preserving observed geometry while freely imagining occluded structures.
• LLM-Augmented Open-Vocabulary Conditioning: Pairing CLIP continuous embeddings with LLM-generated synonym expansions bridges fragmented academic taxonomy splits and supports direct text-to-3D-scene synthesis through automated layout drafting.

Limitations & Future Work

• Reliance on Accurate Camera Poses and Fusion: The pipeline assumes reliable volumetric TSDF inputs. Severe tracking drift or multi-path depth sensor artifacts can introduce erroneous surfaces into the valid observation mask, propagating geometry distortion into the completion.
• Absence of Surface Appearance and Photometric Textures: Seen2Scene focuses on geometric reconstruction and SDF field modeling, omitting RGB radiance or PBR material generation, which requires cascading downstream texture diffusion models.
• Sensitivity in Expansive Unconditioned Voids: While CFG permits layout-free inference, completing very large unscanned volumes without 3D bounding box cues can occasionally yield over-smoothed or generic surfaces.

Related Work & Insights

• vs SG-NN (CVPR 2020): SG-NN uses sparse 3D CNNs for deterministic self-supervised scan completion, leading to oversmoothed predictions and zero sample diversity (\(\text{TMD}=0.00\)). Seen2Scene introduces generative flow matching, achieving far superior geometric fidelity (CD \(2.05 \times 10^{-2}\) vs \(10.77 \times 10^{-2}\)) and multi-modal completion (\(\text{TMD}=3.33\)).
• vs NKSR (CVPR 2023): NKSR is a neural kernel surface reconstruction method that fits implicit fields to local point clouds without generative priors, failing to synthesize entirely occluded furniture (CD \(5.22 \times 10^{-2}\)). Seen2Scene imagines physically plausible unseen structures from learned real-world priors.
• vs LT3SD (CVPR 2025) / BlockFusion (ToG 2024): Previous generative scene models require complete synthetic 3D scenes (3D-FRONT) and struggle on real-world sensor data. Seen2Scene pioneers visibility-guided masked flow matching, bridging the gap between raw partial scans and complete generative world modeling.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering visibility-guided flow matching and masked sparse VAE for training directly on incomplete real-world 3D scans.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across three major datasets (ScanNet++, ARKitScenes, 3D-FRONT) spanning completion and generation tasks with detailed ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear motivation, and cohesive presentation from sensor physics to flow matching.
• Value: ⭐⭐⭐⭐⭐ Resolves a fundamental bottleneck in 3D scene generative modeling by removing the dependency on synthetic ground truth, paving the way for real-world physical world models.