Pano3D: Unified 3D Reconstruction and Panoptic Segmentation

Conference: ECCV2026
arXiv: 2606.14307
Paper: Project Page victorbbt.github.io/Pano3D
Code: Project Page (Google DeepMind × École des Ponts)
Area: 3D Vision / Panoptic Segmentation
Keywords: Feedforward Reconstruction, Panoptic Segmentation, Set-based Mask Decoder, Joint Training, Multi-view Consistency

TL;DR

Pano3D directly appends a Mask2Former-style set-based mask decoder to a Feedforward Reconstruction Model (FRM, e.g., MUSt3R / Pi3), and jointly fine-tunes the geometric decoder using both geometric and semantic losses. This represents the first approach to simultaneously output dense point clouds and 3D panoptic segmentation in a single feedforward pass without requiring external 2D models or post-processing clustering. It significantly outperforms state-of-the-art (SOTA) methods in semantic segmentation mIoU on ScanNet (+16.6), ScanNet200, and ScanNet++.

Background & Motivation

Background & Limitations of Prior Work: Feedforward Reconstruction Models (FRMs) represented by DUSt3R, MASt3R, MUSt3R, VGGT, and Pi3 have demonstrated that Transformers can directly regress dense pointmaps from a set of unposed RGB images without requiring camera intrinsics or extrinsics, opening the door to unified 3D scene understanding. However, these models only capture geometry ("where") but lack semantic understanding ("what"). Existing methods that attempt to add semantic capabilities to FRMs fall into two categories, both of which are suboptimal: one freezes the backbone and appends external 2D features (e.g., PanSt3R uses frozen MUSt3R + DINOv2 features; SIU3R splits early into geometric and semantic branches), relying on hand-crafted consistency regularization or QUBO post-processing to ensure multi-view mask consistency. The other adds a dense contrastive head (e.g., UNITE adds a DPT head on top of VGGT; IGGT uses SAM2 features), requiring HDBSCAN unsupervised clustering during inference to group features into instance masks, which often yields coarse boundaries.

Key Challenge: To obtain multi-view consistent 3D instance masks without introducing heuristic pipelines of clustering, post-processing, and external models—the former demands precise boundaries and consistency, whereas each component of the latter sacrifices accuracy and end-to-end differentiability.

Goal: The goal of this paper is to unify 3D reconstruction and 3D panoptic segmentation into a single, end-to-end trainable feedforward framework, completely eliminating clustering and post-processing.

Core Idea & Key Insight: The core idea supporting this goal stems from a key insight: the information required to "recognize objects" is already implicitly embedded inside the feedforward reconstruction model. The encoder phase of an FRM carries monocular 2D information, while the cross-view decoder detects spatial boundaries and 3D structural coherence during iterative refinement. Together, they constitute a strong representation of objects. Therefore, there is no need for external semantic models; simply appending a set-based panoptic head based on "learnable queries" (like Mask2Former) to the FRM output, and allowing semantic gradients to flow back to fine-tune the geometric decoder, enables native, feedforward prediction of instance masks and categories. This defines the alignment-free (no alignment, no clustering, no post-processing) pipeline of Pano3D.

Method

Overall Architecture

The input to Pano3D is a set of unposed and unordered RGB images \(\mathcal{I}=\{I_1,\dots,I_N\}\), and the output consists of a dense point cloud, semantic categories for each pixel, and multi-view consistent 3D instance masks. The pipeline is divided into four components: ① A pretrained FRM (MUSt3R or Pi3) serves as the geometric backbone, freezing its monocular encoder and fine-tuning its cross-view decoder to output cross-view consistent geometric latent features \(F_i\), from which the 3D head regresses the point cloud \(X_i\) and confidence \(C_i\). ② The Geometric Feature Bridge concatenates the frozen encoder features \(E_i\) and fine-tuned decoder features \(F_i\), processing them through an MLP to generate patch features \(\mathcal{G}_i\) for cross-attention with queries. These features are then upsampled 4x via pixel-shuffle to obtain high-resolution mask features \(\mathcal{M}_i\). ③ The Set-based Mask Decoder uses \(K\) learnable object queries to perform set prediction over the entire sequence. Each query tracks a single 3D instance across all views. The dot product of the query with a frozen CLIP text embedding yields the category. ④ Joint Training unfreezes the geometric decoder, allowing semantic gradients to flow back. The geometric loss \(\mathcal{L}_{geo}\) and semantic loss \(\mathcal{L}_{sem}\) are optimized jointly, with a gradient-scaling mechanism to protect the geometry. Since masks and point cloud pixels are aligned, lifting the masks directly to 3D yields the 3D panoptic segmentation in a single feedforward pass.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    IN["Unposed RGB Sequence<br/>{I₁…Iₙ}"] --> ENC["Frozen Monocular Encoder<br/>→ Eᵢ"]
    ENC --> DEC["Cross-view Geometric Decoder<br/>(FRM, Fine-tunable) → Fᵢ"]
    DEC --> H3D["3D Head<br/>Point Cloud Xᵢ + Confidence Cᵢ"]
    subgraph BR["Geometric Feature Bridge"]
        direction TB
        DEC --> GI["Concatenate Eᵢ+Fᵢ → 𝒢ᵢ"]
        GI --> MI["pixel-shuffle ×4 → Mask Features ℳᵢ"]
    end
    GI --> SEG["Set-based Mask Decoder<br/>K Object Queries"]
    SEG -->|"query·Text Embedding 𝒞"| CLS["Semantic Category"]
    SEG -->|"query·ℳᵢ Dot Product"| MASK["Multi-view Consistent Instance Masks"]
    H3D --> JT["Joint Training<br/>ℒ_geo + λℒ_sem + Gradient Scaling"]
    MASK --> JT
    H3D --> LIFT["Pixel-aligned Lift → 3D Panoptic Segmentation"]
    MASK --> LIFT

Key Designs

1. Geometric Feature Bridge: Concatenating Monocular Local and Cross-view Consistent Features for Mask Queries

Design Motivation: Relying solely on encoder features lacks multi-view consistency (causing a -3 mIoU drop in ablation studies), whereas using only decoder features degrades the geometry. The authors observe that "object representations are distributed across the entire hierarchy of the geometric backbone." The encoder (often self-supervised and pre-trained with CroCoV2/DINOv2) captures local monocular context, while the decoder captures scene-level consistency and geometric appearance. Therefore, the bridge module concatenates the frozen encoder features \(E_i\) and fine-tuned decoder features \(F_i\) for each frame, passing them through an MLP adapter to generate patch features \(\mathcal{G}_i\) for query cross-attention. Sequential pixel-shuffling then upsamples \(\mathcal{G}_i\) from stride 16 (patch level) to stride 4 to produce mask features \(\mathcal{M}_i\) (\(384\times 512\) input \(\rightarrow 96\times 128\) mask). Crucially, no stop-gradient is applied at the concatenation, allowing \(\mathcal{L}_{sem}\) to backpropagate gradients directly into the geometric decoder via both \(\mathcal{G}_i\) and \(\mathcal{M}_i\) paths. This is a prerequisite for joint training to succeed.

2. Set-based Panoptic Decoder: One Query Tracking One 3D Instance Across the Entire Sequence

Design Motivation: Clustering methods (such as UNITE and IGGT, which rely on HDBSCAN) produce coarse masks and suffer from boundary bleeding into the background. Pano3D adopts the set prediction framework of Mask2Former and extends it to sequences following the ODIN paradigm: it initializes \(K\) learnable object queries \(\mathcal{Q}=\{q_k\}\), where each \(q_k\) acts as a "sequence-level detector." A single query is responsible for tracking the same 3D instance across all \(N\) images. Instead of a standard classification head, categories are determined by the similarity between the query embeddings and a frozen CLIP text embedding matrix \(\mathcal{C}\) (enabling open-vocabulary queries at inference by simply swapping the text matrix). Semantic losses follow standard bipartite matching. Since the Dice loss directly optimizes IoU, mask boundaries are crisper with fewer false positives compared to clustering methods. This explains why its mIoU improvement margin is larger than mAcc (clustering blurs masks into the background, covering ground truth but introducing high false positives, which hurts mIoU but not mAcc).

3. Joint Training + Semantic Gradients as Implicit Object Priors

This is the most counter-intuitive aspect of the paper. Prior works either freeze the FRM or add pixel-wise consistency losses out of fear that semantic training will corrupt geometry. Pano3D does the opposite: unfreezing the geometric decoder \(\Phi_{geo}\) and letting the semantic gradients from bipartite matching flow back into the geometric features allows the network to learn to group pixels into discrete entities. The overall objective is:

\[\mathcal{L}_{total}=\mathcal{L}_{geo}+\lambda\mathcal{L}_{sem}\]

The authors demonstrate that while FRMs are trained to "discard blurry predictions in textureless or reflective regions," instance segmentation can serve as an implicit object prior to mitigate this defect. Ablations show that joint training improves instance localization (AP +4.9, 15.7 vs. 10.8 frozen baseline) even more than semantics, indicating that semantic gradients do not merely inject class labels, but actively refine 3D object boundaries and multi-view consistency, acting as a structural regularization. Qualitatively (Fig. 5), the jointly trained model recognizes glass windows as "blurry regions" instead of walls and exhibits higher confidence at the shaded or reflective centers of objects.

4. Segmentation Gradient Scaling: A Fuse for the Geometric Decoder

Unfreezing the geometric decoder poses a risk: the authors experimentally observe that the magnitude of semantic gradients is typically an order of magnitude larger than that of geometric gradients. Direct backpropagation would heavily perturb the weights of the cross-view decoder. To address this, the gradient ratio between the two tasks is measured without the multiplier, and then a constant scaling factor \(\gamma=0.1\) is applied to the segmentation gradient before it enters the geometric decoder. Ablations (Tab. 6) show that this scaling has minimal impact on segmentation metrics (mIoU -1) but significantly improves geometry (per-view/multi-view inlier +3%), acting as a safety valve for geometric fidelity.

Loss & Training

The geometric loss \(\mathcal{L}_{geo}\) is a confidence-weighted dense point cloud regression \(\sum_i\sum_p C_{i,p}\lVert X_{i,p}-\hat{X}_{i,p}\rVert_2 - \alpha\log(C_{i,p})\) (the exact form varies depending on the FRM: MUSt3R uses a reference-frame global point cloud, while Pi3 uses scale-invariant local point clouds + affine-invariant poses). On the semantic side, the standard classification, Dice, and BCE losses from Mask2Former are adopted, with weights of 2, 2, and 5, respectively. The geometric loss is weighted 10x relative to the overall segmentation loss. The mask decoder supervises queries at every layer for faster and more stable convergence. For the MUSt3R memory-based backbone, a specialized two-stage training is designed (Update: builds geometric memory and initializes queries; Render: renders new frames using the same batch of query states). The masks from both stages are concatenated along the view axis for a single bipartite matching, forcing the same query to detect the same instance across both stages—this is key for memory-based models to maintain instance consistency. The model is trained using AdamW on TPU v6, with 30k steps for ScanNet and 50k steps for ScanNet200/++.

Key Experimental Results

Main Results: 3D Semantic Segmentation (UNITE Protocol, RGB-only Input)

Dataset	Metrics	Pano3D(MUSt3R)	Runner-up(UNITE)	Gain
ScanNet	mIoU₃D	65.3	48.7	+16.6
ScanNet	mAcc₃D	76.6	68.3	+8.3
ScanNet200	mIoU₃D	24.6	14.5	+10.1
ScanNet++	mIoU₃D	29.7	21.6(PanSt3R)	+8.1

In class-agnostic 3D instance segmentation, Pano3D also leads, achieving an AP50 of 38.9 on ScanNet (outperforming UNITE's 29.6 by +9.3). The authors honestly present a caveat: on ScanNet++, which contains a massive number of instances, Pano3D's AP25 is lower than UNITE's. This occurs because AP25 rewards detecting more small objects with relaxed precision. Clustering-based methods can group small 3D clusters and detect more objects, whereas query-based methods miss them entirely if a query is not matched; "detections are more accurate, but small objects tend to be missed when the count is extremely high."

In the voxelized joint evaluation of IGGT (where True Positives require both correct geometry and semantics): after switching to a stronger Pi3 backbone, Pano3D(Pi3) achieves a 2D mIoU of 52.5 and a 3D mIoU of 29.5 on ScanNet++, while achieving a geometric AbsRel of 2.44, comparable to or better than IGGT (2.61). This demonstrates that "stronger FRMs naturally yield features more conducive to segmentation."

Ablation Study (ScanNet Validation Set)

Configuration	mIoU₃D	AP₃D	AbsRel↓	Explanation
Frozen Geo + Sem Head	61.9	10.8	5.2	Frozen backbone, training only the semantic head
FT Semantics Only	n/a	n/a	156.3	Training only semantics collapses the geometric space entirely
Pano3D (Joint G+S)	65.3	15.7	3.5	Joint training improves both geometric fidelity and instance localization
Encoder Features \(E_i\) Only	60.4	7.8	3.4	Lacks multi-view consistency; -3 mIoU
Decoder Features \(F_i\) Only	63.8	13.9	3.7	Degrades geometric refinement capability

Key Findings

• Joint training yields the greatest contribution: training semantics alone causes the AbsRel to surge from 3.5 to 156.3 (complete geometric collapse), whereas joint G+S training preserves geometric quality while elevating the AP from 10.8 to 15.7. This proves that semantic gradients act as an "implicit structural regularization" that primarily assists instance localization rather than mere semantic classification.
• The two levels of the feature bridge are both indispensable: using only encoder features sacrifices multi-view consistency (-3 mIoU), whereas using only decoder features hurts geometry. Concatenation achieves the best of both worlds.
• Gradient scaling serves as a geometric fuse: setting \(\gamma=0.1\) incurs only a -1 mIoU drop in segmentation but boosts geometric inliers by +3%. While no scaling (1.0) achieves acceptable performance for both tasks, the authors conservatively select 0.1 to ensure geometric robustness.
• Sensitivity to-loss ratios: if the geometric weight is too weak (\(\lambda_{geo}=\lambda_{sem}\)), both geometry and semantics degrade; if it is too strong (20x), geometry slightly improves at the cost of semantics. 10x is the sweet spot.
• Query count scales with category complexity: on ScanNet++ (characterized by more categories and dense per-scene instances), 200 queries outperform 100, while increasing to 400 yields diminishing returns (saturation).

Highlights & Insights

• The insight that "object recognition is already embedded in the reconstruction model" is elegant: it reframes "adding semantics" from "attaching an external semantic system" to "awakening existing object representations within the backbone," completely removing the heuristic pipeline of clustering + post-processing + external 2D models. This is a highly reusable methodology—any scenario where "a large model implicitly contains structural information, but downstream tasks extract it using hard clustering" could benefit from adapting the "append learnable queries + backpropagate task gradients" scheme.
• Counter-intuitively unfreezing the geometric decoder: while prior works focus on "protecting geometry" by freezing the backbone, this work demonstrates that letting semantic gradients fine-tune the geometry does not degrade the geometric quality. Instead, the introduced object priors enhance geometric robustness in textureless and reflective regions. This suggests that multi-task learning is not a zero-sum game.
• Single query tracking a 3D instance + pixel-aligned lifting: multi-view consistency of instances is natively guaranteed via "set prediction + single bipartite matching," without relying on explicit consistency losses or QUBO formulations. This makes the engineering pipeline remarkably clean.
• Gradient magnitude diagnosis: first logging the gradient norms of both tasks reveals a difference of one order of magnitude. This motivated scaling the semantic gradient—this "measure first, then treat" multi-task balance strategy is highly practical and transferable to any dense/sparse hybrid geometric-semantic training.

Limitations & Future Work

• Query "bleeding" in crowded scenes (acknowledged by the authors, Fig. 6): when many objects are present, the model tends to merge separate objects (e.g., two chairs or two tables) into a single mask, leading to missed detections under high instance density. This is an inherent limitation of query-based detection, where clustering methods generally perform better at retrieving small, dense clusters.
• Geometry is slightly bottlenecked by semantics: under joint training, the per-view depth quality experiences a minor drop (slightly blurry predictions). Although scene-level MVD improves, the absolute geometric accuracy remains slightly lower than that of the original "geometry-only" FRM. Furthermore, the MUSt3R backbone (450M) is inherently weaker than VGGT (1B+), leaving geometric accuracy constrained by the backbone size.
• Limited domain generalization: training is restricted to the indoor ScanNet family, causing failures in outdoor or out-of-distribution (OOD) categories (e.g., "tree", "person"). The open-vocabulary capability works reliably only when test data is close to the training distribution.
• Still a "fine-tuning of existing geometric backbones": the authors point out that ideally, a foundation model should be trained from scratch to jointly optimize geometry and semantics; the current formulation is a transitional stage that grafts semantics onto a geometry-first backbone. Future directions: larger and stronger FRM backbones + larger-scale diverse vocabulary datasets + query deduplication/expansion mechanisms for crowded scenes.

Related Work & Insights

• vs UNITE: UNITE appends a dense DPT head on top of VGGT and distills a massive amount of teacher information (instance, semantics, and depth), relying on contrastive clustering (HDBSCAN) during inference. Pano3D optimizes the target task directly through a set-based decoder without clustering. This yields crisper masks, explaining why Pano3D's mIoU improvement (+16.6) is significantly larger than its mAcc improvement (as clustering blurs masks into the background, generating high false positives).
• vs PanSt3R: PanSt3R places a Mask2Former head, DINOv2 features, and QUBO post-processing on top of a frozen MUSt3R, while requiring the manual selection of a subset of "memory frames." Pano3D's global queries focus on the entire sequence and fine-tune the geometric decoder—achieving full scene coverage and more consistent multi-view integration, whereas static frame selection inevitably misses objects in unselected views.
• vs IGGT: IGGT utilizes SAM2 features, supports open-vocabulary queries, and relies on inference-time clustering + 2D re-projection + external semantic models, making its pipeline heavy and not purely feedforward. Pano3D achieves single-pass, pure feedforward inference; its in-domain training yields stronger semantic performance, though IGGT exhibits broader open-vocabulary generalization.
• vs ODIN (2-stage: reconstruction followed by segmentation): feeding noisy point clouds predicted by MUSt3R into the SOTA ODIN for 3D segmentation yields an mIoU of only 39.2 (whereas ODIN achieves 69.3 using ground-truth depth), because noisy point clouds fail to establish reliable 3D neighborhood correspondences. Pano3D achieves 56.1 mIoU end-to-end, proving that "decoupled reconstruction-segmentation" pipelines collapse under unposed noise, and that joint feedforward modeling is the correct path.

Rating

• Novelty: ⭐⭐⭐⭐⭐ The insight that "object representations are already implicit within the FRM, and can be awakened by letting semantic gradients flow back through queries" is elegant and powerful. Unfreezing the geometric decoder is counter-intuitive yet successfully demonstrates multi-task synergy.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across two evaluation protocols (UNITE mesh and IGGT voxel), three datasets, and two backbones (MUSt3R/Pi3). Features comprehensive ablations (training schemes, feature bridge, loss ratios, gradient scaling, query counts, and 2-stage baselines) while honestly reporting limitations such as the AP25 drop.
• Writing Quality: ⭐⭐⭐⭐ The structure is clear, motivations progress logically, and the figures align well with the text. Formulas and appendices are comprehensive; however, some two-stage details of the memory model are mathematically dense and require careful cross-referencing with the diagrams.
• Value: ⭐⭐⭐⭐⭐ Unifying 3D reconstruction and panoptic segmentation into a single feedforward pass while substantially outperforming SOTA. The paper points to a future where foundation models should jointly train geometry and semantics from scratch, which is highly inspiring for unified 3D scene understanding.

Related Papers

• [ICML 2026] EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation
• [ICCV 2025] PanSt3R: Multi-view Consistent Panoptic Segmentation
• [AAAI 2026] UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning
• [ICLR 2026] FastAvatar: Towards Unified and Fast 3D Avatar Reconstruction with Large Gaussian Reconstruction Transformers
• [CVPR 2026] OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting