YouTube-Occ: Learning Indoor 3D Semantic Occupancy Prediction from YouTube Videos¶
Conference: ECCV2026
Paper: ECCV official page
Area: 3D Vision
Keywords: indoor semantic occupancy, uncalibrated video, knowledge distillation, voxel-anchored Gaussianization, class prototypes
TL;DR¶
YouTube-Occ converts real-estate videos into indoor geometric and semantic pseudo-labels, then pretrains occupancy networks through differentiable rendering and dual-alignment feature distillation; with only 10% of NYUv2 labels for fine-tuning, web-video pretraining improves Symphonies from 17.43 to 22.52 mIoU (+5.09), whereas direct pseudo-label supervision yields 37.38 rather than 40.17 mIoU when the web dataset replaces NYUv2 in joint training with Occ-ScanNet.
Background & Motivation¶
Indoor semantic occupancy prediction determines which locations in 3D space are empty and which semantic categories occupy the rest, requiring both geometry recovery and inference about occluded scene structure. MonoScene, Symphonies and related methods continue to improve network architectures, but supervision remains a bottleneck: NYUv2 contains only 1,449 images, while Occ-ScanNet labels rely on RGB-D scans, camera trajectories and scene-completion resources from CompleteScanNet. Indoor scanning and annotation are expensive and privacy-sensitive, making larger manual collections difficult to scale.
Real-estate walkthroughs offer another source of data: relatively smooth camera motion, mostly static scenes and diverse layouts. Dust3R can estimate geometry from uncalibrated images, while SAN supplies category masks, making automated processing feasible. However, plausible-looking pseudo-labels are not necessarily suitable for training occupancy networks directly. Monocular reconstruction has scale ambiguity, and segmentation predictions can disagree with the target taxonomy; the paper's zero-shot and joint-training experiments expose substantial negative transfer.
Core Idea: use web videos to learn 3D representations rather than treating them as substitutes for accurate occupancy ground truth. Establish approximate geometry, render student voxel features back into images, distill 2D teacher knowledge through same-frame regions and cross-scene class prototypes, and finally fine-tune on real target-domain labels.
Method¶
Overall Architecture¶
The offline stage estimates cameras, point clouds, depth, pixel semantics and voxel pseudo-labels from indoor YouTube clips, producing 100,372 frames across 5,241 scenes. Online pretraining does not treat these voxel categories as exact supervision. Instead, it uses the associated geometry and semantic regions to establish feature correspondences between a 3D student and a 2D teacher.
The student is an existing occupancy network that takes RGB images and produces 3D representations; DINOv2 provides dense teacher features from the same image, and SAN supplies shared category masks. Voxel-anchored Gaussianization projects student features into 2D for intra-frame and cross-scene alignment. Fine-tuning retains pretrained network weights and trains the occupancy head from scratch. Dashed edges in the diagram denote pretraining supervision; the teacher branch is absent at test time.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Indoor YouTube videos"] --> B["Automated Data Construction"]
B --> C["Training frames and<br/>estimated geometry"]
C --> D["Student occupancy network"]
C --> T["DINOv2 teacher features<br/>SAN category masks"]
D --> E["Voxel-Anchored Gaussianization"]
E --> F["Dual-Alignment Feature Distillation"]
T -.->|Same-frame regions and cross-scene prototypes| F
F -.->|Updates during pretraining only| D
D -->|Transfer pretrained weights| G["Fine-tune with real labels<br/>Train occupancy head from scratch"]
G --> I["Fine-tuned occupancy network<br/>Semantic occupancy output"]
H["Test RGB image"] --> I
Key Designs¶
1. Automated Data Construction: obtaining approximate geometry and semantic correspondences for training
The authors collect videos from real-estate channels, download and split them into clips, and remove outdoor content using a Places scene classifier. This is not an arbitrary sample of web videos: the selected footage is relatively static and smooth, making it favorable for 3D reconstruction. Dust3R jointly estimates intrinsics, camera poses and point clouds, followed by heuristic cleaning, z-up scene alignment and approximate metric normalization under a 2.8 m wall-height assumption. This provides a scale reference for fixed-resolution voxelization, not a guarantee of accurate building dimensions.
SAN takes the benchmarks' 11 semantic category names directly and generates pixel masks without elaborate prompt tuning. Pixel semantics are assigned to reconstructed points, which are voxelized at 0.08 m using majority-class voting within each voxel. A volume in front of the current camera is then cropped to form frame-level occupancy labels. Unlike Occ-ScanNet labels that benefit from scans and CAD-based completion, these reconstructions predominantly describe visible surfaces, leaving complete occupancy in unseen regions unreliable. Automation removes per-frame manual annotation, but not geometric, taxonomic or visibility assumptions.
2. Voxel-Anchored Gaussianization: constraining rendering freedom so 3D features can receive 2D supervision
Teacher features occupy a 2D pixel grid, whereas student representations occupy 3D voxels, preventing direct location-wise comparison. Each voxel becomes a Gaussian primitive: its position is fixed at the voxel coordinate, its features come from the student representation, its opacity is predicted by an MLP, and its scale and rotation are fixed to the voxel size and identity matrix. Differentiable rasterization with the estimated camera produces a feature map at the corresponding viewpoint and propagates gradients back to the student.
The purpose is not to optimize a freely deformable Gaussian scene, but to provide a constrained voxel-to-pixel interface. Features and opacity are still learned, so this module is neither fully deterministic nor parameter-free. Fixed scale and rotation limit the freedom to alter geometry merely to satisfy a 2D target. Table 6 reports drops of 1.07 mIoU percentage points with direct projection and 1.76 with learnable scale and rotation, supporting the constrained geometric prior.
3. Dual-Alignment Feature Distillation: same-frame regions provide local correspondences, while prototypes provide cross-scene consistency
Intra-frame alignment applies masked average pooling to DINOv2 features using SAN category masks, producing category-region embeddings. The rendered student features are pooled with exactly the same masks. The region contrastive loss treats student and teacher embeddings of the same category in the same frame as positive correspondences, contrasting them against other category regions. A region here is a category mask, not necessarily an individual object instance: several chairs in one image may contribute to one chair embedding. Pooling reduces reliance on individual boundary pixels but can dilute local detail.
Within-frame alignment alone cannot ensure that chairs in different houses share a stable representation. The framework therefore maintains one prototype per category, updated online from teacher features and category masks with an exponential moving average; the default momentum is 0.999. A prototype contrastive objective pulls student regions toward their corresponding category prototypes and separates them from other prototypes. The two objectives preserve image-specific local correspondences and aggregate cross-scene semantics, respectively. Because prototypes originate from the teacher rather than manually labeled occupancy, they may reduce—but cannot guarantee removal of—teacher bias.
Equations (2)–(4) in the cached text have substantial extraction damage, including incomplete brackets, temperature terms and operators. This note does not reconstruct the authors' exact formulas; refer to the original paper. The text and figure establish that both losses are contrastive and that the total objective sums the two.
A Worked Example¶
Consider an illustrative real-estate clip showing a sofa, wall and floor. Offline reconstruction aligns the scene, normalizes scale using the 2.8 m wall-height assumption and forms geometry at a 0.08 m voxel resolution. The SAN sofa mask participates in pseudo-label construction and later pools both teacher and student 2D features during pretraining.
The student predicts voxel features from one frame, Gaussianizes them and renders them back into that view. Its pooled sofa-region representation must approach both the same image's DINOv2 sofa representation and the sofa prototype accumulated from other training scenes. The learned object is a transferable 3D representation, not a memorized pseudo-occupancy ground truth. The full occupancy network is subsequently fine-tuned on NYUv2 or Occ-ScanNet with a newly initialized head; test time requires neither the original video nor the teacher and prototype-alignment branches.
Loss & Training¶
The framework adapts existing occupancy architectures. The paper identifies the 3D feature dimension as the main interface difference between pretraining and fine-tuning, with the occupancy head trained from scratch; it does not freeze the entire student and train only the new head. Experiments cover MonoScene, Symphonies and EmbodiedOcc, but Table 2 is not a complete three-architecture-by-two-benchmark matrix: pretrained MonoScene results are listed only for NYUv2.
Pretraining sources must be distinguished. Table 3 compares YouTube-Occ, its 50K subset and in-domain target-dataset pretraining. The best Symphonies results in Table 2 numerically match the in-domain rows of Table 3 and must not all be attributed to web-video data. Both training stages use AdamW with an initial learning rate of 2e-4 and a multi-step decay factor of 0.1: 30 epochs for NYUv2 and 10 for Occ-ScanNet and YouTube-Occ, using four RTX A6000 GPUs and batch size 4.
Key Experimental Results¶
Main Results¶
All datasets use 60 × 60 × 36 grids at 8 cm resolution, with 13 label states: 11 semantic categories, free and unknown. NYUv2 has 795/654 training/test images and Occ-ScanNet has 45,755/19,764. The following mIoU and IoU values are percentages reported by the paper. Under the field's convention, mIoU averages semantic-category overlap and IoU measures class-agnostic occupancy overlap; the paper does not expand their exact definitions.
The following selection from Table 3 uses Symphonies throughout and explicitly separates web-video and in-domain pretraining. Gains are mIoU percentage points relative to random initialization in the same row.
| Target dataset and label fraction | No pretraining mIoU | YouTube-Occ-50K mIoU | YouTube-Occ mIoU | Web-video gain | In-domain pretraining mIoU |
|---|---|---|---|---|---|
| NYUv2, 5% | 15.52 | 18.25 | 18.81 | +3.29 | 19.87 |
| NYUv2, 10% | 17.43 | 22.36 | 22.52 | +5.09 | 23.58 |
| NYUv2, 100% | 29.70 | 30.87 | 31.45 | +1.75 | 31.65 |
| Occ-ScanNet, 10% | 34.18 | 36.26 | 36.54 | +2.36 | 37.80 |
| Occ-ScanNet, 100% | 47.71 | 48.15 | 48.47 | +0.76 | 49.38 |
With all labels available, YouTube-Occ pretraining gives IoU values of 52.88 on NYUv2 and 60.63 on Occ-ScanNet, versus 52.15 and 61.25 for in-domain pretraining. The ordering of pretraining sources is not identical across mIoU and IoU, so one metric alone cannot establish universal superiority.
Table 2 also supports cross-architecture applicability: on NYUv2, MonoScene improves from 26.09 to 27.02 mIoU and EmbodiedOcc from 19.14 to 20.18; on Occ-ScanNet, EmbodiedOcc improves from 44.69 to 46.17. In the settings corresponding to Table 3's in-domain rows, Symphonies reaches 31.65 on NYUv2 and 49.38 on Occ-ScanNet, which the paper compares with the strongest earlier baselines, ISO at 31.25 and RoboOcc at 47.67. These results demonstrate framework adaptability, not that every number comes from cross-domain YouTube-Occ pretraining.
Table 1 tests direct pseudo-label supervision: the pseudo-labels themselves achieve 10.06 mIoU on Occ-ScanNet, while zero-shot testing after training only on YouTube-Occ gives 8.41. Joint training with YouTube-Occ and Occ-ScanNet gives 37.38, below 40.17 for joint training with NYUv2 and Occ-ScanNet. Their IoU values, however, are 55.06 versus 51.97; the supported conclusion is poorer semantic transfer, not degradation on every metric.
Ablation Study¶
Table 5 ablates the two losses with Symphonies. Given the source distinction above, these are the framework's benchmark experiments.
| Alignment configuration | NYUv2 mIoU | NYUv2 IoU | Occ-ScanNet mIoU | Occ-ScanNet IoU |
|---|---|---|---|---|
| No pretraining | 29.70 | 49.91 | 47.71 | 59.89 |
| Region loss only | 31.12 | 52.09 | 48.15 | 60.31 |
| Prototype loss only | 31.24 | 51.76 | 48.76 | 60.47 |
| Both | 31.65 | 52.15 | 49.38 | 61.25 |
Table 6 further examines the rendering interface and prototype mechanism on NYUv2.
| Configuration | mIoU | IoU |
|---|---|---|
| Default: Gaussianization, momentum 0.999 | 31.65 | 52.15 |
| Direct 3D projection | 30.58 | 50.83 |
| Learnable scale and rotation | 29.89 | 50.53 |
| Prototype momentum 0.9 | 30.37 | 50.34 |
| Semantic threshold 0.3 | 30.42 | 50.30 |
Key Findings¶
- The prototype loss alone outperforms the region loss alone in mIoU on both benchmarks, while combining them improves further. Cross-scene statistics and same-frame correspondences are complementary rather than interchangeable.
- Learning Gaussian scale and rotation reduces NYUv2 mIoU from 31.65 to 29.89, a loss of 1.76 percentage points; direct projection loses 1.07 points. Additional learnable freedom does not help this interface.
- Full YouTube-Occ outperforms its 50K subset in mIoU at every label fraction in Table 3, but not in every IoU result: at 20% Occ-ScanNet labels, the values are 53.57 versus 54.19. Two data scales establish a trend, not a well-supported scaling law.
- Some gains in Table 3 differ from subtraction of the displayed values: with all Occ-ScanNet labels, YouTube-Occ gives 60.63 IoU versus 59.89, a displayed-value difference of +0.74, whereas the table prints +0.73. This note preserves the original metrics rather than silently making all reported values agree.
Highlights & Insights¶
- Data value is separated from label accuracy. Pseudo-labels that perform poorly as direct supervision can still accompany useful representation-learning data; this finding applies to the tested strategies, not to all hard pseudo-label supervision.
- Voxel-anchored Gaussianization connects a 2D teacher to existing 3D networks without requiring wholesale architecture replacement. The reusable component is a constrained feature-rendering interface, not a training-free deterministic projector.
- SAN masks organize both same-frame alignment and cross-scene prototype updates. Shared category partitions direct both objectives toward the same semantic space, but also expose them to the same segmentation-teacher biases.
- The value of engineering constraints is supported by the fixed-scale/rotation and 0.999-versus-0.9 momentum ablations. The 2.8 m wall height remains an approximate pipeline assumption; the paper does not provide enough ablation evidence to establish its superiority over alternative scale-estimation strategies.
Limitations & Future Work¶
- The authors explicitly identify invisible or occluded voxels, extension to multi-view or temporal networks, and removal of the wall-height assumption as unresolved issues. A 2.8 m height is only a scale prior and can fail in tall or atypical spaces.
- The smooth, static nature of real-estate footage is also a selection bias. Reliability under dynamic household activity, low robot viewpoints or strong motion blur is not directly evaluated. Publicly accessible videos are not automatically unrestricted redistribution assets; avoiding manual annotation does not establish data compliance.
- Region averaging and a single prototype per category can compress within-class geometric diversity. Instances, scales and visibility states are not explicitly separated. Multiple prototypes or reconstruction-confidence-weighted distillation are plausible extensions, but the paper does not test them.
- The main paper lacks multi-seed variance estimates, extensive data-scaling curves and foundation-model replacement ablations. Damaged formula extraction also limits exact reconstruction of the training objective from the cached text; this note is not a code-level reproducibility verification.
Related Work & Insights¶
- vs Occ-ScanNet/ISO: their label construction relies on ScanNet RGB-D data and camera information plus CompleteScanNet completion resources. YouTube-Occ explores a complementary, lower-annotation-cost source and still fine-tunes with real target-domain labels rather than replacing these benchmarks entirely.
- vs SelfOcc, S4C and GaussTR: these methods also use rendering or foundation-model supervision. This paper emphasizes uncalibrated indoor video and model-agnostic pretraining. Its discussion of calibration and platform assumptions is a comparison of applicability conditions, not a cross-domain failure experiment for every outdoor method.
- vs Pri3D, HASSC and GaussRender: Table 4 reports NYUv2 Symphonies mIoU of 28.40, 30.70 and 30.37, respectively, versus 31.65 for this framework and 29.70 for the original baseline. These results support the objective design in this setting, not the claim that Pri3D fails at its original task of using 3D priors for 2D representation learning.
- vs YouTube-VLN: both mine real-estate videos, but YouTube-VLN targets vision-and-language navigation. The present pipeline—clip splitting, filtering, reconstruction, segmentation and voxelization—also offers a reference for other indoor 3D tasks such as depth estimation and surface reconstruction.
Rating¶
- Novelty: 4/5. The combination of uncalibrated indoor web data and dual-alignment pretraining is valuable, although most building blocks originate in existing reconstruction, segmentation and distillation methods.
- Experimental Thoroughness: 4/5. Two benchmarks, multiple architectures, few-shot settings and component ablations provide useful coverage; multi-seed statistics and more systematic pipeline analysis remain missing.
- Writing Quality: 3.5/5. The problem-solution relationship is clear, but in-domain and web-video results require careful separation, some table gains disagree with displayed values, and the cached equations have extraction damage.
- Value: 4/5. A reusable pretraining path for label-scarce indoor 3D perception; the strongest evidence concerns low-label transfer, not wholesale replacement of real 3D annotation.