TASE: Truncation-Aware Semantic Embeddings for 3D Scene Understanding and Editing¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Autonomous Driving
Keywords: 3D Gaussian Splatting, 3D Scene Editing, Truncation-Aware Semantic Embeddings, Matryoshka Representation Learning, Cross-View Consistency
TL;DR¶
TASE projects pretrained 2D visual features into a truncation-aware, positional-bias-free latent space via a Matryoshka autoencoder and lifts them into 3DGS, enabling text-driven 3D scene editing with explicit control over semantic abstraction and substantial geometric changes.
Background & Motivation¶
High-fidelity semantic 3D scene representations such as 3D Gaussian Splatting (3DGS) are essential for robotics, autonomous driving simulation, and digital twin reconstruction. Beyond passive reconstruction, flexible scene editing—such as altering specific road actors or simulating diverse environmental conditions—is vital for constructing corner-case scenarios in simulation. Existing 3D scene editing techniques predominantly condition generative models on rendered RGB images or depth maps, iteratively updating 3DGS parameters via score distillation or pseudo multi-view synthesis. However, these direct image or depth conditionings tie the editing process rigidly to the source scene's preexisting geometry, preventing substantial structural modifications such as transforming a slender bicycle into a heavy motorcycle.
Lifting dense 2D open-vocabulary features (such as DINO and CLIP) into 3D space offers a promising avenue for semantic manipulation. Yet, directly lifting high-dimensional 2D representations remains hindered by severe technical hurdles. High feature dimensionality leads to prohibitive memory and computational overheads, while existing compression methods like discrete quantization or scene-specific autoencoders lack a principled mechanism to continuously modulate semantic abstraction—the spectrum from coarse categorical layouts to fine-grained instance details. Crucially, pretrained 2D vision backbones inherently retain strong 2D absolute positional biases (such as grid-coordinate dependencies), which trigger catastrophic cross-view inconsistencies, blurring, and floaters when backprojected and fused across multi-view 3D radiance fields.
The core tension lies in developing a compact semantic space that eliminates 2D positional biases while simultaneously allowing continuous, controllable trade-offs between scene fidelity and generative freedom. Core idea: train a lightweight symmetric autoencoder on pretrained DINOv3 features using Matryoshka representation learning and scale-translation consistency losses, creating 64-dimensional truncation-aware semantic embeddings (TASE) that enable a fine-tuned ControlNet to execute substantial geometric modifications in 3DGS through progressive channel scheduling.
Method¶
Overall Architecture¶
The TASE framework consists of four interconnected stages: projecting pretrained 2D visual features into a truncation-aware embedding space via a bias-free autoencoder, lifting these embeddings into 3DGS scenes through differentiable splatting and manifold regularization, segmenting target objects in 3D space via anchor similarity propagation, and performing progressive scene editing using a ControlNet conditioned on TASE with an artifact-mitigating scheduling strategy.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["2D Input / Rendered Images"] --> B["1. Truncation-Aware Autoencoder<br/>DINOv3 features + Matryoshka masking + de-biasing"]
B --> C["2. 3DGS Semantic Field Lifting<br/>Feature splatting + smoothness + Mahalanobis regularization"]
C --> D["3. Similarity-Based 3D Segmentation<br/>Multi-view anchor matching + spatial label propagation"]
D --> E["4. Truncation-Conditioned ControlNet & Editing<br/>FLUX.1 diffusion guidance + progressive channel unmasking"]
E -->|Iterative 3DGS parameter optimization| C
Key Designs¶
1. Truncation-Aware Autoencoder: Hierarchical Ordering and Positional De-Biasing To resolve high feature dimensionality, rigid abstraction, and multi-view inconsistencies simultaneously, the method trains a symmetric autoencoder consisting of single-block Transformers that compresses frozen DINOv3 features \(F_o \in \mathbb{R}^{H \times W \times N_o}\) into a compact hidden embedding \(F_h \in \mathbb{R}^{H \times W \times N_h}\) with \(N_h = 64\). The reconstruction objective balances cosine similarity and mean squared error: $$ \ell_r(\mathbf{f}o, \mathbf{f}_r) = \lambda}} \left(1 - \frac{\mathbf{fo^\top \mathbf{f}_r}{|\mathbf{f}_o||\mathbf{f}_r|}\right) + \lambda_r|_2^2 $$ To enforce channel ordering, Matryoshka representation learning is applied during training by uniformly sampling truncation levels from }} |\mathbf{f}_o - \mathbf{f\(\mathcal{T} = \{2^1, 2^2, \dots, 2^6\}\) and zero-masking suffix channels (\(\mathtt{mask}(\mathbf{f}_h, t)\)), requiring earlier channels to capture dominant high-level semantics. To eliminate 2D spatial coordinate cues that impair 3D multi-view consistency, a scale- and translation-consistency loss \(\mathcal{L}_{\text{cons}}\) is introduced. This forces feature maps extracted from random image crops to match the corresponding spatial crops of feature maps extracted from the uncropped image, decoupling semantic representations from absolute image coordinates.
2. 3DGS Semantic Field Lifting: Differentiable Splatting and Manifold Regularization To incorporate TASE into 3D space, each Gaussian primitive \(\mathcal{G}_i\) is augmented with a learnable 64-dimensional semantic attribute \(\mathbf{f}_i\) alongside standard geometric and radiance parameters. The feature field is rendered using identical point splatting and \(\alpha\)-compositing as RGB rendering to match rendered feature maps \(\mathcal{F}_h^{\text{rendered}}\) against pre-extracted 2D target maps \(\mathcal{F}_h^{\text{image}}\) via \(\mathcal{L}_{f_h}\). Because semantic maps have coarser spatial resolution than RGB pixels, an explicit \(k\)-nearest neighbor smoothness loss \(\mathcal{L}_s\) is applied across adjacent Gaussians to suppress high-frequency aliasing: $$ \mathcal{L}s = \frac{1}{N} \sum_j) $$ Furthermore, to prevent semantic vectors from drifting off the valid feature manifold during unconstrained gradient updates, a Mahalanobis distance regularization loss }^N \frac{1}{k} \sum_{j \in \mathtt{kNN}(\mathcal{G}_i, \mathcal{G})} \mathcal{L}_r(\mathbf{f}_i, \mathbf{f\(\mathcal{L}_{\text{maha}}\) penalizes deviations from the empirical mean and covariance computed over the autoencoder's training distribution, preserving downstream editing and segmentation reliability.
3. Similarity-Based 3D Segmentation: Anchor Matching and Spatial Propagation For localized object manipulation, target primitives must be cleanly separated from the background without corrupting surrounding infrastructure. When a user provides one or more clicks on the target object in 2D views, the system computes the median feature vector across the selected regions in \(\mathcal{F}_h^{\text{rendered}}\) to serve as reference semantic anchors \(\mathbf{f}_k^{\text{anchor}}\). An initial binary mask \(\delta_i\) is determined via cosine similarity thresholding against all Gaussian primitives: $$ \delta_i = \bigvee_k \left( \frac{\mathbf{f}_i^\top \mathbf{f}_k^{\text{anchor}}}{|\mathbf{f}_i| |\mathbf{f}_k^{\text{anchor}}|} > \tau_s \right) $$ To eliminate false negatives caused by local specularities or shadowing, labels are iteratively propagated to neighboring Gaussians under a relaxed similarity threshold \(\tau_p < \tau_s\). A spatial \(k\)-nearest neighbor majority voting filter subsequently prunes isolated outliers. Crucially, the similarity evaluation directly supports channel truncation, enabling users to isolate entire semantic categories with fewer channels or discriminate fine-grained object instances with fuller channels.
4. Truncation-Conditioned ControlNet & Progressive Editing Schedule The 2D generative component adapts a ControlNet architecture based on FLUX.1[dev]'s Vision Transformer backbone, taking bilinearly interpolated TASE feature maps directly into latent space. During training, random suffix channel masking trains the control branch to condition on varying semantic abstraction levels. To perform 3D editing with substantial topological restructuring, the optimization deploys a progressive curriculum schedule: - Initial Iterations (Aggressive Restructuring): Input feature maps are truncated to only 2 or 4 channels, providing minimal geometric conditioning and allowing prompt tokens to dominate, driving the ControlNet to synthesize entirely new geometries (e.g., transforming a bicycle into a dirt bike); - Later Iterations (Fidelity Refinement): The retained channels \(t\) are progressively scaled up to 64, while current 3DGS RGB renders with controlled noise injection are integrated into the pipeline to refine surface textures and harmonize lighting. To counteract tearing, over-densification, and geometric floaters during severe structural changes, the ControlNet undergoes a dedicated fine-tuning stage on 50,000 synthetic image pairs corrupted with random 3DGS splat perturbations (inspired by Difix3D+), enabling the diffusion prior to actively heal rendering artifacts.
Loss & Training¶
The overall training pipeline proceeds in three distinct phases: 1. TASE Autoencoder Training: Trained on ImageNet-1k for 1 epoch using an Adam optimizer with a learning rate of \(10^{-4}\) under the combined objective \(\mathcal{L}_{\text{TASE}} = \tilde{\mathcal{L}}_r + \lambda_{\text{cons}} \mathcal{L}_{\text{cons}}\). 2. ControlNet Training and Fine-Tuning: Trained on ImageNet-1k for 1 epoch with a cosine learning rate schedule peaking at \(10^{-5}\), followed by 4 epochs of fine-tuning on 50,000 perturbed pairs with learning rates of \(10^{-6}\) (ControlNet) and \(10^{-7}\) (MViT). 3. 3DGS Scene Optimization: Gaussian scenes are updated via: $$ \mathcal{L}{\text{splat}} = \lambda} \mathcal{L{l1} + \lambda}} \mathcal{L{\text{SSIM}} + \lambda_f \mathcal{L}} + \lambda_s \mathcal{Ls + \lambda $$ For local edits, dilated 2D projections of segmented Gaussians act as inpainting masks, while weight decay is applied to Gaussian opacities }} \mathcal{L}_{\text{maha}\(\alpha_i\) and base spherical harmonics \(\hat{c}_{0,i}\) to prune obsolete geometries.
Key Experimental Results¶
Main Results¶
Quantitative evaluations across diverse scene editing benchmarks measure CLIP Directional Similarity (alignment between image-space edit trajectory and text prompt) and user study preference percentages evaluating structural fidelity in geometry and appearance. Baseline comparisons include Direct Gaussian Editing (DGE), GaussianEditor (GE), EditSplat (ES), and DreamCatalyst (DC).
| Method | CLIP dir. sim. ↑ | Geometry Preference ↑ | Appearance Preference ↑ | Note |
|---|---|---|---|---|
| DreamCatalyst [ICLR'25] | 0.0985 | — | — | SDS-based; struggles with localized geometric adaptation |
| GaussianEditor [CVPR'24] | 0.1179 | 16.7% | 8.8% | Coarse geometry control; produces severe distortions under large shifts |
| DGE [ECCV'25] | 0.1247 | 19.1% | 18.7% | Multi-view iteration fails to alter underlying 3D structures |
| EditSplat [CVPR'25] | 0.1492 | — | — | Attention-guided optimization lacking controllable abstraction |
| TASE (Ours) | 0.1542 | 64.3% | 72.5% | Outperforms all baselines with commanding lead on major geometric edits |
Ablation Study¶
The ablation investigates the choice of dimensionality reduction (MTASE autoencoder vs. standard linear PCA), the scale-translation consistency loss \(\mathcal{L}_{\text{cons}}\), and the generative fine-tuning stage (FT) across reconstruction metrics (PSNR, SSIM, feature loss \(L_f\)) and editing alignment (CLIP dir. sim.).
| Config | PSNR ↑ | SSIM ↑ | Feature Loss \(L_f\) ↓ | CLIP dir. sim. ↑ | Note |
|---|---|---|---|---|---|
| PCA | 22.64 | 0.73 | 0.0879 | 0.1067 | Linear projection fails to maintain 3D geometric stability |
| PCA + FT | — | — | — | 0.1365 | Fine-tuning partially mitigates localized surface artifacts |
| MTASE (w/o \(\mathcal{L}_{\text{cons}}\)) | 23.89 | 0.76 | 0.1016 | 0.0852 | Retains 2D positional bias; severe multi-view inconsistency |
| MTASE + FT (w/o \(\mathcal{L}_{\text{cons}}\)) | — | — | — | 0.0905 | Multi-view tearing causes optimization divergence |
| MTASE + \(\mathcal{L}_{\text{cons}}\) | 24.64 | 0.78 | 0.0120 | 0.1385 | Consistency loss suppresses feature loss by nearly an order of magnitude |
| MTASE + \(\mathcal{L}_{\text{cons}}\) + FT (Full Model) | 24.64 | 0.78 | 0.0120 | 0.1542 | Optimal visual quality, geometric convergence, and prompt adherence |
Key Findings¶
- Consistency loss is vital for 3D multi-view lifting: Incorporating \(\mathcal{L}_{\text{cons}}\) causes the 3DGS feature reconstruction loss \(L_f\) to plummet from 0.1016 to 0.0120. Without removing 2D positional bias, view-dependent coordinate features prevent coherent convergence during 3D splatting.
- Channel truncation governs semantic abstraction: Truncating to 2 channels enables drastic object reconfiguration matching the text prompt (e.g., transforming a bear into a rottweiler), whereas preserving 64 channels tightly retains the exact structural geometry of the source entity.
- Geometric fine-tuning repairs rendering defects: Fine-tuning ControlNet on perturbed Gaussian renders explicitly teaches the diffusion backbone to heal ragged boundaries and floater artifacts introduced during topology restructuring.
Highlights & Insights¶
- Bridging Matryoshka learning and 3D generative control: Adapting channel-ordered embeddings provides an intuitive, continuous dial for balancing source preservation against prompt-guided restructuring, unlocking large-scale 3D geometry manipulation.
- Self-supervised spatial de-biasing for 2D backbones: By enforcing crop-level latent consistency without requiring multi-view or 3D training data, the method neatly cleanses pretrained ViT features of harmful spatial coordinate biases.
- Dynamic curriculum scheduling for 3D radiance fields: Progressively ramping up feature dimensionality while annealing primitive opacities mirrors the classical artistic progression from rough volumetric blocking to detailed surface finishing.
Limitations & Future Work¶
- Densification hyperparameter sensitivity: Aggressive structural expansions (e.g., expanding a thin bike frame into a bulky car body) require careful tuning of 3DGS cloning and splitting thresholds to avoid under-densified cavities or memory-heavy Gaussian clusters.
- Entanglement of color and semantic channels: Because TASE is learned from self-supervised visual features, subtle localized color prompts can occasionally cause semantic bleeding into adjacent background primitives.
- Computational overhead of iterative diffusion: While 3DGS rendering is real-time, executing multi-step diffusion sampling with FLUX.1 across multiple viewpoints during the editing loop demands substantial GPU memory and compute time.
Related Work & Insights¶
- vs GaussianEditor / DGE: While previous state-of-the-art editors rely on RGB or depth inputs that hard-lock scene geometry, TASE provides flexible semantic guidance that permits drastic topological changes while preserving surrounding scene context.
- vs LangSplat / Semantic Gaussians: Prior semantic 3DGS approaches primarily focus on static scene query and open-vocabulary segmentation using compressed CLIP features; TASE pioneers a channel-ordered, bias-free representation explicitly engineered as an interface for generative diffusion control.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ (Pioneering integration of Matryoshka representation learning and bias-free semantic embeddings into controllable 3DGS editing)
- Experimental Thoroughness: ⭐⭐⭐⭐☆ (Strong quantitative and qualitative evaluations on local and global scene edits with rigorous component ablations)
- Writing Quality: ⭐⭐⭐⭐⭐ (Cohesive structure, lucid methodology explanation, and clear motivation)
- Value: ⭐⭐⭐⭐⭐ (Offers high-impact utility for autonomous driving simulation, robotics training environments, and interactive digital twin asset modification)