ReGen3D: Generalizable Unified Representation Learning for 3D Understanding¶
Conference: ECCV 2026
Paper: ECCV 2026
Code: To be confirmed
Area: 3D Vision
Keywords: Representation Generalization / Unified 3D Learning / Representation Shift / Shared Memory Interaction / Parameter-Efficient Fine-Tuning
TL;DR¶
Addressing the vulnerability of existing 3D models tied to a single geometric format under representation shifts, this paper formalizes the new setting of Representation Generalization (RG) and presents ReGen3D, a unified framework integrating Unified Representation Tokenization (URT), Shared Memory Interaction (SMI), and Efficient Representation Adapter (ERA) to achieve cross-representation pretraining, decoupled single-format test-time inference, and lightweight adaptation to unseen geometric primitives.
Background & Motivation¶
Three-dimensional geometric representations have diversified extensively across discrete point clouds, voxel grids, continuous surface meshes, and modern 3D Gaussian primitives, each encoding distinct structural inductive biases such as permutation invariance, regular volumetric metrics, topological connectivity, or directional radiance fields. However, prevailing 3D perception models remain anchored to the single-representation paradigm: their network architectures, feature aggregation operators, and learning targets are intrinsically engineered for a single geometric modality. Once the representation shifts at test time, model accuracy drops precipitously, even when the underlying object-level semantics and physical boundaries remain completely identical.
This structural vulnerability differs fundamentally from conventional domain generalization, which assumes a fixed input structure and focuses on statistical distribution shifts such as sensor noise or environmental changes. Meanwhile, multimodal cross-modal learning emphasizes semantic complementarity across text, vision, and geometry rather than intra-3D structural variation. Previous efforts addressing multiple 3D representations suffer from two primary limitations: first, they rely on isolated, representation-specific branches with late fusion or contrastive objectives, failing to reconcile structural discrepancies within a shared representation space; second, they enforce rigid multi-branch inference coupled to training-time configurations, rendering them unable to handle single-representation test inputs or scale efficiently to novel geometric primitives.
To bridge this foundational gap, the paper formalizes Representation Generalization (RG), a setting requiring unified 3D models to jointly exploit complementary cues during pretraining, execute decoupled inference from any single available modality, and scale gracefully to unseen representations without full backbone retraining. Core idea: decouple geometric primitives into spatial coordinates and padded attribute vectors via Unified Representation Tokenization, exchange cross-representation geometric priors through learnable shared memory slots, and enable parameter-efficient adaptation to unseen representations using low-rank adapters over a frozen backbone.
Method¶
Overall Architecture¶
ReGen3D consists of three synergistic modules: Unified Representation Tokenization (URT) at the input stage, Shared Memory Interaction (SMI) within the feature backbone, and Efficient Representation Adapter (ERA) for unseen geometric modalities. During pretraining, the model receives heterogeneous 3D observations \(\mathcal{S}\) of a sample, projects them into a uniform token dimension \(D\) via URT, concatenates them with a set of learnable global memory slots, and facilitates bidirectional cross-attention within a shared Transformer backbone. The resulting latent representations feed into task-specific decoding heads for self-supervised geometric and attribute reconstruction. For downstream classification and segmentation, the pretrained encoder is finetuned with supervised objectives. When extending to an unseen representation, the shared Transformer backbone and memory bank remain frozen, while lightweight LoRA branches adapt the tokenization and attention pathways.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Heterogeneous 3D Representations<br/>(Points / Voxels / Meshes / 3D Gaussians)"] --> URT["Unified Representation Tokenization<br/>Decoupled FPS/KNN and shared linear projection"]
URT --> SMI["Shared Memory Interaction<br/>Global learnable memory bank and context concatenation"]
SMI --> BB["Shared Transformer Backbone<br/>Cross-representation knowledge accumulation"]
BB -->|Pretraining Stage| Rec["Reconstruction Heads<br/>Chamfer Distance CD + Attribute MSE"]
BB -->|Downstream Finetuning| Down["Downstream Task Heads<br/>Shape classification & part segmentation"]
Unseen["Unseen Geometric Representation<br/>(Novel geometric primitive format)"] --> ERA["Efficient Representation Adapter<br/>URT alignment + Backbone LoRA adaptation"]
ERA -.->|Direct integration with frozen backbone| BB
Key Designs¶
1. Unified Representation Tokenization: Decoupled Spatial Grouping and Unified Attribute Projection Rather than naively collapsing diverse geometric primitives into unorganized point sets—which would destroy native priors such as face normals, volumetric occupancies, or Gaussian opacities—URT represents any primitive in representation \(r_k\) as a unified attribute vector \(a_i^{(k)} = [g_i^{(k)}; s_i^{(k)}] \in \mathbb{R}^{d_a}\). Here, \(g_i^{(k)} \in \mathbb{R}^3\) denotes normalized geometric center coordinates, while \(s_i^{(k)}\) aggregates representation-specific attributes. Missing attributes across different modalities are zero-padded to a shared dimension \(d_a\). URT decouples geometric neighborhood construction from attribute mapping: \(G\) geometric anchors are sampled via Farthest Point Sampling (FPS) strictly over \(\{g_i^{(k)}\}\) and grouped via K-Nearest Neighbors (KNN) in \(\mathbb{R}^3\), ensuring uniform spatial receptive fields across all modalities. Primitive attributes are mapped into a unified dimension \(D\) by a shared linear projection \(\phi: \mathbb{R}^{d_a} \to \mathbb{R}^D\) and aggregated with a learnable representation embedding \(e_k\): $\(z_g^{(k)} = \rho\left(\{ \phi(a_j^{(k)} - a_{c_g}^{(k)}) \mid j \in \mathcal{N}_g(c_g) \}\right) + e_k\)$ where \(\rho(\cdot)\) denotes a permutation-invariant aggregation operator (e.g., Max Pooling). This formulation guarantees that zero-padded channels produce no spurious activation under \(\phi\), preserving format-specific structural traits within an aligned sequence \(Z^{(k)} \in \mathbb{R}^{G \times D}\).
2. Shared Memory Interaction: Decoupling Single-Representation Inference via Latent Prior Accumulation Standard pairwise cross-attention scales quadratically with the number of modalities and crashes when test-time inputs omit certain training streams. SMI resolves this by maintaining a persistent memory bank \(M \in \mathbb{R}^{T \times D}\) comprising \(T\) learnable slots. For any active subset of representations \(\mathcal{S}\), SMI builds a global context by concatenating active token sequences with \(M\): $\(\bar{Z}_{\mathcal{S}} = \text{Concat}\left(\{Z^{(j)}\}_{j \in \mathcal{S}}, M\right)\)$ Each active representation stream updates its tokens via cross-attention with itself as Query and the global context as Key and Value: $\(\hat{Z}^{(k)} = Z^{(k)} + \text{Attn}\left(Q=Z^{(k)}, K=\bar{Z}_{\mathcal{S}}, V=\bar{Z}_{\mathcal{S}}\right)\)$ During multi-representation training, \(M\) accumulates invariant topological priors across modalities. At test time, when only a single representation \(r_k\) is provided, the context automatically contracts to \(\bar{Z}_{\{k\}} = \text{Concat}(Z^{(k)}, M)\), preserving identical operator mechanics without structural alteration. The memory slots act as semantic anchors pulling single-modality and unseen tokens toward the shared geometric manifold.
3. Efficient Representation Adapter: Parameter-Efficient Adaptation to Unseen Representations Retraining the entire network when encountering a novel representation \(r_{K+1}\) is computationally prohibitive and undermines the premise of a representation-agnostic foundation. ERA preserves the pretrained URT embedding layers, SMI modules, and Transformer backbone in a strictly frozen state, injecting low-rank adaptation (LoRA) branches into key parameters. For any pretrained weight matrix \(W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}\), the adapted forward pass computes: $\(W' = W + B A, \quad B \in \mathbb{R}^{d_{\text{in}} \times r}, A \in \mathbb{R}^{r \times d_{\text{out}}}, \quad r \ll \min(d_{\text{in}}, d_{\text{out}})\)$ LoRA parameters are injected into URT projection layers, Transformer self-attention blocks, and downstream task heads, while only the reconstruction decoder is retrained from scratch for reconstruction tasks due to format-specific output dimensions. Because URT and SMI align incoming tokens to the shared latent geometry, ERA requires minimal trainable parameters to bridge format-specific domain gaps.
Loss & Training¶
The framework follows a two-stage training scheme. Stage 1 executes self-supervised reconstruction pretraining on available representations by minimizing joint attribute and geometric reconstruction errors: $\(\mathcal{L}_{\text{rec}} = \frac{1}{|\mathcal{S}|} \sum_{k \in \mathcal{S}} \left( \frac{1}{N_k}\sum_{i=1}^{N_k} \|\hat{a}_i^{(k)} - a_i^{(k)}\|^2 + \lambda \cdot \text{CD}\left(\hat{P}^{(k)}, P^{(k)}\right) \right)\)$ where CD denotes the Chamfer Distance and \(\lambda\) balances geometric and attribute terms. Stage 2 performs downstream finetuning with standard cross-entropy loss \(\mathcal{L}_{\text{task}} = \frac{1}{|\mathcal{S}|} \sum_{k \in \mathcal{S}} \ell(\hat{y}^{(k)}, y)\) for classification and segmentation. No manual pairwise contrastive loss is required, as structural alignment emerges organically from shared supervision and memory-mediated interactions.
Key Experimental Results¶
Main Results¶
Evaluations are conducted on the newly introduced RepShift-3D benchmark spanning ModelNet40 and ShapeNet-Part across four geometric modalities: Point (P), Voxel (V), Mesh (M), and 3D Gaussian (G). Under the Leave-One-Representation-Out (LORO) protocol, three representations are utilized during training, and evaluation is carried out for both single-modality training representations and the held-out unseen modality. The table below presents reconstruction Chamfer Distance (\(\text{CD} \times 10^{-3}\), lower is better) and classification accuracy (Cls Acc %, higher is better) on training modalities:
| Method | Native Rep. | VMG Training (CD ↓ / Cls ↑) | PMG Training (CD ↓ / Cls ↑) | PVG Training (CD ↓ / Cls ↑) | PVM Training (CD ↓ / Cls ↑) |
|---|---|---|---|---|---|
| PointNet | Point | 144.2 (M) / 74.9% | 148.5 (M) / 76.0% | 43.4 (V) / 77.1% | 43.3 (V) / 76.9% |
| DGCNN | Point | 138.6 (M) / 70.2% | 135.8 (M) / 73.7% | 35.4 (V) / 83.8% | 43.4 (V) / 79.7% |
| Point-MAE | Point | 41.2 (M) / 71.4% | 42.5 (M) / 68.4% | 26.0 (V) / 86.0% | 26.8 (V) / 75.3% |
| MeshMAE | Mesh | 102.3 (M) / 71.2% | 108.8 (M) / 70.9% | 35.1 (V) / 74.2% | 35.4 (V) / 74.4% |
| ShapeSplat | Gaussian | 44.2 (M) / 79.2% | 43.6 (M) / 80.9% | 45.6 (V) / 83.6% | 41.9 (V) / 81.7% |
| Naive Baseline | Multiple | 41.1 (M) / 70.1% | 42.0 (M) / 62.2% | 26.6 (V) / 84.3% | 26.5 (V) / 74.9% |
| ReGen3D (Ours) | Multiple | 22.1 (M) / 83.2% | 23.6 (M) / 82.1% | 25.2 (V) / 86.8% | 28.0 (V) / 83.4% |
Generalization to the held-out unseen representation under the LORO setting demonstrates significant improvements over baseline models:
| Method | Native Rep. | Unseen Point (CD ↓ / Cls / Seg) | Unseen Voxel (CD ↓ / Cls / Seg) | Unseen Mesh (CD ↓ / Cls / Seg) | Unseen Gaussian (CD ↓ / Cls / Seg) |
|---|---|---|---|---|---|
| PointNet | Point | 58.4 / 43.3% / 39.7% | 61.2 / 48.1% / 40.1% | 206.2 / 26.2% / 39.0% | 491.4 / 24.0% / 44.6% |
| DGCNN | Point | 48.3 / 45.1% / 46.1% | 53.2 / 48.3% / 38.6% | 226.2 / 46.3% / 35.9% | 58.7 / 35.2% / 39.7% |
| Point-MAE | Point | 52.4 / 42.6% / 24.9% | 55.3 / 36.8% / 24.6% | 59.2 / 30.6% / 26.9% | 63.1 / 25.2% / 27.7% |
| MeshMAE | Mesh | 64.2 / 32.2% / 42.5% | 71.7 / 30.9% / 42.1% | 121.3 / 28.3% / 26.5% | 48.8 / 38.5% / 44.2% |
| ShapeSplat | Gaussian | 54.5 / 61.2% / 46.7% | 55.7 / 58.8% / 47.4% | 52.4 / 29.9% / 51.5% | 52.2 / 45.2% / 50.1% |
| Naive Baseline | Multiple | 55.6 / 42.1% / 25.5% | 57.2 / 36.2% / 22.1% | 59.1 / 29.4% / 25.1% | 63.4 / 23.4% / 26.7% |
| ReGen3D (Ours) | Multiple | 34.5 / 78.4% / 75.2% | 33.1 / 77.6% / 76.6% | 32.1 / 67.5% / 62.0% | 32.2 / 85.1% / 56.6% |
Ablation Study¶
The table below ablates the individual contributions of URT, SMI, and ERA under the PMG training configuration (Point, Mesh, Gaussian) with Voxel held out as the unseen modality:
| URT | SMI | ERA | Seen P Rec (CD ↓) | Seen M Cls (Acc %) | Seen G Seg (mIoU %) | Unseen V Rec (CD ↓) | Unseen V Cls (Acc %) | Unseen V Seg (mIoU %) |
|---|---|---|---|---|---|---|---|---|
| - | - | - | 34.4 | 62.2% | 48.3% | 57.2 | 36.2% | 22.1% |
| ✓ | - | - | 27.2 | 76.9% | 51.2% | 55.7 | 51.8% | 52.4% |
| ✓ | ✓ | - | 23.4 | 82.1% | 58.9% | 50.6 | 72.6% | 58.6% |
| ✓ | ✓ | ✓ | 23.4 | 82.1% | 58.9% | 33.1 | 77.6% | 76.6% |
Key Findings¶
- SMI serves as the central conduit for cross-modal transfer: Introducing URT alone yields modest gains on held-out Voxel reconstruction (CD dropping from 57.2 to 55.7). However, integrating SMI leverages the global memory bank to elevate unseen Voxel classification accuracy from 51.8% to 72.6%, validating that memory-accumulated priors bridge cross-format statistical discrepancies.
- ERA isolates unseen representation adaptation: In the presence of URT and SMI, adding ERA leaves training-set metrics entirely untouched (maintaining 82.1% classification and 58.9% segmentation on seen formats) while drastically reducing unseen Voxel CD from 50.6 to 33.1 and pushing segmentation mIoU from 58.6% to 76.6%.
- Robustness under real scans and geometric perturbations: Under the VMG setting with raw LiDAR ScanObjectNN point clouds treated as unseen inputs, ReGen3D achieves 40.1 CD and 70.6% classification accuracy. Furthermore, under 10% Gaussian noise, \(\pm 20^\circ\) rotation, \(\pm 0.1\) translation, and \(\pm 10\%\) scaling perturbations, accuracy remains resilient at 75.3%, 78.2%, 79.9%, and 78.3%, respectively.
Highlights & Insights¶
- Conceptual Shift to Representation Generalization: The paper establishes that geometric encoding formats constitute a primary axis of domain discrepancy in 3D perception, elevating intra-3D format shifts from engineering hurdles to a formal machine learning paradigm.
- Decoupled Geometric Patching via Coordinate Priming: URT neatly bypasses the dilemma between information-destructive point cloud flattening and modality-specific architectures by anchoring FPS and KNN strictly to 3D centroid coordinates while embedding physical attributes into a unified dimension.
- Star-Topology Shared Memory Interaction: SMI avoids brittle pairwise cross-modal attention in favor of a memory-centered star topology, solving the longstanding dilemma of combining multi-modal training with flexible single-modality deployment.
Limitations & Future Work¶
- Loss of Explicit Connectivity Priors: By relying on coordinate-based FPS and KNN neighborhoods, URT inevitably discards explicit topological edge connectivity and face normal graphs inherent to clean surface meshes.
- Complexity in Gaussian Radiance Modeling: Recovering view-dependent spherical harmonics and radiance parameters in 3D Gaussians remains substantially harder than geometric coordinate prediction, calling for appearance-dedicated modeling in future iterations.
- Residual Fine-Tuning Overhead for New Modalities: While ERA circumvents full network fine-tuning, adapting to unseen representations still demands a small set of target-modality samples and gradient optimization steps, stopping short of pure zero-shot plug-and-play inference.
Related Work & Insights¶
- vs Point-MAE / MeshMAE: Masked autoencoding approaches are restricted to homogeneous geometric formats (e.g., pure point sets or pure triangle meshes). ReGen3D bridges heterogeneous geometric regimes, improving single-modality reconstruction through cross-representation pretraining.
- vs Voxel-Mesh Network & Paired Multi-Modal Fusion: Prior multi-representation networks enforce synchronized multi-branch inputs at inference. ReGen3D decouples testing requirements through SMI, seamlessly executing inference from any single available stream.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the new setting of Representation Generalization with an elegant, principled framework (URT, SMI, ERA).
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Outstanding empirical coverage across four distinct geometric formats, LORO splits, real LiDAR scans, and geometric stress tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear narrative architecture, and comprehensive ablation insights.
- Value: ⭐⭐⭐⭐⭐ Offers crucial architectural foundational insights toward universal 3D foundation models.