Skip to content

3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/yueen-ma/3D-MoE
Area: LLM Reasoning
Keywords: Spatial Intelligence, Mixture of Experts, Routing Curriculum, Embodied AI, Rectified Flow

TL;DR

3D-MoE establishes a sparse 3D multimodal foundation model by cultivating specialist and fusion experts via a two-phase routing curriculum with penalty-adaptive load balancing, seamlessly integrating a single-step rectified flow action head (Pose-DiT) to excel across 3D vision-language reasoning benchmarks and real-time embodied robot manipulation.

Background & Motivation

Spatial intelligence, which unifies 3D visual perception, spatial reasoning, and physical common sense, represents the critical next frontier for artificial general intelligence beyond purely verbal large language models. While contemporary vision-language models (VLMs) have made tremendous strides in multimodal understanding, their persistent reliance on 2D images imposes fundamental geometric bottlenecks. Perspective images are inherently lossy projections of the 3D world that strip away metric depth, fine-grained object bounds, and global coordinate frames, leaving models ill-equipped for precise physical reasoning and manipulation. To circumvent these constraints, recent 3D VLMs connect 3D point cloud encoders directly to dense Transformer backbones; however, scaling dense architectures over lengthy 3D geometric tokens incurs prohibitive computation and latency costs that impede interactive deployment.

Mixture-of-Experts (MoE) architectures offer an appealing path toward scaling model capacity by sparsely activating only a top-k subset of parameters per token. In multimodal settings, MoE provides the structural benefit of mitigating cross-modal interference by routing distinct modalities to dedicated expert parameters, bypassing the capacity competition endemic to shared feed-forward layers. Nevertheless, existing multimodal MoE models remain trapped in a routing dilemma. Deterministic routing statically partitions experts into rigid modality-specific or fusion groups, which preserves stability during training but cripples dynamic cross-modal interaction and flexibility. Conversely, unconstrained probabilistic routing empowers routers to learn dynamic token allocations from scratch, yet warm-starting from dense language models frequently causes catastrophic expert collapse, where routers prematurely route vision tokens to a narrow subset of language-biased experts while completely neglecting others.

To transcend the trade-off between deterministic rigidity and probabilistic instability, this paper introduces a unified curriculum that progressively bridges the two paradigms. Core idea: progressively guide sparse expert specialization via a two-phase routing curriculum—transitioning from deterministic hard partition boundaries to differential boundary relaxation—complemented by penalty-adaptive load balancing and single-step rectified flow action prediction, unifying 3D spatial reasoning with high-frequency embodied control.

Method

Overall Architecture

3D-MoE integrates three primary components: a 3D visual perception module, a sparse MoE Transformer backbone, and a diffusion-based continuous action head termed Pose-DiT. The input scene point cloud is partitioned into object-level clusters processed by PointNet++ and spatially contextualized via a Spatial Transformer to generate visual tokens, which are concatenated with textual instruction tokens before feeding into the 3D-MoE backbone. Inside the backbone, interleaved MoE layers sparsely route representations across specialist and fusion experts; the final contextual representations either pass through an autoregressive language generation head for 3D vision-language tasks or condition the Pose-DiT module to generate sequential 6D pick-and-place end-effector poses for embodied robotic control.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input 3D Scene Point Cloud & Language Instructions"] --> B["3D Vision Perception & Contextual Encoding<br/>PointNet++ Object Features + Spatial Transformer"]
    B --> C["Specialist Warmup Phase<br/>Hard Boundary Penalty lambda_0 Isolates Modalities"]
    C --> D["Differential Boundary Relaxation Phase<br/>Dual-Rate Cosine Decay Cultivates Fusion & Preserves Specialists"]
    D --> E["Penalty-Adaptive Auxiliary-Loss-Free Balancing<br/>Dynamic Target Frequency Prevents Bias Divergence"]
    E --> F["Explicit Spatial Routing Signals<br/>Inject Object Dimensions & Relative Layouts into Router"]
    F --> G["Backbone Multimodal Hidden Representations"]
    G -->|3D-VL Tasks| H["Autoregressive Text Generation Head<br/>3D Scene Captioning & 3D Question Answering"]
    G -->|Embodied Tasks| I["Pose-DiT Single-Step Rectified Flow Action Head<br/>Spatiotemporal Attention Predicts 6D Pick-and-Place Poses"]

Key Designs

1. Routing Curriculum: Specialist Warmup and Differential Boundary Relaxation Because the FFN weights of the backbone are initialized by replicating dense LLM layers, all experts inherit a heavy linguistic bias that destabilizes random probabilistic routing upon the initial injection of visual tokens. To prevent expert collapse, the curriculum introduces a two-phase schedule. In the specialist warmup phase (lasting until step \(T_{\text{warm}}\)), experts are strictly partitioned into language-specialist and vision-specialist sets, enforced by a heavy boundary-crossing penalty \(\lambda_i^m(t) = \lambda_0 \gg 0\) (default \(\lambda_0 = 8.0\)) whenever a token attempts to enter an opposite-modality expert, compelling vision experts to develop robust spatial representations in isolation. Subsequently, the differential boundary relaxation phase lowers these partition barriers to cultivate multimodal fusion experts while stripping away potentially suboptimal manual priors. The penalty for designated fusion experts \(\lambda_{\text{fus}}(t)\) decays rapidly via a cosine schedule to zero by step \(T_{\text{relax}}^{\text{fus}}\), encouraging designated language and vision experts to meet in the multimodal midground. Simultaneously, the penalty for dedicated specialists \(\lambda_{\text{spec}}(t)\) decays much more conservatively to a non-zero residual floor \(\lambda_{\text{spec}}^{\min} = 1.0\) at step \(T_{\text{relax}}^{\text{spec}}\), ensuring the network permanently retains deep, uncorrupted modality-specific specialists.

2. Penalty-Adaptive Auxiliary-Loss-Free Balancing Conventional auxiliary-loss-free balancing schemes assume a uniform token assignment target across all experts, but when non-uniform routing penalties \(\lambda_i^m(t)\) are deliberately imposed by the curriculum, standard bias adaptation actively counteracts the intentional penalties, triggering unbounded bias accumulation and training instability. To reconcile load balancing with curriculum penalties, the method constructs a dynamic penalty-adaptive target frequency \(\tau_i(t)\): $\(\tau_i(t) = \sum_{m \in \{\text{lang}, \text{vis}\}} \rho_m(t) \cdot \text{softmax}_i(-\lambda^m(t))\)$ where \(\rho_m(t)\) is the exponential moving average (EMA) token ratio for modality \(m\), and \(\lambda^m(t)\) is the vector of penalties for that modality across all experts. The balancing bias is updated dynamically against empirical deviations: \(b_i(t+1) = \gamma b_i(t) + \alpha (\tau_i(t) - u_i(t))\), with damping factor \(\gamma \lesssim 1\) bounding maximum drift. During inference, the learned balancing bias is frozen as a static parameter, eliminating runtime overhead while maintaining balanced sparse computation.

3. Explicit Spatial Routing Signals As boundary relaxation unfolds and learnable routing weights \(W_r\) take control of token allocation, standard token hidden states \(x\) alone can lack geometric salience for complex 3D reasoning. To provide structural routing priors, the router incorporates an explicit spatial descriptor \(g(x) \in \mathbb{R}^S\) capturing 3D bounding dimensions and relative displacement vectors toward neighboring objects (set to zero for linguistic tokens). The routing logit is formulated as: $\(z_i(x, t) = W_{r,i}^\top \text{LN}(x \parallel g(x)) - \lambda_i^m(t) + b_i(t)\)$ This explicit spatial grounding acts as an inductive bias that steers tokens exhibiting intricate topological dependencies toward specialized geometric experts, which proves particularly vital for resolving complex spatial reasoning in manipulation tasks.

4. Single-Step Spatiotemporal Action Prediction via Pose-DiT Conventional diffusion-based policies rely on multi-step DDPM/DDIM reverse denoising or flow matching steps, introducing latency bottlenecks that are ill-suited for closed-loop real-time robotic execution. 3D-MoE transforms into a 3D Vision-Language-Action (VLA) policy by attaching Pose-DiT, optimized via the rectified flow formulation that straightens the probability trajectories between noise distribution \(\mu_0\) and 6D pose distribution \(\mu_1\): $\(\mathcal{L}_{\text{RF}} = \int_0^1 \mathbb{E}\left[ \|(X_1 - X_0) - v(X_t, t)\|^2 \right] dt\)$ Owing to straight transport paths that avoid trajectory crossings, Pose-DiT generates high-precision continuous 6D end-effector actions in a single Euler integration step during inference. Furthermore, Pose-DiT employs decoupled spatial and temporal self-attention blocks across the pick-and-place pose sequence \((L_{\text{temp}}, 2, 6)\) conditioned on the penultimate 3D-MoE hidden states, enforcing continuous spatiotemporal physical coherence.

Loss & Training

The architecture is trained in a structured three-stage pipeline: - Stage I (Vision Alignment & Routing Curriculum): Pretrains the PointNet++ encoder and linear projection on Cap3D and LEO-Align with the dense LLM frozen, then replicates FFN weights into an 8-expert MoE to execute the routing curriculum across 10 epochs (specialist warmup in epochs 6-7; differential relaxation across epochs 8-10). - Stage II (3D-VL Task Finetuning): Performs end-to-end or LoRA finetuning across Scan2Cap, ScanQA, SQA3D, and LEO-Instruct under autoregressive cross-entropy loss \(\mathcal{L}_{\text{CE}}\), maintaining EMA balancing updates. - Stage III (Embodied Task Finetuning): Connects the Pose-DiT module and finetunes on LoHoRavens expert manipulation trajectories for 20 epochs, minimizing the rectified flow velocity field objective \(\mathcal{L}_{\text{RF}}\).

Key Experimental Results

Main Results

The model is rigorously evaluated across standard 3D vision-language benchmarks (Scan2Cap, ScanQA, SQA3D) and the challenging LoHoRavens tabletop manipulation benchmark. In 3D-VL tasks, 3D-MoE outperforms dense 7B baselines while activating only a fraction of their parameters; in embodied tasks, it establishes new state-of-the-art success rates with single-step action sampling.

Table 1: 3D Vision-Language Benchmark Performance

Model Activated (Total) Params Scan2Cap (CIDEr) Scan2Cap (BLEU-4) ScanQA (CIDEr) ScanQA (EM@1) SQA3D (EM@1)
Scan2Cap (GPT-3) - 35.2 22.4 - - 41.0
3D-VisTA - 66.9 34.0 69.6 22.4 48.5
LEO 7B 72.4 38.2 101.4 24.5 50.0
Chat-Scene 7B 77.1 36.3 87.7 21.6 54.6
3D-LLaVA 7B 78.8 36.9 92.6 - 54.5
3D-MoE 1B×8 (Top-2) 1.6B (4.1B) 68.6 34.3 93.5 21.7 48.4
3D-MoE 1.5B×8 (Top-2) 2.1B (5.6B) 71.8 35.7 97.2 23.9 52.0
3D-MoE 3B×8 (Top-2) 4.3B (10.6B) 82.2 40.6 102.2 28.2 57.8

Table 2: LoHoRavens Robotic Manipulation Benchmark and Diffusion Latency

Model Action Interface Sampling Steps Diffusion Latency (ms) Pick-Zone Success Pick-Bowl Success Stack Success Reason-Spatial Reason-Arith. Reason-Color
LLaVA-NeXT (2D) Text 6D Pose - - 0.37 0.40 0.09 0.46 0.00 0.14
LEO (3D Dense) Text 6D Pose - - 0.69 0.80 0.23 0.71 0.17 0.20
OpenVLA (2D) Direct Regression - - 0.69 0.63 0.29 0.49 0.09 0.11
\(\pi_0\) (2D Flow) Flow Matching 10 333.55 0.80 0.57 0.43 0.69 0.11 0.26
3D-MoE + Pose-DiT Rectified Flow 1 17.08 1.00 1.00 0.63 1.00 0.80 0.86

Ablation Study

Table 3: Ablation Analysis on Routing Curriculum, Balancing, and Architecture (3D-MoE 1B×8)

Category Ablation Setting Scan2Cap CIDEr ScanQA CIDEr SQA3D EM@1 LoHoRavens Arithmetic Reason
Full Model Default 3D-MoE (Curriculum, Adaptive Balancing, \(g(x)\)) 68.6 93.5 48.4 0.80
Routing Curriculum W/o differential relaxation (all experts become fusion) 65.4 91.6 45.8 -
Routing Curriculum W/o relaxation (rigid static boundaries retained) 67.6 91.3 44.9 -
Routing Curriculum W/o curriculum (standard top-k MoE from step 0) 63.9 88.6 44.8 -
Load Balancing W/o penalty-adaptive balancing (uniform target conflict) 61.9 88.9 45.6 -
Load Balancing W/o load balancing (\(b_i = 0\), balancing disabled) 59.8 83.5 44.3 -
Spatial Signals W/o spatial signals \(g(x)\) (tokens only) 67.8 91.8 47.6 0.66 (drop 0.14)
Action Head W/o rectified flow (100-step DDPM, latency 1505 ms) - - - 0.40 (drop 0.40)
Action Head W/o ST attention (vanilla non-decoupled DiT) - - - 0.74 (drop 0.06)

Key Findings

  • The routing curriculum is indispensable against expert collapse: Training standard top-k MoE without the curriculum precipitates a severe drop on ScanQA CIDEr from 93.5 to 88.6 and degrades SQA3D by 3.6%, demonstrating that unguided routing immediately after initialization results in destructive modality competition.
  • Adaptive balancing resolves internal optimization tensions: Imposing penalty-guided curriculum without adapting the auxiliary-loss-free target leads to bias explosion and severely degrades performance (Scan2Cap CIDEr plunges from 68.6 to 61.9), while disabling balancing entirely degrades CIDEr to 59.8 due to expert starvation.
  • Embodied control relies on deep visual specialists: Quantitative routing analysis reveals that in Stage III embodied tasks, 43.0% of vision tokens are processed by vision specialists (\(\mathcal{E}_{\text{vis}}^{\text{spec}}\)), compared to 36.2% in 3D-VL tasks. In the deepest layers, the top routing pathways converge almost entirely to vision specialists, whereas 3D-VL tasks route over 60% of computation to fusion experts to handle nuanced cross-modal alignment.
  • Rectified flow unlocks real-time diffusion action control: Pose-DiT reduces the action head sampling latency to 17.08 ms, enabling total end-to-end policy inference in 117 ms (~8.5 Hz) on an RTX 4080 SUPER, while outperforming multi-step flow matching baselines on intricate reasoning manipulation tasks.

Highlights & Insights

  • Annealing-inspired boundaries unify stability and expressiveness: Initiating with deterministic boundaries anchors foundational representation learning, while cosine relaxation seamlessly fosters emergent cross-modal fusion experts without sacrificing specialized modality channels.
  • Dynamic target balancing eliminates optimization divergence: Formulating balancing targets as an EMA-weighted softmax over negative penalties mathematically prevents the router bias from opposing intentional curriculum penalties, offering an elegant template for any non-uniformly constrained MoE.
  • Rectified flow breaks the diffusion frequency barrier in robotics: Demonstrating that straightened transport paths enable single-step Euler sampling for continuous 6D poses proves that diffusion-based action policies can achieve high-frequency closed-loop control without sacrificing multi-modal generative fidelity.

Limitations & Future Work

  • Perception remains bound to discrete point clouds: The visual front-end uses PointNet++ over segmented point clusters, which may discard fine-grained surface normals, textures, and continuous volumetric radiance capturable by 3D Gaussian Splatting or neural radiance fields.
  • Embodied evaluation is constrained to simulated tabletop manipulation: All robotic experiments are conducted within the LoHoRavens tabletop environment under suction-gripper kinematics, leaving real-world hardware deployment, tactile feedback integration, and dexterous multi-finger grasping unverified.
  • Hyperparameter search space across curriculum phases: The interaction between decay horizons and penalty magnitudes was explored with coarse sweeps; scaling to extreme expert counts (e.g., 64+ experts) may require automated adaptive scheduling.
  • vs LEO / 3D-LLaVA: Both rely on dense 7B Transformer architectures where every token activates all parameters, creating severe computational bottlenecks; 3D-MoE achieves superior 3D-VL and manipulation performance with only 1.6B active parameters.
  • vs OpenVLA / \(\pi_0\): Conventional VLAs process 2D RGB frames that inherently lack true 3D spatial metric depth and topological awareness; 3D-MoE natively integrates 3D point cloud structures and explicit geometric routing vectors, coupled with single-step rectified flow action generation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ The combination of specialist warmup with differential boundary relaxation and single-step rectified flow pose generation introduces compelling original concepts to spatial AI.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Spans 3D-VL benchmarks, long-horizon manipulation reasoning, real hardware inference profiling, and rigorous layer-wise routing distribution audits.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear theoretical motivation, coherent prose transitions, and self-contained mathematical precision.
  • Value: ⭐⭐⭐⭐⭐ Sets an architectural benchmark for developing compute-efficient, real-time spatial foundation models bridging 3D reasoning and robotic action.