LaMP: Learning Vision-Language-Action Policies with 3D Scene Flow as Latent Motion Prior

Conference: ECCV 2026
Paper: ECCV Official
Project: https://summerwxk.github.io/lamp-project-page/
Area: Robotics & Embodied AI
Keywords: robotic manipulation, 3D scene flow, vision-language-action model, latent motion prior, flow matching

TL;DR

LaMP proposes a dual-expert VLA framework that leverages a conditional flow-matching Motion Expert to generate one-step partially denoised 3D scene flow hidden states, injecting them into an Action Expert via gated cross-attention to substantially enhance physical grounding and OOD robustness without requiring inference-time depth sensors.

Background & Motivation

Humans rarely perform physical manipulation tasks by reflexively translating 2D retinal visual signals straight into joint-level actuator commands. Instead, people inherently maintain an intuitive mental model of dynamic physical interactions—anticipating which visual entities will meet, where contact will occur, and how geometric motion unfolds over time before grasping an object or closing a drawer. In contrast, mainstream Vision-Language-Action (VLA) models (such as OpenVLA and π0) enforce an end-to-end mapping from 2D vision-language representations directly to low-level robot actions. This paradigm forces policies to infer complex 3D physical interactions implicitly from sparse action labels alone, resulting in brittle execution under novel viewpoints, camera rotations, or object rearrangements.

Prior efforts to bridge this 2D-to-3D representation gap introduced intermediate modalities with distinct trade-offs. Optical flow and visual trace-based VLAs (e.g., FlowVLA, TraceVLA) exploit 2D motion cues for temporal alignment, yet lack the explicit metric depth and clearance awareness essential for contact-rich assembly. Point-cloud diffusion policies (such as 3D Diffusion Policy and RISE) operate in metric 3D space, but ingest geometric structures as static snapshot representations rather than modeling continuous spatiotemporal dynamics. Video world models (such as F1 and FLARE) synthesize future frames in pixel space, dedicating substantial representational capacity to visual appearance rendering instead of actionable kinematic dynamics. Furthermore, standalone 3D trajectory predictors lack pre-trained VLM semantic backbones for open-vocabulary task generalization.

Dense 3D scene flow should serve not merely as an observable execution side-effect, but as an explicit, active latent motion prior that guides policy synthesis. However, full multi-step generative reconstruction of 3D motion introduces prohibitive latency for closed-loop control, and naive concatenation of geometric features into VLM backbones risks representational collapse of pre-trained semantic knowledge. Core idea: build a decoupled dual-expert framework where a flow-matching Motion Expert generates one-step partially denoised 3D scene flow representations, adaptively injected into an Action Expert via zero-initialized gated cross-attention to equip policies with geometric foresight and spatiotemporal physical priors at zero test-time depth cost.

Method

Overall Architecture

LaMP employs a dual-expert architecture comprising a Motion Expert (a 300M CogVideoX-style 3D Transformer) that captures dynamic geometric foresight and an Action Expert (a 300M policy network) that predicts continuous robot actions, both conditioned on a frozen pre-trained Qwen3-VL (4.4B) backbone. Given RGB observations and language instructions, the VLM extracts high-level multimodal semantic embeddings. Conditioned on these features, the Motion Expert conducts a single ODE denoising step to extract latent motion states containing future spatiotemporal geometric trends. A single-layer gated cross-attention module then injects these motion representations into the VLM features, which condition the Action Expert to generate continuous action chunks via flow-matching denoising.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Observation Ot + Instruction l"] --> B["VLM Backbone Encoding<br/>Qwen3-VL extracts semantic features zt"]
    B --> C["Dense 3D Scene Flow Representation<br/>TraceForge offline labels + camref frame"]
    C --> D["One-Step Denoising Motion Expert<br/>Conditional flow matching predicts velocity field + zm"]
    D --> E["Gated Cross-Attention Motion Guidance<br/>Zero-initialized scalar gating injects zm into zt"]
    E --> F["Action Expert Action Denoising<br/>Flow matching predicts action chunk At:t+H"]

Key Designs

1. Dense 3D Scene Flow Representation: Viewpoint-Decoupled Spatiotemporal Motion Space

Addressing the limitations of depth-deprived 2D optical flow and sparse contact-agnostic traces, LaMP formulates latent motion as dense 3D scene flow across a regular spatial grid. The model samples a \(K_h \times K_w\) grid (with \(K = 20 \times 20 = 400\) keypoints) on the initial camera image plane and tracks each point across \(T=32\) future timesteps. For every point, it predicts displacement increments \(\Delta p = (\Delta u, \Delta v, \Delta d)\), where \((\Delta u, \Delta v)\) indicates 2D displacement in pixel coordinates and \(\Delta d\) captures depth change, yielding a motion tensor \(M_t \in \mathbb{R}^{K \times T \times 3}\). All trajectories are transformed into a shared reference camera frame \(cam_{ref}\) to compensate for camera motion. Offline dense spatiotemporal correspondences are extracted via TraceForge combined with sensor depth or monocular depth estimation, entirely eliminating human 3D labeling costs and allowing inference to operate on pure RGB inputs without depth sensors.

2. One-Step Denoising Motion Expert: Actionable Dynamic Foresight with Minimal Latency

Iterative multi-step sampling in diffusion or flow-matching models incurs latency that undermines real-time robotic control. LaMP sidesteps this bottleneck through a one-step partial denoising mechanism. The Motion Expert adopts a CogVideoX-style 3D Transformer that groups \(2 \times 2\) spatial keypoints into patch tokens (\(10 \times 10\) spatial tokens per frame) conditioned on VLM last-layer features \(z_t\). During training, given ground-truth motion \(M_t\) and Gaussian noise \(\epsilon \sim \mathcal{N}(0, I)\), the noisy state is formed via linear interpolation \(M^\tau = (1-\tau)\epsilon + \tau M_t\) (\(\tau \in [0, 1]\)), optimized with conditional flow matching:

\[\mathcal{L}_{\mathrm{motion}} = \mathbb{E}_{\tau, M_t, \epsilon} \left\| v_\theta(M^\tau, \tau, z_t) - (M_t - \epsilon) \right\|_2^2\]

During policy rollout, instead of running a full ODE trajectory to reconstruct clean point clouds, the model initializes pure noise \(M^0 \sim \mathcal{N}(0, I)\) at \(\tau = 0\) and takes a single step of a 10-step ODE solver to reach \(\tau = 0.1\). The intermediate hidden state \(z_m \in \mathbb{R}^{B \times N \times d_m}\) from the Transformer is directly extracted. This partially denoised state captures coarse-grained task-relevant physical motion trends while adding only 1.35× inference latency and 1.15× memory overhead over the base VLA, bypassing compounding generative artifacts.

3. Gated Cross-Attention Motion Guidance: Preserving Semantics While Injecting Spatial Precision

Directly concatenating 3D motion features across intermediate VLM layers degrades the pre-trained visual-semantic representations. LaMP incorporates a single-layer gated cross-attention mechanism at the final VLM layer. Motion hidden states are linearly projected and normalized as \(\tilde{h}_{\mathrm{motion}} = \mathrm{LN}(W_{proj} z_m)\), interacting with VLM representations \(z\) via:

\[z_{\mathrm{guided}} = z + \sigma(g) \cdot \mathrm{CA}\left(\mathrm{LN}(z), \tilde{h}_{\mathrm{motion}}\right)\]

where \(\mathrm{CA}\) is multi-head cross-attention, \(g\) is a learnable scalar initialized to 0 across all heads, and \(\sigma(\cdot)\) is the Sigmoid activation bounding the gate to \((0, 1)\). At the onset of training, the near-zero gate ensures that the policy relies on robust pre-trained multimodal representations. As training progresses, the network adaptively scales up 3D motion guidance when precise contact and spatial clearances are required, preventing representation collapse in purely semantic reasoning tasks.

Loss & Training

LaMP employs a two-stage decoupled training protocol: - Stage 1 (Motion-Prior Pretraining): The Motion Expert is trained on 1.6M observation-language-motion triplets across diverse robot embodiments (LIBERO, BridgeV2, DROID, and InternData-A1) optimizing the flow-matching objective \(\mathcal{L}_{\mathrm{motion}}\). - Stage 2 (Motion-Guided Policy Learning): Both the pre-trained Qwen3-VL backbone and the Motion Expert are frozen. Only the gated cross-attention module and the Action Expert are trained using the action flow-matching objective:

\[\mathcal{L}_{\mathrm{action}} = \mathbb{E}_{\tau, a, \epsilon^a} \left\| v_\phi(a^\tau, \tau, z_{\mathrm{guided}}) - (a - \epsilon^a) \right\|_2^2\]

Training is executed across 16 NVIDIA H100 GPUs using the starVLA codebase, guaranteeing modular and embodiment-agnostic policy transfer.

Key Experimental Results

Main Results

Evaluated on the LIBERO simulation benchmark (50 episodes per task) and SimplerEnv-WidowX (trained on real-world BridgeV2 data, evaluated zero-shot on simulated WidowX), LaMP demonstrates substantial gains over state-of-the-art generalist, video-based, and geometry-aware VLAs.

Benchmark / Suite	Metric	LaMP (Ours)	Second Best	Best Prior Baseline	Gain
LIBERO-Spatial	Success Rate (%)	99.4	π0.5 (98.8)	98.8	+0.6
LIBERO-Object	Success Rate (%)	99.8	GeoVLA (99.0)	99.0	+0.8
LIBERO-Goal	Success Rate (%)	97.4	OpenVLA-OFT (97.9)	97.9	-0.5
LIBERO-Long	Success Rate (%)	96.7	GeoVLA (96.6)	96.6	+0.1
LIBERO Average	Success Rate (%)	98.3	GeoVLA (97.7)	97.7	+0.6
SimplerEnv: Stack Block	Success Rate (%)	75.0	FlowVLA (62.5)	62.5	+12.5
SimplerEnv: Put Carrot	Success Rate (%)	66.7	F1 (70.8)	70.8	-4.1
SimplerEnv: Put Spoon	Success Rate (%)	79.1	UniVLA (83.3)	83.3	-4.2
SimplerEnv: Put Eggplant	Success Rate (%)	95.8	UniVLA / FlowVLA (100.0)	100.0	-4.2
SimplerEnv Average	Success Rate (%)	79.2	FlowVLA (74.0)	74.0	+5.2

Ablation Study

Ablations on SimplerEnv-WidowX and zero-shot out-of-distribution (OOD) perturbations in LIBERO-Plus highlight the necessity of each architectural component:

Configuration	SimplerEnv Avg (%)	Stack Block Success (%)	LIBERO-Plus Avg (%)	Note
LaMP (Full Model)	79.2	75.0	79.3	3D scene flow + Gated Cross-Attention + 1-step denoising
w/o Motion Prior (No-Motion)	56.3	25.0	71.6	Drops by 22.9% on SimplerEnv; lacks 3D foresight
2D Optical Flow Prior	66.7	58.3	-	Depth dimension masked; lacks metric contact awareness
Direct Feature Addition (Add)	56.3	33.3	-	Ungated injection harms semantics when noise occurs
Concatenation + MLP	62.5	45.8	-	Feature distortion severely degrades precision tasks

Key Findings

• 3D geometric foresight curbs long-horizon error compounding: On LIBERO-Long, removing the Motion Expert causes success rates to drop sharply from 96.7% to 78.2% (-18.5%). On SimplerEnv's contact-critical Stack Block task, omitting geometric foresight causes a dramatic performance collapse from 75.0% to 25.0% (-50.0%).
• Explicit depth is indispensable for contact reasoning: While 2D flow performs reasonably on coarse pick-and-place tasks (e.g., Put Eggplant, 91.7% vs 95.8%), it lags 3D scene flow by 16.7 percentage points on high-precision stacking (58.3% vs 75.0%).
• Gated cross-attention prevents representation pollution: Naive Add and Concat+MLP fusion degrade performance on Stack Block to 33.3% and 45.8%, respectively. The zero-initialized scalar gate protects pre-trained VLM weights and activates motion guidance adaptively.
• Superior OOD generalization across kinematics and viewpoints: On LIBERO-Plus, LaMP reaches 79.3% average success rate (a 9.7% margin over OpenVLA-OFT at 69.6%). Under Robot Kinematic Perturbations, LaMP achieves 69.6% compared to the next-best baseline π0-Fast at 21.6% (+48.0%), demonstrating that camera-frame motion priors transfer across arm configurations effectively.

Highlights & Insights

• One-step denoised latent states as action priors: Eliminating expensive full ODE trajectory sampling and extracting intermediate states at \(\tau = 0.1\) delivers physical foresight at only 1.35× inference latency, making flow-based world modeling practical for high-frequency control.
• Unified camera-frame spatiotemporal scene flow: Formulating motions as \((\Delta u, \Delta v, \Delta d)\) in a shared \(cam_{ref}\) frame decouples physical dynamics from embodiment-specific joint configurations, enabling scalable cross-robot training across 1.6M multi-embodiment transitions.
• Stabilizing multimodal adaptation via zero-initialized gating: The scalar-gated cross-attention offers a reliable blueprint for integrating physical and geometric priors into pre-trained foundation models without catastrophic semantic forgetting.

Limitations & Future Work

• Fixed spatial grid and temporal horizon: The current implementation fixes keypoints to 400 and temporal horizon to \(T=32\), which may be sub-optimal for delicate manipulation of tiny parts or long-term multi-minute planning.
• Reliance on offline pseudo-ground-truth labeling: Model training relies on TraceForge-derived trajectories. Developing self-supervised scene flow pre-training directly on raw in-the-wild video would further enhance data scalability.
• Single-layer late injection: Injecting motion exclusively into the final VLM layer avoids semantic corruption, but exploring hierarchical, multi-scale geometric integration across shallower visual layers remains an open direction.

Related Work & Insights

• vs FlowVLA / TraceVLA: Prior flow-guided policies operate in 2D pixel space and lack metric depth; LaMP grounds predictions in screen-aligned 3D \((\Delta u, \Delta v, \Delta d)\), resolving line-of-sight distance ambiguities in contact-rich tasks.
• vs F1 / mimic-video: Video world-model VLAs predict full future RGB frames, dedicating excessive capacity to texture and lighting; LaMP isolates actionable motion dynamics for direct policy conditioning.
• vs 3D-VLA / 3D Diffusion Policy: Earlier 3D models ingest static point-cloud frames; LaMP is the first to introduce generative flow-matching spatiotemporal scene flow as a universal cross-embodiment latent motion prior.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering use of one-step flow-matching 3D scene flow as an explicit latent motion prior for VLAs]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across LIBERO 4 suites, LIBERO-Plus 7 OOD axes, SimplerEnv, and real-world Franka/Flexiv robots]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, rigorous mathematical grounding, and comprehensive empirical analyses]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical and effective blueprint for resolving the 2D-to-3D representation mismatch in robot manipulation foundation models]

Related Papers

• [ECCV 2026] LEAP-VLA: Latent-Enhanced Action Prototyping via Continuous Residual Latent Spaces for Vision-Language-Action Models
• [ECCV 2026] VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
• [ECCV 2026] SAFE-Pruner: Semantic Attention–Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation
• [CVPR 2026] Boosting Vision-Language-Action Finetuning with Feasible Action Neighborhood Prior
• [ECCV 2026] Tactile Modality Fusion for Vision-Language-Action Models