RoMan-4D: Learning Robot Arm Manipulation from 4D World Models¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://gem-4d.github.io/
Area: Robotics & Embodied AI
Keywords: 4D world model, embodied AI, flow matching, geometry distillation, inverse dynamics

TL;DR¶

RoMan-4D (GEM-4D) introduces an asymmetric dual flow-matching framework that distills 4D geometry foundation model representations into a video DiT backbone during training with zero inference overhead, coupled with an adaptive confidence-gated inverse dynamics system to derive executable 6-DoF robot arm manipulation trajectories from physical-consistent video rollouts.

Background & Motivation¶

General-purpose robotic manipulation policies typically rely on massive real-world demonstrations and embodiment-specific fine-tuning, incurring prohibitive data collection costs. Video world models offer a promising alternative by synthesizing language-conditioned future visual observations, providing a task- and embodiment-agnostic visual substrate for planning. However, existing video diffusion models predominantly optimize pixel- or latent-space reconstruction objectives. These objectives form an inherently many-to-one mapping, where drastically different configurations of 3D scene depth, camera trajectory, and object deformations can yield virtually identical 2D pixel projections. Consequently, while generated videos appear visually photorealistic, they frequently suffer from non-rigid body deformations, contact point drift, and severe spatiotemporal depth inconsistencies.

When downstream policy extraction modules attempt to derive executable robot actions from such rollouts, any corruption of inter-frame surface correspondences directly leads to tracking failures and invalid end-effector pose estimation. Prior attempts to resolve this issue by imposing explicit geometric supervision (such as TesserACT) force generative models to jointly predict RGB, depth maps, and surface normals. This paradigm not only demands scarce, large-scale multi-modal 3D annotations but also compromises the expressive capacity of pretrained video backbones while substantially inflating inference compute and latency.

Inter-frame pixel correspondence is mathematically governed by camera intrinsics, camera motion, scene depth, and object motion vectors (3D scene flow). Pretrained 4D geometry foundation models (such as PAGE-4D and Depth Anything V3) have already internalized these comprehensive geometric relationships within their latent representations. Core idea: distill dense 4D correspondence priors from a frozen geometry foundation model into the intermediate feature representations of a video DiT via an asymmetric dual flow-matching framework, enforcing strict physical correspondence consistency at zero inference cost and pairing it with a confidence-gated adaptive inverse dynamics system to robustly extract 6-DoF robot manipulation trajectories.

Method¶

Overall Architecture¶

The system (termed GEM-4D in the paper) operates across two stages: geometry-enhanced velocity alignment during training and an Adaptive Inverse Dynamic System (AIDS) during deployment. In training, the framework adopts an asymmetric dual flow-matching architecture comprising a conditional Video DiT and an auxiliary Geometry DiT. The video branch receives noised video latents and outputs video velocity fields, while its intermediate representations serve as the exclusive conditioning input for the geometry branch to predict the velocity of geometry latents extracted by a frozen 4D foundation model. The geometry branch is entirely discarded at inference, preserving single-stream generative efficiency. During deployment, the generated correspondence-consistent video rollout is fed into AIDS, which performs 3D scene grounding, dual-criterion confidence-gated tracking, geometry-kinematics pose fallback, and grasp synthesis to output executable robot joint trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial Image Observation + Text Prompt"] --> B["Geometry Flow Feature Distillation<br/>Asymmetric Dual Flow Matching DiT"]
    B --> C["Correspondence-Consistent Video Rollout"]
    C --> D["Dual-Criterion Confidence-Gated Tracker<br/>Gradual Drift vs. Catastrophic Collapse"]
    D --> E["Geometry-Kinematics Fallback & Grasp Synthesis<br/>Depth Centroid + GraspGen + IK Solver"]
    E --> F["Executable 6-DoF Robot Control Trajectory"]

Key Designs¶

1. Geometry Flow Feature Distillation: Constraining Inter-Frame Physical Correspondences at Zero Inference Cost

To address the inability of standard generation losses to penalize geometric violations, the authors formalize inter-frame pixel projection: a 3D point $\mathbf{X}_t$ projected to pixel $\mathbf{p}_t$ at frame $t$ moves to $\mathbf{p}_{t+1}$ at frame $t+1$ under camera intrinsics $\mathbf{K}$, relative extrinsic pose $(\mathbf{R}_{t\to t+1}, \mathbf{T}_{t\to t+1})$, depth $\mathbf{D}(\mathbf{p}_t)$, and dynamic scene flow $\Delta\mathbf{X}_t$: $$\mathbf{p}_{t+1} \sim \mathbf{K} \left[ \mathbf{R}_{t\to t+1}\,\mathbf{D}(\mathbf{p}_t)\,\mathbf{K}^{-1}\,\mathbf{p}_t + \mathbf{T}_{t\to t+1} + \Delta\mathbf{X}_t \right]$$ When internal representations accurately capture these four factors, correspondence is uniquely constrained with zero remaining degrees of freedom. A frozen 4D geometry foundation model $G$ extracts dense representations $\mathbf{g}_0 = G(\{\mathbf{I}_t\}_{t=0}^T)$ from input frames. During video flow matching, intermediate representations $\mathbf{m}_t$ from the Video DiT are fed as conditioning into a lightweight Geometry DiT to predict the velocity field of noised geometry latents $\mathbf{g}_t$. The geometry branch has no access to raw pixels or depth maps; it relies entirely on $\mathbf{m}_t$. Consequently, $\mathbf{m}_t$ is forced to encode 3D geometry and motion dynamics alongside visual appearance. At inference time, the geometry branch is discarded, allowing the Video DiT to generate geometry-consistent rollouts with zero additional computation.

2. Dual-Criterion Confidence-Gated Tracker: Disentangling Gradual Drift from Abrupt Collapse

Extracting end-effector (EE) motion from predicted videos is often hindered by residual visual artifacts. Conventional methods either rely on naive continuous tracking (which suffers from severe drift accumulation) or repeatedly trigger heavy VLM re-detection (which is computationally expensive and causes temporal jitter). The authors design a confidence-gated mechanism: anchor keypoints $\mathcal{V}_{t_0}$ are sampled from the initial EE mask obtained via SAM 2, and CoTracker3 tracks the surviving subset $\mathcal{V}_t$. Two signals are tracked: anchor retention ratio $s_t = |\mathcal{V}_t| / |\mathcal{V}_{t_0}|$ and frame-to-frame change $\Delta s_t = s_t - s_{t-1}$. When $s_t < \tau$ with smooth variation, it signals natural cumulative drift, prompting keypoint resampling from the latest valid mask. When $\Delta s_t < -\delta$ drops sharply, it detects an abrupt collapse due to generative artifacts or occlusions, triggering Qwen3.5-VL for semantic re-grounding.

3. Geometry-Kinematics Fallback and Grasp Synthesis: Ensuring Reliable Kinematic Execution

Given the EE mask, FoundationPose predicts 6D poses $(\mathbf{R}_{\text{ee}}^t, \mathbf{T}_{\text{ee}}^t)$ and confidence $\kappa_t$. If confidence drops below threshold $\kappa^*$ or inter-frame motion jumps abruptly ($\|\mathbf{T}_{\text{ee}}^t - \mathbf{T}_{\text{ee}}^{t-1}\|_2 > \epsilon_t$ or $SO(3)$ geodesic distance $d_{\text{geo}}(\mathbf{R}_{\text{ee}}^t, \mathbf{R}_{\text{ee}}^{t-1}) > \epsilon_R$), the system activates a geometry-kinematics fallback. Translation is recovered by back-projecting depth pixels within the EE mask to compute their 3D centroid, while rotation is estimated via spherical linear interpolation (SLERP) across the nearest valid frames. On the complete trajectory, the pose closest to the target object is designated as a reference, GraspGen proposes 6-DoF grasp candidates on the object point cloud, and the candidate with minimal translation and geodesic rotation error is chosen as optimal target $\mathbf{T}_{\text{grasp}}^*$. Inverse kinematics (IK) finally translates the smoothed trajectory into an executable joint action sequence.

Loss & Training¶

The model is trained end-to-end with a joint flow-matching objective: $$\mathcal{L} = \mathcal{L}_{\mathrm{FM}}^{\text{vid}} + \alpha \mathcal{L}_{\mathrm{FM}}^{\text{geo}}$$ where the video and geometry flow-matching losses are defined as: $$\mathcal{L}_{\mathrm{FM}}^{\text{vid}} = \mathbb{E}_{\mathbf{z}_t, t} \left[ \|\mathbf{v}_\theta^{\text{vid}}(\mathbf{z}_t, t, c) - \mathbf{v}^*(\mathbf{z}_t, t)\|_2^2 \right]$$ $$\mathcal{L}_{\mathrm{FM}}^{\text{geo}} = \mathbb{E}_{\mathbf{g}_t, t} \left[ \|\mathcal{M}(\mathbf{v}_\psi^{\text{geo}}(\mathbf{g}_t, t, \mathbf{m}_t)) - \mathbf{v}^*(\mathbf{g}_t, t)\|_2^2 \right]$$ The gradient with respect to the video backbone parameters $\theta$ decomposes into an appearance term and a geometry-induced term: $$\nabla_\theta \mathcal{L} = \nabla_\theta \mathcal{L}_{\mathrm{FM}}^{\text{vid}} + \alpha \cdot \frac{\partial \mathcal{L}_{\mathrm{FM}}^{\text{geo}}}{\partial \mathbf{m}_t} \cdot \frac{\partial \mathbf{m}_t}{\partial \theta}$$ Via backpropagation through $\mathbf{m}_t$, the geometry loss acts as a representation-level regularizer, eliminating internal representations that fit pixel appearance while violating underlying 3D physical constraints.

Key Experimental Results¶

Main Results¶

The method is evaluated across 400 unseen samples on the real-world Droid dataset and 780 unseen samples on the synthetic RLBench benchmark.

Domain	Method	FVD ↓	SSIM ↑	PSNR ↑	AbsRel ↓	$\delta_1$ ↑	$\delta_2$ ↑	Chamfer L1 ↓	Tracking $\delta_{\text{vis}}^{\text{avg}}$ ↑
Real (Droid)	CogVideoX	35.56	75.91	20.18	22.33	68.32	83.17	0.2670	66.22
Real (Droid)	Wan 2.2-14B	33.43	76.24	20.70	21.39	71.18	84.35	0.2349	68.18
Real (Droid)	TesserAct	33.28	75.66	20.08	22.07	66.80	82.60	0.2630	67.14
Real (Droid)	Geometry-Forcing	33.17	76.12	20.53	21.96	69.74	83.83	0.2443	67.97
Real (Droid)	GEM-4D (Ours)	31.82	82.05	21.11	20.13	78.19	88.21	0.2001	71.23
Sim (RLBench)	CogVideoX	40.21	75.51	20.03	15.41	70.99	92.90	0.2913	58.32
Sim (RLBench)	Wan 2.2-14B	49.20	73.01	19.87	17.81	67.07	90.16	0.1762	61.99
Sim (RLBench)	TesserAct	41.97	76.72	19.71	16.02	69.26	93.03	0.1813	61.15
Sim (RLBench)	Geometry-Forcing	34.06	77.92	19.48	15.34	68.96	92.80	0.1488	60.84
Sim (RLBench)	GEM-4D (Ours)	27.94	80.27	23.36	14.11	74.13	95.01	0.0702	68.18

In robotic task success evaluation, human studies on real-world Droid tasks show GEM-4D substantially outperforming TesserAct: AUTOLab (75% vs. 58%), CLVR (83% vs. 65%), RAIL (87% vs. 59%), and Numbered Rubbish (78% vs. 21%). On RLBench simulation tasks, GEM-4D achieves 63%–82% success rates across all 7 manipulation benchmarks, whereas TesserAct fails on several tasks (0%–49%) due to trajectory discontinuities.

Ablation Study¶

The ablation study analyzes the role of different geometry supervision paradigms and foundation model priors:

Domain	Config	FVD ↓	SSIM ↑	PSNR ↑	AbsRel ↓	$\delta_1$ ↑	$\delta_2$ ↑	Chamfer L1 ↓	Note
Real	GEM-4D (Full model)	31.82	82.05	21.11	20.13	78.19	88.21	0.2001	Full 4D dynamic geometry distillation
Real	GEM-4D (Dep)	32.91	78.58	20.75	20.89	74.60	86.67	0.2229	Supervised by explicit depth flow matching
Real	GEM-4D (VGGT)	33.68	75.89	20.64	21.73	71.03	83.80	0.2370	Uses static/quasi-static VGGT prior
Real	Wan 2.2-14B	33.43	76.24	20.70	21.39	71.18	84.35	0.2349	Baseline fine-tuned without geometry
Real	CogVideoX	35.56	75.91	20.18	22.33	68.32	83.17	0.2670	Baseline fine-tuned without geometry

Key Findings¶

Dynamic 4D priors outperform static geometry: Incorporating VGGT features slightly degrades performance compared to pure depth supervision. Because VGGT is trained primarily on static scenes, it lacks motion dynamics and introduces conflicting priors for dynamic hand-object interactions.
Implicit representation distillation beats multi-head generation: Unlike TesserAct which enforces multi-task output constraints, GEM-4D guides the internal representation space, retaining visual generation diversity while boosting point tracking accuracy ($\delta_{\text{vis}}^{\text{avg}}$ from 67.14% to 71.23%).
Robustness of confidence-gated inverse dynamics: Disentangling gradual drift from catastrophic failure and adding pose fallback allows open-loop video execution to tolerate minor generation noise, solving complex multi-stage tasks such as Slide Puzzle.

Highlights & Insights¶

Geometry as a training-time regularizer: Rather than expanding output modalities to RGB-D-Normal, GEM-4D demonstrates that distilling foundation geometry features into intermediate DiT representations provides an elegant, zero-inference-overhead physical regularizer.
Dual-criterion failure mode isolation: Structuring tracking interventions into lightweight resampling for drift and VLM semantic re-grounding for abrupt collapse provides a practical blueprint for video-to-action systems.
Scalable embodied foundation paradigm: Demonstrates that embodied planning can leverage internet-scale video generators without extensive embodiment-specific real-world action data, bridging the sim-to-real gap via geometric grounding.

Limitations & Future Work¶

Dependency on geometry foundation model quality: Severe visual artifacts, reflective surfaces, or thin objects that confuse the geometry teacher directly propagate misleading supervision to the video backbone.
Open-loop generation vulnerability: Predicting entire future rollouts open-loop cannot accommodate unexpected dynamic collisions during execution.
Future work: Incorporating closed-loop diffusion model predictive control (Diffusion MPC) to continuously integrate real-time sensor feedback into streaming DiT latents for interactive replanning.

vs. TesserAct: TesserAct requires joint RGB-D-normal output prediction with heavy compute and multi-modal annotation; GEM-4D retains a single-stream RGB output with zero extra inference compute while producing superior geometric fidelity.
vs. Geometry-Forcing: Geometry-Forcing applies simple feature alignment on static representations; GEM-4D derives a mathematically grounded flow-matching loss over 4D dynamic scene representations.
vs. Track2Act / UniPi: UniPi extracts actions purely from 2D pixel space and Track2Act relies on 2D tracking; GEM-4D provides 3D-grounded, metric-scale 6-DoF control through geometric pose optimization and CAD alignment.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant formulation utilizing 4D geometry foundation models as training-time flow-matching regularizers.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluation spanning Droid real-world benchmark, RLBench simulation, and physical UF robot arm execution.
Writing Quality: ⭐⭐⭐⭐⭐ Cohesive mathematical derivation and transparent system architecture.
Value: ⭐⭐⭐⭐⭐ Sets a new benchmark for physically grounded video world models in embodied robotic manipulation.