---

title: >- [Paper Note] PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation description: >- [ECCV 2026][3D Vision][Physical World Model] PhysMani couples a physics-principled 3D Gaussian world model learning a divergence-free velocity field with a future-aware action policy, establishing superior dynamic manipulation performance. tags: - ECCV 2026 - 3D Vision - Physical World Model - Dynamic Manipulation - 3D Gaussian Splatting date: 2026-09-19 content_hash: c8342f8fd5593bb9

PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/vLAR-group/PhysMani
Area: 3D Vision
Keywords: Physical World Model, Dynamic Manipulation, 3D Gaussian Splatting, Velocity Field, Embodied AI

TL;DR

Addressing the failure of existing VLAs and video world models to manipulate fast-moving targets due to the absence of explicit 3D geometry and physical laws, PhysMani couples an online-optimized divergence-free 3D Gaussian velocity field with a future-aware action policy, achieving state-of-the-art dynamic manipulation success rates across 16 benchmark tasks and real-world robot experiments.

Background & Motivation

In recent years, vision-language-action models (VLAs) and video-generative world models have driven significant breakthroughs in embodied AI, allowing robots to acquire versatile behaviors across complex tabletop and household environments. Despite these remarkable strides, prevailing manipulation frameworks overwhelmingly focus on static or quasi-static object interactions, leaving the manipulation of fast and dynamically moving targets largely underexplored. In practical dynamic tasks—such as catching a thrown ball, loading items onto moving conveyors, or placing cups onto rotating racks—targets exhibit agile and complex linear, circular, or rotational trajectories. Under such settings, reactive control policies fall short; the robot must proactively forecast future state dynamics in 3D metric space and generate coordinated, low-latency trajectories.

Existing attempts to tackle dynamic tasks via video world models or large-scale VLAs encounter three critical roadblocks: First, video world models operate primarily in 2D image pixel space and lack explicit 3D spatial geometry and depth awareness, preventing metric-accurate end-effector motion planning. Second, while generated future video frames appear visually compelling, they frequently violate fundamental conservation and physical dynamics laws, producing hallucinations, unnatural deformations, or impossible motion paths. Third, chaining heavy visual backbones and multimodal LLMs incurs severe inference latency (often hundreds of milliseconds to several seconds), which is fundamentally prohibitive for time-sensitive dynamic interactions.

The key insight of this work is to eliminate bulky offline physical simulation and large model inference bottlenecks by formulating a lightweight, differentiable 3D Gaussian velocity field capable of rapid online optimization directly from multi-view streaming observations. Core idea: develop a physics-principled 3D Gaussian world model that continuously infers a divergence-free Gaussian velocity field online, and seamlessly feed the forecasted future 3D dynamics into a Rectified Flow action policy via learnable token-based local cross-attention.

Method

Overall Architecture

PhysMani consists of two parallel and decoupled modules: a physics-principled 3D Gaussian world model and a future-aware action policy model. The world model continuously processes multi-view streaming RGB-D observations, maintaining canonical 3D scene geometry while learning a 6-DoF divergence-free physical velocity field for each 3D Gaussian kernel. The action policy model fuses sparse scene point cloud features, task instructions, and local future dynamics representations from the world model, denoising the robot end-effector keypose trajectories via Rectified Flow.

The end-to-end dataflow pipeline is depicted below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view RGB-D Streams + Task Instruction"] --> B["Canonical 3D Gaussian Initialization<br/>Back-project point cloud and fit static appearance"]
    B --> C["Online Optimization of Future Dynamics<br/>Interleaved mid-point rollout with joint geometry-velocity optimization"]
    C --> D["Divergence-Free Gaussian Velocity Field<br/>Infer 6-DoF basic velocity vector Vt per Gaussian"]
    D --> E["Local Neighborhood Dynamics Aggregation<br/>KNN offset and velocity projection via learnable query token L"]
    E --> F["Rectified Flow Policy Denoising<br/>Condition action trajectory denoising on future-aware tokens"]
    F --> G["Inverse Kinematics Joint Execution"]

Key Designs

1. Canonical 3D Gaussian Initialization: Establishing a Metric Geometric Anchor At time \(t=0\), given multi-view RGB-D observations \(\{(I^c_0, D^c_0)\}_{c=1}^C\) from \(C\) calibrated stationary cameras, the system initialises a set of canonical 3D Gaussian kernels \(\mathcal{G}_0\) via the unprojected sparse point cloud. Each Gaussian kernel encapsulates 3D position \(g_0\), quaternion rotation \(r_0\), 3D scale \(s_0\), opacity \(\sigma\), and color coefficients \(c\). Following 3D Gaussian Splatting, the canonical kernels are optimized via differentiable rendering under photometric \(\ell_1\) and structural \(\ell_{ssim}\) losses against time \(t=0\) views, providing a high-fidelity, explicit 3D geometric base for subsequent temporal dynamic evolution.

2. Online Optimization of Future Dynamics: Decoupling Deformation for Real-time Streaming Adaptation Prior 3D Gaussian physical dynamics frameworks (such as FreeGave) rely on auxiliary neural deformation fields and expensive offline optimization loops, making real-time dynamic robot control impossible. PhysMani strips away auxiliary deformation branches and reformulates the process into an online streaming optimization algorithm. Upon receiving new streaming RGB-D frames at time \(t\), the system uses the velocity network \(f_{vel}\) to drive Gaussians from previous step \(t'\) to \(t\) using the interleaved mid-point numerical method. It then runs \(T\) gradient iterations to jointly optimize Gaussian position, orientation, and velocity MLP weights under rendering losses, followed by \(T'\) refinement iterations with the velocity network frozen to absorb minor photometric drifts. Furthermore, by keeping the Gaussian count constant (\(H=30000\)) and reusing static computation graphs, CUDA kernel launch overhead is minimized, reducing online dynamics update time to approximately 200 ms (\(T=50, T'=7\)).

3. Divergence-Free Gaussian Velocity Field: Enforcing Incompressible Physical Continuity To guarantee that forecasted 3D motions strictly obey physical principles rather than mere visual plausibility, the velocity MLP \(f_{vel}\) outputs six basic velocity components for each Gaussian kernel: \(V_t = [v_t^x, v_t^y, v_t^z, w_t^z, w_t^y, w_t^x]\), corresponding to three linear velocities and three angular velocities. The spatial velocity field \(v(g_t, t)\) is then analytically composed via coordinate basis matrix \(\mathcal{B}(g_t) \in \mathbb{R}^{6 \times 3}\):

\[v(g_t, t) = V_t \cdot \mathcal{B}(g_t), \quad V_t = f_{vel}(g_0, t)\]

Because the basis matrix \(\mathcal{B}(g_t)\) is mathematically constructed with anti-symmetric coordinate operators, the resulting 3D velocity field rigorously satisfies the zero-divergence condition (\(\nabla \cdot v = 0\)). This guarantees volume-preserving, physically realistic motion extrapolation without unnatural clustering or non-physical tearing over extended horizons.

4. Local Neighborhood Dynamics Aggregation: Learnable Token-based Feature Fusion The policy model builds upon 3D FlowMatch Actor (3DFA), taking a sparse 3D point cloud \(P_t \in \mathbb{R}^{4096 \times 3}\) sampled from depth sensors. To integrate continuous Gaussian velocity field \(D_t \in \mathbb{R}^{H \times 6}\) into discrete point tokens, the model queries the \(K\) nearest 3D Gaussians \(\{g_t^k\}_{k=1}^K\) for each point \(p_t\) via KNN. It computes relative spatial offsets \(\Delta p = p_t - g_t^k\) and indexes the corresponding velocity vectors \(\hat{D}_t\). After projecting \(\Delta P\) and \(\hat{D}_t\) into feature tokens \(\Delta \bar{P} \in \mathbb{R}^{4096 \times K \times 120}\) and \(\bar{D}_t \in \mathbb{R}^{4096 \times K \times 120}\) via separate MLPs, a global learnable vector \(L \in \mathbb{R}^{1 \times 120}\) is introduced as Query, while \(\Delta \bar{P}\) and \(\bar{D}_t\) serve as Key and Value in three cross-attention blocks. The resulting dynamics tokens \(\tilde{D}_t \in \mathbb{R}^{4096 \times 120}\) are added to visual-language tokens \(\tilde{F}^{vis}_t\), endowing action prediction with explicit foresight of moving targets.

Loss & Training

During online world model optimization, the photometric and structural rendering loss is minimized:

\[\mathcal{L}_{world} = (1 - \lambda)\mathcal{L}_1(I_{render}, I_{gt}) + \lambda \mathcal{L}_{ssim}(I_{render}, I_{gt})\]

The policy model is trained end-to-end using flow matching imitation learning. Along the linear interpolation trajectory \(\tau^i = (1 - i)\epsilon + i\tau^0\) between sampled Gaussian noise \(\epsilon \sim \mathcal{N}(0, 1)\) and clean demonstration \(\tau^0 = (a^{pos}_{t:t+J}, a^{rot}_{t:t+J})\) with time step \(i \sim \sigma(\mathcal{N}(0, 1))\), the loss supervises the vector field \(V_{flow}\) and gripper openness \(f_{open}\):

\[\mathcal{L}_{policy} = \|V_{flow}(A, o, l, \tau^i, \epsilon) - (\tau^0 - \epsilon)\|_2^2 + \text{BCE}(f_{open}(A, o, l, \tau^i), a^{open}_{t:t+J})\]

Key Experimental Results

Main Results

On the proposed PhysMani-Bench comprising 16 dynamic manipulation tasks, all methods were trained on a unified multi-task dataset using identical action representations. Table 1 lists the mean success rate (SR, %) and breakdown across the 8 primary dynamic task families under normal and high-speed configurations.

Method	Mean SR (%)	Beat Buzz	Insert Peg	Drop to Hoop	Pick Cube	Push Button	Deposit Rubbish	Place Cup	Remove Cup
Act3D	27.1 ± 2.7	46.7 ± 5.9	58.3 ± 4.7	6.7 ± 1.2	56.7 ± 6.6	49.2 ± 11.2	81.7 ± 5.1	0.0 ± 0.0	21.7 ± 8.5
\(\pi_{0.5}\)	8.3 ± 0.2	16.7 ± 3.1	0.8 ± 1.2	0.8 ± 1.2	18.3 ± 10.5	34.2 ± 4.7	8.3 ± 1.2	0.0 ± 0.0	22.5 ± 6.1
ManiGaussian	22.5 ± 0.8	40.0 ± 8.9	35.8 ± 4.2	6.7 ± 6.2	32.5 ± 12.7	18.3 ± 2.4	67.5 ± 2.0	1.7 ± 1.2	61.7 ± 10.5
3DDA	35.1 ± 1.7	60.0 ± 4.1	68.3 ± 6.6	9.2 ± 2.4	63.3 ± 6.6	39.2 ± 6.6	67.5 ± 4.1	15.8 ± 3.1	58.3 ± 3.1
3DFA	37.8 ± 0.9	82.5 ± 2.0	38.3 ± 6.2	30.8 ± 3.1	70.0 ± 6.1	55.8 ± 4.2	53.3 ± 4.2	17.5 ± 4.1	63.3 ± 2.4
3DFA-OF	37.5 ± 1.0	79.2 ± 4.7	45.0 ± 5.4	27.5 ± 5.4	60.8 ± 5.9	54.2 ± 3.1	58.3 ± 4.2	14.2 ± 5.1	58.3 ± 11.2
PhysMani (Ours)	45.9 ± 0.8	78.3 ± 4.2	37.5 ± 6.1	71.7 ± 3.1	84.2 ± 3.1	57.5 ± 3.5	60.0 ± 2.0	15.0 ± 3.5	61.7 ± 6.6

In high-speed task variants (H), PhysMani maintains superior robustness, scoring 46.7 ± 4.2% on Drop to Hoop (H) compared to 9.2 ± 3.1% for 3DFA and 0.0% for \(\pi_{0.5}\), and achieving 46.7 ± 4.2% on Deposit Rubbish (H) compared to 19.2 ± 4.2% for 3DFA.

Furthermore, quantitative evaluation of future frame prediction across 16 tasks (Table 2) demonstrates that PhysMani maintains trajectory errors (Traj. Err.) of 0.008 m, 0.039 m, and 0.074 m at the 1st, 5th, and 10th future frames (50 ms, 250 ms, and 500 ms horizons), whereas ManiGaussian incurs an error of 0.388 m at the very first step.

Ablation Study

The ablation experiments isolate the impact of the 3D scene velocity components \(D_t\) and the learnable query token \(L\) across all 16 benchmark tasks.

| Configuration | Mean SR (%) | Beat Buzz | Insert Peg | Drop to Hoop | Pick Cube | Push Button | Deposit Rubbish | Place Cup | Remove Cup | |---|---|---|---|---|---|---|---|---| | (1) Removing \(D_t\) | 37.9 ± 0.1 | 76.2 ± 1.2 | 43.8 ± 6.2 | 30.0 ± 2.5 | 65.0 ± 2.5 | 47.5 ± 2.5 | 65.0 ± 0.0 | 21.2 ± 1.2 | 58.8 ± 1.2 | | (2) Removing \(L\) (self-attention) | 40.8 ± 1.1 | 78.8 ± 3.8 | 31.2 ± 1.2 | 66.2 ± 1.2 | 72.5 ± 0.0 | 58.8 ± 3.8 | 62.5 ± 10.0 | 10.0 ± 2.5 | 58.8 ± 8.8 | | Full PhysMani | 45.4 ± 0.9 | 76.2 ± 3.8 | 43.8 ± 1.2 | 70.0 ± 2.5 | 85.0 ± 0.0 | 53.8 ± 1.2 | 61.2 ± 1.2 | 16.2 ± 1.2 | 50.0 ± 2.5 |

Key Findings

• Omitting the 3D velocity components \(D_t\) causes the mean success rate to drop sharply from 45.4% to 37.9%, regressing directly to the static 3DFA baseline level (37.8%). Tasks requiring precise temporal-spatial interception (such as Drop to Hoop dropping from 70.0% to 30.0%) suffer catastrophic degradation, verifying that 3D physical future dynamics are essential for dynamic manipulation.
• Replacing the learnable query token \(L\) with unconstrained self-attention degrades the success rate by 4.6 percentage points (from 45.4% to 40.8%), demonstrating that using a compact query vector to compress local neighborhood dynamics effectively eliminates spatial noise and point cloud sparsity issues.
• Sensitivity analysis over optimization iterations \(T \in \{10, 20, 30, 40, 50\}\) shows remarkable stability (mean SR stays within 44.7% to 46.0%), confirming that the divergence-free analytical formulation provides a powerful inductive bias that allows accurate dynamics forecasting even under tight computational budgets.

Highlights & Insights

• Analytical Divergence-Free Parameterization: By defining basis matrix \(\mathcal{B}(g_t)\) using anti-symmetric coordinate operators, the MLP outputs only 6 basic parameters while guaranteeing zero divergence across 3D space, sidestepping computationally prohibitive partial differential equation solvers in standard fluid/rigid simulators.
• Static Graph Re-utilization for Real-time Execution: Maintaining a constant number of 3D Gaussians (\(H=30000\)) enables binding optimization to static computational graphs, drastically cutting kernel launch overhead and bringing per-frame physical optimization latency down to ~200 ms.
• Decoupled Metric Architecture: The design neatly bridges differentiable neural rendering with robot action policies via point cloud neighborhood cross-attention, making the 3D world model directly portable to other point cloud or voxel-based policy architectures without redesigning the backbone.

Limitations & Future Work

• Author-Acknowledged Limitations: PhysMani-Bench currently comprises 16 tasks; while covering multiple motion patterns and speeds, it remains smaller than massive static benchmarks. Additionally, the divergence-free formulation targets continuous rigid and quasi-rigid physical motions, leaving non-rigid deformations, fluid splashing, and topological fractures unmodeled.
• Identified Latency Bottlenecks: With world model optimization taking ~200 ms and policy inference taking 272.8 ms, closed-loop control runs at 3–5 Hz. For ultra-high-speed scenarios (e.g. competitive table tennis or high-speed ball catching), the policy must rely on action buffers and temporal ensembling, where transient collision recovery remains challenging.
• Future Directions: A promising avenue is to isolate moving foreground objects from static backgrounds through hierarchical 3D Gaussian clustering, running physical optimization only on active dynamic clusters to push updates to 50 Hz, while distilling Rectified Flow denoising steps down to 1–2 steps.

Related Work & Insights

• vs ManiGaussian: While ManiGaussian also applies dynamic 3D Gaussians to robot manipulation, it relies on unconstrained time-conditioned deformation fields without explicit physical laws and only predicts a single future frame, suffering an enormous 0.388 m trajectory error on PhysMani-Bench. PhysMani's divergence-free formulation supports 10-step accurate future rollouts (0.008 m 1-step error), doubling the average manipulation success rate (45.9% vs. 22.5%).
• vs 3DFA / 3DFA-OF: 3DFA is an effective 3D action diffusion policy for static tasks, but frequently misses dynamic targets due to zero velocity awareness. Merely appending 2D optical flow (3DFA-OF) yields 37.5% SR—failing to outperform 3DFA (37.8%)—showing that 2D projected motion cues lack the essential 3D physical depth needed for spatial interception.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates an online divergence-free 3D Gaussian world model that explicitly drives future-aware dynamic robotic manipulation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thorough evaluation across a 16-task dynamic benchmark, numerical trajectory and frame prediction error metrics, detailed ablations, and real-world robot deployment.
• Writing Quality: ⭐⭐⭐⭐⭐ Highly structured presentation, clean mathematical formulation, and unambiguous pipeline explanations.
• Value: ⭐⭐⭐⭐⭐ Bridges the critical gap between static robot manipulation and high-speed dynamic interaction in real physical environments.