Towards Generalizable Robotic Manipulation in Dynamic Environments

Conference: ECCV 2026
arXiv: 2603.15620
Code: https://github.com/H-EmbodVis/DOMINO
Area: Robotics / Embodied AI
Keywords: Dynamic manipulation, Vision-Language-Action models, Optical flow, Future prediction, Bimanual manipulation

TL;DR

To address the limitation that existing VLAs can only grasp static objects and fail when facing moving targets, this work constructs DOMINO, a dynamic manipulation benchmark containing 117k trajectories across 35 tasks. Furthermore, it proposes the PUMA architecture, which equips VLAs with "historical optical flow + object-level future feature prediction," achieving a 6.3% absolute success rate increase on dynamic tasks.

Background & Motivation

Vision-Language-Action (VLA) models have progressed rapidly in robotic manipulation in recent years. Models such as OpenVLA, π0, and RDT can map "observed images + a language instruction" directly into control commands for robotic arms, forming the prototype of general embodied intelligence. However, mainstream VLAs almost entirely operate under a "static world" assumption: objects sit still on a table, and the robot only needs to reach and grasp in static environments. Real-world scenarios are far more dynamic—parts on an assembly line move, items handed over by humans move, and robots must simultaneously perceive in real time and predict motion to grasp objects accurately. The authors term this requirement dynamic manipulation and point out that it has long been a neglected gap: on one hand, there is a lack of large-scale dynamic manipulation data; on the other hand, mainstream VLA architectures themselves lack dynamic perception and motion prediction capabilities.

The core difficulty in data collection lies in the strict spatial-temporal synchronization required between the moving target and the motor actions of the robotic arm. Human teleoperation or scripted policies struggle to react precisely to a continuously moving target, resulting in the vast majority of existing embodied datasets being confined to static tasks. On the architectural side, the authors debunk two seemingly reasonable intuitions using a set of oracle experiments. First, fine-tuning standard VLAs on dynamic data yielded an average success rate increase of less than 3%, indicating that this is not a simple data distribution shift but rather an architectural bottleneck. Second, even if ground-truth future trajectories of the objects are directly provided to the model, the overall success rate only sees marginal improvement. This occurs because the model lacks historical frame observations; although it can follow the trajectory, it fails to understand the underlying physical dynamics of the target, degrading into mechanically "following the path," which instead interferes with closed-loop manipulation. These two findings jointly point to a Key Challenge: dynamic manipulation requires both extracting "how the object is currently moving" from the history and predicting "where it will go next" in the future, yet single-frame-observation VLAs fail on both ends.

Consequently, the authors deconstruct dynamic perception into two goals that must be achieved simultaneously: scene-centric historical dynamics capture and object-centric future motion prediction. Core Idea: Use optical flow between historical frames to supply the VLA with "explicit motion perception," and introduce a set of learnable world queries during training to align with object-level future features extracted by DINO. This internalizes the ability to predict the target's future state into the shared representation, thereby allowing the model to make proactive interactions with zero additional overhead during inference.

Method

Overall Architecture

This work, DOMINO, stands on two pillars: the DOMINO benchmark (data + evaluation) and the PUMA architecture (a dynamic-aware VLA). On the benchmark side, based on the SAPIEN physics engine and RoboTwin 2.0, a low-cost, scalable dynamic data generation pipeline is established to produce 35 dynamic tasks, 5 robot platforms, and over 110k expert trajectories. The task difficulty is parameterized by a scalar "dynamic coefficient \(\alpha\)" (denoted as DOMINO@\(\alpha\), where \(\alpha\) is the maximum target velocity, and \(\alpha=0\) degenerates to static). Tasks are also categorized into three levels based on motion complexity: Level 1 (constant speed, predictable), Level 2 (polynomial high-order curves), and Level 3 (piecewise random sudden changes), accompanied by a multi-dimensional metric system beyond binary success rates (including Success Rate SR and Manipulation Quality Score MS).

On the architectural side, PUMA is built upon Qwen3-VL. It takes a history of observations \(o_{t-h:t}\) of length \(h+1\) along with a language instruction as input, sharing a backbone to optimize two heads simultaneously: an action policy \(\pi_\phi\), which outputs an action chunk \(\hat{a}_{t:t+K-1}\) of length \(K\) per forward pass, and an auxiliary future feature predictor \(\psi_\omega\), supervised only during training, which outputs auxiliary features \(z_{t+1:t+N}\) encoding the target's future motion. The design features two key aspects: on the encoding side, historical frames are compressed to compute optical flow maps, providing the policy with explicit motion cues (scene-level historical dynamics); on the decoding side, a dedicated set of world queries is used to implicitly predict the future representation of the target object, supervised by similarity metrics against ground-truth features extracted from a frozen DINO (object-level future dynamics). Since the future predictor is inactive during inference, PUMA achieves dynamic perception capabilities with almost zero additional latency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Multi-view current observation<br/>+ historical frames + language instruction"] --> B["Dataset Construction: DOMINO<br/>3-stage spatio-temporal synchronized generation of 117k trajectories"]
    B --> C["Scene-level Historical Dynamics Encoding<br/>Compress historical frames → compute optical flow → feed to Qwen3-VL"]
    C --> D["Object-level Future Representation Prediction<br/>World Query aligns with DINO future features"]
    D --> E["Dual-Query Decoding<br/>Action Query yields action chunks<br/>World Query supervised during training"]
    E -->|Future predictor inactive during inference| F["Output: Bimanual action chunks<br/>a(t) … a(t+K-1)"]

Key Designs

1. DOMINO Three-Stage Spatio-Temporal Synchronized Data Generation: Aligning Moving Targets and Robotic Arm Actions Seamlessly

The fundamental difficulty in generating dynamic data lies in the necessity of strict spatial-temporal synchronization between the target and the robotic arm because both are in motion. Directly teleoperating a robotic arm to interact with a moving object in real time is nearly impossible for human operators, yielding mostly useless data. The authors' ingenious solution is to solve this synchronization problem in reverse: first, in the static dry-run stage, a target grasping pose is randomly sampled, and the task is executed in a static environment to precisely record the execution time \(T_{exec}\) required by the robotic arm. Second, in the kinematic reverse calculation stage, based on this execution time and the specified motion trajectory, the initial position and velocity of the object are reverse-calculated so that it arrives precisely at the grasping point when the robotic arm reaches it. Finally, in the synchronized dynamic execution stage, the target object is instantiated as a kinematic rigid body in SAPIEN, and the robotic arm replays the verified action plan in synchronization with the moving target. Consequently, action and motion are naturally aligned, avoiding real-time reaction uncertainties. Taking Level 1 as an example, the velocity magnitude \(v\) is sampled from a half-normal distribution \(|\mathcal{N}(0.8\alpha, \alpha/3)|\) and clipped to \([0.01, \alpha]\), the direction angle is uniformly sampled, and the starting position is directly reverse-calculated by:

\[\mathbf{p}_{\text{start}} = \mathbf{p}_{\text{end}} - \mathbf{v} \cdot T_{\text{sec}}\]

For Levels 2/3, polynomial fitting of control points and piecewise Dirichlet time-partitioning are used to generate high-order and sudden-change trajectories. The entire pipeline runs on SAPIEN + RoboTwin 2.0 and features collision detection (once the gripper contacts the target, it transitions from a kinematic body to a dynamic rigid body, ensuring realistic grasp forces), enabling low-cost, scalable mass production of high-quality expert demonstrations.

2. Scene-Level Historical Dynamics Encoding: Visualizing "Motion" Explicitly via Optical Flow Rather Than Stacked Raw Frames

Standard VLAs only observe a single frame and are inherently unable to interpret where an object is moving. A simple remedy is to stack historical frames and feed them into the network, but the authors found that forcing the network to implicitly infer frame-to-frame changes increases the learning burden and yields poor results (in the ablation study, using raw historical frames actually reduced the success rate from the 10.86% baseline to 8.15%). PUMA's approach is as follows: sample \(h\) third-perspective historical frames with a fixed step, perform spatial compression, and compute optical flow maps between the compressed frames (using the Farnebäck algorithm for dense optical flow mapped to HSV—where hue encodes motion direction and value encodes motion magnitude, and then convert to RGB). These optical flow maps are then processed by Qwen3-VL's vision encoder alongside the current multi-view images. The benefit of optical flow is that it expresses the motion state explicitly and intuitively. The policy does not need to struggle to guess motion from raw pixel stacks; it can directly estimate the object's motion trend. To control overhead, the resolutions of the history and optical flow maps are compressed to 64×64, and a disk-based caching mechanism is utilized (saved as compressed NumPy arrays after the first epoch and read directly in subsequent epochs). In practice, optical flow computation accounts for only 3.3% of the single-step inference latency.

3. Object-Level Future Representation Prediction: Implicitly Predicting Target Destination via World Queries Supervised Only During Training

Having historical data alone is insufficient; to catch a moving target, the policy must predict forward where it will go next. Existing world-model approaches (e.g., DreamVLA) mostly predict scene-level global dynamic regions, but the authors argue that dynamic manipulation truly requires object-centric prediction to isolate the target's trajectory from irrelevant scene dynamics. Specifically, during training, \(N\) future frames are sampled at fixed intervals. The manipulated object specified in the language instruction is grounded using frozen GroundingDINO + SAM2 to obtain the mask \(\mathcal{B}\). Then, the patch tokens are extracted using a frozen DINO encoder \(\mathcal{E}\), and mask average pooling \(\mathcal{P}\) is applied to obtain the object-level future ground-truth feature:

\[\mathbf{f}_{t+i}=\mathcal{P}\!\left(\mathcal{E}(I_{t+i}), \mathcal{B}(I_{t+i}, p)\right), \quad i=1,\dots,N.\]

Then, \(N\) learnable world queries are introduced to aggregate spatial-temporal context and predict the target's future representation \(z_{t+i}\) in the latent space. A cosine similarity loss is employed to pull the prediction closer to the ground-truth \(f_{t+i}\). The core merit of this design is that "supervision is applied only during training": it forces the shared representation to learn how the target will move in the future, acting as a forward-looking regularization for the action policy. During inference, future frames are completely unnecessary, and the predictor is inactive, allowing the model to enjoy dynamic prediction capabilities with zero additional computational overhead. Ablation results show that adding this auxiliary prediction (\(N=2\)) improves the success rate from 11.71% (with only optical flow) to 14.80%, and extending the prediction horizon to \(N=4\) achieves the optimal success rate of 17.20%.

Loss & Training

PUMA is trained end-to-end, with the total loss formulated as the weighted sum of two terms: the action policy is supervised by an \(\ell_1\) regression loss to predict action chunks:

\[\mathcal{L}_{action} = \frac{1}{K} \sum_{i=0}^{K-1} \| \hat{\mathbf{a}}_{t+i} - \mathbf{a}^*_{t+i} \|_1,\]

and the auxiliary future predictor minimizes the cosine distance between the predicted representation and the object-level ground-truth feature:

\[\mathcal{L}_{world} = \frac{1}{N}\sum_{i=1}^{N}\left(1-\frac{\mathbf{z}_{t+i}^{\top}\mathbf{f}_{t+i}}{\|\mathbf{z}_{t+i}\|_2\,\|\mathbf{f}_{t+i}\|_2}\right),\]

combining into \(\mathcal{L}_{total} = \mathcal{L}_{action} + \lambda\,\mathcal{L}_{world}\), where \(\lambda\) balances the impact of the dynamic task (set to 0.05 in the implementation). The policy is initialized from Qwen3-VL-4B. The action head is an MLP predicting 14-dimensional absolute joint actions with a future action window of 15 steps. It uses 4 world queries with history and future windows of 4 frames and step size 4, and future features are extracted using frozen DINOv2-B/14. Full-parameter fine-tuning is conducted with DeepSpeed ZeRO-2 on 8×A100s.

Key Experimental Results

Main Results

Comparing against SOTA on [email protected] (Level 1, clean setting, Aloha-AgileX robot, with all VLAs fine-tuned on the 35 dynamic tasks):

Method	Backbone	SR (%)	MS
OpenVLA	Prismatic	1.54	6.10
RDT-1B	DiT	5.34	17.71
π0	PaliGemma	8.17	23.96
π0.5	PaliGemma	9.63	26.17
OpenVLA-OFT	Qwen3-VL*	10.86	30.49
PUMA (Ours)	Qwen3-VL	17.20	34.97

PUMA achieves a success rate of 17.20%, substantially outperforming the runner-up OpenVLA-OFT (Qwen3-VL version) at 10.86% and \(\pi0.5\) at 9.63%, and obtains the highest MS score of 34.97. The "absolute gain of 6.3%" mentioned in the TL;DR is relative to the strong baseline using the same backbone.

Performance collapse from static to dynamic (Level 1) explains why the problem is challenging:

Method	Static S→S	Dynamic S→D (Zero-shot)	Dynamic D→D (Fine-tuned)
ACT	27.7	6.5	9.4
OpenVLA-OFT	17.5	6.7	9.1
π0.5	44.8	7.5	9.6

\(\pi0.5\) drops directly from 44.8% in the static setting to 7.5% in the zero-shot dynamic setting, and fine-tuning with dynamic data only recovers it to 9.6%—confirming that "this is an architectural bottleneck rather than a data issue."

Ablation Study

Config	Hist. Rep.	Aux. Pred.	N	SR (%)	MS
Baseline	✗	✗	-	10.86	30.49
+ Historical Optical Flow	✓	✗	-	11.71	31.02
+ Optical Flow + Future Prediction	✓	✓	2	14.80	32.74
+ Historical Raw Frames + Future Prediction	✓	✓	2	8.15	28.62
+ Optical Flow + Future Prediction	✓	✓	4	17.20	34.97

Key Findings

• Optical flow vs. raw frames is the bottleneck: Adding future prediction with raw historical frames (8.15%) performs worse than the baseline (10.86%), whereas optical flow (14.80%) yields performance gains. This proves that "forcing the network to infer motion from raw pixel stacks" is a dead end, and explicit optical flow representation is key.
• Longer future prediction horizons lead to more stable performance: Increasing \(N\) from 2 to 4 improves the success rate from 14.80% to 17.20%, suggesting that a longer prediction horizon helps establish a more robust understanding of future trajectories.
• Dynamic data benefits static tasks: Policies trained only on dynamic data transfer zero-shot to static scenarios and even outperform static-trained policies in certain tasks (e.g., Adjust Bottle: 65% vs. 47% on OpenVLA-OFT). Combined training on both static and dynamic data further increases PUMA's dynamic SR from 14.80% to 19.71% (+4.91%). The authors explain that static data provides stable structural priors, while dynamic data breeds reactive dexterity, making them complementary.
• Real-world transferability: On the bimanual AgileX Piper, 5 dynamic tasks with 50 demonstrations per task were chosen. After fine-tuning from the simulation checkpoint using LoRA, PUMA achieves a 42% average SR, far outperforming \(\pi0.5\) (24%) and ACT/RDT (<10%).
• Almost zero overhead: PUMA takes 103.7ms (9.6Hz) per step on a single RTX 4090 GPU, comparable to \(\pi0.5\). The optical flow computation only accounts for 3.3% of the execution time, making the overhead of the dynamic perception module negligible.

Highlights & Insights

• "Reverse-calculated" data synchronization is the most elegant engineering insight of the entire paper. Instead of forcing the robotic arm to chase a moving target in real time (which makes high-quality data collection nearly impossible), it measures the execution time in static dry-runs and then reverse-calculates where the object must start. This transforms the dynamic synchronization dilemma into a deterministic kinematics reverse calculation problem. This approach can be applied to any data collection scenario that requires precise alignment between the agent and environmental events.
• Imposing auxiliary supervision only during training is highly cost-effective: the world queries internalize "predicting the future" into the shared representation, and the prediction head is discarded during inference, yielding forward-looking capabilities for free. This "heavy during training, light during inference" auxiliary task paradigm is valuable for latency-sensitive robot policies.
• Debunking prior assumptions with two oracle experiments before proposing theories: Instead of stacking modules from the start, the authors rule out the shortcuts of "only data matters" and "only future trajectory is needed" via two experiments—namely, "fine-tuning only yields a 3% gain" and "feeding ground-truth future trajectories only brings marginal improvements." This solidifies the motivation for the proposed method by finding that both history and future must be prioritized.
• Optical flow as an explicit motion prior is validated as effective in VLA. Encoding directions/magnitudes using HSV and converting them to RGB allows direct feed into the vision encoder with virtually no architectural changes. This trick of "visualizing motion as an image" can be integrated into other single-frame policies at low cost.

Limitations & Future Work

• The vast majority of main experiments are conducted under Level 1 (constant speed), a clean setting, a single Aloha-AgileX platform, and \(\alpha=0.1\). Level 2/3 high-order/sudden-change dynamics are only validated on a small subset, and the absolute success rate remains low (with PUMA achieving only 4.6% under Level 3), which is far from practical.
• The data originates entirely from simulation (SAPIEN). Real-world experiments rely on "dragging objects with a transparent string" to simulate Level 1 linear motion, which fails to reproduce high-order or sudden-change trajectories, resulting in limited real-world coverage of dynamic complexity.
• Object-level supervision depends on GroundingDINO + SAM2 to ground the target from instructions. If instruction parsing fails, it falls back to using the complete sentence as a prompt, making it sensitive to instruction wording and grounding quality.
• Future prediction remains an implicit feature alignment. The authors themselves describe PUMA as an "exploratory attempt" that does not explicitly model physical dynamics (such as collisions and contact forces), leaving it insufficient to handle truly random sudden target changes.

Related Work & Insights

• vs. DreamVLA / Scene-level World Models: These models predict global scene transitions or robot kinematics, neglecting the dynamics of individual objects, which are critical for reactive planning. PUMA uses world queries for object-level future feature prediction, isolating target trajectories from irrelevant scene dynamics to achieve a finer granularity.
• vs. VLAs with Temporal Context / Memory Mechanisms (e.g., MemoryVLA, TraceVLA): These methods primarily track task progress for long-horizon tasks and struggle with the high-frequency motion estimation required for dynamic manipulation. PUMA uses optical flow to provide explicit high-frequency motion cues specifically for moving targets.
• vs. Benchmarks like RoboTwin 2.0 / LIBERO / RLBench: Existing simulation benchmarks are mostly built on the "static world" assumption (where state transitions are completely driven by the robot) and cannot evaluate manipulation of independently moving targets. DOMINO is the first benchmark that parameterizes difficulty with a dynamic coefficient \(\alpha\) and categorizes tasks by motion complexity to specifically evaluate dynamic bimanual manipulation.
• vs. Oracles directly receiving ground-truth future trajectories: This paper demonstrates that providing future trajectories in isolation causes policies to overfit to "blind tracing," while a lack of history interferes with closed-loop controls. This serves as a reminder for future work that future cues must be anchored in historical physical context to be effective.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to systematically define and benchmark dynamic manipulation. The reverse-calculated data synchronization and object-level future prediction designs are highly novel.
• Experimental Thoroughness: ⭐⭐⭐⭐ Main experiments, ablations, cross-difficulty tests, real-world evaluations, and latency analyses are all provided, though the coverage of high-order dynamics and real-world dynamic complexity is somewhat thin.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear logical chain, proving theories by first debunking prior assumptions with two oracle experiments. The finding-style summaries make it highly readable.
• Value: ⭐⭐⭐⭐⭐ Fills the gap of VLAs "only being able to grasp static objects." Open-sourcing the benchmark, data, and architecture provides a substantial boost to the embodied AI community.

Related Papers

• [CVPR 2026] AdaDexTrack: Dynamic Modulation for Adaptive and Generalizable Dexterous Manipulation Tracking
• [ICLR 2026] VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation
• [ICLR 2026] Test-Time Mixture of World Models for Embodied Agents in Dynamic Environments
• [ECCV 2026] MuSix: Multi-scale Mixture of World Models for Embodied Agents in Evolving Environments
• [AAAI 2026] Dexterous Manipulation Transfer via Progressive Kinematic-Dynamic Alignment