ChronoFlow Policy: Unifying Past-Future Interaction Flow in Visuomotor Policy Learning

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/The-kamisato-Sii/ChronoFlow-Policy
Area: Robotics / Embodied Intelligence
Keywords: visuomotor policy, interaction flow, historical memory, 3D point tracking, diffusion policy

The code URL is supplied by the paper, which promises a code and model release; release availability has not been checked online. The official title omits Current, whereas the PDF title uses Past-Current-Future.

TL;DR

ChronoFlow-Policy represents past, current, and future object-gripper motion as unified 3D keypoint trajectories, jointly trains future-flow recovery and action prediction, reaches average success rates of 72% on MetaWorld and 66% on RoboTwin 2.0, and improves final-stage real-world Swap-Easy success from 20% without history to 93% with history.

Background & Motivation

A point-cloud snapshot can tell a robot where a toy is now, but not necessarily which coaster it originally occupied, whether it has already been swapped, or which folding stage a towel has reached. Strong 3D imitation policies such as DP3 and RISE provide spatial perception, yet action-only supervision does not guarantee that their representations preserve these interaction histories. Historical trajectories can reduce state ambiguity, but do not by themselves explicitly constrain the object changes that should happen next.

A complementary approach predicts the future, for example through object-pose trajectories in MBA or dense scene flow in 3D-FDP. Pose trajectories are compact, but a single rigid pose cannot describe local towel deformation. Dense flow is expressive, but also includes background motion that may be irrelevant to the task. Object motion, gripper approach and contact, and task stage are different aspects of the same interaction; modeling them separately can lose their relationship.

This paper expresses that relationship through trackable 3D points: object points describe how the manipulated entity changes, gripper points describe how the robot participates, history records what has happened, and future trajectories supply structured supervision for action learning. Core idea: use the same object-gripper trajectory representation for memory and anticipation, then decode actions from a shared representation trained to predict future interactions rather than fitting actions directly from the current observation alone.

Method

Overall Architecture

Inputs comprise the current colored point cloud from one RGB-D camera, proprioception when available, and historical interaction trajectories accumulated from past observations. The scene point cloud has shape \(N\times6\), with 3D coordinates and RGB at every point and a potentially time-varying point count. It remains a separate observation condition rather than being completely replaced by sparse keypoints. Outputs are a future action chunk and predicted object-gripper trajectories.

The pipeline constructs interaction keypoint flow, encodes historical and noisy future flow into compact tokens, and denoises them under the current 3D observation condition. Both decoded future trajectories and hidden states enter the action decoder. This is neither full-video generation followed by planning nor a hand-designed controller that simply tracks predicted points.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB-D observation<br/>and gripper pose"] --> B["Interaction Keypoint Flow"]
    B --> C["Temporal Interaction Encoding<br/>History and noisy future"]
    A --> D["Current 3D observation encoding"]
    C --> E["Shared Denoising and Dual Decoding"]
    D --> E
    E --> F["Future interaction trajectories<br/>and action chunk"]
    F --> G["Execute first four steps<br/>Update observation and history"]

Key Designs

1. Interaction Keypoint Flow: describe the robot and objects in one spatial-temporal representation

Gripper and object points come from different sources but share coordinate semantics. Canonical gripper keypoints are fixed relative to the tool center point, or TCP. Applying the TCP rigid transformation at each timestep produces temporally consistent world-coordinate trajectories. For objects, SAM-2 segments task-relevant entities in the initial observation; farthest point sampling, or FPS, builds a candidate pool for each object, and TAPIP3D tracks the points over time. This is not a separate contact detector that automatically discovers contact events. Instead, coupled trajectories carry cues about approaching, grasping, and transporting objects.

Let \(N_g\) and \(N_o\) be the gripper and object point counts. Their union describes each timestep, while the future tensor preserves point identity, time, and 3D coordinates:

\[ P_t=P_t^g\cup P_t^o,\qquad P_t^g\in\mathbb{R}^{N_g\times3},\quad P_t^o\in\mathbb{R}^{N_o\times3},\quad P_{t:t+H}\in\mathbb{R}^{(N_g+N_o)\times H\times3}. \]

The window notation follows the paper; the tensor contains \(H\) future steps. Unlike a rigid object pose, a point set can represent different motions at different locations on a towel. Unlike dense scene flow, it removes many irrelevant scene points. Gripper trajectories are structurally aligned with actions, so future supervision asks not only where an object will go but how the robot will participate. Nevertheless, sparse points are not necessarily contact points, and the method does not explicitly enforce physical contact consistency.

2. Temporal Interaction Encoding: query history and noisy futures with the same interaction queries

Historical flow contains trajectories that have already occurred. During training, noise is added to demonstrated future trajectories; at inference, the future starts from Gaussian noise. A multi-head cross-attention encoder reads the concatenation of history and noisy future. History is therefore not simply a stack of complete point-cloud frames: it shares keypoint semantics with the future being predicted. Two observations with the same current object positions but different preceding interactions can consequently lead to different action conditions.

Learnable interaction queries have shape \(Q\in\mathbb{R}^{(N_g+N_o)\times D}\), and the output tokens have shape \(Z_t^k\in\mathbb{R}^{(N_g+N_o)\times D}\). Each compact token corresponds to an object or gripper keypoint, and \(k\) denotes the noise step. This compression avoids making the downstream denoiser directly process the complete spatial-temporal trajectory matrix while retaining point identities. The cache does not specify fixed point counts, the history-window length, or embedding width, so these symbols should not be mistaken for published numerical hyperparameters.

The current point cloud has an independent encoding path: a PointNet-style encoder in simulation and a sparse 3D encoder in real-world experiments. Available proprioception is encoded by an MLP and fused into the observation feature. History explains task stage, the current cloud supplies spatial conditions, and the noisy future specifies the generative target; these inputs meet in conditional denoising.

3. Shared Denoising and Dual Decoding: expose both explicit interaction predictions and hidden states to actions

The backbone can be Unet1D or a Diffusion Transformer. It receives observation features, the noise step, and interaction tokens, progressively producing clean interaction representations. A lightweight Transformer trajectory decoder expands each point token into \(H\cdot3\) coordinate values, then reshapes them into the future trajectory tensor. The action decoder reads both these coordinates and the shared hidden states, retaining control information that sparse points alone might not fully express.

The inference-side action relationship is legible in the source:

\[ \hat a_{t:t+H}=A_\phi\bigl([Z_t^0;\hat P_{t:t+H}]\bigr). \]

The brackets indicate concatenated inputs, \(Z_t^0\) denotes clean interaction tokens, and \(\hat P\) denotes decoded future trajectories. Unlike independent action and auxiliary-prediction heads, action generation explicitly uses the trajectory output without being restricted to it. The cached policy-factorization equation and several diffusion equations contain missing symbols, so this note does not reconstruct their complete probability expressions, noise schedules, or implementation-specific concatenation axes.

Completing all denoising steps at every training update would be expensive. Instead, the action decoder is trained with partially denoised tokens \(Z_t^{k-1}\) and an approximate trajectory \(\tilde P_{t:t+H}\) reconstructed from an intermediate noisy state. Inference uses fully denoised outputs. This connects action supervision to the shared representation without requiring the complete sampling chain for each optimization step; it does not imply that deployment skips denoising altogether.

A Worked Example

In real-world Swap-Hard, two toys must exchange their original coaster locations. Once toy A has been moved to an intermediate position, a current image alone may not reveal A's original coaster or reliably distinguish an unfinished swap from the stage requiring retrieval of the intermediate toy. Historical object trajectories preserve A's origin, while gripper trajectories provide evidence of the transfer just completed.

The model encodes that history together with the current point cloud and noisy future flow, generates the next object-gripper motion, and decodes 8 action steps. Only the first 4 steps are executed before replanning from a new observation; asynchronous TAPIP3D updates the online history. This is a task-level explanation of the reported mechanism, not a new experiment or an assumption about unpublished point counts or coordinates. Full CFP achieves 61% at the final stage, versus 11% without history, showing that memory helps while the difficult final stage remains far from solved.

Loss & Training

Training combines an interaction-flow diffusion objective with an action MSE objective. The former trains the shared backbone to recover interaction dynamics; the latter makes partially denoised representations and approximate trajectories predictive of expert actions. The weighted objective described explicitly in the prose can be summarized as:

\[ \mathcal L=\mathcal L_{\mathrm{action}}+\lambda_{\mathrm{CF}}\mathcal L_{\mathrm{CF}}. \]

This expression summarizes the textual description. Cached equations (5) through (8) are incompletely extracted, so they do not support reconstructing the exact noise loss, trajectory-recovery formula, numerical weight, or optimizer configuration. The text also uses both diffusion and flow-matching terminology; this note does not infer a particular implementation from damaged equations.

All experiments use an action horizon of 8, a current-observation horizon of 1, and receding-horizon execution of the first 4 steps. Training randomly subsamples object keypoints and applies the same spatial transformation jitter to trajectories and scene point clouds, preserving their spatial correspondence. The authors also mention dropout, history truncation, and using the same imperfect tracking pipeline during training and inference. These choices discourage dependence on fixed tracks, but their probabilities and perturbation magnitudes are not specified in the cache.

Key Experimental Results

Main Results

Simulation covers 14 tasks, with 7 each from MetaWorld and RoboTwin 2.0. The following entries are the reported averages in Tables 2 and 3, not averages recomputed from rounded task-level results.

Method	MetaWorld average success (%)	RoboTwin 2.0 average success (%)
DP3	30	43
3D-FDP	34	47
MBA	47	56
CFP (Unet)	72	66
CFP (DiT)	70	63

These simulation tasks are largely Markovian, so the comparison mainly tests future interaction supervision rather than establishing the necessity of long-term history. MetaWorld uses 3 seeds, evaluates 20 episodes every 200 training epochs, averages the top 5 success rates, and then aggregates across seeds. RoboTwin evaluates 100 episodes every 500 epochs and averages the top 3 success rates without multiple seeds. These are not unselected estimates from a fixed final checkpoint.

The real platform comprises a Flexiv Rizon arm, a Robotiq 2F-85 gripper, and one fixed top-down RealSense D415. Each task provides 50 expert demonstrations, with training on an RTX 3090. Each policy normally receives 15 evaluation trials per task, increased to 18 for Swap-Hard. The next table extracts only final-stage success rates from Table 5: completion of the entire task in one execution without resets, not an average over stages.

Method	Breakfast II (%)	Towel II (%)	Pour II (%)	Swap-Easy III (%)	Swap-Hard III (%)
RISE	53	40	47	13	11
3D-FDP	67	53	47	33	17
HistRISE	67	47	67	80	56
CFP w/o past	67	87	60	20	11
CFP	80	87	80	93	61

MBA is evaluated only on Breakfast, where final-stage success is 73%, versus CFP's 80%; it is not a baseline evaluated uniformly across all real-world tasks. Adding history improves CFP by 73 percentage points on Swap-Easy and 50 points on Swap-Hard, but leaves Towel unchanged at 87%. The benefit of history is clearly task-dependent.

Ablation Study

The following Fold Towel Stage II values are explicitly stated in the ablation prose. Other figure values without an unambiguous textual correspondence are not transcribed.

Configuration	Stage II success (%)	Drop from full model (percentage points)
Full CFP	87	0
Without ChronoFlow trajectory supervision	67	20
Without object trajectories	47	40
Without gripper trajectories	80	7
Without random keypoint sampling	27	60
Without interaction-flow spatial augmentation	6	81

Key Findings

• Object trajectories are particularly important for towel deformation, but gripper trajectories are not redundant: removing them costs 40 and 7 percentage points, respectively. This ordering is evidence from a specific deformable-object task, not a universal ranking of components.
• Random sampling and shared spatial augmentation have larger effects than removing auxiliary supervision. Strong performance therefore depends on the combination of representation and robust training, rather than solely on the idea of unifying history and the future.
• Deployment runs at 12.18 Hz without history tracking, 0.93 Hz with synchronous TAPIP3D, and 5.82 Hz with asynchronous tracking. Asynchrony substantially reduces the bottleneck but remains slower than omitting tracking; these are inference frequencies, not low-level robot servo rates.

Highlights & Insights

• The auxiliary objective shares spatial semantics with control. It predicts robot-object motion rather than arbitrary visual reconstruction, and the action decoder explicitly consumes those predictions, connecting supervision to control more directly.
• One representation supports both memory and prediction. Historical trajectories encode event order, while future trajectories express anticipated geometric change, avoiding incompatible state descriptions for the two temporal directions.
• Sparsity does not eliminate current global perception. Keypoint flow handles compact interaction modeling, while the separate point-cloud encoder retains environmental conditions. This division is more cautious than replacing perception entirely with tracked points.

Limitations & Future Work

• The authors acknowledge segmentation and tracking noise, drift, and missing keypoints. Severe occlusion or persistent tracking failure still degrades performance; confidence-aware trajectories, failure detection, and uncertainty-aware control are natural next steps.
• Real-world evaluation covers 5 tasks with only 15 or 18 trials each, and the table does not report confidence intervals. CFP's 61% versus HistRISE's 56% on Swap-Hard is a small gap and does not establish statistical significance from rounded percentages alone.
• Main results average selected best checkpoints, and RoboTwin lacks multi-seed evaluation. Stronger verification should report a fixed checkpoint-selection rule, final-checkpoint results, and between-seed variance.
• Deformable-object evidence centers on towel folding, as do the component ablations. This does not establish generality to arbitrary flexible objects or universal contact reasoning.
• Asynchronous tracking still has overhead, and the paper does not fully analyze stale-history effects on closed-loop control. Tracking update rate, occlusion duration, and action speed deserve a joint evaluation.
• The cache contains damaged equations and does not specify keypoint counts, history length, or the loss weight. It supports a mechanism-level explanation, not a complete reproducible training configuration.

Related Work & Insights

• Versus DP3 / RISE: These methods primarily use 3D observations and action supervision; CFP adds future dynamics targets with explicit entity semantics. The gains come with segmentation, tracking, and trajectory-construction costs.
• Versus HistRISE / TraceVLA: Historical traces help retain object motion; CFP additionally couples object and gripper trajectories and predicts future interaction. HistRISE remains strong on both Swap tasks, indicating that history itself explains a substantial share of the improvement.
• Versus MBA: Pose trajectories naturally describe rigid motion, whereas CFP's point flow can express local deformation but depends more strongly on persistent point identity. The representation trade-off cannot be settled by a single task's success rate.
• Versus 3D-FDP: Dense scene flow preserves broader motion information, while CFP focuses on task-relevant interaction. It remains worth testing whether sparse selection becomes a liability when target segmentation fails or important environmental dynamics are omitted.

Rating

• Novelty: 4/5. Unifying object-gripper motion across history and future, then exposing the shared representation to actions, is a meaningful combination; tracking and diffusion building blocks come from prior work.
• Experimental Thoroughness: 4/5. The paper includes 14 simulated and 5 real-world tasks, with history and component ablations, but real-world sample sizes, best-checkpoint reporting, and cross-task ablation coverage remain limited.
• Writing Quality: 3/5. Representation, architecture, and real-world tasks align well, but damaged cached equations, mixed diffusion terminology, and missing protocol details limit precise interpretation.
• Value: 4/5. The approach offers practical ideas for history-dependent manipulation and deformable objects; its gains should be understood alongside tracking costs and robust-training requirements.