Trajectory-Level Continuous Action Representation for Robotic Manipulation¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Robotics & Embodied AI
Keywords: Continuous Action Representation, Trajectory-Level Modeling, Frequency Decoupling, Embodied Control, Visuomotor Policy
TL;DR¶
To resolve the rigid entanglement between control frequency and action representations in visuomotor systems, CAT encodes action trajectories within a fixed real-time interval into a constant set of continuous latent tokens with frequency-aware rotary position embeddings and asymmetric causal decoding, consistently outperforming timestep-level and discrete VQ baselines across simulation and real-world long-horizon manipulation tasks.
Background & Motivation¶
Visuomotor policy learning requires predicting action trajectories where task-critical dynamics are unevenly distributed across time. In typical manipulation tasks, only brief, localized trajectory segments decide success or failureโsuch as contact transitions, switches from free-space reaching to constrained interaction, or fine corrective adjustments. The remainder of the trajectory predominantly comprises smooth, repetitive free-space motion. When modeling capacity is spread uniformly across the discrete temporal grid, smooth portions dominate gradients and dilute the learning signal of consequential interaction moments.
This representational bottleneck is heavily amplified by control frequency. Under conventional timestep-level discretization or action chunking, representational length is tied directly to the temporal sampling grid. Scaling execution frequency from 10Hz to 50Hz proportionally increases sequence length, introducing substantial temporal redundancy. While approaches based on predefined temporal parameterizations (such as DCT bases or B-splines) prevent length expansion, their representational capacity remains strictly constrained by the chosen basis functions, struggling to generalize across arbitrary control rates and sharp contact transitions.
This paper tackles the challenge by unbinding action representation from discrete temporal steps and directly modeling trajectories over continuous physical time. Core idea: propose CAT, a trajectory-level continuous action representation framework that encodes action sequences within a fixed real-time window into a constant budget of continuous register tokens, normalizes physical temporal coordinates via frequency-aware rotary position embeddings (F-RoPE), and enforces temporal directionality through asymmetric causal decoding alongside trajectory-level contrastive regularization.
Method¶
Overall Architecture¶
CAT provides a compact, frequency-decoupled continuous action latent space for continuous generative policies (such as Flow-Matching or Diffusion-based VLAs). Within a fixed real-time window (\(T = 1\text{ s}\)), regardless of whether the sampling frequency \(f_s\) is 10Hz or 50Hz (where the actual number of timesteps \(N = f_s T\) varies), CAT compresses the full trajectory into a fixed budget of \(K\) continuous latent register tokens \(\{z_i\}_{i=1}^K\).
The architecture comprises two main stages: a frequency-aware Transformer encoder that injects Frequency-aware Rotary Positional Embeddings (F-RoPE) to align time indices into a shared temporal manifold and aggregates trajectory semantics into learnable register tokens; and an asymmetric causal Transformer decoder that allows target action embeddings to attend to register tokens while enforcing temporal causality among themselves. The flow diagram below illustrates the end-to-end representation learning, flow-matching policy generation, and multi-frequency continuous decoding:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Actions & Control Frequency<br/>{a_t}, frequency f_s, duration T=1s"] --> B["Frequency-Aware Rotary Positional Embedding (F-RoPE)<br/>Normalized time coordinates t/f_s"]
B --> C["Structural Decoupling Representation<br/>Learnable registers aggregate to K continuous latent tokens"]
C --> D["Trajectory-Level Contrastive Regularization<br/>L_rec + ฮป_reg * L_reg optimization"]
D --> E["Flow-Matching Latent Space Policy Denoising<br/>Predict velocity fields on compact latent tokens"]
E --> F["Asymmetric Causal Decoding<br/>Cross-step attention with causal masking to recover actions"]
F --> G["Multi-Frequency Robot Low-Level Execution"]
Key Designs¶
1. Structural Decoupling: severing representational dimensionality from timestep density Existing action tokenizers or timestep-level continuous predictors scale their sequence length with execution sampling frequency. CAT eliminates this coupling by projecting trajectories within a fixed real-time duration (\(T=1\text{ s}\)) into a fixed number of continuous latent tokens. Given an action sequence \(\{a_t\}_{t=1}^N\) where \(N = f_s T\), CAT introduces \(K\) learnable continuous register tokens \(\{z_i\}_{i=1}^K\) (empirically set to \(K=4\)): $\(\{z_i\}_{i=1}^K = f_\theta(\{a_t\}_{t=1}^N, \{t / f_s\}_{t=1}^N)\)$ Because \(K \ll N\) and \(K\) remains constant across sampling rates, downstream generative policies only predict this compact continuous representation rather than high-dimensional step-by-step trajectories, dramatically reducing the generative burden and cumulative drift at high control frequencies.
2. Frequency-Aware Rotary Positional Embedding: unifying physical time across control rates Under differing control rates, identical discrete step indices represent completely different physical timestamps, causing standard positional embeddings to fracture temporal semantics. CAT introduces Frequency-aware Rotary Positional Embeddings (F-RoPE) by scaling discrete step index \(t\) by control rate \(f_s\), producing physical time coordinates \(t / f_s\) with spacing \(1 / f_s\). For step \(t\) and sampling rate \(f_s\), the \(i\)-th dimension of the rotary embedding is formulated as: $\(\text{F-RoPE}\left(\frac{t}{f_s}\right) = \left[ \sin\left(\omega_i \frac{t}{f_s}\right), \cos\left(\omega_i \frac{t}{f_s}\right) \right]_{i=1}^{d/2}\)$ where \(\omega_i\) denotes the frequency coefficient and \(d\) is embedding dimensionality. Injected into Query and Key vectors throughout the encoder and decoder, F-RoPE embeds control frequency directly into the representation space, establishing a unified continuous coordinate system across diverse control rates.
3. Asymmetric Causal Decoding: reconciling trajectory-level guidance with execution causality When reconstructing continuous actions from continuous latents \(\{z_i\}_{i=1}^K\), unconstrained bidirectional attention allows non-causal backward information leakage, impairing deployment stability. CAT constructs an asymmetric attention mask within the Transformer decoder: the input consists of \(K\) register tokens and \(N\) learnable action embeddings \(\{e_t\}_{t=1}^N\). All action embeddings freely attend to all register tokens to extract global trajectory conditioning, whereas attention among action embeddings is strictly constrained to be causal along the temporal dimension (\(t' \le t\)). This setup anchors action generation to global planning while preserving strict temporal directionality during execution.
4. Trajectory-Level Contrastive Regularization: preserving discriminability under tight token budgets Compressing high-frequency manipulation trajectories into very few continuous tokens under pure MSE reconstruction loss \(\mathcal{L}_{\text{rec}} = \frac{1}{N}\sum_{t=1}^N \|a_t - \hat{a}_t\|_2^2\) can cause latent collapse, where dynamically distinct trajectories overlap in latent space. To ensure an expressive and discriminative latent structure, CAT introduces trajectory-level contrastive regularization. Concatenating register tokens yields trajectory vector \(\bar{z} = [z_1 \parallel z_2 \parallel \dots \parallel z_K]\). Over a mini-batch of size \(B\), the regularization objective is: $\(\mathcal{L}_{\text{reg}} = \frac{1}{B} \sum_{i=1}^B \log \left( 1 + \sum_{j \ne i}^B \exp\left( - \frac{\mathcal{D}(\bar{z}_i, \bar{z}_j)}{\eta} \right) \right)\)$ where \(\mathcal{D}(\cdot, \cdot)\) is a distance metric and \(\eta\) is temperature. Optimized under \(\mathcal{L}_{\text{CAT}} = \mathcal{L}_{\text{rec}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}}\) (with \(\lambda_{\text{reg}}=0.1\)), this penalty disperses semantically differing trajectories across latent space while maintaining intra-trajectory fidelity.
Loss & Training¶
CAT operates in a two-stage training paradigm: 1. Action Representation Pretraining: On demonstration datasets (e.g., Open X-Embodiment), the autoencoder (Encoder + Decoder) is trained end-to-end with \(\mathcal{L}_{\text{CAT}} = \mathcal{L}_{\text{rec}} + \lambda_{\text{reg}}\mathcal{L}_{\text{reg}}\). 2. Policy Flow-Matching: Downstream VLA policies (e.g., SmolVLA, \(\pi_0\)) model the continuous latent tokens \(Z = \{z_i\}_{i=1}^K\) using flow matching. The model learns a velocity field \(v_\psi(Z_\tau, \tau, c)\) under intermediate interpolation \(Z_\tau = (1-\tau)Z_0 + \tau Z_1\). At inference time, the policy generates clean latents \(Z_0\) in few denoising steps, and CAT's decoder reconstructs smooth continuous actions at target frequency \(f_s\).
Key Experimental Results¶
Main Results¶
1. Evaluation on the LIBERO Benchmark (Table 1)¶
Evaluated across Spatial, Object, Goal, and Long-horizon task suites under identical architectures and training regimes:
| Paradigm / Method | Spatial | Object | Goal | Long | Avg. Success Rate (%) |
|---|---|---|---|---|---|
| Autoregressive-based VLA | |||||
| Octo | 78.9 | 85.7 | 84.6 | 51.1 | 75.1 |
| OpenVLA | 84.7 | 88.4 | 79.2 | 53.7 | 75.9 |
| SmolVLA-FAST | 87.0 | 93.0 | 90.0 | 68.0 | 84.5 |
| \(\pi_0\)-FAST | 96.4 | 96.8 | 88.6 | 60.2 | 85.0 |
| SmolVLA-VQ-VLA | 90.0 | 94.0 | 86.0 | 71.0 | 85.3 |
| OmniSAT | 94.1 | 98.7 | 94.6 | 86.0 | 93.4 |
| Diffusion / Flow-based VLA | |||||
| Diffusion Policy | 78.3 | 92.5 | 68.3 | 50.5 | 72.4 |
| 4D-VLA | 93.8 | 92.8 | 95.6 | 86.5 | 92.2 |
| CogACT | 97.2 | 98.0 | 90.2 | 88.8 | 93.2 |
| SmolVLA (Baseline) | 90.0 | 96.0 | 92.0 | 71.0 | 87.3 |
| SmolVLA-CAT (Ours) | 95.0 | 97.0 | 94.0 | 77.0 | 90.8 (+3.5) |
| \(\pi_0\) (Baseline) | 96.8 | 98.8 | 95.8 | 85.2 | 94.2 |
| \(\pi_0\)-CAT (Ours) | 98.2 | 99.8 | 95.6 | 92.4 | 96.5 (+2.3) |
2. Multi-Frequency Evaluation on RoboTwin 2.0 (Table 3 excerpt)¶
Evaluation across 8 manipulation tasks under 10Hz, 16.7Hz, 25Hz, and 50Hz control rates (success rate in %):
| Task Name | 10Hz (Base / Ours) | 16.7Hz (Base / Ours) | 25Hz (Base / Ours) | 50Hz (Base / Ours) | Task Avg. (Base / Ours) |
|---|---|---|---|---|---|
| Grab Roller | 17 / 47 | 46 / 66 | 49 / 58 | 56 / 71 | 42.0 / 60.5 (+18.5) |
| Press Stapler | 33 / 52 | 36 / 52 | 19 / 50 | 50 / 48 | 34.5 / 50.5 (+16.0) |
| Place Container Plate | 43 / 71 | 64 / 57 | 50 / 76 | 46 / 40 | 50.8 / 61.0 (+10.2) |
| Lift Pot | 13 / 25 | 26 / 28 | 28 / 29 | 20 / 27 | 21.8 / 27.3 (+5.5) |
| Adjust Bottle | 76 / 68 | 77 / 78 | 69 / 72 | 78 / 86 | 75.0 / 76.0 (+1.0) |
| Rotate QRcode | 8 / 18 | 12 / 8 | 8 / 12 | 1 / 9 | 7.3 / 11.8 (+4.5) |
| Frequency Average (%) | 35.3 / 45.6 (+10.3) | 42.1 / 46.1 (+4.0) | 37.9 / 48.1 (+10.2) | 40.0 / 45.3 (+5.3) | - |
3. Real-World Long-Horizon Robotic Manipulation (Table 4)¶
Evaluated on an ARX R5 robot arm across 4 complex long-horizon tasks (normalized task score out of 100):
| Model | Stack-5 (5-cube tower) | Drawer (multi-stage place/pick) | Rope (two-ring threading) | Flower (two roses into vase) | Avg. Score (%) |
|---|---|---|---|---|---|
| SmolVLA (Baseline) | 29 | 49 | 62 | 40 | 45.0 |
| SmolVLA-CAT (Ours) | 43 (+14) | 61 (+12) | 70 (+8) | 68 (+28) | 60.5 (+15.5) |
Ablation Study¶
Ablations on LIBERO validate key hyperparameter choices (Table 5):
| Ablation Dimension | Configuration | Spatial | Object | Goal | Long | Avg. Success Rate (%) |
|---|---|---|---|---|---|---|
| Number of Registers \(K\) | \(K=2\) | 91.0 | 98.0 | 91.0 | 72.0 | 88.0 |
| \(K=4\) (Default) | 95.0 | 97.0 | 94.0 | 77.0 | 90.8 | |
| \(K=6\) | 92.0 | 98.0 | 93.0 | 74.0 | 89.3 | |
| \(K=8\) | 90.0 | 96.0 | 92.0 | 71.0 | 87.3 | |
| Reg. Loss Weight \(\lambda_{\text{reg}}\) | \(\lambda_{\text{reg}}=0\) (no reg.) | 91.0 | 98.0 | 93.0 | 75.0 | 89.3 |
| \(\lambda_{\text{reg}}=0.1\) (Default) | 95.0 | 97.0 | 94.0 | 77.0 | 90.8 | |
| \(\lambda_{\text{reg}}=0.2\) | 92.0 | 97.0 | 92.0 | 76.0 | 89.3 | |
| \(\lambda_{\text{reg}}=0.3\) | 90.0 | 95.0 | 91.0 | 76.0 | 88.0 | |
| Decoder Attention | Full Attention | 90.0 | 95.0 | 93.0 | 76.0 | 88.5 |
| Causal Attention | 95.0 | 97.0 | 94.0 | 77.0 | 90.8 (+2.3) |
In addition, comparing positional encoding schemes across multiple control rates on RoboTwin 2.0 (Figure 5) reveals that F-RoPE achieves 46.1% average success rate whereas standard Learned PE drops severely to 19.5% (-26.6 points), demonstrating the essential role of explicit physical time normalization.
Key Findings¶
- Latent capacity is non-monotonic: Success rate peaks at \(K=4\) (90.8%) and drops to 87.3% at \(K=8\). Over-allocating latent tokens introduces fitting noise and weakens regularized latent compactness.
- Contact-critical tasks gain the most: Across varying frequencies, tasks dominated by contact transitions and alignment (Grab Roller +18.5 points, Press Stapler +16.0 points) exhibit major gains, confirming CAT's advantage in preserving localized crucial dynamics.
- Optimal control frequencies vary across task dynamics: Not all tasks prefer high execution frequencies (e.g., Place Container Plate excels at 25Hz with 76% success, versus 40% at 50Hz). Decoupling action modeling from execution frequency allows adapting control rates to task physical properties.
Highlights & Insights¶
- Control frequency decoupling: Rather than indexing tokens by discrete time steps, CAT operates over continuous physical time intervals with fixed latent tokens, decoupling control hardware frequency from policy sequence modeling.
- Unified time coordinates via F-RoPE: Scaling step index by sampling frequency (\(t / f_s\)) into rotary embeddings maintains consistent geometric curvature in frequency space across arbitrary control rates.
- Asymmetric causal decoding: Allowing action embeddings to view all register tokens while enforcing strict causal temporal masking among actions prevents backward temporal leakage during physical rollout.
Limitations & Future Work¶
- Fixed real-time horizon: The current framework standardizes on a fixed window of \(T = 1\text{ s}\). Dynamic tasks with sub-second impacts or extended quasi-static motions may benefit from adaptive physical time horizons.
- Cross-embodiment generalization: While tested across simulation and real ARX arms, multi-embodiment transfer spanning mobile bases, multi-finger hands, and diverse compliance characteristics remains open for exploration.
Related Work & Insights¶
- vs FAST / VQ-VLA: VQ-based approaches discretize trajectories into symbolic tokens, incurring quantization error and discontinuous action jumps; CAT maintains a fully continuous latent space that preserves smooth dynamics.
- vs B-Spline / DCT (e.g., OmniSAT / FreqPolicy): Fixed parametric bases assume rigid analytical curves; CAT learns data-driven continuous latent representations via self-attention without predefined functional constraints.
Rating¶
- Novelty: โญโญโญโญโ (Decouples control frequency from action tokenization using continuous trajectory latents and F-RoPE)
- Experimental Thoroughness: โญโญโญโญโญ (Extensive simulation across LIBERO, MimicGen, and RoboTwin 2.0 multi-frequency splits, plus 4 real-world robotic tasks)
- Writing Quality: โญโญโญโญโญ (Crisply structured methodology, clear mathematical formulations, and thorough ablations)
- Value: โญโญโญโญโญ (High utility for modern VLA policy deployment on diverse physical robot hardware with variable control rates)