Skip to content

LARY: A Latent Action Representation Yielding Benchmark

Conference: ECCV 2026
Paper: ECCV Official
Code: https://meituan-longcat.github.io/LARYBench/
Area: Robotics & Embodied AI
Keywords: Latent Action, Embodied AI, Benchmark, Vision-Language-Action, Motion Regression

TL;DR

Meituan introduces LARYBench, the first unified benchmark decoupling latent action representation evaluation from downstream policy learning across 1.2M video clips and 151 action classes, revealing that off-the-shelf general visual encoders without action supervision systematically outperform specialized embodied latent action models.

Background & Motivation

Leveraging large-scale, unannotated human demonstration videos to resolve the acute robotic "data island" challenge—where real robot teleoperation data remains prohibitively scarce—has emerged as a foundational frontier in Vision-Language-Action (VLA) learning. To map continuous perceptual video streams into embodiment-agnostic control representations, Latent Action Models (LAMs) compress inter-frame visual transitions into discrete codebooks or continuous embeddings to serve as universal atomic control tokens. However, the field has long suffered from the lack of a rigorous, standardized methodology to evaluate the intrinsic representation quality of these latent action spaces. Most prior works evaluate latent actions solely via end-to-end task success rates on downstream policies or qualitative clustering visualizations, deeply confounding representation fidelity with downstream policy capacity.

Current evaluations fail to span heterogeneous robotic embodiments, diverse multi-stage tasks, and multi-granularity action primitives. Moreover, existing embodied manipulation video datasets suffer from imprecise temporal segmentation boundaries and noisy semantic action labels. It has remained largely unclear whether learned latent representations genuinely capture high-level behavioral intent ("what to do") and low-level physical dynamics ("how to do it"), or merely exploit static visual shortcuts and environmental background distractors.

This paper addresses these deficiencies by establishing a decoupled two-tier evaluation framework directly probing semantic separability and physical trajectory regression on a unified multi-embodiment foundation. Core idea: build LARYBench, a benchmark encompassing over 1.2M video clips and 151 action categories across 11 robotic platforms, and evaluate semantic separability and continuous trajectory regression to reveal that general self-supervised visual encoders systematically surpass specialized embodied action models.

Method

Overall Architecture

LARYBench directly probes the latent action space \(\mathcal{Z}\) by decoupling it from policy learning. Given consecutive visual observations \(o_{1:T}\), a latent action model maps visual transitions into latent embeddings \(z \in \mathcal{Z}\). The evaluation framework formalizes two complementary probing pathways: high-level semantic intent decoding \(f_{\text{sem}}: \mathcal{Z} \to \mathcal{C}\) (via multi-layer attentive probes for classification accuracy) and low-level physical control mapping \(f_{\text{dyn}}: \mathcal{Z} \to \mathcal{A}\) (via residual MLP action experts for continuous trajectory regression). The benchmark comprehensively spans 11 representative models across four architectural paradigms: Embodied LAMs, General Semantic Encoders, Generative Pixel Encoders, and General LAMs adapted from frozen vision backbones.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Multi-Embodiment Video & Trajectory Corpus<br/>(1.2M video clips / 11 platforms / 151 action classes)"] --> B["Automated Data Engine<br/>Spatiotemporal segmentation + semantic alignment"]
    B --> C["Latent Action Extraction<br/>(4 paradigms / 11 models latent space Z)"]
    C --> D["Hierarchical Probing & Assessment System"]
    subgraph D["Hierarchical Probing & Assessment System"]
        direction TB
        E["Hierarchical Semantic Probing Protocol<br/>4-layer attentive probe (Top-1 Acc)"]
        F["Physical Execution Mapping Assessment<br/>Residual MLP action expert regression (Normalized MSE)"]
    end
    E --> G["Empirical Insights & Diagnostic Findings<br/>(Latent beats pixel / General encoders outperform embodied LAMs)"]
    F --> G

Key Designs

1. Automated Data Curation Engine: Cross-Embodiment and Multi-Granularity Action Alignment To overcome imprecise temporal boundaries and inconsistent annotations in existing human and robot video datasets, the authors construct an automated curation pipeline driven by a vision-language model with strong spatio-temporal reasoning capabilities (Doubao-1.5-pro-vision API). The engine executes automated temporal segmentation, generates dense video descriptions, verifies video-verb semantic consistency, and undergoes human quality audits. This establishes a unified taxonomy comprising 28 kinematic-level atomic primitives and 145 task-level composite behaviors. The curated corpus aggregates 1.2M+ short clips (over 1,000 hours), 620K image pairs, and 595K continuous robot trajectories across 11 physical morphologies (including Franka 7-DoF arms, bimanual 12-DoF platforms, and 16-DoF AgiBot G1 semi-humanoids) and first-person human interactions.

2. Hierarchical Semantic Probing Protocol: Multi-Granularity Intent Decoding with Dynamic Sampling To quantify whether latent representations preserve granular motion nuances alongside abstract semantic behavior, the protocol introduces multi-granularity evaluations. At the atomic level (Atomic Robot), 28 kinematic primitives are derived from LIBERO demonstrations by thresholding directional translational velocities along the \(z\)-axis and gripper actuations. At the composite level (Composite Human and Composite Robot), 145 task classes capture long-horizon interactions. Because source datasets differ widely in frame rates and execution speeds, naive uniform frame sampling misses critical motion transitions; hence, a Motion-Guided Sampler (MGSampler) dynamically samples 9 informative transition frames. Continuous pairwise latent features are then evaluated using a 4-layer attentive probe with a multi-head optimization strategy.

3. Physical Execution Mapping Assessment: Absolute and Relative Trajectory Regression While semantic probing inspects "what to do", robotic execution requires fine-grained physical grounding of "how to do it". The benchmark builds a lightweight Action Expert consisting of a 2-block residual MLP with a hidden dimension of 4096. Given latent actions derived from frame pairs separated by interval \(s=5\), the expert regresses continuous end-effector action chunks of size \(s\) across single-arm setups (CALVIN, VLABench) and complex bimanual environments (RoboCOIN, AgiBotWorld-Beta). The evaluation disentangles Absolute Regression (predicting global world-frame poses) from Delta Regression (predicting local relative increments), tracking both Normalized MSE and unnormalized physical SE(3) error metrics.

4. Feature-Level General LAM Training: Transferring General Visual Priors to Latent Action Spaces To bridge the gap between pure visual representation and discrete action quantization, the paper proposes General Latent Action Models. Instead of training latent models from raw pixels on small-scale embodied datasets (the paradigm of LAPA and UniVLA), General LAMs freeze high-capacity pretrained vision encoders (such as DINOv2, DINOv3, SigLIP 2, or MAGVIT2) and train the discrete VQ-VAE bottleneck from scratch on large-scale internet video datasets comprising diverse human and non-human motions. Systematic ablations reveal that scaling sequence length to 49 and constraining codebook capacity to 64 completely prevents codebook collapse, ensuring 100% utilization while maintaining physical fidelity.

Key Experimental Results

Main Results

The benchmark systematically compares 11 models across semantic action classification and low-level control regression. General vision foundation models (such as V-JEPA 2.1 and DINOv3), trained without explicit action supervision, consistently outperform domain-specific embodied LAMs by a substantial margin.

Model Paradigm Params (M) Semantic Classification (Avg. Acc↑) CALVIN Absolute (MSE↓) VLABench Absolute (MSE↓) AgiBot Absolute (MSE↓) Delta Regression (Avg. MSE↓)
V-JEPA 2.1 Semantic Encoder 304.68 76.29% 0.18 0.04 0.28 0.50
DINOv3 Semantic Encoder 303.13 72.63% 0.20 0.06 0.24 0.63
Wan2.2 Pixel Encoder 704.69 66.58% 0.38 0.09 0.40 0.84
FLUX.2-dev Pixel Encoder 84.05 45.24% 0.56 0.32 0.63 0.84
LAPA Embodied LAM 343.80 19.13% 0.94 0.90 1.00 0.94
UniVLA Embodied LAM 287.75 18.82% 0.82 0.73 0.97 0.82
villa-X Embodied LAM 238.71 23.85% 0.86 0.71 0.97 0.82
LAPA-DINOv3 General LAM 472.45 62.73% 0.50 0.22 0.84 0.57
LAPA-DINOv2 General LAM 473.69 57.41% 0.55 0.27 0.84 0.60

Ablation Study

Ablations on quantization hyper-parameters within the LAPA-DINOv3 framework on AgiBotWorld-Beta demonstrate the sensitivity of the action bottleneck (default: codebook size = 64, sequence length = 49, latent dimension = 256).

Ablation Dimension Setting Recon Loss↓ Codebook Utilization (%) Composite Human Acc↑ Composite Robot Acc↑ AgiBot Regression MSE↓
Codebook Size 8 0.00808 100.0% 71.31% 64.89% 0.88
64 (default) 0.00751 100.0% 70.15% 64.04% 0.83
256 0.00758 89.5% 69.84% 63.82% 0.85
Sequence Length 16 0.00908 1.6% (collapsed) 69.23% 63.33% 0.79
49 (default) 0.00751 100.0% 70.15% 64.04% 0.83
64 0.00773 79.7% 71.37% 64.70% 0.72
Latent Dimension 32 0.01141 3.1% (collapsed) 60.94% 57.69% 0.87
64 0.00967 100.0% 66.94% 63.07% 0.83
256 (default) 0.00751 100.0% 70.15% 64.04% 0.83
512 0.00732 1.6% (collapsed) 71.25% 64.97% 0.80

Key Findings

  • General Visual Priors Inherently Encode Action Dynamics: Without any robot trajectory supervision, self-supervised semantic encoders (V-JEPA 2.1, DINOv3) achieve top semantic classification accuracy (76.29%) and lowest absolute trajectory error (0.18-0.19 MSE), whereas specialized embodied LAMs suffer severe degradation (18.82%-23.85% accuracy and ~0.90 MSE).
  • Latent-Based Spaces Outperform Pixel Reconstruction Spaces: Feature-level self-supervised representations decisively outperform pixel-level video generative encoders (Wan2.2, FLUX), confirming that embodied physical control aligns naturally with high-level spatio-temporal abstractions rather than microscopic pixel synthesis.
  • Delta Formulations Narrow Paradigm Gaps: Under Delta Regression, General LAMs trained on inter-frame changes close the performance gap with pure semantic encoders (trailing by only 0.07 MSE versus 0.42 MSE in Absolute Regression), demonstrating that pairwise temporal modeling naturally captures differential physical displacement.
  • Temporal Robustness Over Expanded Horizons: When the sampling stride expands from 5 to 30 on VLABench, purely spatial models like FLUX collapse catastrophically (MSE increases from 0.04 to 0.62), whereas LAM architectures maintain consistent stability (MSE varies by <0.08), confirming that latent action representations preserve coherent dynamic trajectories.

Highlights & Insights

  • Decoupled Evaluation Paradigm: Establishes a standardized framework separating the intrinsic representational quality of latent actions from the stochastic optimization of downstream reinforcement or imitation learning policies.
  • Successful Transfer of Web-Scale Visual Priors: Proves that grafting frozen visual foundation models onto VQ-VAE architectures yields General LAMs that substantially outperform natively trained embodied models on diverse activities.
  • Quantization Stability Frontier: Identifies specific codebook and sequence bottlenecks where quantization collapses (such as 1.6% codebook utilization under short sequences or intermediate dimensions), providing practical architectural guidelines for future discrete latent action models.

Limitations & Future Work

  • Omission of Dexterous Articulation: Due to severe occlusion and the ill-posed nature of estimating multi-joint finger kinematics from monocular RGB video, hand joint states in datasets like RoboCOIN were masked, leaving micro-manipulation and tactile contact dynamics unassessed.
  • Embodiment Domain Shift in General LAMs: While General LAMs trained on web videos excel at human actions, their performance experiences a mild drop on robotic embodiments, highlighting the need for cross-embodiment domain alignment techniques.
  • Discrete vs. Continuous Trade-Off: Although discrete action tokens seamlessly integrate into autoregressive LLM backbones, continuous embeddings retain higher geometric fidelity; hybrid representations bridging both regimes warrant further investigation.
  • vs LAPA / Moto / LAPO: Prior unsupervised LAMs trained discrete codebooks directly on limited robot or human video; LARYBench demonstrates that small-scale native training suffers from severe representational collapse, whereas leveraging web-scale pretrained visual backbones yields far superior generalization.
  • vs UniVLA / villa-X: Existing VLA frameworks enforce low-level action constraints early in the representation learning process; this benchmark shows that premature task-specific conditioning restricts visual flexibility compared to unconstrained self-supervised representations.
  • vs V-JEPA 2 / DINOv3: Evaluates general video and image foundation models under rigorous embodied physical benchmarks for the first time, establishing their direct applicability as universal control representations for physical AI.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Decouples policy learning to create the first rigorous latent action benchmark across 151 action classes and 11 robotic embodiments.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 11 models, 4 paradigms, semantic/control tasks, multi-stride temporal horizons, and quantization ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear conceptual hierarchy, robust analytical framework, and transparent error decomposition.
  • Value: ⭐⭐⭐⭐⭐ Provides critical architectural guidance for foundational VLA development, advocating a strategic shift toward general visual representation alignment.