VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model

Conference: ECCV2026
Paper: ECCV Paper
Authors: Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, Zhibo Chen
Code: https://github.com/ginwind/VLA-JEPA
Area: Robotics & Embodied AI
Keywords: vision-language-action models, latent actions, latent world models, joint-embedding prediction, human video pretraining

TL;DR

VLA-JEPA generates latent actions from a vision-language model that sees only initial images and instructions, learns manipulation semantics by predicting future V-JEPA2 states, and generates robot actions through flow matching; Tables 1 and 3 report success rates of 97.2% on LIBERO and 78.1% on LIBERO-Plus.

Background & Motivation

Robot action trajectories are expensive, whereas human manipulation videos are abundant. A common approach therefore extracts latent actions from adjacent video frames, trains a vision-language-action model (VLA) to predict them, and finally maps them to robot controls. However, visual change does not necessarily describe an executable operation: camera motion, tablecloth texture, and background objects can be more prominent than hand-object interaction. Objectives centered on pixel reconstruction or frame differences may teach what changed visually without identifying the operation that changes the state.

The paper also questions latent-action extraction that receives both current and future frames. An extractor can compress the future image into its latent variable, obtaining a low prediction loss without learning to infer transitions from present conditions. The authors describe this shortcut as information leakage. This is a mechanistic hypothesis about training behavior, not a causal conclusion established solely by downstream success rates.

VLA-JEPA changes the route by which future information enters training instead of designing another discrete action codebook. Future frames supply latent-state supervision, action representations come from current visual and language conditions, and a world model tests whether these representations support temporal prediction. Core Idea: train latent actions to explain future state transitions, rather than to obtain a low loss by encoding a future image that the extractor has already seen.

Method

Overall Architecture

The inputs are multi-view RGB images at the initial time step and a language instruction; the output is a continuous robot action trajectory. Figure 2 first learns state prediction from human videos and then jointly learns state prediction and action generation on robot data. Experimental pretraining also includes action-labeled DROID data, so the entire pretraining phase is not action-free.

The model constructs world states and latent actions, predicts subsequent states through a time-causal world model, and additionally uses the latent actions for conditional action decoding during robot training. The world model primarily supplies a learning constraint. Deployment directly generates actions through the action head rather than generating a goal image and performing energy-minimization planning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Initial multi-view images<br/>Language instruction"] --> State["World States and Latent Actions"]
    State --> World["Time-Causal World Model"]
    Target["Encoded video states<br/>Training targets and history"] --> World
    State --> Action["Conditional Action Decoding"]
    World --> Align["Next latent state<br/>Alignment loss"]
    Action --> Output["Continuous action trajectory<br/>Robot action supervision"]

Key Designs

1. World States and Latent Actions: separate the environment from its anticipated transition

A V-JEPA2 video encoder supplies world states, with representations from different views at a given time aggregated by concatenation. The resulting unified world state is a multi-view feature representation, not an explicit 3D map or an identified set of physical state variables. A video-pretrained encoder moves supervision away from pixel details such as color and texture toward more abstract temporal information. Whether the representation fully removes nuisance variation remains an empirical question.

The other branch uses Qwen3-VL; the backbone-controlled baseline is explicitly based on Qwen3-VL-2B. The VLM receives initial images, the instruction, and learnable <latent_i> queries. Each query represents a time-step transition and is replicated \(K\) times to adjust representation capacity. All latent actions are generated from the initial conditions without appending later ground-truth images to the VLM. The model must therefore infer possible operations using its vision-language priors instead of copying the future.

These latent actions are not robot end-effector poses. Human videos contain no robot control labels, so the learned variables are conditioning representations useful for predicting state changes. Action magnitudes and control correspondences in the robot coordinate system must be learned from labeled data in the next stage. Observing human videos is not equivalent to directly acquiring robot dynamics.

2. Time-Causal World Model: allow supervised history without looking ahead

An autoregressive Transformer world model receives sequences of world states and latent actions. State and latent-action tokens within a time step use full bidirectional attention. Across time steps, a causal mask permits access only to the current and preceding steps. A future-state prediction can therefore use preceding states but cannot attend to states beyond its prediction boundary.

Training uses teacher forcing: the world model receives encoded ground-truth historical states rather than relying entirely on its own rolled-out predictions. Two boundaries must be distinguished. The VLM latent-action branch always sees only initial images; the world model can receive ground-truth states that occur after the initial observation but precede the current prediction target. Saying that no module ever receives later frames during training would misrepresent Section 3.2.

A JEPA alignment objective compares predicted states with V-JEPA2 target states and jointly optimizes the world model and VLM. The cached extraction of Equation (4) does not preserve the distance operator, so this note does not invent a mean-squared-error or cosine loss. The main text also does not sufficiently specify encoder freezing, stop-gradient, or teacher-update details. Those implementation choices cannot be inferred merely from the JEPA label.

3. Conditional Action Decoding: connect video transitions to a control distribution

Learnable <action> queries are appended after the latent-action queries. Through causal attention, they aggregate images, instructions, and preceding latent actions into the action-head condition \(z_a\). Rather than hard-decoding each latent token into a discrete control command, the model conditions continuous trajectory generation on transition semantics, allowing it to use representations learned across video domains.

The action head uses conditional flow matching. During training, a ground-truth action trajectory is linearly interpolated with Gaussian noise, and the head predicts the velocity from noise toward that trajectory. During inference, the velocity field is integrated with the conditioning representation held fixed to generate an action sequence. The following standard expressions are reconstructed from the prose accompanying Equations (6) and (7); their cached mathematical layout is damaged, so the PDF remains authoritative for the original notation:

\[ a_t=(1-t)\epsilon+t\,a_{0:H},\qquad \epsilon\sim\mathcal{N}(0,I),\quad t\in[0,1]. \]

\[ \mathcal{L}_{\mathrm{FM}}=\mathbb{E}_{a_{0:H},\epsilon,t}\left[\left\|v_\theta(a_t,t\mid z_a)-(a_{0:H}-\epsilon)\right\|_2^2\right]. \]

Here \(t\) is the flow interpolation time, not a video-frame index; \(H\) denotes the action prediction horizon, and \(a_{0:H}\) follows the paper's trajectory notation. The cache does not reliably preserve the interpolation-time sampling distribution and does not specify deployment integration steps, so these hyperparameters are not supplied. Robot supervision connects latent representations to actual controls, while state prediction continues to constrain their temporal semantics.

A Worked Example

Consider an illustrative instruction to pick up an object from a table and place it at a specified location. Initial multi-view images establish the visible object and gripper state, while language specifies the goal. Without seeing the actual subsequent frames, latent-action queries represent anticipated transitions. During training, the world model uses historical states from the video to predict the next latent state and aligns it with the target encoding.

For human videos, this already provides a learning signal without requiring robot end-effector coordinates. For robot demonstrations, the appended action queries also condition the flow-matching head to learn the actual trajectory. Deployment uses the current input and instruction to generate actions directly. This example explains module interaction and is not an additional experimental trial reported by the paper.

Loss & Training

Action-free human videos use the world-model alignment objective, whereas action-labeled robot data use the joint objective in Equation (8). Following its textual description:

\[ \mathcal{L}_{\mathrm{robot}}=\mathcal{L}_{\mathrm{FM}}+\beta\mathcal{L}_{\mathrm{WM}}. \]

\(\beta\) weights the state-prediction term. Its numerical value, \(K\), the learning rate, batch size, and complete training-step configuration are absent from the cached main text. The two stages mean pretraining and downstream fine-tuning, not that fine-tuning updates only an action head: the method explicitly describes end-to-end joint optimization of both objectives.

Pretraining uses Something-Something-V2 human hand-object interaction videos and DROID robot data, followed by environment-specific fine-tuning. The paper lists LIBERO, BridgeV2, and self-collected real-world data, and reports using 8 NVIDIA A100 GPUs for all experiments. The supplementary material is not included in this cache, preventing verification of finer training-budget details.

Key Experimental Results

Main Results

The table retains aggregate success rates from Tables 1 through 3, in %. These columns are not the same metric on a shared task set. LIBERO evaluates 50 episodes per task and 500 per suite; LIBERO-Plus aggregates seven perturbation dimensions, and SimplerEnv uses the visual matching setting.

Method	LIBERO Avg, Table 1	LIBERO-Plus Total, Table 3	Google Robot Avg, Table 2	WidowX Avg, Table 2
UniVLA	95.2	42.9	Not reported	42.7
OpenVLA-OFT	97.1	69.6	Not reported	41.8
QwenVLA	96.0	69.6	Not reported	Not reported
VLA-JEPA	97.2	78.1	65.2	57.3
VLA-JEPA, without human video	96.1	62.9	78.4	57.3

The LIBERO result of 97.2% exceeds OpenVLA-OFT by 0.1 percentage points and QwenVLA by 1.2 percentage points; LIBERO-10 reaches 95.8%. The larger gain is on LIBERO-Plus: 8.5 percentage points above OpenVLA-OFT's 69.6%, although this does not imply winning every perturbation category.

Real-world experiments use a Franka Research 3 with a Robotiq 2F-85 gripper, 100 demonstrations covering 3 pick-and-place tasks, and evaluations on in-distribution tasks, unseen tasks, and changed object layouts. The text reports the best performance for in-distribution and layout-shift settings and second-best for unseen tasks. The bar values in Figure 4a are not reliably extracted, so per-setting success rates are not invented here.

Ablation Study

The following success rates, in %, come from Tables 1 and 3. The QwenVLA family controls the backbone, but plain QwenVLA uses only robot data. Its difference from the complete model therefore cannot be attributed entirely to the JEPA loss.

Config	LIBERO Avg	LIBERO-Plus Total	Supervision or data difference
QwenVLA	96.0	69.6	Robot action supervision
QwenVLA + ViT	96.4	72.6	Continuous future visual features
QwenVLA + VQGAN	93.7	50.5	Discrete future tokens
QwenVLA + LAM	95.9	Not reported	Latent-action-model encodings
VLA-JEPA, without human video	96.1	62.9	Human video pretraining removed
VLA-JEPA	97.2	78.1	Full setting

Table 4 separately studies the prediction horizon by directly fine-tuning on LIBERO with other hyperparameters held constant and an action horizon of 8. Here \(T\) denotes the future video prediction horizon and must not be converted into seconds without a sampling interval. This is not the full pretrained setting in the preceding table.

Video prediction horizon \(T\)	LIBERO Avg success (%)	LIBERO-10 success (%)
4	94.8	89.0
8	96.1	94.0
16	95.5	92.2

Key Findings

• Human videos primarily help perturbation robustness: LIBERO-Plus rises from 62.9% to 78.1%, a 15.2-percentage-point gain, while LIBERO only rises from 96.1% to 97.2%. Google Robot instead drops from 78.4% to 65.2%, showing that these data do not universally improve control.
• Figure 1 reports 79.5% on LIBERO-Plus, inconsistent with Table 3's 78.1%; this note uses Table 3. Its entries establish wins on Robot, Language, Background, and Layout, four categories. QwenVLA's Light result is 96.8%, above VLA-JEPA's 95.6%, so the text's claim of winning five of seven categories is not repeated.
• WidowX's 57.3% matches LAPA trained on successful in-simulation demonstrations, but differing data conditions prevent an equal-generalization interpretation. Avg/Total entries are retained as reported rather than forced into an unverified equally weighted average of the displayed subtask values.

Highlights & Insights

• Placing future information on the supervision side while restricting latent-action inputs constrains shortcuts more directly than reducing latent capacity alone. The important property is that representations must support prediction without already seeing the answer, not simply that the encoder is stronger.
• Keeping video learning objectives during robot fine-tuning prevents a complete disconnect between pretrained action representations and final control. Flow matching handles continuous trajectories, while latent prediction provides temporal constraints; the objectives serve distinct roles.
• Human videos appear more useful for stabilizing existing skills under visual disturbances than for replacing action labels. This distinction informs data allocation because actual control correspondences still require high-quality robot demonstrations.

Limitations & Future Work

• The authors acknowledge that human videos lack actionable trajectories and cannot directly teach robot action dynamics. Unseen-task generalization is also weaker than \(\pi_{0.5}\) in real deployment, despite more stable trajectories and fewer safety-boundary violations. This is not a formal safety guarantee.
• This note's assessment: attention visualizations and downstream comparisons support greater focus on manipulation, but do not independently prove recovery of true controllable factors. Stronger isolated ablations should control data volume, target encoders, and future-information pathways.
• The figure/table discrepancies and textual inconsistencies above coexist with insufficient variance estimates, confidence intervals, and per-setting real-world values for fine-grained comparisons. A 0.1-percentage-point LIBERO advantage should not be presented as a statistically established lead.
• The cached equations are partially damaged, and implementation details are deferred to unavailable supplementary material. Reproduction should verify the original losses, teacher updates, video sampling intervals, data mixing ratios, and action integration settings instead of treating missing values as standard defaults.

Related Work & Insights

• vs LAPA / UniVLA: These methods learn latent actions from frame transitions and use them to guide policies. VLA-JEPA instead generates latent actions from initial conditions and trains them through future-state prediction, emphasizing control over how future information enters learning.
• vs V-JEPA2-AC: VLA-JEPA uses VLM language and world knowledge and directly predicts actions without a goal image or energy-minimization planning. It still needs action-labeled robot data to learn the control mapping.
• vs QwenVLA + ViT / + VQGAN: Continuous visual-feature prediction also helps, whereas discrete future tokens perform worse in these tables. The baselines should therefore not all be described as pixel reconstruction; the more useful question is which target representations retain manipulation-relevant transitions.

Rating

• Novelty: 4/5. The connection between input-restricted latent actions, JEPA state supervision, and control training is clear, although the individual components have precedents.
• Experimental Thoroughness: 3/5. Three simulation evaluation settings and real robots provide breadth, but data confounds, numerical inconsistencies, and limited statistics weaken conclusions.
• Writing Quality: 3/5. The motivation is clear, but readers must resolve the module boundaries of leakage prevention and several result statements.
• Value: 4/5. The paper provides a concrete interface for learning VLA representations from human videos and useful evidence that robustness gains do not imply universal control gains.