IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZGC-EmbodyAI/IntentVLA
Area: Robotics & Embodied AI
Keywords: Robot Manipulation, Vision-Language-Action Models, Intent Modeling, Observation Aliasing, Flow Matching
TL;DR¶
Addressing the issue where current-frame observation aliasing under partial observability causes chunked policies to resample conflicting intents across adjacent steps, IntentVLA extracts compact camera and register tokens from recent visual history via a frozen VGGT encoder, injects them into the Qwen3-VL backbone via gated cross-attention along with a global intent summary token, and conditions a DiT flow-matching action head, substantially improving rollout stability and task success across AliasBench, SimplerEnv, LIBERO, and real-world manipulation.
Background & Motivation¶
In large-scale human-collected robot manipulation datasets, trajectories naturally exhibit rich multimodality: even under identical language instructions, different human demonstrators or the same demonstrator across different episodes may pursue alternative feasible paths, task phases, or execution strategies. Crucially, while demonstrations are multimodal across episodes, they remain locally committed within any given episodeโonce a demonstrator initiates a specific sub-task or path, consecutive action chunks stay strictly consistent with that choice. Most existing vision-language-action (VLA) models, such as OpenVLA and \(\pi 0\), infer action chunks conditioned solely on the current visual frame and language instruction.
Under real-world partial observability, relying exclusively on single-frame conditioning frequently induces short-horizon observation aliasing: two distinct execution phases (e.g., transporting bread toward a skillet for cooking vs. returning cooked bread from the skillet to a plate) can yield near-identical visual observations at the gripper level. Because frame-only policies cannot observe the episode-level commitment established in earlier steps, repeated chunk generation during receding-horizon execution often samples contradictory intents across adjacent decision steps, triggering severe inter-chunk conflicts, physical oscillation, and catastrophic rollout failures.
The objective of this work is not to eliminate behavioral multimodality, but to condition action chunk generation on recent temporal context to preserve the commitment already active in the current episode. Core idea: extend VLA action generation to condition on recent visual history, leveraging a frozen geometric visual foundation model to extract compact intent tokens and fusing them with the current visual-language context via gated cross-attention to anchor within-episode intent commitments.
Method¶
Overall Architecture¶
The overall architecture of IntentVLA decouples current multimodal understanding from recent visual history modeling across three coordinated stages: current vision-language context encoding, short-horizon intent extraction, and intent-conditioned action chunk generation. The system takes as input the current observation image \(o_t\), language instruction \(\ell\), and a finite visual history window \(h^K_t = o_{t-K:t-1}\) from the head camera; it outputs a future continuous action chunk \(\tau_t \in \mathbb{R}^{H \times d_a}\) over horizon \(H\). The current frame and instruction are encoded into visual-language tokens using a Qwen3-VL 4B backbone. In parallel, the history frames are processed by a frozen VGGT-1B encoder, retaining only camera and register tokens that capture 3D viewpoint dynamics and global inter-frame relations. These history tokens are fused into current features via gated cross-attention, accompanied by a pooled global intent-evidence token, forming a condition context that drives a Diffusion Transformer (DiT) flow-matching head.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
subgraph Input["Multimodal Inputs"]
direction TB
IH["Recent Visual History<br/>Ot-K:t-1 (Head Camera)"]
IC["Current Observation Ot<br/>+ Instruction โ"]
end
IH --> B["Compact History Intent Encoding<br/>Frozen VGGT extracts camera & register tokens"]
IC --> Q["Current VL Representation<br/>Qwen3-VL extracts current tokens Ft"]
B --> M["Gated Cross-Attention Fusion & Intent Summary<br/>Gated residual attention injection + compact intent token"]
Q --> M
M --> D["Flow-Matching Action Chunk Generation<br/>DiT predicts velocity field & integrates via Euler solver"]
D --> OUT["Output Action Chunk<br/>ฯt โ โ^(Hรda)"]
Key Designs¶
1. Compact History Intent Encoding: Geometric Token Selection Over Raw Stacking
Feeding multi-frame visual history directly into a high-capacity VLM as extra context tokens leads to severe GPU memory consumptionโoften triggering out-of-memory (OOM) errors during trainingโand risks diluting attention away from fine-grained manipulation targets. To circumvent this, IntentVLA employs an independent frozen VGGT-1B encoder \(g_\phi\) operating exclusively on recent head-camera frames. For each history frame, rather than retaining dense patch tokens, the model extracts only 1 camera token and 4 register tokens:
The camera token captures ego-motion and relative viewpoint shifts induced by robot arm movement, while the register tokens summarize cross-frame temporal evolution and global scene geometry. These features are projected into the action model's hidden dimension via learned linear projections and layer normalization to yield a fine-grained token sequence \(\tilde{U}_t = \text{LN}(W_h U_t)\) and a compact summary vector \(e_t = W_e \bar{e}_t\). This selective compression reduces peak forward GPU memory by 5.55 GiB compared to raw 16-frame inputs while retaining critical geometric intent cues.
2. Gated Cross-Attention Fusion & Intent Summary: Smooth Conditioning without Semantic Degradation
To ensure temporal intent evidence informs action generation without corrupting the semantic reasoning priors encoded in the pre-trained VLM, IntentVLA designs a gated cross-attention mechanism. Taking the current vision-language tokens \(F_t \in \mathbb{R}^{N \times d}\) from Qwen3-VL as Query and the projected history tokens \(\tilde{U}_t\) as Key and Value:
where \(\alpha\) is a learnable scalar parameter and \(\sigma(\cdot)\) is the Sigmoid gate function, initialized near zero to safeguard training stability. Furthermore, the compact history summary \(e_t\) is reshaped into a single token \(e^{\text{tok}}_t \in \mathbb{R}^{1 \times d}\) and appended to the fused sequence, yielding the final intent context \(C_t = [F'_t; e^{\text{tok}}_t]\). This dual formulation provides both fine-grained token-level cross-attention and a unified global intent anchor for subsequent downstream action sampling.
3. Flow-Matching Action Chunk Generation: Continuous Velocity Field Optimization
Given the conditioning context \(C_t\), the policy generates continuous action trajectories using a Diffusion Transformer (DiT) flow-matching action head. Compared to discrete tokenization, flow matching provides smooth continuous dynamics well suited for receding-horizon control. During training, an interpolated chunk \(X_s = (1 - s)\epsilon + s\tau_t\) is constructed between ground-truth trajectory \(\tau_t\) and Gaussian noise \(\epsilon \sim \mathcal{N}(0, I)\) at flow time \(s \in [0, 1]\), optimizing the velocity field \(\hat{V}_\theta(X_s, s \mid C_t)\):
At test time, \(C_t\) is computed once per decision step, and action chunks are generated by integrating the conditional velocity field from pure Gaussian noise using an Euler solver.
A Worked Example¶
Consider the "Cook Bread and Plate It" task in AliasBench, where the robot must move bread into a skillet to cook and subsequently transfer it back to a plate. When the gripper holds the bread mid-air above the workspace, the single-frame visual appearance is identical across both phases, leaving a frame-conditioned policy unable to determine whether the target is the skillet or the plate. In IntentVLA, the history window \(h^K_t\) captures the preceding 16 frames in which the gripper lifted the bread from the skillet. The frozen VGGT extracts this upward motion through camera and register tokens, which gated cross-attention fuses into the active context. Anchored by the resulting intent representation, the DiT action head generates chunks consistently driving the gripper toward the plate, eliminating the inter-chunk hesitation and mode switching observed in single-frame baselines.
Key Experimental Results¶
Main Results¶
The authors evaluate IntentVLA on AliasBenchโa 12-task benchmark spanning four distinct ambiguity familiesโtrained under a standardized budget of 30K steps on 16 NVIDIA H100 GPUs. Evaluations are also conducted across standard manipulation benchmarks including SimplerEnv (WidowX robot) and LIBERO (4 suites, with special focus on LIBERO-Long).
| Benchmark / Suite | Metric | IntentVLA (Ours) | Qwen3-VL-GR00T (Frame-only) | MemoryVLA | Relative Gain |
|---|---|---|---|---|---|
| AliasBench (Back-and-Forth) | Success Rate (%) | 49.3 | 6.0 | 13.3 | +43.3% vs. GR00T |
| AliasBench (Crossing-Path) | Success Rate (%) | 74.7 | 15.7 | 22.7 | +59.0% vs. GR00T |
| AliasBench (Bimanual) | Success Rate (%) | 17.0 | 5.5 | 4.0 | +11.5% vs. GR00T |
| AliasBench (Multi-Goal) | Success Rate (%) | 31.3 | 8.7 | 16.7 | +22.6% vs. GR00T |
| AliasBench Average | Average Success (%) | 45.8 | 9.0 | 14.9 | +36.8% vs. GR00T |
| SimplerEnv (WidowX 4 tasks) | Average Success (%) | 72.9 | 65.3 | 71.9 | +7.6% vs. GR00T |
| LIBERO-Long | Success Rate (%) | 97.4 | 92.0 | - | +5.4% vs. GR00T |
| LIBERO-All (4 suites avg) | Average Success (%) | 98.6 | 96.5 | - | +2.1% vs. GR00T |
Ablation Study¶
Ablations on SimplerEnv validate the contributions of the history encoder, temporal window, cross-attention fusion, and compact intent token:
| Variant / Configuration | Stack Block | Put Carrot | Put Spoon | Put Eggplant | Average Success (%) | Note |
|---|---|---|---|---|---|---|
| IntentVLA Full Model | 54.2 | 66.7 | 70.8 | 100.0 | 72.9 | Full temporal fusion + intent token |
| History fusion only (no intent token) | 49.0 | 65.6 | 67.7 | 95.8 | 69.5 | Removes compact summary token \(e^{\text{tok}}_t\) |
| VGGT on current frame only | 30.2 | 61.5 | 72.5 | 94.8 | 64.8 | Confirms single-frame VGGT yields no gain |
| Frame-only Qwen3-VL-GR00T | 18.8 | 59.4 | 83.0 | 100.0 | 65.3 | Baseline with single-frame input |
Additional ablations on AliasBench reveal: 1. Window Length \(K\): Performance peaks at \(K=16\) (45.8%), compared to 39.9% at \(K=8\) and 43.7% at \(K=24\), confirming that an optimal window covers the immediate disambiguating phase transition without introducing stale historical context. 2. Encoder Choice: Replacing VGGT with the video self-supervised encoder V-JEPA 2 drops average success from 45.8% to 38.6%, highlighting the importance of VGGT's geometric pre-training and alternating attention for manipulation intent.
Key Findings¶
- Reduced Inter-Chunk Inconsistency (ICC-L2): Measuring squared L2 action discrepancy on overlapping timesteps during receding-horizon replanning, IntentVLA achieves an average ICC-L2 of 0.181 versus 0.219 for the strongest feasible history baseline (a 17.6% relative reduction), while reducing 90th-percentile tail inconsistency by 21.7%.
- Task Sensitivity: Large gains occur on geometrically sensitive and transition-heavy tasks like Stack Green Block (+35.4 points). Conversely, on Put Spoon on Towel, performance dips slightly (83.0% \(\to\) 70.8%), indicating that in strictly static, non-aliased tasks with small objects, visual history can mildly dilute spatial focus.
- Dual-Arm Real-World Validation: On a dual-arm Franka setup performing multi-snack cleanup (50 trials per model), IntentVLA achieves an expected 1.86 snacks packed per rollout, outperforming Qwen3-VL-GR00T (1.20), \(\pi 0.5\) (1.44), and MemoryVLA (1.54).
Highlights & Insights¶
- Geometric Token Sparsification: Restricting history representation to 1 camera and 4 register tokens per frame extracts critical 3D dynamic cues without memory explosion, providing a practical blueprint for history conditioning in resource-constrained VLAs.
- AliasBench Ambiguity Benchmark: Systematically categorizes observation aliasing into back-and-forth, crossing-path, bimanual, and multi-goal paradigms, exposing a fundamental failure mode in contemporary frame-conditioned policies.
- Non-Destructive Gated Conditioning: The gated residual cross-attention mechanism acts as a minimally intrusive adapter, preserving base VLM reasoning while providing the temporal steering necessary for smooth trajectory rollout.
Limitations & Future Work¶
- Local Scope: The short-horizon visual history window covers approximately 0.5 seconds (\(K=16\) at 30 FPS), meaning it cannot resolve ambiguities requiring long-term episodic recall or recovery from severe out-of-distribution drift.
- Single Camera History: History encoding is currently restricted to head-camera observations, omitting wrist-camera feedback that could aid fine-grained local alignment and insertion.
- Future Integration: Combining short-horizon geometric intent conditioning with a hierarchical high-level semantic planner represents a promising avenue for unified long-horizon physical intelligence.
Related Work & Insights¶
- vs. MemoryVLA: MemoryVLA relies on a growing memory bank with vector-averaging consolidation; in aliased settings where states are visually near-identical but semantically opposite, vector averaging collapses distinct intents, leading to a much lower 14.9% success rate on AliasBench.
- vs. Raw History-as-Context: Feeding 8 or 16 raw history frames directly into the VLM causes severe OOM; uniformly subsampling 4 frames achieves only 28.1% success, showing that naive image concatenation cannot match structured geometric token extraction.
- vs. DIAL & Latent Intent VLAs: Prior intent-based VLAs infer latent intentions from single frames using forward world models; IntentVLA demonstrates that ground-truth physical history provides a significantly more reliable anchor than hallucinated latents under severe perceptual aliasing.
Rating¶
- Novelty: โญโญโญโญโ Pinpoints the critical problem of within-episode intent aliasing and solves it with an elegant geometric token compression scheme.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage across the custom AliasBench, SimplerEnv, LIBERO, RoboCasa, and dual-arm real-robot rollouts.
- Writing Quality: โญโญโญโญโญ Rigorous problem formulation, precise mathematical definitions, and lucid empirical analysis.
- Value: โญโญโญโญโญ Essential guidance for embodied AI practitioners seeking to stabilize chunked receding-horizon manipulation policies.