EvoVLA: Self-Evolving Vision-Language-Action Model

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Official ID: 3353
Citations: 10 (provided in the task, recorded on 2026-09-17)
Area: Robotics & Embodied AI
Keywords: vision-language-action models, stage hallucination, stage-aligned rewards, relative-pose exploration, long-horizon memory

TL;DR

EvoVLA combines stage-aligned rewards, pose-based exploration, and selective memory on OpenVLA-OFT to reduce states that look complete but are physically incomplete, improving Discoverse-L average success from 59.0% to 69.2% and reducing hallucination rate from 38.5% to 14.8%.

Background & Motivation

Vision-language-action (VLA) models turn images and instructions into robot actions, but long-horizon manipulation cannot rely exclusively on trajectories encountered during imitation learning. A convenient reinforcement learning recipe freezes a vision-language model (VLM) and rewards similarity between the current image and a target description such as a completed grasp. Similar appearance, however, does not imply the same physical state: a gripper hovering above an object may also receive a high grasp-completion score.

The paper calls this disagreement between reward evaluation and physical stage completion stage hallucination. It is not merely an incorrect language response: a policy can exploit evaluator blind spots during closed-loop interaction and repeatedly reach high-scoring states without making progress. Ordinary positive-negative descriptions distinguish completion from noncompletion without covering subtle near misses. Pixel-based curiosity may reward lighting or camera changes, while compressed histories can obscure which stages have already been completed.

The authors therefore retain the policy backbone and jointly revise reward evaluation, exploration, and history use. Core Idea: identify deceptive near misses with counterfactual hard negatives, direct exploration through relative gripper-object poses, and use stage-relevant memory to gate learning-progress rewards so that high returns better track physical progress.

Method

Overall Architecture

Inputs are multi-view observations, task language, and interaction history; outputs are robot actions. The policy follows OpenVLA-OFT, using SigLIP and DINOv2 visual components with Llama-2 7B. A separate frozen CLIP ViT-B/16 provides image-text evaluation rather than serving as the policy updated by PPO. Here, self-evolution primarily means further training the policy and auxiliary modules from interaction feedback, not modifying the architecture online.

Data preparation collects 50 scripted simulation trajectories per task, stores them in RLDS, applies task-aligned normalization, and runs video-driven VLM prompting to produce a unified stage dictionary. Each stage contains a completion description, a mutually exclusive negative, and a counterfactual hard negative. Gemini 2.5 Pro is the backend used in the paper, which also discusses open-source alternatives. Simulation and corresponding real tasks share stage semantics; an unseen task receives a new dictionary generated from 50 teleoperated demonstrations.

During interaction, SAR supplies stage-progress rewards, POE generates curiosity and base learning progress from relative-pose prediction, and Long-Horizon Memory retrieves history and gates the latter signal. These terms combine with sparse task rewards to form PPO returns, after which the updated policy collects further interaction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Observations, instruction<br/>and stage dictionary"] --> Policy["OpenVLA-OFT policy<br/>and environment interaction"]
    Policy --> SAR["SAR"]
    Policy --> POE["POE"]
    POE --> Memory["Long-Horizon Memory"]
    Policy -->|Policy latents and history| Memory
    SAR --> Update["Reward combination<br/>and PPO update"]
    POE -->|Curiosity| Update
    Memory -->|Gated progress| Update
    Update --> Policy

Key Designs

1. SAR: include nearly successful failures in reward comparisons

Stage-Aligned Reward (SAR) compares an image against both ordinary and hard negatives, rather than asking only whether it resembles success. An ordinary negative can describe noncompletion; a hard negative describes an easily confused state such as grasping the wrong object. Generation prioritizes spatial and contact predicates over appearance attributes such as color, keeping evaluation focused on interaction conditions.

The evaluator computes CLIP similarities to the three descriptions, subtracts the larger negative similarity from the positive similarity, and applies temperature scaling and a sigmoid. Hovering or grasping the wrong object should therefore lose credit when it matches a hard negative, even if it also resembles the positive description. CLIP remains frozen: the mechanism changes the preferences conveyed through PPO returns rather than directly training a new visual contact detector.

Single-frame scores can still fluctuate. SAR applies an exponential moving average and uses the temporal difference of the smoothed active-stage score as its stage reward. It rewards improvement rather than indefinitely occupying a high-scoring image. Stage transitions also require sustained progress within a sliding window of length 8. The main paper does not fully specify the window threshold and implementation, so this should not be reduced to advancing whenever one frame scores above 0.7.

This mechanism depends on a correct stage dictionary. The authors describe automated validation and refer semantic perturbation and alternative-backend tests to the supplement. The available cache contains only the main paper and references, so no numerical results from those tests are supplied here. Counterfactual descriptions constrain visual rewards but do not guarantee that CLIP reliably recognizes every contact relation.

2. POE: focus curiosity on gripper-object geometry

Pose-Based Object Exploration (POE) encodes the object's transformation relative to the end effector as a six-dimensional state: three translation coordinates and a three-dimensional axis-angle rotation. A forward model predicts the next relative pose from the current pose and action; an inverse model predicts the action from consecutive poses. Both are lightweight MLPs with two layers of 256 units each.

Curiosity comes from forward prediction error, encouraging exploration of unfamiliar gripper-object configurations. Separately, the method smooths the forward loss and takes the positive part of its decrease between successive times as base learning progress. Unlike rewarding large errors, this term asks whether the model is learning new interaction dynamics. Curiosity enters the reward mixture directly, whereas base progress passes through memory gating; the two signals should not be conflated.

Because the state represents relative geometry, lighting changes alone should not create a novel pose, reducing irrelevant pixel-driven exploration. During real-robot training, relative poses come from AprilTag. This also qualifies the self-supervision claim: the method still needs usable object poses, and six-dimensional pose alone does not encode contact forces, stable placement, or task ordering. It cannot serve as an independent success criterion.

3. Long-Horizon Memory: preserve stage-relevant history and gate learning progress

Long-Horizon Memory pools the policy language backbone's hidden states into a current representation, adds temporal positional encodings to stored history, and retrieves the Top-K relevant items with attention. Retrieved items are prepended as independent context tokens instead of averaging adjacent history. This preserves details such as which block was previously grasped. The main paper does not specify reproducible memory-capacity or K values, so none are assumed here.

A learned gate also fuses the current representation with the weighted retrieved history. Beyond contextual fusion, this gate multiplies the base learning-progress reward from POE, making that reward depend on historical context. The intended effect is to suppress unstable progress signals associated with repeated failures or oscillation. This is neither a new classifier for every SAR image-text score nor a replacement for sparse task-success labels.

The fused representation is written back to selected memory items, and capacity management evicts entries using utility based on usage frequency, recency, and redundancy. The paper describes stop-gradient and separate optimizers to discourage reward exploitation, but these measures do not theoretically guarantee that the policy cannot exploit prediction errors. Memory and POE are complementary: one preserves temporal task context, while the other grounds exploration in states less sensitive to appearance changes.

A Worked Example

Consider the sequential stacking task in Figure 7: place the left block on the middle block, then place the right block on top. The following explains how the mechanisms interact; it is not an additional measured reward trajectory.

If the gripper opens before contact, the image may resemble completed placement, but SAR's hard negatives should lower the score for near misses such as absent contact or unstable placement. Smoothing and sustained-progress checks further prevent one visually favorable frame from immediately triggering a stage transition.

As the robot adjusts the relative gripper-block pose, POE rewards exploration of unfamiliar configurations. Improving forward predictions then produces base progress. Memory retrieves previously completed subtasks to help distinguish a completed first layer from continued correction of that layer, while gating the progress reward. Final success still depends on physical task-completion criteria, not on assumptions made in this explanatory walkthrough.

Loss & Training

Following the main paper's verbal definition of Equation (1), the combined reward can be written clearly as:

\[ \tilde r_t=(1-\rho)r_t^e+\rho\bigl(r_t^{\mathrm{stage}}+r_t^{\mathrm{cur}}+r_t^{\mathrm{prog}}\bigr),\qquad \rho=0.6. \]

The extrinsic term is the sparse task reward; the intrinsic terms are SAR, POE curiosity, and memory-gated progress. Two critics estimate extrinsic and intrinsic returns separately, and GAE advantages are mixed in the same proportions. Training also includes forward/inverse-model mean squared errors, value losses, and a policy-entropy term. Several cached equations have damaged operators, so the incomplete total-loss expression and unspecified hyperparameters are not reproduced or guessed.

Experiments use 8 parallel DISCOVERSE environments, 2 million environment steps per seed, and 3 random seeds. Optional behavior-cloning warm-start is applied consistently within a comparison setting, and all baselines use task-aligned normalization. The cleanest same-backbone comparison is EvoVLA versus OpenVLA-OFT; Octo, OpenVLA, and other policy families should not all be described as having identical backbones.

Key Experimental Results

Main Results

Discoverse-L includes Bridge, Jujube-Cup, and Stack with 74, 19, and 18 prompted stages respectively, and 50 scripted trajectories per task. Bridge's 74-stage dictionary merges micro-adjustments from the simulator's 79 fine-grained stages. Simulator stage events are used only for evaluation diagnostics, not training. Success Rate (SR) measures episodes completed within 400 steps. Evaluation uses 50 held-out-seed episodes per task and reports means over 3 training seeds.

The following rows are selected from Table 1 on page 12. All scores are success percentages; gains are percentage points.

Method	Bridge	Jujube-Cup	Stack	Average
Octo	24.8	33.7	29.1	29.2
OpenVLA	32.6	42.0	37.5	37.4
OpenVLA-OFT	54.1	63.5	59.4	59.0
EvoVLA	65.3	72.6	69.7	69.2
Gain over OpenVLA-OFT	+11.2	+9.1	+10.3	+10.2

Hallucination Rate (HR) is not the fraction of failed episodes. It is the fraction of high-scoring evaluator events whose physical completion condition is false. Restating the conditional counts in Equation (12):

\[ \mathrm{HR}=\frac{\mathbb E[\mathbf 1\{u_k(t)>\theta\ \land\ c_k(t)=0\}]}{\mathbb E[\mathbf 1\{u_k(t)>\theta\}]},\qquad \theta=0.7. \]

Here \(u_k(t)\) is the stage evaluator score and \(c_k(t)\) is the physical-completion indicator accessed only at evaluation. Expectations cover evaluation episodes, timesteps, and active stages. Tables multiply this ratio by 100. Lower HR does not eliminate all failures, and interpretation depends on the high-score denominator and threshold.

Ablation Study

Table 2 on page 12 adds components cumulatively rather than removing each from the full system. Original values are retained below; the final column gives SR/HR percentage-point changes relative to the preceding row.

Cumulative configuration	SR (%)	HR (%)	Change from preceding row
OpenVLA-OFT	59.0	38.5	Not applicable
+ Hard negatives	61.8	31.2	+2.8 / -7.3
+ Temporal smoothing	63.7	23.4	+1.9 / -7.8
+ Long-Horizon Memory	66.1	19.5	+2.4 / -3.9
+ POE, full EvoVLA	69.2	14.8	+3.1 / -4.7

Key Findings

• Hard negatives and temporal smoothing constitute SAR and cumulatively reduce HR from 38.5% to 23.4%, a 15.1-point drop. The full reduction is 23.7 points. Addition order affects marginal gains, so these rows do not directly establish arbitrary independent removal effects.
• Page 13 reports approximately 600,000 environment steps to reach 50% average SR, versus approximately 900,000 for OpenVLA-OFT, yielding 1.5-fold sample efficiency at that target. This measures environment interactions, not wall-clock training speed.
• Figure 6 on page 13 and the text on page 14 report 54.6% average real-robot SR across four tasks versus 43.6% for OpenVLA-OFT; Insert achieves 55.2% versus 41.8%. The first three tasks use direct Sim2Real transfer, but Insert uses 50 teleoperated demonstrations, behavior cloning, and approximately 5,000 PPO steps. It is not zero-shot transfer.
• Page 14 describes multiple batches of at least 20 valid physical trials per model-task pair, without a uniform exact total in the main paper. SR is 68.7%-69.2% over thresholds 0.65-0.75, but changing the threshold also changes HR's conditional denominator.

Highlights & Insights

• Localizing reward exploitation to individual stages makes high-reward physical failure more diagnosable. Hard negatives can target the physical conditions most easily exploited by a robot instead of merely adding more successful demonstrations.
• Prediction error and learning progress are treated separately: the former drives exploration, and the latter receives memory gating. This distinction helps avoid equating persistent unpredictability with persistent learning, although stability still requires empirical verification.
• Memory contributes to reward shaping as well as policy context. A transferable idea is to use history to assess the credibility of a learning signal, not only to generate the next action.

Limitations & Future Work

• Evaluation covers three simulation tasks and one additional real task, concentrated on AIRBOT-Play manipulation. These results do not establish broad generalization across objects, contact conditions, or robot embodiments.
• Stage dictionaries, demonstration videos, and object poses remain important dependencies; real-world POE additionally uses AprilTag. Self-supervision does not mean the absence of external structure or physical-state information.
• The main paper repeatedly refers to the supplement for non-cumulative ablations, dictionary stress tests, and hyperparameters, but those materials are absent from this cache. The claim that removing any module costs at least 2.4 SR points cannot be fully verified from the visible cumulative table alone.
• SAR still delegates spatial and contact interpretation to a visual model, leaving potential systematic blind spots. Independent contact or stability checks, confidence intervals, pose-noise sensitivity, and reward-module computational costs would strengthen the evaluation.

Related Work & Insights

• Versus OpenVLA-OFT: EvoVLA retains action chunking, parallel decoding, and continuous action regression while primarily revising reinforcement learning feedback and auxiliary memory mechanisms. Its strongest attribution is the +10.2 simulation SR points over the same-backbone baseline.
• Versus ICM / RND: These approaches derive novelty from prediction or distillation errors; POE restricts exploration to relative-pose space. Reduced sensitivity to irrelevant visual changes comes at the cost of pose acquisition and geometric-representation assumptions.
• Versus MemoryVLA: Both use history to improve manipulation, but EvoVLA emphasizes coupling selective context to learning-progress reward gating. Describing it merely as adding memory to a VLA misses this distinction.
• Versus RoboCLIP-style VLM rewards: The emphasis is stage-conditioned hard negatives and sustained-progress checks, rather than treating frozen image-text similarity as a reliable physical verifier.

Rating

• Novelty: 4/5. Stage-hallucination diagnosis and the interaction of rewards, exploration, and memory are targeted contributions built from established components.
• Experimental Thoroughness: 3/5. Simulation, real robots, and cumulative ablations are included, but task coverage and main-paper statistical details remain limited.
• Writing Quality: 3/5. The problem and headline results are clear, but cumulative and removal ablations need careful distinction, and several implementation details depend on the supplement.
• Value: 4/5. The work offers actionable mechanisms and diagnostics for long-horizon VLA reward reliability, not a guarantee of physical success.