Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models¶

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/chen-siyao/T2VLA
Area: Robotics & Embodied AI
Keywords: Vision-Language-Action Models, Test-Time Reinforcement Learning, Intrinsic Reward, Dynamic Time Warping, Self-Bootstrapping

TL;DR¶

Addressing the heavy dependence of existing VLA reinforcement learning on external environmental success feedback, this paper demonstrates a strong positive correlation between generative confidence and physical execution success, and introduces T2VLA—an architecture-agnostic test-time RL framework that self-improves via dual-expert bootstrapping and DTW-based intrinsic rewards across discrete and continuous flow-based VLAs.

Background & Motivation¶

Vision-Language-Action (VLA) models have emerged as a foundational paradigm in embodied AI by directly mapping multimodal visual observations and natural language instructions to low-level robotic control. However, conventional supervised fine-tuning (SFT) over static demonstration datasets suffers from prohibitive data collection expenses and severe out-of-distribution generalization bottlenecks. To transcend the performance ceilings of passive imitation learning, reinforcement learning (RL) has become indispensable for driving active, closed-loop environmental exploration. While online RL-VLA approaches have demonstrated impressive capabilities, existing methods universally depend on explicit external supervision signals—such as predefined binary success flags from simulators, specialized critic networks, or costly human annotations. This severe reliance on external reward evaluators confines robotic learning to artificial sandboxes and prevents real-world robots from achieving truly autonomous, lifelong self-evolution.

In pure language reasoning and code generation tasks, the "Era of Experience" has been unlocked by test-time reinforcement learning (Test-Time RL), where models self-improve on unlabeled inputs via self-consistency majority voting. Nevertheless, transplanting this test-time self-improvement to physical manipulation introduces a fundamental challenge: whereas mathematical or code problems yield discrete, unique ground-truth answers that enable simple majority verification, robotic manipulation requires generating continuous trajectories across multi-modal, non-convex action spaces. Multiple diverse spatial paths and velocity profiles can accomplish identical manipulation objectives, depriving the robot of an unambiguous objective ground truth for self-reflection.

This paper tackles this challenge through an insightful empirical discovery: across thousands of discrete-action VLA rollouts, trajectory-level generation confidence (quantified as the length-normalized mean action log-probability) exhibits a remarkably strong positive correlation with physical execution success. In essence, a model's internal generative certainty serves as an effective implicit estimator of physical feasibility. Core idea: leverage internal VLA generative confidence as an intrinsic evaluation anchor, establish a self-bootstrapping mechanism combining task-conditioned local pseudo-experts with a dynamic global expert pool, and formulate sequence-level DTW geometric similarities as dense intrinsic rewards under GRPO to achieve completely autonomous test-time policy evolution without external reward feedback.

Method¶

Overall Architecture¶

T2VLA establishes an architecture-agnostic test-time reinforcement learning pipeline designed to eliminate the need for external environment rewards $R_{\text{env}}(\tau)$ by replacing them entirely with an intrinsic reward $R_{\text{self}}(\tau)$ derived from model-internal signals. Given visual observations and language instructions, the framework executes closed-loop actions on continuous robotic manipulators. During each test-time interaction iteration, the workflow operates through three coordinated stages: first, the model executes exploratory rollouts under its current policy and evaluates length-normalized confidence scores suited to its action representation; second, a confidence-driven dual-expert bootstrapping mechanism dynamically mines on-policy local pseudo-experts while curating a priority-based global expert memory; third, a Dynamic Time Warping (DTW) module computes temporal-shift-tolerant geometric similarities, which are smoothly interpolated via Min-Max normalized confidence weights to supply dense advantages for policy updates via Group Relative Policy Optimization (GRPO).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Environment Observations & Instructions<br/>Multimodal State Perception"] --> B["Length-Normalized Confidence Estimation<br/>Mean Log-Prob / Flow Denoising Likelihood"]
    B --> C["Dual-Expert Bootstrapping Mining<br/>Local Pseudo-Expert + Dynamic Global Expert Pool"]
    C --> D["DTW Hybrid Similarity Intrinsic Reward<br/>Non-linear Time Warping + Min-Max Smooth Weighting"]
    D --> E["Group Relative Policy Optimization GRPO<br/>Group Advantage Normalization + Clipped Surrogate Update"]
    E -->|Continuous Iterative Policy Evolution| A

Key Designs¶

1. Length-Normalized Confidence Estimation: Horizon-Invariant Intrinsic Quality Metric

To evaluate the quality of exploratory rollouts in the complete absence of environmental completion signals, a reliable scalar score must be extracted directly from internal generation statistics. In discrete autoregressive VLAs, directly summing token log-probabilities introduces an undesirable horizon bias, unfairly penalizing extended multi-stage manipulation trajectories and causing policies to degenerate toward trivial, truncated failures. T2VLA formulates length-normalized confidence estimation tailored to both discrete and continuous VLA paradigms. For discrete-token architectures (such as OpenVLA-OFT), padding tokens beyond environment termination are strictly discarded, and confidence is computed as the mean action log-probability over the effective horizon $T_i$: $$c_i^{\text{disc}} = \frac{1}{T_i} \sum_{t=1}^{T_i} \log \pi_\theta(a_{i,t} \mid s_{i,t})$$ For continuous flow-matching models (such as $\pi_0$ and $\pi_{0.5}$), intermediate generation steps do not produce categorical logits. Instead, T2VLA computes the Gaussian transition log-likelihood $\ell_{i,j}^{\text{flow}}$ at selected denoising step $k$ across action-chunk horizons and dimensions, normalizing over valid prediction steps $L_i$ to yield $c_i^{\text{flow}} = \frac{1}{L_i}\sum_{j=1}^{L_i} \ell_{i,j}^{\text{flow}}$. This formulation projects probability densities onto a standardized scale comparable across heterogeneous execution horizons, providing an objective metric for subsequent autonomous expert election.

2. Dual-Expert Bootstrapping Mining: Balancing Aggressive Exploration with Historical Stability

Relying solely on the most confident trajectory in a single exploratory batch captures recent behavioral discoveries, but remains highly fragile: if an entire batch suffers from poor exploratory rollouts, treating the local best as an undisputed reference inevitably accumulates errors and triggers policy collapse. Conversely, relying exclusively on static historical demonstrations freezes policy evolution and prevents the model from discovering superior novel trajectories. To overcome this tension, T2VLA introduces a confidence-driven dual-expert bootstrapping mechanism. Exploratory trajectories within a batch $\mathcal{D}$ are first partitioned by task instruction $l$ into sub-batches $\mathcal{D}_l$. Within each subset, the trajectory with the highest internal confidence is designated as the task-conditioned Local Pseudo-Expert: $$\tau_{local,l}^* = \arg\max_{\tau_i \in \mathcal{D}_l} c_i$$ This local expert provides an agile, on-policy behavioral anchor aligned with the current model distribution. Concurrently, a dynamic Global Expert Pool $\mathcal{P}_l$ of bounded capacity ($K=5$) is maintained for each task. The global pool enforces strict admission criteria, admitting only newly elected local pseudo-experts $\tau_{local,l}^*$ and maintaining trajectories sorted by confidence $c$, automatically evicting outdated, low-performing trajectories: $$\mathcal{P}_l \leftarrow \text{Top-}K\left(\mathcal{P}_l \cup \{\tau_{local,l}^*\}\right)$$ This dual synergy guarantees that the guidance references stay both dynamically responsive to emerging breakthroughs and resiliently protected against transient exploratory failures.

3. DTW Hybrid Similarity Intrinsic Reward: Phase-Invariant Spatial Alignment and Smooth Interpolation

In continuous physical control, stochastic disturbances and differing movement cadences frequently produce trajectories that are spatially identical but temporally shifted. Measuring trajectory discrepancy via standard Euclidean distance imposes strict point-wise matching, causing distance penalties to artificially blow up under trivial pacing discrepancies and mislabeling high-quality executions as poor behaviors. T2VLA introduces Dynamic Time Warping (DTW) to non-linearly warp the temporal dimension within normalized $[0, 1]$ action bounds: $$\text{Sim}_{\text{DTW}}(\tau_A, \tau_B) = \frac{1}{1 + \frac{D_{\text{DTW}}(\tau_A, \tau_B)}{\max(M, T)}}$$ To combine immediate on-policy alignment with historical stability, the dynamic hybrid similarity reward $r_i^{\text{sim}}$ interpolates between the local expert and the best-matching global expert via an adaptive weight $w$: $$r_i^{\text{sim}} = w \cdot \text{Sim}_{\text{DTW}}(\tau_i, \tau_{local,l}^*) + (1-w) \cdot \max_{\tau_p \in \mathcal{P}_l} \text{Sim}_{\text{DTW}}(\tau_i, \tau_p)$$ The weight $w = \text{clip}\left(\frac{c_{local,l}^* - c_{\min,l}}{c_{\max,l} - c_{\min,l} + \epsilon}, 0, 1\right)$ continuously normalizes the local expert's confidence against the historical extrema of pool $\mathcal{P}_l$. When the current batch produces an exceptionally confident expert ($w \to 1$), the optimization aggressively pursues novel modes; when confidence is low ($w \to 0$), it smoothly retreats to historical anchors. To prevent catastrophic forgetting, a reference policy penalty yields the final reward: $r_i = r_i^{\text{sim}} - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$.

Loss & Training¶

Given trajectory-level proxy rewards $\{r_1, \ldots, r_N\}$, T2VLA optimizes policy parameters via Group Relative Policy Optimization (GRPO), computing baseline-free advantages directly across sampled rollout groups without requiring a separate value critic: $$A_i = \frac{r_i - \mu(r)}{\sigma(r) + \epsilon}$$ Policy updates maximize the clipped surrogate objective: $$\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^{T_i} \min\left( \rho_{i,t}(\theta) A_i, \; \text{clip}(\rho_{i,t}(\theta), 1-\epsilon_{\text{clip}}, 1+\epsilon_{\text{clip}}) A_i \right)$$ where $\rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid s_{i,t})}{\pi_{\text{old}}(a_{i,t} \mid s_{i,t})}$ represents the importance sampling ratio. The training utilizes AdamW with a peak learning rate of $5 \times 10^{-6}$ and an initial KL penalty coefficient $\beta = 0.02$. OpenVLA-OFT adopts a sampling temperature of 1.6 and a PPO clipping range of $[0.2, 0.28]$, while continuous flow policies ($\pi_0/\pi_{0.5}$) use temperature 1.0, clipping threshold 0.2, and 4 denoising steps.

Key Experimental Results¶

Main Results¶

T2VLA is extensively validated across two major robotic manipulation benchmarks: the comprehensive multi-suite LIBERO benchmark (covering Spatial, Object, Goal, and Long suites), and the challenging dual-arm coordination RoboTwin 2.0 benchmark.

Method	Reward Source	Spatial	Object	Goal	Long	Average Success (%)
Octo [39]	None (SFT)	78.9	84.6	85.7	51.1	75.1
OpenVLA [19]	None (SFT)	84.7	79.2	88.4	53.7	76.5
UniVLA [5]	None (SFT)	96.5	96.8	95.6	92.0	95.2
VLA-RL [27]	Env. Success	90.2	94.3	91.8	82.2	89.6
SimpleVLA-RL [20] (Oracle)	Env. Success	99.4	99.1	99.2	98.5	99.1
EVOLVE-VLA [1]	Learned Critic	95.4	97.4	95.8	94.4	95.8
OpenVLA-OFT (Base)	None (SFT)	91.6	95.3	90.6	86.5	91.0
Ours (OpenVLA-OFT)	Self-Reward	97.7 (+6.1)	99.6 (+4.3)	96.1 (+5.5)	95.3 (+8.8)	97.2 (+6.2)
$\pi_0$ (Base)	None (SFT)	65.3	64.4	49.8	51.2	57.7
Ours ($\pi_0$)	Self-Reward	86.3 (+21.0)	91.0 (+26.6)	82.0 (+32.2)	68.0 (+16.8)	81.9 (+24.2)
$\pi_{0.5}$ (Base)	None (SFT)	84.6	95.4	84.6	43.9	77.1
Ours ($\pi_{0.5}$)	Self-Reward	94.9 (+10.3)	98.4 (+3.0)	91.8 (+7.2)	55.1 (+11.2)	85.1 (+8.0)

On RoboTwin 2.0 bimanual tasks using OpenVLA-OFT, T2VLA demonstrates pronounced gains across various execution horizons: short-horizon tasks (Lift Pot, Beat Hammer) improve from an average of 19.1% to 53.9% (Beat Hammer surging by +39.9% to 68.0%); medium-horizon task (Place Empty Cup) reaches 84.8% (+7.5%); and long/extra-long horizon tasks (Handover Block, Stack Bowls) advance from 36.8% to 51.55%. The overall 5-task bimanual success rate climbs from 37.8% to 59.1%, representing a +21.3% absolute leap.

Ablation Study¶

The ablations investigate the impact of the dual-expert mechanism, expert pool capacity $K$, fusion gating strategies, and trajectory alignment metrics on LIBERO-Long using OpenVLA-OFT.

Component / Strategy	Configuration	LIBERO-Long Success (%)	Phenomenon & Mechanism
Expert Source	Local Expert Only	94.5	Responsive to new modes, but vulnerable to occasional poor batches
Expert Source	Global Expert Only	93.0	Maintains baseline stability, but over-relies on older trajectories
Expert Source	Dual Expert (Ours)	95.3	Optimal synergy: aggressive discovery plus historical baseline anchor
Pool Capacity	$K = 3$	93.4	Insufficient coverage of diverse successful spatial trajectories
Pool Capacity	$K = 5$ (Default)	95.3	Ideal trade-off between behavioral diversity and quality freshness
Pool Capacity	$K = 10$	91.4	Retains stale trajectories from early stages, diluting guidance
Gating Strategy	Static Weighting ($w = 0.5$)	88.3	Early policy collapse ($\dagger$, < 100 epochs) from abrupt updates
Gating Strategy	Max Routing	87.5	Hard switching triggers early policy collapse ($\dagger$) via gradient shocks
Gating Strategy	Adaptive Margin Fallback (AMF)	90.0	Hard thresholding prevents collapse but lacks fine-grained adaptation
Gating Strategy	Sigmoid Soft Gate	92.6	Smooth transition stabilizes training, but lacks extreme-value bounds
Gating Strategy	Min-Max Normalization (Ours)	95.3	Strictly bounded, continuous projection enables smooth expert fusion
Distance Metric	Euclidean Distance	Similarity: 0.6914	Rigid point-wise alignment causes penalty explosion under phase shifts
Distance Metric	Dynamic Time Warping (DTW)	Similarity: 0.9460	Non-linear elastic matching captures true geometric trajectory congruence

Key Findings¶

Smooth continuous gating and dual experts are essential safeguards against unsupervised RL collapse: Both rigid static weights ($w=0.5$) and hard max routing trigger severe early policy collapse ($\dagger$, < 100 epochs). Without external ground-truth rewards, sudden step changes in proxy rewards destabilize policy gradients. Min-Max normalization provides a continuous damping mechanism across non-stationary log-probabilities.
Continuous flow models demonstrate remarkable self-bootstrapping elasticity: On the initially under-trained $\pi_0$ policy, T2VLA achieves an exceptional +24.2% overall gain entirely autonomously (surging +32.2% on LIBERO-Goal), proving that denoising likelihood consistency serves as a robust guidance signal in diffusion/flow models.
The "Prior Threshold" phenomenon in self-bootstrapping: In 1-shot SFT evaluations, T2VLA boosts performance by ~20% across Spatial, Object, and Goal suites, but experiences degradation on LIBERO-Long (dropping from 17.3% to 11.0%). This highlights a critical boundary condition: when base capabilities are too weak to produce even a single viable execution on complex long horizons, self-elected pseudo-experts mislead the policy into uninformative noise and policy drift.

Highlights & Insights¶

Bridging LLM confidence estimation into physical robotic control: This work establishes the first systematic proof that internal generation likelihoods in discrete and continuous VLAs faithfully indicate physical task success, providing a scalable foundation for reward-free autonomous learning.
Decoupling confidence ranking from geometric behavioral reward: Directly using scalar confidence as a reward invites reward hacking and distribution collapse. T2VLA smartly downscales confidence to an "expert election criteria", employing DTW spatial geometry as the tangible reward metric.
Compatibility with video world models: Beyond real simulators, T2VLA self-improves inside "imagined" rollouts synthesized by the OpenSora video world model (yielding +2.1% on LIBERO-Spatial), indicating high potential for deployment in generative world simulators.

Limitations & Future Work¶

Cold-start vulnerability under extreme data scarcity: When task complexity is high and initial demonstrations are near zero (such as 1-shot long-horizon tasks), random exploration fails to produce viable paths, causing pseudo-experts to lock onto suboptimal modes.
Kinematic DTW overlooks contact dynamics: DTW is applied to end-effector poses without incorporating contact forces or object deformation, which may limit effectiveness in micro-force assembly tasks.
Future directions: Integrating latent visual state transition verification and contact-aware physical modeling into the self-evaluation loop to overcome cold-start thresholds in complex manipulation settings.

vs EVOLVE-VLA: EVOLVE-VLA relies on a separately trained generalist foundation critic to estimate task progress; T2VLA introduces zero external models or evaluators, deriving self-improvement purely from the base model's own probabilities.
vs SimpleVLA-RL / $\pi$RL: These state-of-the-art online RL methods require continuous access to environment-provided oracle reward flags during training; T2VLA operates without any external rewards while closely approaching oracle performance (97.2% vs 99.1% on LIBERO).
vs Test-Time Adaptation (e.g., V-GPS, TACO): Test-time adaptation methods rely on training-free sampling and re-ranking without modifying weights; T2VLA performs parameter-level Test-Time Learning (TTL) via GRPO, directly updating policy weights.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Establishes the first systematic paradigm leveraging internal VLA generative confidence to drive test-time RL without external rewards.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations covering discrete/continuous paradigms, single/bimanual setups, extreme few-shot regimes, and world model rollouts.
Writing Quality: ⭐⭐⭐⭐⭐ Cohesive prose, mathematically rigorous formulation, and candid failure mode analyses regarding policy collapse and prior thresholds.
Value: ⭐⭐⭐⭐⭐ Eliminating external reward dependencies is pivotal for scaling embodied agents in open-world settings; the insights and methodology provide strong guidance for future work.