PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models¶

Conference: ECCV 2026
arXiv: 2606.22540
Project Page: https://inceptionwang.github.io/PolicyTrim/
Code: None
Area: Robotics / Embodied AI
Keywords: VLA Models, Reinforcement Learning, Robotic Manipulation, Policy Efficiency, Action Chunk Expansion

TL;DR¶

PolicyTrim proposes a two-stage GRPO post-training framework. By utilizing dynamic execution window exploration to expand reliable action chunk lengths and incorporating step-saving rewards alongside group-anchoring stability regularization to eliminate redundant physical steps, it improves the end-to-end deployment speed of VLA robots by up to 5.83x without altering the architecture or requiring additional demonstration data.

Background & Motivation¶

Vision-Language-Action (VLA) models unify visual perception, language understanding, and action generation into an end-to-end framework, becoming a paradigm-level solution for general robotic manipulation. To reduce the computational overhead of large visual-language backbones, the community has carried out extensive work around computational efficiency—techniques like visual token pruning, quantization, and KV-cache reuse continuously reduce single-step inference latency. However, a more fundamental issue has been ignored: the efficiency of the policy itself, i.e., how many policy inference calls are required in total to complete a task. This is jointly determined by two factors: the length of the reliably executable actions within each predicted action chunk, and the total physical steps needed to complete the task. Current VLA policies suffer from rapid quality degradation at the tail end of the predicted action chunk, forcing the system to conservatively execute only the first few actions before retrieving new observations. Meanwhile, policies trained via imitation learning naturally tend to generate redundant corrective steps. Experimental results clearly demonstrate this Key Challenge: simply forcing the execution of longer action chunks (increasing chunk length from 5 to 20) drops the task success rate from 97.8% to 94.1%, while the physical steps increase rather than decrease—prediction errors from tail degradation force the robot to repeatedly correct itself, which in turn makes the execution path longer.

Existing RL-based post-training methods (such as VLA-RL, Co-RFT) generally use binary success rewards, which brings two deep limitations. First, once the policy success rate reaches a high level, the intra-group reward variance collapses rapidly, causing the advantage estimation to lose discriminability, and learning stagnates. Second, binary rewards do not distinguish between "completing in 80 steps" and "completing in 120 steps," and completely fail to incentivize the model to expand its reliable prediction range—the policy learns whether it can do it, but never learns how to do it more efficiently. Interestingly, repeating rollouts of the same policy on the same task shows that compact execution paths do physically exist; they just happen to appear by chance under the current paradigm. This indicates that there is a massive space for efficiency optimization within the policy, but it lacks a guiding signal to systematically extract it.

The Key Insight of this paper is that policy efficiency and computational efficiency are completely orthogonal optimization axes. The former provides multiplicative speedups by reducing the number of inference calls, yielding compounding gains when overlaid with any single-step inference acceleration methods. PolicyTrim decouples policy efficiency optimization into two progressive stages: first, dynamic execution window exploration is used to help the model maintain reliable prediction quality over longer action chunks; then, step-saving rewards combined with stability regularization are employed to eliminate redundant physical steps. Core Idea: Decompose VLA policy efficiency optimization into a two-stage GRPO post-training framework of "first expanding reliable execution chunk length, then reducing redundant physical steps," using customized rewards to respectively incentivize long-chunk reliable prediction and compact execution trajectories, achieving up to 5.83x end-to-end speedup without sacrificing task success rate.

Method¶

Overall Architecture¶

PolicyTrim is a two-stage RL post-training framework based on GRPO (Group Relative Policy Optimization) for progressive optimization of pre-trained VLA policies, aiming to reduce the total number of inference calls required to complete a task. At any decision step, the VLA policy generates an action chunk of length \(H\) based on the current visual observation and language instruction. Due to unreliable prediction quality at the tail end, the system typically executes only the first \(h = \lfloor\gamma \cdot H\rfloor\) actions (\(\gamma \in (0,1]\) is the acceptance rate) and then retrieves new observations for the next round of inference. Although this conservative strategy ensures physical stability, it wastes a large number of potentially reliable predictions.

Stage 1 of PolicyTrim allocates different acceptance rates \(\gamma_i\) to each trajectory within a group to parallelly probe the prediction reliability of each chunk position, and guides the policy to evolve towards longer reliable chunks using a combined reward. Based on the expanded chunks, Stage 2 uses step-saving rewards to encourage successful trajectories to take fewer physical steps, while applying group-anchoring stability regularization to prevent the policy from collapsing into unreplicable shortcuts. The two stages are executed sequentially—first making predictions further and more reliable, then making execution more compact—ultimately drastically reducing inference calls.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pre-trained VLA Policy<br/>Action Chunk Capacity H"] --> B["Stage 1: Reliable Action Chunk Expansion"]
    B --> B1["G Trajectories in Group<br/>Assign γ_1,...,γ_G\nAuto-form Multi-acceptance Rate Scan"]
    B1 --> B2["Conditional Reward R_ext\nI_succ · (R_succ + β·γ)\nGRPO Group Relative Advantage"]
    B2 --> C["Policy Learns Longer Reliable Chunk\nh: 5→10→15 (π_0.5)"]
    C --> D["Stage 2: Redundant Step Reduction"]
    D --> D1["Step-saving Reward R_step\nFewer Steps to Complete = Higher Reward"]
    D1 --> D2["Group-anchoring Regularization P_stab\ntanh Penalty for Deviation from Group Consensus"]
    D2 --> E["Efficient Deployment Policy\nLong Chunk + Fewer Steps + High Success Rate"]
    E --> F["End-to-End 5.83× Speedup\nOverlay with VLA-Cache → 2.48×"]

Key Designs¶

1. Dynamic Execution Window Exploration: Multi-Acceptance Rate Parallel Probing of Reliable Boundary within Group

The core challenge of Stage 1 is that we want to know whether predictions are reliable under different chunk lengths without paying extra rollout costs for it. PolicyTrim utilizes the group sampling mechanism of GRPO—assigning a different acceptance rate \(\gamma_i\) (sampled from a discrete set \(\Gamma = \{\gamma_1, \gamma_2, \dots, \gamma_M\}\)) to each of the \(G\) trajectories within a group. This forms a multi-acceptance rate scanner where one trajectory executes 5 steps before re-observation, another executes 10 steps, and another executes all 20 steps. The economy of this design lies in that the GRPO group sampling itself is a comparative experiment of multi-horizon reliability, consuming no extra environment interaction budget. Ablation studies strongly support the necessity of this design: when fixing \(\gamma = 1.0\), the success rate plummets from 99.1% to 94.4%, while dynamic window exploration under the same average chunk length maintains the success rate at 98.8%. The key difference lies in intra-group contrast: successful long chunks receive high rewards, while failed long chunks receive zero, naturally guiding the policy to learn which chunk positions remain reliable and optimize in that direction.

2. Conditional Horizon Reward: Preventing Misdirection when No Successful Trajectory Exists

The reward design for reliable chunk expansion faces a subtle anti-pattern: if no trajectory within a group successfully completes the task, the objective horizon reward \(R_{horizon} = \beta \cdot \gamma\) would still yield a positive value for longer trajectories, mistakenly pushing the policy toward long but unreliable execution windows. PolicyTrim's solution is elegant: it introduces a success indicator \(I_{succ}\), defining the joint reward as \(R_{ext}(\tau_i) = I_{succ}(i) \cdot (R_{succ}(\tau_i) + \beta \cdot \gamma_i)\). The horizon component is added if and only if the trajectory is successful; otherwise, the reward for failed trajectories is always zero. This conditional gating ensures that the horizon reward only incentivizes longer chunks under the premise of "already successful," avoiding blindly extending the chunk when the model does not yet possess reliable prediction capabilities. Within the same GRPO group, this design naturally creates a contrastive learning effect—successful long chunks receive significantly higher rewards than successful short chunks, and failed trajectories all get zero—guiding the advantage estimation to evolve toward longer reliable chunks.

3. Step-saving Reward and Group-anchoring Regularization: A Balanced Mechanism to Prevent Greedy Shortcut Collapse

The goal of Stage 2 is to minimize physical steps, but directly minimizing steps can lead to devastating side effects. First, the step-saving reward is defined as: \(R_{step}(\tau) = \max(0, S_{base} - S(\tau)) / S_{base}\), where \(S_{base}\) is set to approximately 1.3 times the average success steps of the initial policy. When using this reward alone, due to the high variance of initial steps, occasional low-cost short trajectories receive abnormally high returns, prompting the policy to speculate through unreplicable "lucky" shortcuts—dropping the success rate sharply from 97.8% to 93.7%. To address this, group-anchoring stability regularization is introduced:

\[P_{stab}(\tau_i) = \lambda_{stab} \cdot \tanh\!\left( \frac{|S(\tau_i) - \mu_{group}|}{\max(\sigma_{group}, \sigma_{floor})} \right)\]

where \(\mu_{group}\) and \(\sigma_{group}\) are the mean and standard deviation of completed steps of successful trajectories within the group. This penalty suppresses trajectories that deviate significantly from the group consensus, ensuring that the policy converges to stable, reproducible, and compact execution patterns rather than relying on occasional low-probability events. With the anchoring regularization, the task success rate recovers from 93.7% to 97.5%, while the average steps further decrease from 81.7 to 61.6—creating a win-win scenario that improves both success rate and execution efficiency, with the training curve rising steadily compared to the sharp drops without regularization.

Loss & Training¶

Both stages use GRPO optimization without a critic model or an extra reward model. Intra-group rewards are normalized to calculate relative advantages, and the policy is updated using a clipped surrogate objective with a KL penalty. The two stages are trained sequentially, with each stage taking up to 500 epochs; the reference policy is frozen after Stage 1 before entering Stage 2. Default group size is \(G = 8\), the horizon reward coefficient is \(\beta = 0.8\), and the stability regularization coefficient is \(\lambda_{stab} = 0.2\).

Key Experimental Results¶

Main Results¶

Results of \(\pi_{0.5}\) and OpenVLA-OFT on LIBERO

Model	Subset	Success Rate (Baseline → PolicyTrim)	Physical Steps (Baseline → PolicyTrim)	Chunk Length	Speedup
\(\pi_{0.5}\)	Spatial	97.8% → 97.8%	108.3 → 59.8	5→15	5.43×
\(\pi_{0.5}\)	Object	99.1% → 98.5%	125.0 → 64.3	5→15	5.83×
\(\pi_{0.5}\)	Goal	98.7% → 98.8%	110.6 → 63.5	5→15	5.23×
\(\pi_{0.5}\)	Long	93.0% → 93.3%	249.8 → 171.8	5→10	2.91×
OpenVLA-OFT	Spatial	98.6% → 98.8%	111.2 → 62.1	8→8	1.79×

Cross-benchmark Cross-model Summary

Model	Benchmark	Success Rate	Step Changes	Speedup
\(\pi_{0.5}\)	ManiSkill	88.1% → 89.8%	45.2 → 38.3	2.36×
\(\pi_{0.5}\)	Meta-World	65.1% → 65.4%	66.3 → 52.6	2.52×
GR00T	LIBERO-Spatial	91.4% → 92.0%	67.2 → 56.6	2.37×

Ablation Study¶

Configuration	Success Rate	Physical Steps	Chunk Length	Speedup	Description
Baseline	97.8%	108.3	5	1.00×	No optimization
+ Chunk Expansion	97.2%	113.8	15	2.86×	Chunk length tripled, steps increased by 5.5
+ Step Reward (w/o Reg)	93.7%	81.7	5	1.32×	Success rate plummeted by 4.1%
+ Step Reward + Reg	97.5%	61.6	5	1.75×	Regularization completely restores success rate
Full (Two-stage + Reg)	98.3%	59.8	15	5.43×	All components optimal

Key Findings¶

Reliable chunk expansion alone pushes the execution window from 5 steps to 15 steps (3x), but because tail prediction residuals amplify over long-term execution, physical steps actually increase from 108.3 to 113.8. This is direct evidence for the ablation and highlights the necessity of Stage 2: long chunks cannot replace step optimization, and both must be processed progressively.
Using the step-saving reward alone causes the success rate to plummet (97.8% → 93.7%), while adding group-anchoring regularization fully recovers it (97.5%) and further reduces steps to 61.6. Regularization simultaneously improves success rate and efficiency; this win-win effect stems from suppressing "lucky shortcuts" to guide the policy to learn reproducible, compact execution patterns.
Dynamic window exploration (intra-group multi-\(\gamma\) parallel scanning) holds a significant advantage over the fixed \(\gamma\) scheme: Fixed \(\gamma = 1.0\) yields a success rate of only 94.4%, whereas dynamic exploration under the same 15-step chunk length maintains 98.8%. Parallel multi-horizon probing provides critical support for stability.
When orthogonally combined with VLA-Cache, a joint speedup of 2.48× is achieved on LIBERO-Object, which far exceeds the 1.26× of VLA-Cache alone. This is the first experimental validation of the feasibility of dual-axis optimization: "computational efficiency \(\times\) policy efficiency."
Under visual distribution shifts (Gaussian blur, random occlusion), PolicyTrim not only maintains a higher success rate (88.4% vs 79.6%) but also requires fewer physical steps (77.6 vs 128.9), showing that the learned high-efficiency execution strategy is more robust to perceptual degradation.

Highlights & Insights¶

Perspective-level Contribution: Shifts the focus of "deployment speed" from "how fast is a single inference step" to "how many total inference steps are required." It identifies two orthogonal optimization axes—computational efficiency \(\times\) policy efficiency. Although this observation is intuitive, the VLA community has not systematically explored it before, representing the most fundamental academic contribution of this work.
Free Probing via GRPO Group Structure: Stage 1's dynamic window exploration cleverly utilizes the inherent multi-trajectory parallel nature of GRPO groups. By turning the group into a multi-acceptance rate reliable-horizon scanner, it gains comparison signals across multiple execution lengths without adding extra rollout budgets, yielding an extremely economical and elegant design.
Gated Insight of Conditional Horizon Reward: The horizon reward is activated only if there is at least one successful trajectory in the group. This highlights a subtle but important principle: "only after ensuring success are we qualified to discuss efficiency." Without this gating, policies would be misled toward long but unreliable chunks.
Group Consensus as a Natural Anchor: The beauty of group-anchoring regularization lies in using group consensus instead of an absolute threshold as an adaptive anchor, which automatically relaxes constraints as the group converges, while using a tanh function to bound the penalty. This adaptive regularization design is more robust than fixed penalty coefficients.

Limitations & Future Work¶

Architecture Dependency: Stage 1's chunk expansion is ineffective for parallel decoding architectures (e.g., OpenVLA-OFT)—their placeholder token mechanism suffers severe accuracy degradation even with a small increase in chunk length. Autoregressive VLA models also cannot directly apply Stage 1. Future work could embed modeling support for long-chunk reliable predictions directly into the pre-training phase.
Requirement for High-Quality Simulation: Both stages of RL require intensive environment interaction (\(\approx 68\) hours \(\times\) 8× H100 GPUs), which remains costly for tasks lacking high-fidelity simulators or direct physical robot training.
Uncertainty in Sim-to-Real Transfer: In actual deployment, PolicyTrim completes RL post-training in simulation and then transfers to the real world through SFT on a small number of real demonstrations. The paper does not systematically analyze to what extent the efficiency patterns learned by RL are preserved after SFT.
Unverified Scenarios: The experiments focus on tabletop manipulation tasks. Generalization capabilities to more complex scenarios such as multi-step long-horizon tasks, dynamic human-robot collaboration, and deformable object manipulation still require further validation.

vs VLA-RL / Co-RFT / SimpleVLA-RL: These methods also apply RL to VLA post-training, but they all rely on binary success rewards, completely lacking incentives for execution efficiency. PolicyTrim is the first RL post-training framework to systematically optimize policy efficiency as an explicit objective, and its reward design heuristic can be directly extended to other control tasks requiring efficiency constraints (e.g., navigation, assembly).
vs Computational Acceleration Methods (SPVLA / VLA-Cache / SpecVLA): These focus on reducing single-step inference latency but cannot reduce the number of inference calls. PolicyTrim is orthogonally complementary—combining them yields multiplicative speedups, providing a complete dual-axis roadmap for VLA deployment optimization.
vs Action Chunk Prediction Degradation Studies (MoH / VLA Knows Its Limits): These works analyze and quantify chunk tail prediction degradation, but remain at the diagnostic stage. PolicyTrim is the first RL-based solution specifically targeting this issue, closing the loop from identifying bottlenecks to eliminating them.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [First to systematically define the VLA policy efficiency dimension and provide a complete optimization scheme; the perspective is novel and directly addresses deployment bottlenecks]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Cross-evaluated on 3 architectures \(\times\) 3 simulation benchmarks \(\times\) real-world platform \(\times\) visual perturbations \(\times\) orthogonal combination with computational acceleration; deep ablation experiments reveal individual components' behaviors]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, progressive method, multi-layered experimental validation, and rich appendix; the logical chain is complete]
Value: ⭐⭐⭐⭐⭐ [Orthogonal to all computational acceleration methods, directly multiplying VLA deployment efficiency, with an impact spanning the entire VLA deployment pipeline]