Skip to content

SAFE-Pruner: Semantic Attention–Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

Conference: ECCV 2026
Paper: ECCV 2026
Area: Robotics & Embodied AI
Keywords: VLA inference acceleration / visual token pruning / semantic attention consistency / training-free / robotic manipulation

TL;DR

SAFE-Pruner observes that VLA models attend to the same semantic entities across control timesteps (semantic attention consistency), and therefore uses the per-layer attention saliency cached at the most recent key timestep to forecast the deep-layer saliency that is not yet computable for the current frame; the forecast is fused with the current shallow-layer saliency for top-K token pruning, and a shallow-attention drift detector adaptively refreshes the key timestep, yielding up to 1.89x speedup with less than 1.5% drop in success rate without any training or architectural change.

Background & Motivation

Vision-language-action (VLA) models unify visual perception, language understanding, and action generation into a single policy and, building on pretrained vision-language backbones and large-scale robot demonstration datasets, already deliver end-to-end manipulation across diverse tasks and environments; OpenVLA, OpenVLA-OFT, CogACT, and π0.5 represent the mainstream architectural families. Visual tokens, however, dominate sequence length, and the resulting attention cost makes per-forward latency hard to reconcile with the real-time demands of closed-loop control — the arm is still moving and objects are still being occluded, so a late action can be useless or even unsafe. Distillation, quantization, and layer pruning do bring efficiency gains, but they typically require retraining or architectural modification, which makes transferring them to a new backbone expensive; visual token pruning, by contrast, simply discards redundant patches at inference time and is thus training-free and plug-and-play.

Existing VLA token pruning methods (FastV, SparseVLM, DivPrune, VLA-Cache, VLA-Pruner) mostly base their pruning decisions on shallow-layer signals: they score tokens by shallow cross-attention weights or token similarity, keep the top-K visual tokens, and discard the rest. The problem is that this discard is irreversible — a dropped token is gone for every subsequent layer. Yet VLA attention is coarse-to-fine: shallow layers attend broadly and diffusely, and only deep layers converge on the truly critical regions. The paper quantifies this on OpenVLA-OFT: once the pruning rate exceeds 70%, the probability that early-saliency-only pruning removes "core tokens" of other layers (the top 10% most important tokens of each layer) rises sharply (0-50% is a stable region, 50-70% a transition region, and 70-90% a loss-surge region). In other words, the pruning decision is made at the very moment of least certainty, while being held responsible for the entire downstream reasoning chain. Recent work (SpecPrune-VLA, action-aware dynamic pruning) tries to estimate late-stage saliency as a remedy, but performs that estimation on the token subset that has already been pruned, so the genuinely critical tokens were irreversibly removed at the shallow layer and the estimate is corrupted from the start.

This is the core tension the paper addresses: avoiding mis-pruning requires knowing which tokens deep layers will attend to, but pruning happens in the shallow layer where those layers have not been computed yet. SAFE-Pruner's way out comes from an observation about the temporal evolution of VLA attention — semantic attention consistency: between adjacent control timesteps of the same task, especially in the more concentrated deep layers, attention stays concentrated on patches carrying the same semantic information (for instance the object to be grasped) even when those patches have moved substantially in space (the gripper travels from the middle to the right to grasp the soup and then to the left to place it, yet attention remains anchored near the gripper/object rather than at a fixed coordinate). Since attention follows semantics rather than pixels, the deep-layer saliency already computed on a historical frame can serve as a forecast of the current frame's deep-layer saliency. Core idea: turn the difficulty of "the current frame's deep attention is unknowable" into a solvable one by transferring per-layer saliency from a historical key timestep across frames via semantic token correspondence — fuse the transferred deep saliency with the shallow saliency to prune, and use cosine drift of shallow attention to refresh the key timestep, so that a single pruning decision sees both the present and the future.

Method

Overall Architecture

SAFE-Pruner is a training-free pruning shell wrapped around the existing VLA forward pass: the input is a text instruction plus a stream of observation frames, and the output is still one action per control timestep. It intervenes at the shallow layer \(\ell_s\) of each forward pass and decides which visual tokens continue through the remaining Transformer layers. Three things chain together: key timesteps turn history into a usable saliency prior, a fusion step combines that prior with the current frame's shallow evidence into a single pruning score, and an attention-drift detector decides when the prior must be rebuilt.

For a single control timestep \(t\): the shallow-layer attention saliency \(\mathcal{S}^{\ell_s}_t\) is computed as usual (the only signal available without extra cost); the per-layer saliency cached at the most recent key timestep \(\tau_{\mathrm{ref}}\) is then transferred to the current frame through semantic nearest-neighbour matching and averaged over a set of deeper layers \(\mathcal{L}_d\) to yield the forecast deep saliency; finally the current shallow saliency is compared with the transferred shallow saliency, and if their cosine similarity falls below a threshold \(\gamma\) the attention is deemed to have drifted and the old prior untrustworthy, so \(t\) is declared a key timestep — no pruning, a full-token forward pass, and the true per-layer saliency written back to the cache. Otherwise \(t\) is an ordinary frame: the top-K visual tokens by the fused score are kept, and only those \(K\) tokens run through the remaining layers to produce the action. Key timesteps are the anchors of the whole mechanism: they pay the cost of a full forward pass in exchange for a cross-layer, pruning-uncorrupted ground-truth saliency that subsequent ordinary frames borrow repeatedly.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Instruction + current frame<br/>multimodal token sequence"] --> B["Future-aware saliency forecast<br/>transfer the reference frame's per-layer saliency"]
    B --> C["Two-stage saliency fusion<br/>shallow + forecast deep"]
    C --> D["Adaptive key timestep refresh<br/>triggered when similarity drops below the threshold"]
    D -->|key timestep| E["Skip pruning<br/>full-token inference, refresh the cache"]
    D -->|non-key timestep| F["Keep top-K tokens<br/>continue inference after pruning"]
    E --> G["Output action"]
    F --> G
    E -.->|refresh reference keyframe τ_ref| B

The order in the figure — future-aware saliency forecast, two-stage saliency fusion, adaptive key timestep refresh — matches the three key designs below, and is also the actual order of the decisions taken within one timestep.

Key Designs

1. Future-aware saliency forecast: transfer the historical key timestep's per-layer saliency to the current frame through semantic correspondence, filling in the deep-layer information that is invisible at pruning time

Pruning is irreversible and deep-layer saliency cannot be computed at the shallow layer, so the forecast needs a source that has already computed its deep layers and has not been corrupted by pruning. The paper therefore splits timesteps into two classes. At key timesteps \(\mathcal{T}_{\mathrm{key}}\) no pruning is performed at any layer; the full forward pass runs and the visual-token saliency vectors of every layer, \(\{\mathcal{S}^{\ell}_{\tau}, \ell = 0, \ldots, L-1\}\), are cached. An ordinary timestep \(t\) then takes the most recent key timestep earlier than itself as reference, \(\tau_{\mathrm{ref}} = \max\{\tau \in \mathcal{T}_{\mathrm{key}} : \tau < t\}\). Because the reference and current frames need not have tokens in one-to-one correspondence (the view moves, objects get occluded), the scores cannot simply be copied wholesale; instead, each visual token of the current frame is matched to the semantically closest token in the reference frame — cosine distance between hidden states serves as the measure of semantic proximity — establishing a cross-frame token correspondence, and the reference frame's per-layer saliency at that token is carried over to produce \(\{\hat{\mathcal{S}}^{\ell}_t\}\). Since what needs forecasting is the deep stage, only a set of deeper layers \(\mathcal{L}_d\) (e.g. the last few layers) is considered and averaged:

\[ \hat{\mathcal{S}}^{\mathcal{L}_d}_t = \frac{1}{|\mathcal{L}_d|}\sum_{\ell \in \mathcal{L}_d} \hat{\mathcal{S}}^{\ell}_t \in \mathbb{R}^{L_{\mathrm{vis}}} \]

The reason this works is precisely the semantic attention consistency observed above: attention is anchored to what a patch is, not where it is, so even after the gripper has moved a long way, the patches around it in the current frame are still semantically closest to the gripper patches of the reference frame and the transferred saliency still points at the right targets. Conversely, if the model's attention really followed spatial coordinates, this correspondence would collapse immediately — which is why the paper treats semantic consistency as the load-bearing assumption rather than an empirical trick. Compared with work such as SpecPrune-VLA that also estimates late-stage saliency, the decisive difference is that the prior comes from a full-token key timestep rather than an already-pruned subset, so an early pruning error is never amplified down the line.

(The symbols are re-typeset from the original; ⚠️ refer to the original paper.)

2. Two-stage saliency fusion: let the directly measured but short-sighted shallow score and the forward-looking but indirect deep forecast jointly decide what to keep

The transferred deep score sees the future, but it is second-hand information carried over from another frame, so an imperfect correspondence or a slightly stale reference injects noise. The current frame's shallow saliency is the opposite: measured on the spot, exactly right for the shallow layer, yet it is itself the short-sighted signal responsible for mis-pruning. Neither works alone, so a single scalar combines them into the final token-importance score:

\[ \mathfrak{s}_t = (1-\lambda)\,\mathcal{S}^{\ell_s}_t + \lambda\,\hat{\mathcal{S}}^{\mathcal{L}_d}_t \]

Here \(\lambda\) weights the foresight: as \(\lambda \to 0\) the method degenerates into pure shallow-layer pruning (the FastV / VLA-Pruner family), while \(\lambda \to 1\) trusts the cross-frame prior entirely. The top-K visual tokens by \(\mathfrak{s}_t\) are kept and the rest discarded, and the remaining layers are computed only over those \(K\) tokens. The fusion happens at the score level rather than the feature or token level: everything retained is still the current frame's own token, the reference frame injects no visual content, and hence the spatial layout that action regression depends on is left intact. It also adds no forward-pass overhead — only one weighted sum over two sets of already-computed scalars.

(The notation for \(\mathfrak{s}_t\) and the admissible range of \(\lambda\) are re-typeset from the original; ⚠️ refer to the original paper.)

3. Adaptive key timestep refresh: use cosine drift of shallow attention to swap out an outdated reference frame at subtask boundaries

The quality of the key timestep set \(\mathcal{T}_{\mathrm{key}}\) directly determines the quality of the forecast. The naive baseline is to sample key timesteps at a fixed interval, but semantic attention consistency does not hold unconditionally: the visualizations show that at boundaries between subtasks — switching from "pick up" to "place" — attention abruptly shifts from the manipulated object to the target container (frame 12 still on the object, frame 13 clearly biased toward the basket). At such a moment the old prior is telling the story of a different subtask, and a fixed interval may both straddle the transition point and waste several full-token forward passes during stable stretches. The paper's argument is that the validity of a prior correlates far more with whether two control timesteps belong to the same subtask than with their temporal distance, and whether a subtask has switched is readable from the attention distribution itself. A very cheap proxy signal therefore performs the detection: the cosine similarity between the current frame's shallow saliency \(\mathcal{S}^{\ell_s}_t\) and the shallow saliency transferred from the reference frame \(\hat{\mathcal{S}}^{\ell_s}_t\),

\[ \kappa_t = \frac{\left\langle \mathcal{S}^{\ell_s}_t,\ \hat{\mathcal{S}}^{\ell_s}_t \right\rangle}{\left\|\mathcal{S}^{\ell_s}_t\right\|_2 \left\|\hat{\mathcal{S}}^{\ell_s}_t\right\|_2} \]

When \(\kappa_t < \gamma\) an attention shift is declared and \(t \in \mathcal{T}_{\mathrm{key}}\): pruning is skipped, a full-token forward pass produces the action, and the true per-layer saliency overwrites the cache so that \(\tau_{\mathrm{ref}}\) advances to the current frame. Otherwise the original reference is reused for transfer and pruning. Compared with fixed-interval selection, this on-demand refresh both avoids error propagation at subtask boundaries (success rate rises from 95.8% to 96.4% in the ablation) and reduces the number of key timesteps because stable stretches no longer force a full-token forward pass (FLOPs fall from 2.236 T to 1.722 T and latency from 51.98 ms to 37.22 ms) — the step that makes accuracy and efficiency improve in the same direction rather than trading off. One caveat: \(\kappa_t\) detects the fact that the current attention deviates from the historical prior, not the reason for it. A forecast error and a genuine attention shift look identical to this statistic and both trigger a refresh — conservative from a safety standpoint, but frequent refreshes dilute the speedup.

A Worked Example

Take the long-horizon episode visualized in the paper (frames 0/7/12/13/15): the gripper moves right to grasp the soup and then left to place it. Walking through the decisions: frame 0 has no history and is treated as a key timestep — no pruning, a full-token forward pass, and the visual-token saliency of every layer written into the cache, after which \(\tau_{\mathrm{ref}} = 0\). Frames 1-12 are ordinary frames: for each visual token of the current frame, the semantically closest token in frame 0 is found by hidden-state cosine distance, its per-layer saliency is carried over and averaged over the deeper layer set, and the result is fused with the shallow saliency computed for the current frame; the top-K tokens are kept and only those run through the remaining layers. During this stretch the gripper moves from the middle to the right and then to the left — a large spatial displacement — but attention stays anchored to the gripper and the object, so \(\kappa_t\) stays above \(\gamma\) and the reference remains frame 0 with no refresh needed.

Frame 13 is different: the subtask switches from "pick up" to "place", the centre of attention moves from the object to the basket, the current shallow attention is clearly no longer consistent with the prior transferred from frame 0, and \(\kappa_{13} < \gamma\) triggers a refresh. Frame 13 is declared a key timestep, pruning is skipped, one full forward pass produces the action, and the cache is overwritten with frame 13's per-layer saliency, so \(\tau_{\mathrm{ref}} \leftarrow 13\). Frames 14 and 15 then continue as ordinary pruned frames referencing frame 13. Only frames 0 and 13 paid the cost of a full-token computation; every other frame ran in the pruned state, and the basis of every pruning decision contained both what the shallow layer currently sees and what the deep layers will attend to next.

(The frame indices and scenario are taken from Figure 3 of the original paper; the specific values along the way illustrate the flow.)

Loss & Training

SAFE-Pruner is training-free: it introduces no learnable parameters, defines no loss function, and modifies no architecture. At inference time it merely reads attention weights and truncates the token set at a shallow layer, so it applies directly to any existing VLA backbone (the paper instantiates it on OpenVLA, OpenVLA-OFT, CogACT, and π0.5) and stacks with orthogonal acceleration such as quantization (combined with HiF8 quantization it cuts FLOPs by 56.4% and GPU memory by 38.1% with essentially unchanged success rate).

Only four quantities need to be set: the fusion weight \(\lambda\), the drift threshold \(\gamma\), the deep layer set \(\mathcal{L}_d\) used for averaging, and the number of tokens kept per frame \(K\) (equivalently the pruning rate). The paper does not report concrete values for these in the main text, nor sensitivity curves for \(\lambda\) and \(\gamma\) (⚠️ refer to the original paper / appendix). The one necessary empirical convention is that key timesteps cost a full forward pass, so the reported latency is averaged over all control timesteps, key and non-key alike, and already includes every overhead introduced by the method — otherwise the comparison with baselines would not be fair.

Key Experimental Results

Main Results

The evaluation covers four VLA backbones (OpenVLA, OpenVLA-OFT, CogACT, π0.5). Baselines include the general VLM acceleration methods FastV, SparseVLM, and DivPrune, plus the VLA-specific efficiency methods VLA-Cache and VLA-Pruner, all tuned under the same setup. LIBERO comprises 4 suites × 10 tasks × 50 episodes = 2000 episodes, and latency is measured on an NVIDIA RTX 4090 (per-timestep inference latency of the VLM backbone). For brevity the table lists, per backbone, only vanilla, the strongest acceleration baseline, and Ours:

Model Method Spatial Object Goal Long Avg. SR (%) FLOPs (T) Latency (ms)
OpenVLA Vanilla 84.6 86.6 78.2 53.2 75.7 1.862 48.24
OpenVLA VLA-Pruner 82.0 84.4 77.8 52.6 74.2 0.793 36.47
OpenVLA Ours 82.2 84.8 77.6 52.0 74.2 0.742 32.18
OpenVLA-OFT Vanilla 98.4 98.2 96.4 94.2 96.8 3.970 70.25
OpenVLA-OFT VLA-Pruner 97.4 93.0 94.0 92.0 94.1 2.234 54.53
OpenVLA-OFT Ours 98.0 98.0 96.2 93.4 96.4 1.722 37.22
π0.5 Vanilla 98.2 98.0 98.0 91.8 96.5 2.115 35.28
π0.5 VLA-Pruner 95.8 97.2 95.6 88.0 94.2 1.583 32.64
π0.5 Ours 97.0 98.8 96.0 90.2 95.5 1.482 24.33

The gains are clearest on OpenVLA-OFT: FLOPs drop from 3.970 T to 1.722 T (-56.6%) and backbone latency from 70.25 ms to 37.22 ms (1.89x speedup), while average success rate only falls from 96.8% to 96.4% — 1.9 points above the best acceleration baseline (94.1%, VLA-Pruner). On SIMPLER with CogACT as the backbone, the Visual Matching setting reaches 74.5% average success (vanilla 74.8%) at only 37.4% of the baseline FLOPs, a 1.73x speedup that clearly beats FastV's 1.21x and VLA-Cache's 1.38x; Variant Aggregation reaches 61.9% (vanilla 61.3%) at 36.2% FLOPs retention, a 1.67x speedup. Notably, the accelerated model occasionally slightly exceeds the unpruned baseline, which the authors explain along the lines of VLA-Cache — pruning tokens irrelevant to the task concentrates the model on useful information.

Real-robot experiments run on the Astribot S1 dual-arm platform, with chassis and torso immobilized and control restricted to the 14-DoF arms and grippers; visual observations come from two wrist-mounted cameras and a head-mounted camera tracking the workspace centre. Task-specific π0.5 models are fine-tuned from 200-1000 VR-teleoperated demonstrations, each evaluated over 200 trials, with VLM backbone latency measured on an RTX 3090:

Method Pick and Place Throw Basketball Pack Doll Average FLOPs (T) Latency (ms)
π0.5 94% 77% 73% 81.3% 2.264 80.36
+ FastV 82% 69% 56% 69.0% 1.857 56.93
+ Ours 90% 81% 67% 79.3% 1.543 43.56

The real-robot result is more telling than simulation: while using fewer FLOPs (1.543 T vs 1.857 T) and lower latency (43.56 ms vs 56.93 ms) than FastV, the method is 10.3 points higher in average success rate (79.3% vs 69.0%) and only 2.0 points below the unaccelerated π0.5. The largest gap is on Throw Basketball (81% vs FastV's 69%), which requires precise grasping of small objects — exactly the setting most sensitive to visual detail and most prone to losing critical patches under shallow saliency. The long-horizon multi-stage Pack Doll task also improves from 56% to 67%.

Ablation Study

Four configurations on OpenVLA-OFT + LIBERO: vanilla; shallow-attention-only pruning with no forecast (w/o forecast); forecast enabled but with fixed-interval keyframes (w/o adaptivity); and the full pipeline.

Config Avg. SR (%) FLOPs (T) Latency (ms) Note
Vanilla 96.8 3.970 70.25 unaccelerated
w/o forecast 94.5 2.141 50.31 no future-saliency forecast, degenerates to shallow pruning, -2.3 points
w/o adaptivity 95.8 2.236 51.98 forecast kept but fixed-interval keyframes; +1.3 points, yet higher FLOPs than w/o forecast
Ours 96.4 1.722 37.22 adaptive keyframes deliver both the best success rate and the lowest cost

Key Findings

  • The forecast drives the accuracy gain; adaptive keyframes drive the efficiency gain. Removing the forecast (w/o forecast) drops success to 94.5%, almost exactly matching FastV's 94.5% and 2.141 T in the main table — confirming that "this method without forecasting" really does degenerate into a shallow pruning baseline. Adding the forecast recovers 95.8%, but because keyframes fire at a fixed interval, FLOPs are actually higher than w/o forecast (2.236 T vs 2.141 T): a fixed interval both wastes full-token forward passes and cannot dodge subtask boundaries. Adaptive refresh improves both at once.
  • The speedup is strongly architecture-dependent. On OpenVLA the same method pushes FLOPs from 1.862 T down to 0.742 T with almost no success loss (75.7% → 74.2%, matching VLA-Pruner at lower cost), whereas on OpenVLA-OFT it achieves both a higher absolute success rate (96.4%, 1.9 points above the best baseline) and the largest efficiency gain. OpenVLA-style step-by-step decoding has a much lower absolute ceiling to begin with (75.7% vanilla), so the headroom and perceptual sensitivity differ across backbones — speedup factors should not be compared in isolation.
  • Complementary to quantization. Combined with HiF8 quantization on OpenVLA-OFT, FLOPs drop 56.4% relative to vanilla and GPU memory falls from 17.83 GB to 11.03 GB (-38.1%), with success rate 96.6% essentially matching vanilla's 96.8%. Used alone, the method slightly increases memory (18.31 GB), showing that pruning saves compute rather than memory and that the two are complementary.
  • Shallow pruning fails far more visibly on real hardware. In simulation the gap between FastV and Ours is 1-3 points; on the real robot it widens to 10.3 points and concentrates in vision-critical scenarios (small-object grasping in Throw Basketball). This suggests simulators under-represent the perceptual sensitivity that matters here, and that real-robot validation is not optional for this class of pruning methods.

Highlights & Insights

  • Turning "unknowable future" into "transferable past" is the paper's most elegant move. Pruning happens in the shallow layer yet must answer for the deep layers, a temporal mismatch that is fundamentally unsolvable; rather than adding an auxiliary predictor (which would require training), the authors exploit the observation that attention within one task anchors to the same semantic entity across adjacent frames, converting the problem into cross-frame score transport while keeping the method at zero training cost.
  • Replacing "fixed interval" with "keyframe + on-demand refresh" generalizes well. Any online decision that relies on a historical prior — KV cache eviction, token budgeting for long-video VLMs, temporal ensembling over action chunks — can reuse the pattern: compare a cheap current signal against the prior and pay the expensive rebuild only once the deviation crosses a threshold. It reframes "how long until the prior expires" from a temporal question into a state question.
  • Fusing at the score level rather than the feature level is a deliberately restrained design. Only scalars are transferred; the current frame's tokens and their spatial positions stay untouched, so no visual content from the reference frame leaks into the action-regression input — a boundary that matters a great deal for layout-sensitive manipulation.
  • Training-free and orthogonally composable makes it practically valuable. No architecture change, no retraining, and it stacks with quantization (FLOPs -56.4%, GPU memory -38.1%), which means very low integration cost for an existing VLA deployment pipeline — often more useful than a headline speedup number on its own.

Limitations & Future Work

  • Hyper-parameters and sensitivity analysis are missing from the main text. The concrete values of \(\lambda\), \(\gamma\), the deep layer set \(\mathcal{L}_d\), and the kept-token count \(K\) are not given, nor are sweep curves for \(\lambda\) and \(\gamma\), so readers cannot judge robustness to these choices; if each backbone needs its own setting, the practical convenience of "plug-and-play" is reduced.
  • Drift detection says "something changed", not "why" or "whether the forecast was right". Since \(\kappa_t\) compares the current shallow attention with the transferred same-layer attention, a forecast error and a genuine attention shift are numerically indistinguishable: the former causes unnecessary refreshes (paying the full-token cost for nothing), while the latter can be missed if an incorrect forecast happens to line up. The paper reports neither the actual keyframe trigger frequency nor how it varies with episode length and subtask count, so the worst case (tasks with frequent switching) is hard to bound.
  • Baseline coverage on the real robot is incomplete. Only FastV and vanilla are compared there, with stronger VLA-specific baselines such as VLA-Cache and VLA-Pruner absent; there are also just 3 tasks, 200 trials each, on a single robot platform, so confidence intervals are wide (whether the 67% vs 56% gap on Pack Doll is stable is not reported).
  • Cross-frame correspondence relies on cosine nearest neighbours and is fragile to occlusion and view change. Matching by hidden-state cosine distance implicitly assumes that semantically similar tokens are visible in both frames. When a target is occluded by the arm, leaves the field of view, or the viewpoint changes sharply, the nearest neighbour can match the wrong patch, and the paper offers neither an experiment nor a quantification of this failure mode. A natural improvement is soft matching via attention weighting or optimal transport, with the matching confidence in turn modulating \(\lambda\).
  • The retention rate is fixed although the information each frame needs is not. A stable stretch could probably tolerate more aggressive pruning, while the frames around a subtask transition should be conservative. Since \(\kappa_t\) already characterizes how consistent the current frame is with the prior, it could drive a dynamic retention rate instead of merely a binary refresh trigger.
  • vs FastV: FastV decides at the second layer which visual tokens to drop and never revisits the decision; this paper acknowledges that shallow attention cannot represent deep-layer needs and fills that gap with the deep saliency of historical keyframes. On the real robot the two differ by 10.3 points in average success rate, with lower FLOPs and latency for this paper — the gap comes mainly from accounting (or not) for downstream layers' needs.
  • vs VLA-Cache: VLA-Cache mines cross-timestep token redundancy, caching and reusing visual tokens that change little; this paper caches the attention distribution (saliency prior) rather than the tokens themselves, so there is no risk of stale visual content being fed forward — only the prior of "whom to attend to" is transferred. On SIMPLER the gap is clear: 1.73x vs 1.38x (VM) and 1.67x vs 1.37x (VA).
  • vs VLA-Pruner: temporal-aware dual-level token pruning is the closest competitor; it ties on OpenVLA (74.2%) at a lower cost here (0.742 T vs 0.793 T) and is beaten by 2.3 points on OpenVLA-OFT. The difference is that VLA-Pruner's temporal modelling lives mainly in the pruning level and the time dimension, without explicitly forecasting the current frame's deep saliency distribution.
  • vs SpecPrune-VLA / action-aware dynamic pruning: these also recognize that late layers matter, but estimate late-stage saliency from a token subset already pruned in early layers, so a critical token removed at the shallow layer cannot be recovered; this paper's prior comes from full-token keyframes, which is the most essential accuracy difference.
  • vs SparseVLM / DivPrune: text-guided similarity sparsification and diversity-based selection respectively, both general-purpose VLM acceleration methods; this paper's signals are attention saliency plus cross-timestep temporal priors, a VLA-specific route. The experiments also show that transplanting general methods directly to VLA is markedly more costly (FastV loses 12.3 points on the real robot).

Rating

  • Novelty: ⭐⭐⭐⭐ The observable regularity of semantic attention consistency plus per-layer saliency transfer from historical keyframes is a clear new angle, though top-K pruning, keyframe caching, and cosine drift detection are all existing building blocks.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Four VLA backbones, two simulation benchmarks (LIBERO and SIMPLER), three real-robot tasks on the Astribot S1, plus explicit verification of orthogonality with quantization — unusually broad coverage for this class of work.
  • Writing Quality: ⭐⭐⭐⭐ The observation → mechanism → design argument chain is clear and the failure case (subtask transitions) is turned into design motivation rather than hidden; however key hyper-parameter values and sensitivity analysis are absent from the main text, and some equations are corrupted in the cached text and need checking against the original.
  • Value: ⭐⭐⭐⭐ Training-free, plug-and-play, and composable with quantization, which gives direct engineering value for real-time VLA deployment; marks are lost for incomplete real-robot baselines and for leaving the failure mode of cross-frame matching unexamined.