Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Conference: ECCV2026
Paper: ECCV Paper
Project: IGAR
Area: Robotics & Embodied AI
Keywords: vision-language-action models, linguistic grounding, contradictory instructions, attention sinks, train-free recalibration

TL;DR

ICBench exposes linguistic blindness when VLAs complete the original task despite scene-inconsistent instructions, and train-free attention recalibration with IGAR reduces this erroneous execution, lowering π0's success rate on LIBERO-Goal spatial contradictions from 76.4% to 36.4%, although improvements do not extend to every model and condition.

Background & Motivation

Vision-language-action models (VLAs) map images and natural-language instructions directly to robot actions and are commonly evaluated by manipulation success. However, a tabletop layout may already suggest the most frequent training behavior: when a bowl and a plate are visible, the model places the bowl on the plate. Even if the instruction changes to picking up a white bowl while only a black bowl is present, the robot may still complete the original black-bowl task. A high success rate, normally considered desirable, then exposes the failure of language to constrain actions.

Evaluation with valid instructions cannot readily distinguish understanding the instruction from inferring the task through visual priors. This paper therefore fixes the scene and environment dynamics while minimally modifying object attributes, destination attributes, or spatial relations to make the instruction unsatisfiable. This isolates whether semantic constraints change actions without adding perceptual difficulty through camera or object-position changes. Nevertheless, failure under contradiction establishes behavioral sensitivity rather than semantic refusal by itself: control degradation and random failure remain alternative explanations.

The authors connect visual dominance to attention sinks in Transformers: unusually activated tokens may absorb attention and diminish the influence of words specifying objects and constraints. Instead of collecting counterfactual demonstrations and fine-tuning the policy, the paper attempts to recover language capabilities already present in the model. Core Idea: expose linguistic blindness through contradictory instructions in fixed scenes, then selectively reallocate sink attention to non-sink instruction tokens at inference time so that language can influence action generation again.

Method

Overall Architecture

The paper contains a diagnostic benchmark, ICBench, and an intervention, IGAR. ICBench is an external evaluation protocol rather than a runtime contradiction detector. IGAR likewise does not first classify an instruction as executable or impossible; it directly modifies attention during the existing VLA forward pass.

The inputs remain the current observation and user instruction, and the original action module still produces the output. IGAR performs Attention Sink Detection, Grounding Head Selection, and Text Attention Redistribution in sequence while keeping model weights frozen. Every intervention in the diagram occurs at inference time, with no additional training supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Observation + instruction<br/>Frozen VLA forward pass"] --> B["Attention Sink Detection"]
    B --> C["Grounding Head Selection"]
    C --> D["Text Attention Redistribution"]
    D --> E["Original action module<br/>Robot action output"]

ICBench includes four contradiction types. V1 replaces an operand attribute with one absent from the scene; V2 adds an inconsistent destination attribute; V3 changes both operand and destination attributes; V4 replaces a spatial relation. Changing a black-bowl-on-plate instruction to a white-bowl-on-black-plate instruction is V3, provided the specified objects are absent from the fixed scene. A spatial substitution must also be genuinely unsatisfiable in that scene; changing “on” to “under” is not automatically a contradiction in every environment.

Evaluation continues to check completion of the original task, rather than treating the contradictory instruction as another reachable goal. The paper defines the Linguistic Grounding Score (LGS):

\[ \operatorname{LGS}(\tilde{\ell})=\operatorname{SR}(f_\theta,\ell)-\operatorname{SR}(f_\theta,\tilde{\ell}). \]

The first term is original-task success under a valid instruction; the second is original-task success despite a contradictory instruction. When SR is expressed as a percentage, LGS is a percentage-point difference. A value near zero means changing the instruction barely affects execution, while a negative value means measured success is higher under contradiction. This definition does not directly measure semantic understanding or assign hovering and empty grasps an independent safety label.

Key Designs

1. Attention Sink Detection: identify localized spikes instead of treating every high-magnitude token as anomalous

IGAR starts from intermediate hidden states and examines absolute activations across tokens for each feature dimension. The central comparison is the maximum activation against the mean for that dimension. Uniformly large values suggest widespread activation; a few exceptionally large tokens fit the localized-sink hypothesis. The paper first lists a per-token RMS norm, but the subsequent explicit selection rule uses this cross-token spike ratio rather than RMS ranking alone.

Specifically, dimensions whose spike ratio exceeds 3.0 are retained, and the top-5 are selected. A token becomes a sink if its maximum absolute activation over the selected dimensions exceeds 20. Token modality then partitions the detections into visual and text sinks. This requires only the current forward-pass state and no anomaly labels. However, it detects activation structure, not the semantic incorrectness of a particular noun, color word, or preposition.

2. Grounding Head Selection: retain visual interaction without selecting positions overwhelmed by visual sinks

Selection operates on an attention-head/query-position pair within a layer, rather than simply on a whole head or layer; eligible query positions lie beyond the image region. Each candidate must satisfy two conditions: attention assigned to visual sinks must constitute no more than 0.4 of total visual attention, and total visual attention must be at least 0.01. The first condition excludes combinations already dominated by structural sinks, while the second excludes nearly text-only combinations with little cross-modal interaction.

The inequality direction matters. IGAR does not select the most severely visual-sink-dominated heads for forced correction; it strengthens instructional influence where useful visual interaction remains. When no visual sinks are detected, the first condition passes automatically, but the minimum visual-attention requirement still applies. This aims to focus intervention on potentially meaningful cross-modal interactions instead of indiscriminately modifying every attention pathway needed for action decoding.

3. Text Attention Redistribution: return the text-sink budget to ordinary instruction tokens

For selected head/query pairs, the paper attenuates sink attention using a retention factor of 0.6 and distributes the released budget among non-sink text tokens in proportion to their original attention. A distinction is necessary between the overview and the readable equations: although the paper repeatedly emphasizes vision-dominant sinks, Equation (7) explicitly sums over text sinks, and Equation (8) sends the budget to non-sink text tokens. The mechanism should therefore not be presented as an explicitly established transfer of total visual-token attention into text tokens.

The core relationship in Equations (7)–(8) is retained below. \(S_T\) denotes text sinks, \(T_{ns}\) denotes non-sink text tokens, \(A^h_{q,j}\) is the corresponding attention weight, and \(p=0.6\):

\[ \Omega_{hq}=(1-p)\sum_{j\in S_T}A^h_{q,j},\qquad A'^h_{q,j}=A^h_{q,j}+\Omega_{hq}\frac{A^h_{q,j}}{\sum_{j'\in T_{ns}}A^h_{q,j'}+\epsilon},\quad j\in T_{ns}. \]

Proportional redistribution preserves the model's existing ordering of textual attention without training another word-importance scorer. Consequently, a critical constraint word receiving almost no attention does not automatically receive uniform or maximal weight; without text sinks, the released budget is zero. The role of visual sinks in selection must be distinguished from that of text sinks in the budget equation. The paper does not provide enough detail to infer additional cross-modal mass-conservation rules or every implementation branch.

A Worked Example

Suppose the scene contains only a black bowl and an ordinary plate, but the user requests putting a white bowl on a black plate. ICBench labels this V3 and continues to evaluate completion of the original bowl-placement task. Moving the black bowl successfully does not establish satisfaction of the new instruction. IGAR instead reads the current activation spikes and attention, applying redistribution only to head/query pairs that pass both selection conditions.

Consider illustrative weights, not measurements from the paper: a pair allocates 0.20 attention to text sinks, 0.10 and 0.30 to two non-sink text tokens, and 0.40 to all remaining tokens. Retention by 0.6 reduces the text-sink allocation to 0.12, releasing 0.08. Ignoring the numerical stabilizer, the two text tokens receive 0.02 and 0.06 in a 1:3 ratio, becoming 0.12 and 0.36. The increased semantic influence then propagates through the original action network rather than invoking a new stop controller.

This example explains how the attention budget changes, not why the word specifying white must be a recipient or why the robot must stop. Evaluation asks whether original-task completion decreases under contradiction while being retained under valid instructions. Both conditions must be examined together.

Loss & Training

IGAR introduces no loss function, gradient updates, or additional training data. For π0.5 and OpenVLA-OFT, intervention spans the initial 16 layers with shared detection thresholds and selection parameters rather than task-specific tuning. The sensitivity analysis separately varies attenuation strength, head-selection boundaries, and the number of intervened layers.

π0's continuous flow-matching generation pathway receives special treatment. Section 5.1 describes a discrete contrastive formulation with a trigger threshold of 0.15 and an interpolation limit of 0.50, but does not expand the full computation in the paper's main text. Consequently, π0's results should not be interpreted as establishing an attention implementation identical to those of the other two models. The paper specifies the initial 16 layers in one place and invokes mid-level semantic fusion to explain its configuration elsewhere; exact layer indexing requires checking the implementation.

Key Experimental Results

Main Results

Experiments cover 30 simulated manipulation tasks across the LIBERO Spatial, Object, and Goal suites, with 50 independent rollouts per task variant. The main table below selects Goal V1 and V4. SR measures completion of the original task despite contradictory instructions, so lower is better; valid instructions are not the conditions being compared here.

Model	Goal contradiction	Baseline SR (%)	IGAR SR (%)	Baseline LGS	IGAR LGS
π0	V1: operand attribute	90.2	46.4	5.6	49.4
π0	V4: spatial relation	76.4	36.4	19.4	59.4
π0.5	V1: operand attribute	93.8	90.2	3.8	7.4
π0.5	V4: spatial relation	93.6	96.2	4.0	1.4
OpenVLA-OFT	V1: operand attribute	97.8	66.4	0.2	31.6
OpenVLA-OFT	V4: spatial relation	90.2	58.4	7.8	39.6

Values come from original Tables 1 and 2. Table 2 anchors LGS to the unintervened valid-instruction SR in Table 1: for π0, 95.8 − 36.4 = 59.4. Using the intervened valid-instruction SR of 95.6 from Table 3 would instead yield 59.2. The table's original convention is retained here rather than conflating the two definitions.

Model	Mean valid-instruction SR: baseline (%)	Mean valid-instruction SR: IGAR (%)	Change (percentage points)
π0	97.1	96.7	-0.4
π0.5	97.8	96.3	-1.5
OpenVLA-OFT	98.0	98.5	+0.5

The second table reports the three-suite averages from original Table 3. On Object alone, π0.5 drops from 98.4% to 94.4%, a 4.0-percentage-point decrease, so performance cannot universally be described as unaffected. OpenVLA-OFT latency on a single NVIDIA RTX 5090 over 50 measurements rises from 98.7 ± 2.0 ms to 105.6 ± 0.5 ms. The authors report approximately +7.0 ms; subtracting the displayed means gives 6.9 ms.

Ablation Study

The following values come from original Table 4: π0.5 on LIBERO-Goal under ICBench V1, not the π0/V4 condition with the largest improvement above. Attn-to-Text is the paper's reported text-attention quantity. Its aggregation across layers and heads is not explicitly defined in the text and should not be interpreted as a separate language-accuracy measure.

Config	Attn-to-Text	LGS	LGS difference from full IGAR
Baseline	0.08	3.8	-3.6
Full IGAR	0.15	7.4	0.0
Without sink detection	0.11	4.6	-2.8
Without head selection	0.10	4.2	-3.2
Detection only	0.08	3.8	-3.6

Key Findings

• Detection alone does not change behavior; redistribution is necessary. Removing head selection has a slightly larger effect than removing sink detection in this ablation, but this does not establish a universal contribution ranking across models.
• Gains depend strongly on architecture. π0 improves substantially on Goal, whereas π0.5 deteriorates on Goal/V4; its contradictory SR on Spatial/V4 also rises from 97.6% to 99.6%. The paper's description of consistent reduction is stronger than its tabulated evidence.
• Figure 4 reports that overly strong or weak attenuation is suboptimal on OpenVLA-OFT/Goal, and intervention beyond 24 layers can damage deep action representations. Exact curve values cannot be reliably extracted from the text cache, so no numerical table is invented.
• Real-world evaluation uses a Franka Research 3, two Intel RealSense D435 cameras, and π0 for placing a blue cube in a drawer. It shows hovering and empty grasps under contradictions but does not provide sufficient repetition counts or success-rate statistics.

Highlights & Insights

• Holding images fixed while changing language separates manipulation competence from adherence to linguistic constraints. This diagnosis complements ordinary success rates but does not replace safety metrics.
• Selecting head/query pairs before recalibration is more granular than globally strengthening language. Ablations support the joint role of selection and redistribution rather than attributing gains to activation visualization alone.
• A train-free correction suggests existing policies retain language signals that can still be used. Its value is a low-cost intervention, although gains remain constrained by architecture and the original attention distribution.

Limitations & Future Work

• Boundaries presented by the authors: the results discussion acknowledges smaller gains for π0.5, and sensitivity analysis shows that excessive intervention can damage representations. The paper has no dedicated systematic limitations section.
• Assessment in this note: LGS cannot distinguish deliberate refusal from control failure, and empty grasps are not inherently safe. Future evaluations should separately measure refusal correctness, dangerous contact, recovery behavior, and false refusal of valid instructions.
• Assessment in this note: π0 adaptation, text-attention aggregation, and sink-budget descriptions leave reproducibility gaps. Changes in attention and success rates alone do not establish a complete causal account of semantic understanding.
• Assessment in this note: three LIBERO suites and one physical task do not cover open vocabulary, unusual but executable instructions, or long-horizon manipulation. Executable counterfactual controls could rule out a degenerate policy that stops whenever language is unfamiliar.

Related Work & Insights

• vs LIBERO / LIBERO-PLUS: the former supplies a manipulation benchmark, while the latter studies robustness to perturbations including instructions. ICBench deliberately introduces unsatisfiable semantic contradictions, turning high original-task success into a diagnostic warning.
• vs CAST / CounterfactualVLA: these approaches improve instruction following through counterfactual training data or training mechanisms. IGAR leaves weights unchanged and intervenes in the attention of an existing policy at inference time.
• vs SayCan: SayCan constrains action selection by combining skill values with language planning. IGAR modifies internal attention in an end-to-end VLA without explicit skill-feasibility scoring or a refusal module.
• vs attention-sink mitigation: the paper transfers LLM/VLM sink analysis to action generation. Its central validation should concern whether linguistic constraints change physical behavior, not merely whether attention maps look more intuitive.

Rating

• Novelty: 4/5. Combining contradiction diagnosis with inference-time VLA attention intervention is well targeted, although sink redistribution builds on prior work.
• Experimental Thoroughness: 3/5. Three models, 30 tasks, ablations, and a physical demonstration provide breadth, but safety statistics and uncertainty analysis remain limited.
• Writing Quality: 3/5. The problem is intuitive, but some overall claims, budget descriptions, and implementation details require tighter alignment.
• Value: 4/5. The paper warns against equating manipulation success with linguistic grounding and offers a low-cost intervention for further investigation.