VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijacking

Conference: ECCV2026
Paper: ECCV Paper
Area: Robotics & Embodied AI
Keywords: vision-language-action models, visual proprioception, adversarial patches, cross-model transfer, safety evaluation

TL;DR

VLA-Hijack investigates visual self-localization representations as a robustness weakness in vision-language-action models, combining real-arm feature suppression with surrogate embodiment injection to reach 61.64% average task failure across 11 black-box LIBERO transfer models, with substantial architecture-dependent variation.

Background & Motivation

Vision-language-action models (VLAs) turn camera observations and language instructions into robot actions, so visual errors can change closed-loop control rather than merely affect textual answers. OpenVLA discretizes actions into tokens, UniVLA uses latent action representations, and CronusVLA introduces multi-frame continuous prediction; their output encodings differ even for similar tasks. Earlier patch studies often construct perturbations by increasing a surrogate model's action deviation, but a direction that affects one action head need not survive another architecture's control interface. A high task failure rate under white-box access therefore does not establish an equally strong cross-model threat.

The authors start from an observation: some action-space methods produce patterns resembling arms or grippers, while an ordinary arm image can also cause interference. They hypothesize that a model must infer its own arm's location, in addition to recognizing the target object, to translate spatial relationships into motion. Here, visual proprioception means inferring embodiment state from images; it is distinct from hardware proprioceptive sensors such as joint encoders or force sensing. The paper seeks a shared visual dependency upstream of different action heads instead of designing a separate objective for every action encoding.

The difficulty is that adding an arm-like region does not necessarily eliminate the role of the actual arm: a model may continue attending to both. The authors call this competition proprioceptive conflict and study attention to the real arm and acceptance of surrogate features as complementary processes. Attention visualizations, transfer experiments, and ablations support this interpretation, but it should not be taken to mean that every VLA contains a separately observable self-localization module. Core Idea: jointly perturb visual self-localization through real-arm feature suppression and surrogate embodiment injection, testing whether this shared representation is a robustness weakness across different action architectures.

Method

Overall Architecture

The study considers universal, untargeted local patches: the same pattern is used across observations and tasks, language instructions remain unchanged, and the outcome is task non-completion rather than a specified dangerous trajectory. The threat model permits access to one surrogate VLA's internal features and gradients, while target models expose neither internal parameters nor architecture information. Consequently, black-box access describes the transfer stage, not a claim that the patch can be obtained without any white-box model. Inputs include clean observations, associated instructions, and surrogate representations; the offline research stage additionally extracts attention, real-arm masks, and embodiment semantic references. Model parameters are not trained in this procedure; the patch changes instead, distinguishing this setting from training-data poisoning or model backdoors.

The framework uses Attention-Guided Proprioceptive Suppression to reduce the consistency of original visual representations, then introduces surrogate embodiment representations through Multimodal Proprioceptive Injection. The Alternating Injection Schedule switches between semantic and visual injection objectives while retaining suppression throughout. Although the paper calls the complementary objectives Phase I and Phase II, this does not mean permanently disabling one module before running the other in isolation. The resulting fixed patch is evaluated in the target model's visual input; the target continues producing actions under the original instruction, and the environment determines task success. The diagram separates surrogate-side research from target-side evaluation; it does not imply that the target runs segmentation or repeats optimization during execution.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Clean observations and instructions<br/>Surrogate features and embodiment references"] --> SUPPRESS["Attention-Guided<br/>Proprioceptive Suppression"]
    SUPPRESS --> INJECT["Multimodal<br/>Proprioceptive Injection"]
    INJECT --> SCHEDULE["Alternating Injection<br/>Schedule"]
    SCHEDULE -->|Offline surrogate optimization| PATCH["Fixed patch"]
    PATCH -->|Only visual input changes| EVAL["Target VLA closed-loop evaluation<br/>Task failure rate"]
    OBS["Test observations and original instructions"] --> EVAL

Key Designs

1. Attention-Guided Proprioceptive Suppression: reducing the original advantage of real-arm representations

If the objective only encourages an arm-like patch, the actual arm can still provide a more reliable localization signal. The authors therefore extract multi-head self-attention from the visual encoder's final layer and aggregate responses into a spatial saliency map. High-response locations receive larger penalty weights after normalization, and the method compares clean and perturbed features at corresponding positions. The suppression objective reduces weighted cosine similarity, pushing perturbed representations away from the original visual features rather than merely changing the patch region itself. This relies on contextual interaction throughout the encoder: a local input change can affect representations at other positions. It neither erases the actual arm's pixels nor directly changes the robot's physical location.

Weighting is motivated by the possibility that unselective feature deviation could be dominated by unstable background locations that do not determine control. Assigning larger weights to previously attended locations is intended to reduce this shortcut unrelated to embodiment localization. However, attention saliency is not equivalent to an arm segmentation mask: target objects and other important regions can also receive strong responses. Real-arm suppression is therefore the authors' mechanistic interpretation of feature changes, not a property guaranteed by the weighting definition itself. Figure 2 illustrates this interpretation through attention distributions, while the component ablation in Figure 5 tests whether suppression is associated with performance. These forms of evidence complement each other but do not establish replacement of a unique, explicit internal self-localization variable.

2. Multimodal Proprioceptive Injection: constraining both embodiment concepts and visual appearance

Weakening an original representation does not establish that the model will interpret another region as itself. The authors address this gap with semantic concept anchoring and visual prototype projection, both components of Multimodal Proprioceptive Injection. The semantic branch encodes embodiment descriptions into normalized reference embeddings and evaluates patch-token similarity to these references. For each patch token, its objective aggregates the most relevant top-k references instead of forcing every token to match one fixed word. This flexible matching allows different local textures to correspond to different embodiment concepts without compressing diverse mechanical structures into a single language label. The target is representational semantics, not an additional command or prompt inserted into the original task instruction.

Language concepts alone cannot fully describe the current robot's surface material, geometric details, and scene appearance, motivating the visual branch. The authors use SAM 3 to segment the arm in a clean observation, pool clean visual features within the mask, and normalize them into a visual prototype vector. This prototype provides a reference that encourages patch-region features to approach real-arm features in cosine similarity. Projection here means alignment in feature space, not reconstructing an arm or directly changing coordinates through geometric projection. The semantic branch supplies an abstract embodiment-category constraint, while the visual branch supplies an observation-dependent appearance reference; the authors intend their combination to improve transferability. This also introduces dependencies: segmentation quality and the fidelity of surrogate arm representations can affect the reference signal.

3. Alternating Injection Schedule: retaining suppression while switching reference signals

Semantic categories and specific visual appearances do not always favor the same local feature changes, so enforcing both objectives simultaneously may create competition. The paper switches between semantic and visual injection across iteration stages, retaining suppression throughout those switches. Thus, the alternation concerns injection modalities rather than a choice between suppression alone and injection alone. The schedule is intended to prevent one reference from dominating persistently while keeping weakened real-embodiment signals from regaining their original advantage in the other stage. Conceptually, it connects disruption of original localization cues with formation of a surrogate interpretation without depending on the target's action discretization. Table 5 supports alternation over the reported simultaneous-optimization configuration, but the gradient-conflict explanation is not directly quantified there through gradient angles or conflict rates.

A Worked Example

Consider a LIBERO evaluation in which the original language instruction requires moving an object and the clean image contains both the arm and target object. The surrogate-side process records clean features and attention and derives a visual reference from the real-arm region; this information is used only for offline research. Patch-induced representation changes are then constrained by suppression and whichever injection modality is active. After the offline process, the same fixed patch transfers to another VLA, while the language instruction still requests the original task. The target can use a different action head but continues producing actions through its own visual processing and control pipeline. If the robot stalls or handles the wrong object, the environment records task failure; these observations fit the localization-interference account but do not independently rule out other perceptual confusion. This example explains the study's information flow and evaluation boundary, rather than introducing additional attack steps or an operating procedure for physical equipment.

Loss & Training

Suppression reduces weighted similarity between clean and perturbed representations, semantic injection increases alignment with relevant textual references, and visual injection reduces cosine distance to an arm prototype. Optimization occurs only on the accessible surrogate; target VLAs neither undergo adaptive training nor provide internal gradients to that process. The main experiments use a patch covering 5% of input image area, an essential condition when comparing threat strength. That constraint limits visible modification area but does not imply imperceptibility or identical physical area at different camera distances. The cached extraction of Equations (2) through (10) contains incomplete operators or brackets, and the displayed Equation (9) cannot be directly reconciled with its following weight description. Accordingly, this note explains only the objective directions and alternation confirmed by prose, without presenting a guessed reconstruction as the authors' exact total loss. The failure-rate definition below is explicitly specified in the experimental setup and can be retained unambiguously.

Key Experimental Results

Main Results

LIBERO includes Spatial, Object, Goal, and Long suites, each with 10 tasks. Main experiments run 50 rollouts per task, yielding 500 episodes per suite; ablations use 10 rollouts per task, so their sampling budgets and precision differ. The metric is task failure rate, where a higher value means lower task-completion reliability for the evaluated system:

\[ \mathrm{FR}=1-\mathrm{SR}. \]

It includes failures already present without perturbation, rather than attack success measured only on clean-success episodes. Table percentages express this fraction as a percentage, and interpretation also requires the clean baseline for the same model.

Source and setting	Method	Transfer average FR (%)	Averaging scope
Table 1, page 10; within-simulation transfer	Arm image	19.56	11 models excluding the surrogate
Table 1, page 10; within-simulation transfer	UADA1-3	37.53	11 models excluding the surrogate
Table 1, page 10; within-simulation transfer	VLA-Hijack	61.64	11 models excluding the surrogate
Table 2, page 11; Real-to-Sim	Arm image	23.45	All 12 target models
Table 2, page 11; Real-to-Sim	TMA1-7	21.02	All 12 target models
Table 2, page 11; Real-to-Sim	VLA-Hijack	63.68	All 12 target models

Table 1 uses the OpenVLA-Spatial surrogate; the proposed method exceeds UADA1-3 by 24.11 percentage points in average transfer FR. Table 2 uses real BridgeData V2 observations and the OpenVLA-7B surrogate, but evaluation still occurs in LIBERO simulation; the gap over TMA1-7 is 42.66 percentage points. Real-to-Sim is not end-to-end testing on physical robots, and the two tables' averages should not be treated as a direct improvement over an identical target set.

Ablation Study

The following schedule ablation comes from Table 5 on page 12: OpenVLA-Spatial is the surrogate, transfer FR is reported across four UniVLA suites, and the ablation rollout budget applies. Schedule identifiers are retained to interpret the reported experiment, not as recommendations for patch-generation settings.

Config	Spatial FR (%)	Object FR (%)	Goal FR (%)	Long FR (%)	Average FR (%)
Simultaneous optimization, no alternation	69.00	98.00	19.00	79.00	66.25
Alternating period 3	74.00	98.00	20.00	78.00	67.50
Alternating period 5	88.00	100.00	28.00	81.00	74.25
Alternating period 10	80.00	100.00	23.00	85.00	72.00

The best average exceeds the no-alternation configuration by 8.00 percentage points, but the highest Long result occurs at another period; average optimality is not per-suite optimality. Figure 5 reports degradation when suppression or either injection branch is removed; the cache provides no reliably transcribable bar values, so exact decreases are not supplied. The prose on pages 13–14 cross-references Tables 4 and 5 in reverse: their actual captions identify Table 4 as injection weight and Table 5 as alternating period.

Key Findings

Transfer strength depends strongly on architecture: Table 1 reports 100.00% FR for UniVLA-Object but 12.40% for CronusVLA-Object, whose clean baseline is 7.60%. A substantial increase in overall average failure therefore does not imply nearly complete failure on every model or discovery of a universal failure mechanism for all VLAs. Table 3 changes the surrogate to UniVLA variants, with UniVLA-Object reaching 64.35% average transfer FR across 11 models; it does not provide a complete comparison with adapted versions of the same baselines. The paper states that existing action-discretization baselines are incompatible with UniVLA, limiting what this experiment establishes about superiority over alternatives.

Physical experiments are reported in Table 6 on page 14: T1 puts paper into a drawer, and T2 sorts and disposes of trash, with 20 trials per task. OpenVLA and the multi-view, flow-matching architecture \(\pi_0\) are both fine-tuned on collected real data; the table below follows the model columns strictly.

Task	OpenVLA clean FR (%)	OpenVLA perturbed FR (%)	\(\pi_0\) clean FR (%)	\(\pi_0\) perturbed FR (%)
T1	0	100	0.0	85.0
T2	20	100	5.0	100.0
Average	10	100	2.5	92.5

The prose on page 15 describes 100% and 92.5% as averages for T1 and T2, whereas Table 6 identifies them as cross-task averages for the two models. This note preserves the table's meaning and flags the prose inconsistency rather than merging the two interpretations.

Highlights & Insights

• The analysis targets visual dependencies upstream of heterogeneous action heads. For defense, this suggests that inspecting output action distributions alone may miss embodiment-representation errors that arise earlier.
• Suppression and injection are ablated separately, so arm-like patch appearance is not the sole evidence. The more useful question is when original localization cues lose their role in control and whether the system can detect that change.
• Within-simulation transfer, Real-to-Sim, and limited physical trials provide different levels of evidence. Reporting them separately explains coverage more accurately than combining everything into a single real-world success rate.

Limitations & Future Work

• Attention maps and failures support the authors' interpretation, but do not directly measure self-localization error or establish it as the sole mediator; phantom embodiment is a mechanistic hypothesis, not evidence of bodily awareness.
• Three principal simulation architectures and limited physical models do not cover all VLAs, especially systems with more different representations and sensor inputs; weaker CronusVLA results are an important boundary rather than an exception to hide in the average.
• Physical evaluation covers only two tasks with 20 trials each, and the reported table lacks confidence intervals. Coverage of lighting, viewpoint, distance, occlusion, and independent repetitions remains insufficient for general deployment conclusions.
• FR does not distinguish failure severity, such as stalling, picking the wrong object, or leaving a safe trajectory region. Higher FR cannot be converted directly into injury probability or controllability of a specified dangerous behavior.
• Visual–joint-state consistency checks, multi-view corroboration, and safe stopping on anomalies are reader-proposed defense evaluation directions; the paper establishes neither their effectiveness nor a safety guarantee.
• This note is intended for academic understanding of robustness; physical validation belongs in authorized, isolated environments with emergency stopping and supervision, not near people or during live service tasks.

Related Work & Insights

• Relation to UADA, UPA, and TMA: These experimental baselines primarily rely on action-space objectives, whereas VLA-Hijack targets visual features. This changes transfer dependencies but introduces assumptions about internal representation access and visual-reference extraction.
• Relation to OpenVLA, UniVLA, and CronusVLA: These systems represent different action-modeling approaches and serve both as evaluated policies and comparisons for shared visual dependence, rather than new robot policies trained by this paper.
• Relation to LIBERO and BridgeData V2: The former supplies closed-loop simulation tasks and success criteria, while the latter supplies real visual data; real-world data provenance does not automatically make subsequent simulation evaluation a physical validation.
• Defense research implication: Controlled evaluations could jointly report task success, visual self-localization error, and consistency between vision and proprioceptive sensors to test whether a failure chain is interrupted; this is a future research proposal, not a validated result of the paper.

Rating

• Novelty: 4/5. Embodiment-feature suppression and multimodal injection form a focused transfer-mechanism study, although causal evidence for a shared self-localization mechanism remains limited.
• Experimental Thoroughness: 4/5. The study includes 12 simulation models, surrogate changes, and physical trials, but is limited by small-scale physical tasks, baseline compatibility, and incomplete uncertainty reporting.
• Writing Quality: 3/5. The motivation is clear, but table cross-references, physical-average descriptions, and formula completeness in the available cache require careful checking.
• Value: 4/5. The work has clear value for robotic visual robustness evaluation, without establishing failure of every VLA or the availability of a reliable defense.