Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection¶

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Code: https://github.com/HashTAG00002/UI-Injection
Area: LLM Agent
Keywords: GUI robustness, visual grounding, semantic-level UI element injection, black-box search, cross-model transfer

TL;DR¶

The paper distracts GUI agents with real, content-harmless UI icons subject to spatial and semantic constraints, reaching a five-depth miss rate of 32.99% on UI-TARS-1.5-7B through cumulative multi-candidate search and using injected-icon click statistics to expose grounding weaknesses that content safety alignment alone does not address.

Background & Motivation¶

GUI agents must turn natural-language instructions into specific screenshot coordinates: understanding a request involving search, navigation, or a product entry does not guarantee reliable localization among similar buttons, repeated lists, and complex toolbars. As systems move from multi-module pipelines toward end-to-end vision-language models, grounding becomes increasingly important, and its errors cannot be assessed solely by whether a linguistic response follows safety policies.

Existing red-teaming evaluations often use pixel-level adversarial perturbations or malicious prompt injection. White-box perturbation methods require model gradients, limiting their direct applicability to commercial interfaces; prompt injection primarily examines whether an agent follows malicious instructions embedded in untrusted content. This paper investigates a different failure mode: can ordinary icons cause an incorrect action without asking the model to violate a safety rule, simply by changing the visual correspondence between an instruction and its target? The aim is to separate content safety from grounding reliability, not to establish that every existing defense fails.

To avoid counting target occlusion as a meaningful discovery, the method keeps the true target visible and constrains the similarity between injected icons and the target crop. The resulting question concerns semantic competition in an otherwise solvable interface, rather than destruction of the task's basic solvability. Core Idea: let a vision-language model describe which icon to add and where, realize the edit through independent retrieval and constraint checks, and use black-box click feedback to retain the most distracting screenshot across search depths.

Method¶

Overall Architecture¶

The inputs are the original screenshot, the current step instruction, and the ground-truth target box; the outputs are a constrained edited screenshot and the agent's click result. No new victim model is trained, and no victim gradients are accessed. However, the editor knows the task and the true target position, so black-box describes model access rather than an attacker with no task information.

The pipeline combines an Editor, an Overlapper, and a Victim. Task-aware proposals determine icon descriptions and placements; constrained icon retrieval turns those descriptions into auditable local overlays; click feedback and cumulative search determine which screenshot is carried into the next round. Separate L1/L2 criteria distinguish a general miss from an actual click on an injected icon, preventing the two outcomes from being conflated.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Screenshot, instruction, and target box"] --> EDITOR["Task-aware proposals"]
    EDITOR --> OVERLAY["Constrained icon retrieval"]
    OVERLAY --> SEARCH["Click feedback and cumulative search"]
    SEARCH -->|Continue search| EDITOR
    SEARCH --> METRICS["L1/L2 criteria"]
    METRICS --> OUTPUT["Edited screenshot and evaluation results"]

Key Designs¶

1. Task-aware proposals: understand the layout before describing local edits

The Editor defaults to Qwen3-VL-Plus. It reads the current screenshot, instruction, and normalized target box, then outputs a textual icon description and a normalized placement box. The authors do not generate a complete replacement screenshot: full-image synthesis could alter unrelated controls or introduce rendering artifacts, making failures difficult to attribute. Restricting the output to content plus placement preserves the main interface structure and makes the subsequent overlay reproducible.

The editor cannot inspect the icon pool directly, so whether a description retrieves a useful icon is a practical bottleneck. Across roughly 20 pilot samples, the paper reports an average retrieval cosine similarity of approximately 0.22 for naive descriptions, with normalized click displacement below 0.01 in over 95% of attempts. These are small-scale development observations, not full-test-set results. The final prompt organizes six layout strategies involving similar icons, adjacent rows or columns, navigation cues, positional offsets, cross-toolbar placement, and active-state indicators. Each call also receives candidate regions that avoid the target and a distinct diversity token to reduce duplicate parallel proposals. These choices make proposals responsive to the task rather than merely adding random interface clutter.

2. Constrained icon retrieval: make edits realistic without relying on occlusion

The Overlapper uses Qwen3-VL-Embedding to map descriptions and real UI crops into a shared space, retrieves an icon with FAISS cosine nearest-neighbor search, and loads its image from LMDB before resizing and compositing it into the requested region. The pool aggregates mobile, desktop, and multilingual GUI data; Figure 2 reports a final pool of 58,922 elements from eight sources. Offline processing includes size and aspect-ratio checks, alpha-coverage and sharpness filtering, and hash-based deduplication, reducing low-quality and repetitive candidates.

Retrievability does not imply acceptability. Equation (1) excludes edits that substantially overlap the target or nearly duplicate its appearance. The following legible form is reconstructed from the surrounding explanation because the cached equation has damaged formatting; verifiable numerical threshold values are not available in the cache.

\[ \operatorname{IoU}(\hat b,b^*)<\tau_{\mathrm{iou}},\qquad \cos\bigl(\phi(e),\phi(e^*)\bigr)<\tau_{\mathrm{cos}}. \]

Intersection over union between the placement box and the true target box limits spatial interference; embedding cosine similarity between the injected icon and the true target crop limits near-duplication. The latter is distinct from description-to-candidate similarity during retrieval: retrieval should match the description, whereas the non-triviality constraint prevents an almost exact copy of the actual target. Edits violating either condition are discarded and cannot count as valid injections.

3. Click feedback and cumulative search: test several candidates and retain one screenshot per depth

The search uses Depth × Pass@N. At each depth, several independent proposals start from the current screenshot, are overlaid separately, and are evaluated through Victim click predictions. If a candidate includes a valid edit and moves the click outside the target box, ordinary execution stops early. Otherwise, one best candidate is retained, including its previously added icons, for the next depth. This is greedy single-path search, not a full tree that retains every branch, and not independent sampling that resets to the original screenshot after each attempt.

Candidates are ranked lexicographically by five signals: attack success, normalized click distance from the target center, average cosine similarity of applied icons, the number of valid icons, and a pass index used to break ties. The distance is normalized by the screenshot diagonal and supplies a continuous partial-progress signal among unsuccessful candidates. Carrying the screenshot forward preserves existing visual competition so that the next depth can refine it instead of starting over. The paper is less explicit about the comparison pair underlying the average-cosine ranking term than about the two constraints, so no additional equation is supplied for that term.

The main setting allows five depths, three candidates per depth, and at most three proposed icons per candidate; filtering can reduce the accepted count. Both victim and editor calls scale with depth times candidate count, giving at most 15 candidate queries in this setting. This must not be confused with a guarantee that the final image contains 15 icons. Early stopping, constraint rejection, and the icon budget accumulated across attempts all affect actual consumption.

4. L1/L2 criteria: distinguish general misses from direct decoy clicks

L1 success means the predicted click falls outside the true target box, regardless of which element receives the click. L2 requires a click on an injected icon. Under the paper's target-preserving setup, L2 captures a more direct decoy-attraction phenomenon. ASR denotes the corresponding attack success rate, but the denominator matters: the main evaluation retains only samples that the particular victim answers correctly on clean screenshots, rather than reporting unconditional performance over all initial samples.

ASR@D measures whether L1 success has occurred within the allotted search depth. ASR@K instead budgets the cumulative number of accepted non-trivial icons across attempts, so the two axes are not interchangeable. The authors additionally run a full mode that continues after the first L1 success and records L1 and L2 rates on subsequent attempts. This conditional statistic tests whether decoy clicks recur, but it is neither the overall success rate across all samples nor a pure repeatability measurement obtained by evaluating one unchanged screenshot repeatedly.

A Worked Example¶

In Figure 1, the instruction identifies an apartment entry with a displayed price. The Editor examines the entry and surrounding toolbar layout and proposes local edits involving navigation, price-label, and search-related icons. The Overlapper retrieves real crops, applies the constraints, and retains valid overlays. The Victim then predicts a click on the edited screenshot, and the evaluator checks whether it leaves the target box and whether it lands within an added icon.

If a candidate has not yet induced a miss, the search does not discard all progress: it ranks the same-depth candidates by click feedback, preserves the most promising edited screenshot, and generates the next proposals on that image. A miss ends ordinary execution; only the dedicated consistency analysis continues through the remaining budget. This example illustrates cumulative state without assuming an unreported success depth or decoy-click probability for the figure.

Key Experimental Results¶

Main Results¶

The authors build a candidate set from OS-Atlas, SeeClick, AMEX, and ShowUI across mobile, desktop, and web interfaces. Clean-image grounding filters yield 885 valid instances, followed by exclusion of samples each victim already answers incorrectly. The paper reports coverage of 19 models from eight families; the table below includes only results directly verifiable in main-text Table 1. UT-opt. and GO-opt. refer to attacks optimized against UI-TARS-1.5-7B and GUI-Owl-7B, respectively.

Victim	UT-opt. L1-ASR, D=5	GO-opt. L1-ASR, D=5	Random-injection L1-ASR, approximate K=15 comparison
UI-TARS-1.5-7B	32.99%	34.43%	7.58%
GUI-Owl-7B	51.65%	50.96%	23.36%
GUI-Owl-1.5-8B	28.99%	28.86%	7.56%
Qwen3-VL-8B	31.70%	32.87%	8.43%
Claude-Sonnet-4.6	22.24%	21.91%	3.23%

Random injection uses memoryless sampling and requires zero pixel overlap between placement boxes and the true target, but the paper explicitly states that it performs no semantic check. It therefore should not be assumed to use the same semantic constraints as the full method. The paper substitutes icon-budget results at K=3D for the depth comparison, so this table also does not establish perfectly matched query budgets. Claude's 22.24% versus 3.23% is approximately a 6.9-fold difference, subject to that approximation; clean-image eligibility also varies between attack rows.

Ablation Study¶

Main-text Table 5 supplies a more discriminating victim-query-budget-matched comparison. The following results all use the UI-TARS-1.5-7B victim and the UT-opt. column, with the same Overlapper, icon pool, non-triviality constraints, and D=5, pass@3 setting.

Config	L1-ASR	L2-ASR	Factor examined
LLM-Bank-Square-UI	18.0%	8.4%	Fixed descriptor bank and coarse-to-fine spatial search
GPTFuzz-UI	18.3%	8.6%	Descriptor mutation and corpus search
Random-BBox	27.4%	16.3%	Strategic descriptions retained, placements randomized
Full method, Qwen3-VL-Plus editor	33.0%	23.7%	Task-aware descriptions, placement, and cumulative search

The full method exceeds Random-BBox by 5.6 percentage points in L1 and 7.4 points in L2, supporting the importance of placement. Its L1 gaps over the two search baselines are 15.0 and 14.7 points, respectively. However, these baselines change proposal generation and search together, so the entire difference cannot be attributed solely to cumulative carry-forward. Table 5 rounds to one decimal place, explaining 33.0% versus 32.99% in Table 1.

Key Findings¶

The post-first-success statistics in main-text Table 3 provide further evidence about why clicks go wrong. The following selection uses same-victim optimization; the denominator is the set of attempts after the first L1 success, not the original 885 instances.

Victim	Attack	Post-first-success L1	Post-first-success L2
UI-TARS-1.5-7B	Random injection	57.94%	0.75%
UI-TARS-1.5-7B	UT-opt.	58.20%	22.73%
GUI-Owl-7B	Random injection	35.42%	0.45%
GUI-Owl-7B	GO-opt.	52.14%	15.95%

For UI-TARS, the two methods have nearly identical conditional L1 rates but differ by 21.98 percentage points in conditional L2. This supports the interpretation that strategic icons attract clicks more directly than a comparison of general miss rates alone could show.

Main-text Table 4 also shows that changing only the editor moves UI-TARS UT-opt. L1/L2 from 33.0%/23.7% with Qwen3-VL-Plus to 42.4%/30.7% with Claude-Sonnet-4.6. Attack strength therefore depends on editor capability; the default configuration does not establish an absolute upper bound on a victim's vulnerability.

Highlights & Insights¶

Separate harmless content from reliable action. An ordinary icon can alter the visual referent of an instruction, so safety evaluation must inspect whether the action is correct, not merely whether the output contains prohibited content.
Use local, auditable edits. Real crops, explicit placements, and non-triviality constraints make modifications easier to trace. Their value is reducing unrelated changes introduced by full-image generation, not proving perfect equivalence to real application rendering.
Use L2 to explain what L1 misses. Random injection still produces many conditional misses after its first success but almost never directs clicks to the added icon. The metrics therefore capture meaningfully different outcomes.

Limitations & Future Work¶

Strong task-information assumptions. The editor knows the instruction and ground-truth target box, so the results cannot be transferred directly to settings lacking that information. Screenshot overlays do not establish deployment of interactive controls in a real application, much less inevitable failure of an entire multi-step task.
Eligibility and budget conventions limit comparisons. Clean-image capability determines eligibility, and denominators differ across models and attack rows. Qwen2.5-VL-7B qualifies on only approximately 11% of candidates, making its high ASR unsuitable for an unconditional robustness ranking of all models.
Causal language needs restraint. A click on a decoy provides fairly direct behavioral evidence, but conditioning on first success and continuing to edit does not replace a counterfactual removal experiment on a fixed screenshot. Persistent attraction also does not imply persistent control across sessions.
Source verification is bounded. The local cache ends with the references and does not contain the cited Appendices B/C. Numerical constraint thresholds, complete 19-model results, and search-width ablations are therefore not supplied here. The authors suggest cross-modal consistency auditing and attention-region verification as defenses, but the main text reports no measured gains for these defenses.

Versus prompt injection. This method does not rely on malicious text telling the model to replace its goal; it examines competition among interface semantics. That is a different robustness dimension from instruction-source isolation, not evidence that prompt-injection research has lost its relevance.
Versus TAP. The authors borrow structured black-box search but change the search state to a cumulatively overlaid screenshot and retain only one best branch. Persistent visual elements are central to that adaptation.
Versus Square Attack, Sparse-RS, and GPTFuzzer. The Table 5 methods are adaptations to discrete GUI icons and click feedback, not the original methods' performance on their native tasks. Their names should not obscure the changed task and feedback signal.
Versus SeeClick, OS-Atlas, and UI-TARS. These works supply grounding capabilities, data, or victim models; this paper asks how local interface changes disrupt grounding that already succeeds on clean images. Future defenses should report clean-image capability, miss rates, and decoy-click rates together, avoiding apparent robustness obtained simply by refusing to act.

Rating¶

Novelty: 4/5. Constrained semantic distraction with harmless real UI elements defines a clear problem, although the search combines existing models with greedy mechanisms.
Experimental Thoroughness: 4/5. The main text includes multiple models, editor substitutions, query-budget-matched baselines, and L2 analysis, but approximate budget comparisons and conditional denominators complicate interpretation, and the appendix cannot be verified from this cache.
Writing Quality: 4/5. Module responsibilities and algorithm flow are clear, while some claims of near-perfect transfer and causality are stronger than the evidence strictly establishes.
Value: 4/5. The work provides an interpretable evaluation framework for GUI-agent grounding robustness; deployment evidence still requires real interactive environments and defense experiments.