Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

Conference: ECCV2026
Paper: ECCV Official Page · PDF
Area: Agent / GUI Grounding
Keywords: coordinate-token confidence, pseudo-labels, test-time training, negative learning, GRPO

TL;DR

The paper selects a pseudo-label from multiple predictions using coordinate-token confidence, then uses CANL to penalize only predictions far from that anchor; after independent label-free test-time training on each benchmark, Qwen-2.5-VL-7B improves from 88.1% to 92.1% on ScreenSpot-V2 and from 24.9% to 33.8% on ScreenSpot-Pro.

Background & Motivation

GUI grounding converts an instruction such as "click to lock rotation" into a click location on a screenshot. Understanding a button's meaning is only part of the problem: the model must also place its prediction precisely within a dense interface. Supervised fine-tuning needs target-box annotations, while reinforcement learning with verifiable rewards also needs the true box to determine whether a click is correct. Although the optimization objectives differ, both routes depend on annotations. Improving a deployed model on unfamiliar applications and layouts therefore remains constrained by the cost of supervision.

Replacing labels with model predictions is an obvious possibility, but coordinates do not behave like discrete answers that can simply be voted on. Nearby points may belong to the same button, whereas the average of several plausible candidates may land on empty space. Full-response confidence can also be inflated by easy formatting tokens, hiding uncertainty in the actual coordinate digits. Even after selecting the most confident prediction, its surrounding "correct region" can miss the target or fail to match an elongated button. Reinforcing every nearby prediction may merely make the model increasingly certain about the wrong location.

The authors exploit an asymmetry in supervision quality: a target usually occupies only a small part of a large screen, so establishing that a point is correct is difficult, while identifying many incorrect points is comparatively easy. This is an empirical argument based on target sparsity, not a mathematical guarantee that every point far from a pseudo-label is wrong. Core idea: establish a relatively reliable spatial anchor with coordinate-token confidence, then update the policy only from negatives outside its neighborhood to avoid directly reinforcing potentially incorrect positive pseudo-labels.

Method

Overall Architecture

The input is a screenshot and an instruction; the model generates text containing click coordinates. During training, multiple responses are sampled for the same input. Coordinate-Confidence Anchoring selects a pseudo-label, and Normalized-Distance Rewards divide candidates into nearby and distant groups. CAL learns from both groups; CANL adds Negative-Advantage Filtering to retain only the negative policy-learning signal. The output remains a click point, without an additional external object detector or human reward model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Screenshot and instruction"] --> Sample["Sample 16 responses"]
    Sample --> Anchor["Coordinate-Confidence<br/>Anchoring"]
    Anchor --> Reward["Normalized-Distance<br/>Rewards"]
    Reward -->|CAL| Mixed["Positive-negative GRPO"]
    Reward -->|CANL| Negative["Negative-Advantage<br/>Filtering"]
    Mixed --> Update["Update policy and output click"]
    Negative --> Update

Here, "test-time" includes actual parameter updates: the model is adapted independently on each benchmark for one epoch before evaluation. This is not merely sampling several answers and selecting one during inference. Ground-truth boxes determine evaluation correctness but do not construct the adaptation rewards. The main results should therefore be read as label-free target-domain adaptation, not zero-shot generalization to entirely unseen test inputs.

Key Designs

1. Coordinate-Confidence Anchoring: separate predictable output formatting from uncertain location digits

Coordinate field names, punctuation, and explanatory text are often easy to predict. Averaging probabilities over the entire response lets these high-probability components dilute the signal that actually determines the click location. On ScreenSpot-V2, the authors observe that 71.5% of coordinate tokens have probabilities below 0.6, whereas 78.1% of non-coordinate tokens have probabilities above 0.9. They therefore average probabilities only over the tokens representing coordinate values. This is neither a product of full-sequence probabilities nor a geometric agreement score between coordinates.

Let \(x_{\mathrm{img}}\) and \(x_{\mathrm{ins}}\) denote the screenshot and instruction, \(y\) a candidate response, \(\mathcal C(y)\) its coordinate-token positions, and \(k\) the number of those positions. With the typography of the paper's Equation (2) normalized, the definition is:

\[ c_{\mathrm{coord}}(y\mid x_{\mathrm{img}},x_{\mathrm{ins}}) =\frac{1}{k}\sum_{i\in\mathcal C(y)} P(y_i\mid y_{<i},x_{\mathrm{img}},x_{\mathrm{ins}}). \]

The model generates a candidate set for the same input, selects the complete response with the highest score, and extracts its click point as the pseudo-label. The selected point is thus an actual model-generated candidate, rather than an average of several button positions that might not itself be clickable. Confidence serves as a ranking signal here, not a calibrated probability of click correctness. The subsequent analysis also shows that incorrect predictions can become more confident during training.

2. Normalized-Distance Rewards: turn one anchor into group-level feedback without a true box

A single pseudo-label is insufficient for group-relative optimization. CAL divides each candidate's horizontal coordinate by image width and its vertical coordinate by image height, then computes Euclidean distance to the identically normalized anchor. Candidates inside threshold \(\tau\) receive reward 1, and those outside receive reward 0. This is not a raw-pixel radius: normalizing the axes makes the threshold more comparable across resolutions, but on a non-square screenshot the region should not be interpreted as a fixed pixel-space circle.

The reward measures proximity to a point chosen by the model itself, without knowing the true button shape. If the anchor is correct but the target is elongated, some valid clicks can still fall outside the reward region. If the anchor is wrong, its neighboring positive samples may all be incorrect. Equation (4) is corrupted in the cached extraction, so the convention for a distance exactly equal to the threshold cannot be verified. This note describes the reward using the adjacent prose rather than supplying an unrecoverable piecewise equation as though it were intact.

3. Negative-Advantage Filtering: retain the signal for avoiding errors without directly rewarding doubtful positives

CAL uses GRPO's within-group reward standardization: candidates above the group's mean reward receive positive advantages, and candidates below it receive negative advantages. CANL keeps the same anchor and binary rewards but sets the advantages of reward-1 samples to zero, retaining only the negative advantages of reward-0 samples. With \(\mu_R\) and \(\sigma_R\) denoting the group's reward mean and standard deviation, the relationship in Equations (5) and (7) can be written as:

\[ A_i^{\mathrm{CAL}}=\frac{R_i-\mu_R}{\sigma_R}, \qquad A_i^{\mathrm{CANL}}= \begin{cases} 0,&R_i=1,\\ -\mu_R/\sigma_R,&R_i=0. \end{cases} \]

This expression requires \(\sigma_R>0\). The cache does not specify the numerical safeguard for a group with identical rewards, so an implementation detail cannot be inferred. Crucially, CANL computes advantages over the full candidate group before selecting negatives; it does not restandardize within a set containing only reward-0 samples. Doing the latter would eliminate reward variance and lose the learning signal relative to the original group.

Policy optimization still uses GRPO with clipped probability ratios and KL regularization. "Negative-only learning" means the task-reward policy term no longer uses positive advantages to increase the likelihood of doubtful answers. It does not mean all regularization disappears, or that zeroed-out answers can never change probability through shared parameter updates. The mechanism progressively reduces the relative probability of clearly wrong predictions; it does not acquire a supervisor that directly identifies the correct button.

A Worked Example

Figure 2 uses the instruction "click to lock rotation." Among the displayed candidates, [1796, 459] has coordinate confidence 0.42, above the displayed alternatives of 0.27, 0.37, and 0.33, and is selected as the anchor. The figure labels [1875, 466] as a nearby positive and [1743, 787] and [1953, 656] as negatives. These assignments are reported from the figure: the cache does not provide the screenshot dimensions needed to independently recompute normalized distances from the pixel coordinates.

Under CAL, the illustrated positives initially receive an advantage of approximately 1.2 and negatives approximately -0.7. CANL zeros the former and retains the latter. If the anchor happens to be wrong, the method at least avoids actively rewarding the candidates clustered around that wrong anchor. However, if the true target falls in the negative region, incorrect penalization is still possible. This is the distinction between more reliable supervision and perfectly reliable supervision.

Loss & Training

The backbones are the 3B and 7B versions of Qwen-2.5-VL, using the VLM-R1 framework and Flash Attention2. Each benchmark is trained independently for 1 epoch with learning rate \(10^{-6}\), sampling temperature 1.0, top-k 50, top-p 1.0, KL coefficient \(\beta=0.04\), and default distance threshold \(\tau=0.05\).

For each input, 16 responses are first sampled to construct the pseudo-label. CAL randomly downsamples 8 responses to compute advantages. CANL instead computes advantages over all 16 responses, then selects the 8 negative responses farthest from the pseudo-label for policy optimization. The cache does not specify what happens when fewer than 8 negatives exist; resampling or duplication should not be assumed. This sampling difference also means the performance gap between CAL and CANL cannot be attributed entirely to zeroing positive advantages.

Key Experimental Results

Main Results

The metric is accuracy of the predicted click falling within the ground-truth target box, in %. The table selects the aggregate 7B results from Tables 1, 2, and 3. UI-Vision includes only Element Grounding, not Layout Grounding or Action Prediction. Gains are absolute percentage points over the same backbone, not relative percentages.

Benchmark	Qwen-2.5-VL-7B	CAL-7B	CANL-7B	CANL Gain over Backbone
ScreenSpot-V1	84.9	88.6	89.2	+4.3
ScreenSpot-V2	88.1	92.1	92.1	+4.0
ScreenSpot-Pro	24.9	32.7	33.8	+8.9
UI-Vision	15.0	18.6	20.1	+5.1

CANL exceeds CAL by 1.1 percentage points on ScreenSpot-Pro and 1.5 on UI-Vision, supporting the value of negatives on difficult grounding tasks. The 7B methods nevertheless tie on ScreenSpot-V2. GUI-RCPO-7B scores 88.9% on V2 and 25.9% on Pro and also adapts on test sets, making its protocol more comparable. Comparisons with models trained using GUI annotations primarily illustrate annotation efficiency; differences in training data and exposure to evaluation inputs remain important.

Ablation Study

The following selected threshold-analysis rows are reproduced from Table 4, with all accuracies in %. The cached table does not explicitly identify the model size, and its default-threshold Pro results of 33.2/33.7 differ from the corresponding main-table results. They are therefore kept separate rather than assigned to a particular backbone's main experiment.

Method	Threshold	ScreenSpot-V2	ScreenSpot-Pro
CAL	0.01	87.7	33.3
CAL	0.05	88.9	33.2
CAL	0.1	86.9	30.7
CANL	0.01	87.5	30.2
CANL	0.05	88.5	33.7
CANL	0.1	88.5	33.5

These numbers support greater CANL robustness to larger thresholds, but not near-invariance across every threshold: its Pro accuracy changes from 30.2% at 0.01 to 33.7% at 0.05, a 3.5-point difference. CAL drops from 88.9% at the default threshold to 86.9% at 0.1 on V2, showing that expanding the pseudo-positive region can indeed hurt training.

A separate candidate-selection analysis provides another line of evidence. With 12 sampled responses on ScreenSpot-V2, coordinate confidence selects pseudo-labels with 83.9% accuracy, versus 78.6% for full-response confidence, a 5.3-point gap. With 2 responses, the corresponding values are 76.7% and 74.8%. These are pseudo-label selection accuracies, not final accuracy after CAL/CANL training.

Key Findings

• Generalization is not demonstrated solely through target-test-set adaptation. In Table 5, training the 3B model for one epoch on 5k randomly sampled GroundCUA instances gives CANL scores of 86.7%, 33.4%, and 16.7% on V2, Pro, and UI-Vision, against backbone scores of 82.1%, 16.1%, and 12.0%. This supports transfer from external data but does not establish the absence of every possible data overlap.
• Negative learning is not universally superior. On AndroidWorld's 116 tasks, using GPT-4o planning and SeeAct-V to separate planning from grounding, success rates are 25.0% for the backbone, 30.2% for CAL, and 29.3% for CANL. The grounding models were trained on V2, and the results show that relatively reliable positive pseudo-labels remain valuable.
• Increased confidence does not necessarily mean error correction. Both correct and incorrect pseudo-labels become more confident, and some errors far from the true target persist. A more concentrated prediction distribution establishes concentration, not by itself improved accuracy.

Highlights & Insights

• Estimate uncertainty over the task-bearing tokens. Rather than introducing a complex reward network, the paper removes the distortion caused by formatting tokens in the mean. The transferable principle is to identify the output components that determine the action and verify their discriminative value, rather than assuming full-response probability measures task correctness.
• Downgrade the pseudo-label from a positive answer to a spatial reference. CANL still needs an anchor, but does not require its neighborhood to supply trustworthy positives. This reduces dependence on precise positive supervision while retaining the assumption that the negative region rarely covers the true target.
• Examine reward quality alongside end-task outcomes. Candidate-selection analysis, threshold analysis, and AndroidWorld evaluation probe the anchor, the reward, and downstream utility, respectively. Together they explain the mechanism better than a grounding aggregate alone.

Limitations & Future Work

• The authors identify a capability ceiling from missing accurate positive feedback. Continually ruling out errors does not guarantee discovering the correct region, and erroneous pseudo-labels can grow more confident. Adding a small amount of verified positive feedback is a possible direction, but would change the fully label-free setting.
• The geometric premise is not universal. Large targets, elongated targets, and misplaced anchors can put valid clicks in the negative region. Adapting reward regions to element shape is a direction to test, not an implemented component of this paper.
• Evaluation protocols and causal attribution require care. The main results use per-benchmark test-time training, and CAL/CANL also differ in downsampling. A stricter follow-up should fix the candidate set and selection strategy while toggling positive-advantage zeroing, and clearly distinguish target-domain adaptation from unseen-domain generalization.
• The cached evidence has gaps. Equations (4) and (6) are corrupted in extraction, and the referenced Appendix A.5 is absent. Handling zero reward variance, insufficient negatives, and the discrepancy between Table 4 and the main tables cannot be verified further. This note does not invent those implementation details or describe the threshold results as completely stable.

Related Work & Insights

• Versus GUI-RCPO: Both use label-free test-time training, but GUI-RCPO relies on consistency of predicted regions, whereas this paper anchors supervision with coordinate-token confidence. The latter avoids treating predicted-box consistency directly as reliable supervision, while still retaining pseudo-label error.
• Versus GUI-R1 and UI-R1: These RLVR approaches compute rewards from annotations; this paper substitutes spatial-distance proxy rewards. The trade-off is less demand for new annotations at the cost of imperfect supervision and a potential capability ceiling.
• Versus TTRL and negative-learning work: The paper inherits test-time reinforcement learning and negative-only learning, contributing their connection to GUI coordinate confidence and sparse spatial structure. For another action space, the premise that negatives are more reliable than positives should be tested rather than assumed universal.

Rating

• Novelty: 4/5. Combining coordinate-level confidence with negative-advantage filtering fits GUI supervision noise well, although the underlying reinforcement-learning framework is not new.
• Experimental Thoroughness: 4/5. Four grounding benchmarks, external-data transfer, and online tasks provide broad evidence; the threshold-table setup and isolation of individual components could be clearer.
• Writing Quality: 3/5. The central argument is understandable, but the threshold-stability claim is stronger than the table supports; extraction damage separately limits verification of the equations.
• Value: 4/5. A practical route to reducing GUI annotation needs during adaptation, subject to parameter-update costs, incorrect anchors, and missing reliable positive feedback.