Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents¶

Conference: ECCV 2026
arXiv: 2606.31270
Code: https://github.com/snow10072740/Learning_from_Failure
Area: Multimodal VLM / Agent
Keywords: Failure-driven self-improvement, Inference-time optimization, Computer-use agent, GUI agent, Code patch

TL;DR¶

This work proposes a failure-driven inference-time self-improvement loop: an LLM acts as a meta-controller to analyze the failure trajectories accumulated by an agent in GUI tasks, identifying four categories of bottlenecks: grounding errors, capability gaps, insufficient knowledge, and redundant loops. It designs corresponding inference-time optimization strategies for each (visual search / terminal execution / knowledge support / repetition warnings), which are injected into the agent workflow as code patches. This approach improves the success rate of OpenCUA-72B on OSWorld from \(42.3\%\) to \(48.9\%\) (\(+6.6\) points) with zero training overhead, requiring only a \(\sim 8\%\) increase in inference computation.

Background & Motivation¶

Background: Computer-use agents utilize multimodal large language models (VLMs) to operate computers and complete complex tasks like booking tickets or making slides. These systems require a large volume of high-quality interaction trajectory data for training and improvement. Because human annotation of trajectories is expensive and not scalable, the mainstream approach is to let the agent execute tasks autonomously in a verifiable environment (e.g., OSWorld), use an automatic verifier to judge whether the trajectory is correct, and keep successful trajectories for iterative supervised fine-tuning (SFT). This forms a self-improvement loop of "Agent \(\rightarrow\) Environment \(\rightarrow\) Successful Trajectories \(\rightarrow\) Fine-tuning \(\rightarrow\) Improved Agent". Representative works like OpenCUA, UI-TARS, and Mobile-Agent adopt this paradigm.

Limitations of Prior Work: This success-driven loop suffers from a fundamental waste: a large number of failure trajectories are directly discarded. Building verifiable environments itself requires formidable human engineering effort (which, though cheaper than human trajectory annotation, is far from free), meaning discarding failure trajectories wastes more than half of this investment. More importantly, failure trajectories themselves contain rich diagnostic information—each failure trajectory is a diagnostic sample pointing out exactly where the agent failed to ground correctly, which operations repeatedly failed to take effect, and what knowledge was beyond the model's training data. These diagnostic signals are more informative than successful trajectories.

Key Challenge: On one hand, the engineering efforts to build verifiable environments must be fully utilized and not wasted; on the other hand, failure trajectories contain rich diagnostic information but have consistently been discarded.

Goal: To explore a complementary failure-driven self-improvement loop that extracts value from failure trajectories at inference time (rather than during training) to achieve continuous agent improvement with zero additional training cost.

Core Idea: Using an LLM as a meta-controller to analyze failure trajectories, diagnose failure modes, propose inference-time solutions, and automatically generate code patches. After lightweight human verification (with average code modifications under \(3\%\) of the lines), these patches are injected into the agent's workflow, enabling the agent to improve its inference-time behavior without retraining.

Method¶

Overall Architecture¶

The paper proposes a Failure-Case Loop that is complementary to the existing success-driven loop. The core process is divided into three stages:

Stage 1 — Failure Trajectory Collection: The base agent executes tasks in a verifiable environment (OSWorld) to generate diverse execution trajectories. The environment's built-in reward functions automatically evaluate each trajectory, and failure trajectories are retained as material for subsequent analysis and improvement.

Stage 2 — LLM-Guided Diagnosis and Improvement: Each failure trajectory (including task instructions, action history, and intermediate reasoning steps) is sent to the LLM. The LLM reasons about the failure cause in-context, performs a structured diagnosis—summarizing the currently dominant failure modes, proposing corresponding inference-time correction strategies, and automatically generating code patches that can be directly injected into the agent's code. Humans perform only lightweight verification (selecting strategy directions and checking code syntax), and over \(97\%\) of the patches are deployed without modification.

Stage 3 — Iterative Accumulation: The patches are integrated into the agent's inference workflow, and the agent executes in a new round of rollouts. Updated failure trajectories enter the diagnostic loop again. After \(4\) rounds of iterations, the four strategies are sequentially introduced and accumulate their effects.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Base Agent<br/>Runs Tasks"] --> B["Collect Execution Trajectories"]
    B -->|Success Trajectories| C["Send to Existing<br/>Success-Driven Loop<br/>(SFT Fine-tuning)"]
    B -->|Failure Trajectories| D["LLM Diagnosis<br/>Summarize Failure Modes"]
    D --> E["LLM Proposes Inference-Time Solutions<br/>+ Generates Code Patches"]
    E -->|Lightweight Human Verification<br/>97%+ No Changes Needed| F["Inject into Agent<br/>Inference Workflow"]
    F --> A

Key Designs¶

1. Visual Search: Addressing Inaccurate Grounding

In the first round of analysis, the LLM finds that grounding errors are among the most common bottlenecks: the agent frequently fails to locate target elements on complex UI interfaces, especially in high-resolution or visually dense layouts. To address this, a visual search mechanism is designed as a post-verification step for grounding. Whenever the agent performs an operation requiring spatial localization (e.g., click, moveto, dragto), the framework crops a \(400 \times 400\) pixel region centered on the target coordinates and scales it up by \(2\times\). The original action position is marked with a red open circle (radius \(7\text{px}\)) on this region. This annotated local view, along with the task instruction, action history, and reasoning context, is sent back to the agent for verification. If the agent determines the coordinates are incorrect, it can actively correct them before execution. This "execution \(\rightarrow\) verification \(\rightarrow\) correction" closed loop significantly improves grounding accuracy, increasing the success rate from \(41.67\%\) to \(47.22\%\) (\(+5.55\) points) on the OSWorld subset, establishing itself as one of the most substantial contributors among the four strategies.

2. Terminal Execution: Filling the Agent's Operational Capability Gaps

The LLM identifies another systemic weakness: the agent by default uses only low-level GUI operations (clicks, drags) via mouse/keyboard, whereas the operating system offers a more reliable top-level control interface—the terminal. The agent rarely takes the initiative to use the terminal, making tasks that could be done with a single command devolve into error-prone, complex click sequences. To this end, the framework introduces terminal execution capabilities: the agent can open a terminal via the shortcut Ctrl+Alt+T and run commands to search files, install programs, or copy paths. For agents with weaker terminal operating capabilities (e.g., GUI-Owl-32B), the framework forces three additional operational rules: the terminal must be opened via shortcut, correct syntax must be searched before executing a command, and commands must be submitted with Enter. This strategy yields a \(5.52\) percentage point improvement (from \(41.67\%\) to \(47.19\%\)) on the OSWorld subset, comparable to the visual search strategy. This demonstrates that "giving the agent more efficient tools" is just as crucial as "making the agent use existing tools more accurately".

3. Knowledge Support: Compensating for the Agent's Insufficient Domain-Specific Knowledge

The LLM finds that the agent frequently stalls when encountering tasks that exceed its internal knowledge base—such as not knowing specific shortcuts for LibreOffice or being unfamiliar with command-line installation syntax. The agent lacks a mechanism to look up external sources like humans do. Thus, two complementary knowledge acquisition channels are designed: first, a software manual retriever that provides the agent with handpicked shortcuts/operation guides for common software (e.g., Ctrl+D to fill down in LibreOffice Calc, Ctrl+L to align left in Writer), organized as concise reference tables instead of full text; second, a search engine interface allowing the agent to send natural language queries to GPT-5-mini when uncertain, returning brief, actionable information. When the agent detects that an action requires knowledge it lacks, it can actively trigger a query. This strategy alone yields a \(2.77\) percentage point improvement on the OSWorld subset. Typical case: the agent did not know how to resolve the "conda: command not found" error, but learned the correct installation commands after a query search.

4. Repetition Warnings: Breaking the Agent's Redundant Loops

The LLM's analysis shows that the most frequent failure mode is redundant action loops: the agent gets stuck in infinite loops of repeating the same operation without checking if the state has changed, which accounts for \(43\%\) of all failure modes. To address this, a triple repetition detection mechanism is designed to monitor three types of signals within a sliding window of the last \(5\) steps: thinking loops (identical low-level planning statements repeatedly appearing), action loops (identical pyautogui operations frequently repeated), and screen state loops (UI node subtrees extracted from the OSWorld accessibility tree, computing hashes of structural and textual attributes, showing no change for \(3\) consecutive steps). The alarm threshold is met if any signal appears \(\ge 3\) times in the \(5\)-step window. Once triggered, the system appends a prompt to the agent, encouraging it to try alternative strategies (such as initiating search queries, switching to terminal operations, or exploring different UI paths) instead of continuing the unproductive loop. This strategy alone yields a \(2.73\) percentage point boost on the OSWorld subset.

A Complete Example: LibreOffice Impress Font Change Task¶

The task requires "changing all fonts in the presentation to Liberation Sans Narrow". The base agent repeatedly enters the Master View, edits the font, exits, and re-enters the Master View, looping more than 20 times until the 30-step limit is exhausted.

Upon diagnosis, the LLM finds that the root cause is that font editing only takes effect on normal slides, and changes made in the Master View do not propagate to normal slides. The agent lacked awareness of this "scope isolation" and failed to detect that it was repeating an ineffective action. The LLM-proposed fix is to inject an action loop detection module: checking after each step whether high-level action patterns (Enter Master View \(\rightarrow\) Edit \(\rightarrow\) Exit \(\rightarrow\) Re-enter) have appeared \(\ge 2\) times in the last \(5\) steps. Once triggered, the loop stops, and the agent is prompted to try a different strategy.

The human verification step requires only syntax-level checks (modifying less than \(3\%\) of the code lines) before deploying the patch. After deployment, the agent detects the repetitive pattern, actively switches to directly editing normal slides, and successfully completes the font change.

Loss & Training¶

This method does not involve model weight updates or training. All improvements are injected into the agent's workflow as inference-time code patches; hence, there is no training loss function. Regarding inference overhead, it introduces a \(\sim 8\%\) computational increase (primarily from image cropping for visual search and LLM verification calls); however, since the four strategies reduce redundant loops and ineffective actions, the number of interaction steps drops by approximately \(15\%\), resulting in a net improvement in overall efficiency. The choice of meta-controller (the paper compares Claude 4.5 Sonnet, GPT-5.2, Gemini 3 Flash, and Qwen3-VL-32B) has a significant impact on performance, with Claude 4.5 Sonnet being overall the best in terms of completeness in failure mode analysis and code implementation capability.

Key Experimental Results¶

Main Results: OSWorld Benchmark Comparison¶

Model	Steps	Success Rate (%)
Claude 3.7 Sonnet	100	28.0
OpenAI CUA 4o	200	38.1
UI-TARS-1.5	100	42.5
OpenAI CUA o3	200	42.9
Agent S2 w/ Gemini-2.5-Pro	50	41.4
OpenCUA-32B	100	34.5
+ Ours	100	38.2
GUI-Owl-32B	100	19.0
+ Ours	100	21.3
OpenCUA-72B	100	42.3 ± 2.6
+ Ours (Main Result)	100	48.9 ± 1.2

On the full OSWorld test set (100-step limit), this method improves the success rate of OpenCUA-72B from \(42.3\%\) to \(48.9\%\), representing an absolute gain of \(+6.6\) points (a relative improvement of \(+15.6\%\)), outperforming all open-source models. The variance drops from \(\pm 2.6\) to \(\pm 1.2\), indicating that the improved system is more stable.

Ablation Study: Contribution of Individual Strategies¶

Configuration	OSWorld Subset (30 Steps)	Description
OpenCUA-72B Baseline	41.67	Original Model
+ Visual Search	47.22	Resolving grounding errors
+ Terminal Execution	47.19	Filling capability/operation gaps
+ Knowledge Support	44.44	Compensating for professional knowledge shortage
+ Repetition Detection	44.40	Breaking redundant action loops
+ Full Method (All Composed)	52.74	Four strategies take effect synergistically

All four strategies individually contribute positive gains, with visual search and terminal execution having the most significant effects (both roughly \(+5.5\) points). The combined full method achieves \(52.74\%\), far outperforming any single strategy, which shows that the four strategies cover complementary failure modes and substantially strengthen the agent's robustness when integrated.

Cross-Model and Cross-Benchmark Generalization¶

Model	Base	+ Ours	Relative Gain
GUI-Owl-32B (OSWorld)	19.0	21.3	+12.1%
OpenCUA-32B (OSWorld)	34.5	38.2	+10.7%
OpenCUA-72B (OSWorld)	42.3	48.9	+15.6%

Benchmark	Base (Qwen3-VL-32B)	+ Ours	Absolute Gain
OmniACT (Desktop Agent)	4.77	6.90	+2.13
AndroidControl (Mobile GUI)	28.37	36.23	+7.86
ScreenSpotPro (GUI Referring Grounding)	27.50	30.74	+3.24
WebVoyager (Web Interaction)	23.80	27.90	+4.10

Key Findings¶

Visual search and terminal execution are the two largest contributors (each \(+5.5\) points), demonstrating that inaccurate grounding and the lack of system-level tools are the two most prominent performance bottlenecks for GUI agents.
Cross-model generalization is consistent: three models of different architectures/origins achieve approximately \(+10\text{--}15\%\) relative gains, with stronger models benefiting more (\(72\text{B} > 32\text{B}\)), showing that the framework acts as an amplifier of model capabilities rather than just a quick patch.
Failure patches mined from OSWorld transfer successfully to other benchmarks, with AndroidControl (\(+7.86\)) and WebVoyager (\(+4.10\)) seeing the largest gains. This implies that the summarized failure modes (especially capability gaps and redundant loops) possess cross-platform generalizability.
Initial failure modes are dominated by redundant loops (\(43\%\)) and grounding errors (\(24\%\)). After four rounds of iterations, the remaining failures shift toward higher-level cognitive issues—task comprehension (\(12\%\)), CAPTCHA bypass (\(12\%\)), Complex UI recognition (\(24\%\)), and multi-step planning (\(28\%\)). This indicates that while the framework effectively resolves low-level bottlenecks, high-level semantic reasoning challenges still require upgrades to the model's core capabilities.

Highlights & Insights¶

Replacing "Internal Training" with "External Diagnosis": The most significant paradigm shift is correcting the agent's inference-time behavior via LLM failure analysis and code patch generation instead of retraining the model. This is equivalent to equipping the agent with an external-diagnosis-driven behavior correction system, closer to how humans learn from mistakes—we do not rewrite our brains; we devise new strategies.
Systematic Utilization of the "Diagnostic Value" of Failure Trajectories: Previous works discarded failure trajectories as waste. This work proves that they are not noise but structured supervision signals—each failure trajectory is a free, unannotated test case that precisely defines the agent's capability boundaries.
The Four Failure Modes Themselves as Independent Contributions: The four failure modes (inaccurate grounding, capability gaps, insufficient knowledge, and redundant loops) summarized by the LLM are not only inputs to this method but also a systematic taxonomy of current GUI agent bottlenecks, which can be directly referenced for agent debugging, evaluation, and benchmark building.
The "Strong LLM Diagnosis + Weak Agent Execution" Meta-Control Paradigm: Using Claude 4.5 Sonnet as a meta-controller to diagnose failures and formulate strategies, which are then injected into weaker models such as Qwen3-VL-32B. This division of labor bypasses the dilemma of "weak models cannot identify their own errors", presenting broad transferability (e.g., using the strongest model to generate strategies and injecting them into open-source agent pipelines).
Extremely Low Human Verification Rate (<3%): Over \(97\%\) of the LLM-generated patches are deployable without modification, showing that state-of-the-art LLMs as meta-controllers already possess non-trivial cross-codebase diagnostic and patch generation capabilities.

Limitations & Future Work¶

The authors acknowledge that the choice of the meta-controller (LLM) significantly impacts effectiveness: GPT-5.2 and Qwen3-VL identify less comprehensive failure modes, and Gemini 3 Flash exhibits weaker code-implementation capabilities. This indicates that the framework's quality is capped by the capability ceiling of the meta-controller, which may result in compromised performance under weaker LLMs.
While the lightweight human verification (\(<3\%\) modifications) holds true in the OSWorld environment, the distribution of failure modes might vary widely in more complex or unseen environments, potentially requiring more human intervention to ensure the robustness of LLM-generated patches.
Although the \(\sim 8\%\) computational overhead is described as "moderate" in the paper, it can accumulate significantly in large-scale deployments (e.g., running hundreds of agents simultaneously), especially since visual search involves multiple image cropping and LLM verification calls.
As reflected by the shifting distribution of failure modes, the remaining high-level cognitive problems (e.g., task comprehension bias, CAPTCHA bypass) after resolving low-level bottlenecks are harder to fix using "code patches". These issues likely require upgrades to the model's self-contained reasoning capability or dedicated alignment training.
A viable direction for future work is to allow the meta-controller to automatically adjust strategy priorities over multiple rounds, or to train a reward model to automatically sift the most valuable failure trajectories and optimal repair strategies, thus reducing the dependency on full LLM calls in each round.

vs Success-Driven Loop (OpenCUA / UI-TARS / Mobile-Agent): These methods only utilize successful trajectories for SFT fine-tuning. This work operates complementarily—leveraging discarded failure trajectories without increasing training overhead. The two loops can be cascaded: the improved agent of this work generates more successful trajectories \(\rightarrow\) reinforcing the SFT loop.
vs Self-Evolving Agents (SEAgent / SEA / UI-Genie): These methods learn from experience via trial-and-error or iterative enhancement, usually requiring substantial rollouts and training expenses. This work is much lighter—relying on pure inference-time interventions without weight updates.
vs GUI Error Analysis (Zheng et al. / Liu et al. / Wu et al.): Prior works have analyzed failure causes like grounding errors in GUI agents, but this paper goes a step further—translating analysis results into executable code patches and validating their effectiveness.
vs Test-Time Compute Scaling: This method can also be interpreted as a "targeted" test-time compute scaling: instead of undifferentiated token expansion (e.g., chain-of-thought expansion), it injects strategic interventions tailored to specific failure types, yielding higher efficiency.

Rating¶

Novelty: ⭐⭐⭐⭐ [Systematically introduces the "failure-driven" perspective into agent self-improvement, with a relatively novel patch-based inference-time intervention paradigm; the four strategies themselves are not entirely new, but the integration and framework design are elegant]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Contains OSWorld main results, strategy-by-strategy ablation on the subset, cross-model generalization over 3 models, and cross-platform generalization across 4 benchmarks. The verification framework is highly complete]
Writing Quality: ⭐⭐⭐⭐ [The motivational chain is clear (waste \(\rightarrow\) systematic utilization \(\rightarrow\) experimental proof \(\rightarrow\) generalization); the method section contains both structural design and concrete case studies]
Value: ⭐⭐⭐⭐⭐ [Delivers an absolute gain of \(+6.6\) points with zero training overhead, which is perfectly complementary to existing SFT pipelines and directly deployable in current agent systems. The taxonomy of failure modes also possesses independent reference value]