Reward Modeling for Computer-Using Agent from Video Execution¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/limenlp/ExeVRM
Area: Alignment & RLHF
Keywords: Computer-Use Agent (CUA), Execution Video, Reward Modeling, Spatiotemporal Token Pruning, Adversarial Instruction Translation

TL;DR¶

Addressing the reliance of computer-use agent (CUA) evaluation on handcrafted rules or internal reasoning traces, this paper introduces ExeVRM, a model-agnostic reward model evaluated directly on interface execution videos, enabled by the 53k-scale ExeVR-53k dataset, adversarial instruction translation for hard negative synthesis, and spatiotemporal token pruning (STP+TTP) that keeps 720p long-horizon evaluation tractable while outperforming GPT-5.2 and Gemini-3 Pro.

Background & Motivation¶

Computer-use agents (CUAs) are emerging as a core paradigm for automating complex, multi-step workflows across desktop operating systems (e.g., Windows, macOS, Ubuntu) and mobile environments. However, reliably verifying whether an interaction trajectory genuinely satisfies a user's instruction remains a critical bottleneck that severely limits downstream reinforcement learning alignment and automated benchmarking. Most existing evaluation protocols depend on task-specific rule engines, handcrafted verification scripts, or coarse checks on final-state screenshots; the former lack scalability and transferability across open-domain tasks, while the latter easily fail due to the absence of causal intermediate interaction history. To build an objective, universal evaluator across heterogeneous agent architectures, using the observable sequence of interface states—the execution video—serves as the most natural, method-agnostic representation.

Nonetheless, reward modeling from execution video introduces two fundamental challenges. First, desktop interaction videos contain enormous spatiotemporal redundancy: large regions of the interface (e.g., toolbars, taskbars, wallpapers, and application chrome) remain static across dozens of steps, whereas correctness often hinges on minute, localized visual cues (e.g., cursor focus shifts, small text edits, transient dialog boxes, or toggle changes). Naively feeding high-resolution, long-horizon video frames quickly exhausts context memory budgets, while coarse downsampling or sparse subsampling irrevocably destroys these critical cues. Second, existing computer-use datasets are overwhelmingly biased toward successful demonstrations created for behavior cloning, with a severe lack of informative negative trajectories annotated with exact temporal failure divergence points.

To resolve these tensions, this paper proposes an end-to-end framework for video-based reward modeling. By standardizing diverse agent and human interaction logs into a unified video representation, generating hard contrastive negatives via adversarial instruction translation, and developing a spatiotemporal token pruning mechanism tailored to computer interfaces, the approach makes high-resolution long-horizon training and inference practical. Core idea: decouple CUA trajectory evaluation into a pure interface video-to-goal reward prediction problem, establishing the 53k-scale ExeVR-53k dataset and an adversarial negative synthesis strategy, coupled with spatiotemporal token pruning (STP+TTP) that preserves subtle UI transitions to train ExeVRM for accurate binary success judgment and temporal error attribution.

Method¶

Overall Architecture¶

The objective of ExeVRM is to take a natural language user instruction \(X\) along with the agent's keyframe execution video sequence \(V\), directly predict a binary scalar task success label \(\hat{Y} \in \{0, 1\}\), and localize the initial temporal failure window \(\hat{\mathcal{I}} = [\hat{t}_s, \hat{t}_e]\) on erroneous runs. To process high-resolution trajectories spanning up to 100 frames without discarding decisive micro-level UI cues, the framework decomposes into: trajectory video standardization, adversarial instruction translation for negative pair and attribution synthesis, and dual-domain token pruning operating on the frozen visual encoder's feature space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: User Instruction X + Execution Keyframe Sequence V"] --> B["Stage 1: Multi-source Trajectory Videoization & Adversarial Instruction Translation<br/>Unified 1 FPS sequences / VLM-synthesized hard negatives & error step timestamps"]
    B --> C["Stage 2: Spatial Token Pruning (STP)<br/>Local feature graph clustering to filter large homogeneous backgrounds"]
    C --> D["Stage 3: Temporal Token Pruning (TTP)<br/>Dynamic reference cosine similarity to suppress persistent layout redundancy"]
    D --> E["Stage 4: Token Combination & Re-packing<br/>Dual-mask logical AND filtering, projected into LLM token space"]
    E --> F["Output: Task Success Judgment & Initial Error Temporal Attribution<br/>Scalar binary reward + first-failure time window"]

Key Designs¶

1. Multi-source Trajectory Videoization & Adversarial Instruction Translation: Resolving Negative Supervision and Attribution Data Scarcity Addressing the heterogeneity and positive-only bias of existing CUA corpora, this design establishes a standardized execution video representation and produces scalable hard counterexamples. Trajectory logs from AgentNet (human demonstrations), ScaleCUA (mixed grounding/agent runs), and OSWorld (rollouts from 30 distinct CUA systems) are unified into atomic interaction steps, taking a single representative post-action screenshot per step to construct 1 FPS execution summaries across Windows, macOS, Ubuntu, and Android. To synthesize missing negative supervision, inspired by back-translation, adversarial instruction translation prompts a vision-language model (e.g., GPT-5.2) with a valid trajectory segment to generate an alternative instruction that is plausible in the same UI context but semantically violated by the execution. The VLM is forced to provide both a textual justification of the failure and the exact reference step \(t_{\text{ref}}\) where the discrepancy first emerges. Verified with a 100% human pass rate on audit subsets, this strategy yields the 53k-instance ExeVR-53k corpus with reliable temporal attribution labels.

2. Spatial Token Pruning (STP): Connected-Component Filtering of Large Homogeneous Backgrounds To eliminate the massive token burden of static backgrounds and uniform desktop regions, STP builds a spatial feature graph per frame to prune large uninformative components while preserving intricate UI elements. Given patch features \(\mathbf{P}^{(t)} \in \mathbb{R}^{H' \times W' \times D}\) from the frozen visual encoder, horizontal and vertical differences between adjacent patches are computed as \(d_h^{(t)}(i, j) = \|\mathbf{P}^{(t)}_{i,j} - \mathbf{P}^{(t)}_{i,j+1}\|\) and \(d_v^{(t)}(i, j) = \|\mathbf{P}^{(t)}_{i,j} - \mathbf{P}^{(t)}_{i+1,j}\|\). Neighbors within a 4-neighborhood \(\mathcal{N}(i, j)\) are connected if their distance falls below similarity threshold \(\tau_s\):

\[e_{(i,j) \leftrightarrow (i',j')}^{(t)} = \mathbb{I}\left[(i',j') \in \mathcal{N}(i, j) \;\land\; \|\mathbf{P}_{i,j}^{(t)} - \mathbf{P}_{i',j'}^{(t)}\| < \tau_s\right]\]

Applying Union-Find over the graph identifies connected components \(\mathcal{C}^{(t)} = \{C_1^{(t)}, \dots, C_{K_t}^{(t)}\}\). Components whose patch count exceeds size threshold \(\tau_{\text{large}}\) form the redundancy set \(\mathcal{R}^{(t)} = \{C \in \mathcal{C}^{(t)} : |C| > \tau_{\text{large}}\}\), yielding binary spatial mask \(\mathbf{M}_s^{(t)}(i, j) = 0\) (pruned) or \(1\) (retained). This parameter-free grouping effectively drops broad blank canvases while keeping localized UI structures intact.

3. Temporal Token Pruning (TTP): Dynamic Reference Cosine Comparison to Suppress Static UI Structures Because desktop interfaces exhibit localized activity, window chrome, navigation sidebars, and menu headers remain identical across dozens of frames even after spatial pruning. TTP suppresses temporally stationary tokens to focus model attention on active state transitions. For each spatial location \(i\) in token tensor \(\mathbf{V} \in \mathbb{R}^{T \times N \times D}\), a reference token \(\mathbf{v}_i^{(\text{ref})}\) is initialized from the initial frame (\(\mathbf{v}_i^{(0)}\)) and retained unconditionally (\(\mathbf{M}_t(0, i) = 1\)). For subsequent frames \(t\), cosine similarity against the reference token determines retention:

\[\mathbf{M}_t(t, i) = \mathbb{I}\left[\text{sim}_{\text{cos}}\left(\mathbf{v}_i^{(\text{ref})}, \mathbf{v}_i^{(t)}\right) \le \tau_t\right], \quad t \in \{1, \dots, T-1\}\]

When the similarity drops below threshold \(\tau_t\) (signifying a noticeable UI change), the token is retained and the reference state updates dynamically:

\[\mathbf{v}_i^{(\text{ref})} \leftarrow \begin{cases} \mathbf{v}_i^{(t)}, & \text{if } \mathbf{M}_t(t, i) = 1 \\ \mathbf{v}_i^{(\text{ref})}, & \text{otherwise} \end{cases}\]

This stateful sliding reference ensures that each coordinate is continuously checked against its most recent distinct state, successfully discarding persistent layout scaffolding while capturing transient state mutations such as cursor movements, dropdown toggles, and text edits.

Loss & Training¶

During training, the vision encoder \(\mathcal{V}\) and multi-modal projector \(\mathcal{P}\) are frozen, tuning only the autoregressive LLM backbone. Spatial and temporal masks are combined via element-wise logical AND (\(\mathbf{M} = \mathbf{M}_s \land \mathbf{M}_t\)). The retained tokens are packed and mapped to language token space \(\tilde{\mathbf{H}}_V\). Optimization is governed by the joint reward classification and justification generation objective:

\[\mathcal{L}_{\text{rm}} = \ell(\hat{Y}, Y) + \lambda \mathcal{L}_{\text{gen}}\]

Hyperparameters are configured to \(\tau_s = 0.3\), \(\tau_t = 0.9999\), and \(\tau_{\text{large}} = 40\). Models are trained on 8× NVIDIA A100-80GB GPUs using an adapted LLaMA-Factory codebase, fine-tuning Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct with a learning rate of \(5 \times 10^{-6}\) under a cosine decay schedule.

Key Experimental Results¶

Main Results¶

The evaluation on ExeVR-Bench (789 balanced test instances across Ubuntu Agent/Human, Mac/Win, and Android) evaluates binary success classification along with temporal error localization.

Model	Type	Ubuntu(Agent) Acc/Prec/Rec	Ubuntu(Human) Acc/Prec/Rec	Mac/Win Acc/Prec/Rec	Android Acc/Prec/Rec	Overall Acc / Prec / Rec
Gemini 2.5 Pro	Proprietary	84.6 / 85.6 / 82.8	64.5 / 82.2 / 37.0	69.0 / 83.9 / 47.0	74.0 / 81.6 / 62.0	72.8 / 83.5 / 56.7
Gemini 3 Pro	Proprietary	80.4 / 75.0 / 90.0	71.2 / 71.7 / 71.0	75.6 / 75.3 / 76.8	73.5 / 74.7 / 71.0	75.1 / 74.2 / 76.7
GPT-5.2	Proprietary	82.5 / 85.1 / 78.7	74.0 / 84.3 / 59.0	74.5 / 88.9 / 68.7	74.5 / 75.5 / 75.8	75.0 / 82.7 / 66.5
Seed-2.0 Pro	Proprietary	85.1 / 85.1 / 85.1	77.2 / 81.7 / 69.1	81.0 / 86.1 / 74.0	78.0 / 82.6 / 71.0	80.3 / 83.9 / 74.7
InternVL-3.5 8B	Open-source	64.6 / 64.8 / 55.9	52.0 / 52.2 / 48.0	59.5 / 58.6 / 65.0	57.5 / 59.3 / 48.0	56.6 / 58.9 / 55.9
Qwen2.5-VL 7B	Open-source	64.6 / 65.5 / 60.6	63.0 / 65.1 / 56.0	68.0 / 74.3 / 55.0	63.5 / 62.7 / 66.0	64.5 / 66.6 / 59.2
Qwen3-VL 8B	Open-source	71.4 / 74.4 / 64.9	69.0 / 86.5 / 45.0	66.5 / 81.1 / 43.0	64.5 / 71.6 / 48.0	67.7 / 77.6 / 49.9
ExeVRM 4B (Ours)	Custom Open	77.8 / 83.3 / 69.1	81.0 / 82.9 / 78.0	90.0 / 87.7 / 93.0	72.5 / 66.4 / 91.0	80.1 / 79.2 / 82.5
ExeVRM 8B (Ours)	Custom Open	82.5 / 85.9 / 77.7	84.0 / 84.0 / 84.0	89.0 / 85.5 / 94.0	83.5 / 77.2 / 95.0	84.7 / 82.9 / 87.7

In temporal attribution, ExeVRM 8B achieves a tIoU score of 0.3332, markedly superior to GPT-5.2 (0.0855), Gemini 3 Pro (0.0245), and the base Qwen3-VL 8B (0.0862), evidencing its ability to pinpoint the precise steps causing failure.

Ablation Study¶

The ablations evaluate input resolutions, evaluation strategies, and individual pruning components.

Table 1: Effect of Input Resolution on Reward Modeling | Base Model | Input Resolution & Pruning | Accuracy (%) | Precision (%) | Recall (%) | |---|---|---|---|---| | Qwen3-VL 4B | 360p (w/o pruning) | 79.3 | 80.6 | 77.8 | | Qwen3-VL 4B | 720p (w/ STP & TTP) | 80.1 | 79.2 | 82.5 | | Qwen3-VL 8B | 360p (w/o pruning) | 81.5 | 82.5 | 80.5 | | Qwen3-VL 8B | 720p (w/ STP & TTP) | 84.7 | 82.9 | 87.7 |

Table 2: Ablation of Pruning Modules on Qwen3-VL 4B | Configuration | w/ STP | w/ TTP | Accuracy (%) | Precision (%) | Recall (%) | Peak GPU Memory (50 frames) | Per-step Time | |---|---|---|---|---|---|---|---| | Spatial only | ✓ | ✗ | 77.6 | 81.7 | 72.6 | ~56 GiB | 6.2 s | | Temporal only | ✗ | ✓ | 80.3 | 81.3 | 79.3 | ~68 GiB | 7.8 s | | Combined (STP + TTP) | ✓ | ✓ | 80.1 | 79.2 | 82.5 | ~49.5 GiB | 5.8 s |

Key Findings¶

Temporal pruning drives semantic discrimination while spatial pruning delivers memory scalability: TTP-only achieves 80.3% accuracy compared to 77.6% for STP-only, showing that inter-frame transitions hold the core reward signal. However, adding STP slashes 50-frame peak memory from 68 GiB down to 49.5 GiB and elevates recall to 82.5%, proving that joint spatiotemporal pruning is crucial for practical high-resolution training.
Dense execution video outperforms sparse snapshots: Compared against final-frame evaluation (AER: 55-69% accuracy) and head-and-tail snapshots (Simplified Judge: 57-78% accuracy), ExeVRM's dense video context achieves 84-89% accuracy, demonstrating that observing continuous state transitions prevents false positives caused by static deceptive states.
High resolution is essential for subtle UI detection: Raising resolution from 360p to 720p yields a +7.2 percentage point jump in recall for the 8B model (80.5% \(\rightarrow\) 87.7%), confirming that fine-grained UI elements (e.g., small text, cursors, icons) require physical pixel fidelity.

Highlights & Insights¶

Shifting CUA Evaluation to Pure Video Execution: Removes dependence on internal thoughts, proprietary tool-call formats, or brittle execution environment scripts, enabling non-invasive, universal evaluation of arbitrary black-box agents through screen recordings.
Adversarial Instruction Translation: Inverts the generation paradigm by synthesizing context-grounded hard negative instructions and deviation timestamps from valid trajectory segments, solving the persistent shortage of verified negative training data in agent learning.
Interface-Specific Token Pruning: Tailored to UI topology by combining connected-component graph reduction with dynamic reference cosine comparison, effectively decoupling the memory cost of long-horizon 720p video from token sequence explosion.

Limitations & Future Work¶

Admitted Limitations: Operating at 1 FPS keyframes can miss sub-second transient animations, rapid cursor drags, or high-refresh rate transitions; furthermore, on web pages dominated by dynamic video playback or non-homogeneous textures, STP's background grouping heuristic may underperform.
Future Directions: Exploring adaptive frame-rate sampling based on UI motion velocity, and incorporating ExeVRM as an online process reward model (PRM) to guide large-scale reinforcement learning (PPO/GRPO) directly in real desktop environments.

vs AER / Simplified Judge: AER and Simplified Judge rely solely on terminal or initial+terminal screenshots, missing intermediate path deviations; ExeVRM utilizes the entire temporal trajectory to capture causal failures.
vs GUI-Critic-R1 / VAGEN: These methods depend heavily on internal reasoning traces or code-level API logs; ExeVRM treats the agent as a pure black box, inspecting only external visual execution for fair cross-system arbitration.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates video-based reward modeling for CUAs with tailored adversarial synthesis and UI-specific token pruning.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 platforms, comparing multiple proprietary and open models with detailed efficiency ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear structure, rigorous mathematical formulations, and thorough architectural explanations.
Value: ⭐⭐⭐⭐⭐ Provides a foundational, highly scalable evaluator for agent benchmarking and automated reinforcement learning alignment.