CaptchaMind: Training CAPTCHA Solvers via Reinforcement Learning with Explicit Reasoning Supervision¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/AlibabaResearch/captcha-mind
Area: LLM Agent
Keywords: CAPTCHA Solving, Multimodal Agent, Reinforcement Learning, Explicit Reasoning Supervision, Process-Level Reward
TL;DR¶
To resolve the chronic failure of multimodal agents on CAPTCHAs caused by reliance on holistic visual impressions and lack of intermediate reasoning supervision, this paper introduces CaptchaBench (16k programmatically generated instances with process annotations) and CaptchaMind, an RL-based solver that enforces explicit intermediate region localization via a bounding box tool and dense process-level rewards under GRPO, reaching 82.9% average accuracy across eight tasks and transferring effectively to live commercial instances.
Background & Motivation¶
Multimodal agents powered by vision-language models (VLMs) are advancing rapidly toward real-world deployment across web navigation, digital interface automation, and complex human-computer workflows. However, CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) verification systems deployed across modern web architectures constitute a formidable barrier that continually halts end-to-end autonomous agent execution. Contemporary CAPTCHAs have evolved far beyond basic distorted text recognition into intricate challenges requiring fine-grained spatial reasoning, dynamic multi-image comparison, and multi-turn stateful interactions. Recent empirical evaluations on benchmarks such as OpenCaptchaWorld reveal that even the most capable proprietary agent setups succeed only around 40% of the time, falling drastically behind human performance which routinely exceeds 90%.
Existing CAPTCHA-solving efforts predominantly rely on prompting closed-source flagship models such as GPT-5 or Claude-4 alongside external heuristic planning or domain-specific languages (DSL). Such prompting approaches incur steep inference costs, introduce high latency, and lack the fine-grained behavioral controllability necessary for secure, dependable execution. Conversely, training open-source vision-language models for CAPTCHA solving has remained largely unviable due to two critical impediments: first, an acute shortage of large-scale, process-annotated training corpora (existing benchmarks serve purely as evaluation suites); second, conventional reinforcement learning schemes rely strictly on sparse final-outcome rewards, leaving the model's visual reasoning process entirely implicit and black-box, thereby failing to steer the network toward capturing subtle, task-critical visual features against cluttered backgrounds.
The core diagnosis of this paper is that models fail primarily because they depend on holistic, coarse visual impressions rather than attending to task-relevant visual regions at intermediate reasoning phases. Overcoming this requires making intermediate visual reasoning observable and directly supervisable. To bridge this divide, the authors synthesize CaptchaBench, a large-scale benchmark of 16,000 instances with full programmatic process annotations. Core idea: Externalize intermediate visual reasoning into explicit spatial grounding via a bounding box tool, and train the agent using reinforcement learning (GRPO) with multi-level rewards that explicitly penalize or reward intermediate task-relevant region identification, multi-step interaction progress, and task completion, ensuring the model gathers grounded visual evidence before committing to final actions.
Method¶
Overall Architecture¶
CaptchaMind operates on a principled two-stage training paradigm: an initial supervised fine-tuning (SFT) warm-up via behavior cloning to establish foundational multi-turn tool interaction protocols and a think-then-act reasoning pattern, followed by reinforcement learning with explicit reasoning process supervision optimized via Group Relative Policy Optimization (GRPO).
The CAPTCHA-solving interaction is mathematically framed as a partially observable Markov decision process (POMDP). At each interaction step \(t\), the agent receives observation \(h_t\) consisting of current interface screenshots, instructions, and prior interaction history. The agent policy \(\pi_\theta(a_t | h_t)\) first invokes an explicit reasoning action using the bounding_box tool to outline the visual components (such as candidate icon pairs, coordinate anchors, or dice faces) deemed critical for task resolution. The simulation environment cross-references these predicted coordinates with programmatic ground-truth bounding boxes, delivering an immediate reasoning process reward \(R_{\text{process}}\). Conditioned on these grounded visual anchors, the agent executes operational actions (click, drag, enter_number), receiving step-wise interaction feedback rewards \(R_{\text{feedback}}\) and episode-terminating outcome rewards \(R_{\text{outcome}}\) to update the policy parameters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: CAPTCHA Image + Instruction"] --> B["Programmatic Simulator & CaptchaBench Construction<br/>16,000 samples with process-level annotations"]
B --> C["Two-Stage Training Initialization<br/>SFT Behavior Cloning for Think-then-Act"]
C --> D["Explicit Region Grounding Mechanism<br/>bounding_box tool externalizes visual reasoning"]
D --> E["Multi-Level Joint Reinforcement Learning Rewards<br/>Process reward + Feedback reward + Outcome reward"]
E --> F["Output: Precise Action Execution click / drag / enter_number"]
Key Designs¶
1. Programmatic Simulator & CaptchaBench Construction: Overcoming data scarcity with process-verifiable synthesis
Real-world commercial verification services operate under strict black-box regimes, yielding only a binary pass/fail response without exposing step-wise logical errors, while existing public benchmarks are far too small to support post-training. To solve this, the authors engineer a Python-based interactive simulator that renders diverse visual scenes across eight commercial-grade task types (image-switching: connect_icon, coordinates, dart_count, rotation_match; multi-step interaction: click_order, image_select; single-step decision: dice_count, slide_puzzle). The pipeline synthesizes 16,000 training and 1,600 evaluation instances with randomized icon subsets, background textures, spatial placements, and rotation angles. Crucially, the engine synchronously records ground-truth coordinate masks, step-by-step intermediate state transitions, and deterministic rule-derived Chain-of-Thought (CoT) rationales. A human discrimination study with 50 participants yielded an average classification accuracy of 51.04% (\(t(49)=1.37, p=0.178\), Cohen's \(\kappa=0.02\)), empirically validating that synthesized instances are perceptually indistinguishable from live commercial screenshots.
2. Explicit Region Grounding Mechanism: Converting black-box reasoning into supervisable visual evidence
Standard multimodal foundation models often suffer from perceptual hallucination on complex puzzles because they extrapolate solutions directly from overall visual impressions without explicitly inspecting minute visual features. CaptchaMind introduces a non-intrusive bounding_box tool that modifies neither the underlying CAPTCHA puzzle state nor interface dynamics. Instead, it serves as a dual-purpose cognitive scaffold and supervision interface: the model is forced to explicitly externalize its internal attention focus by emitting bounding box coordinates for relevant visual entities prior to executing manipulation actions. This externalization makes the intermediate reasoning chain transparent, permitting programmatic spatial verification and enabling reward assignment directly over intermediate perceptual hypotheses.
3. Multi-Level Joint Reinforcement Learning Rewards: Mitigating reward sparsity with difficulty-adaptive supervision
To overcome the catastrophic sample inefficiency of outcome-only reinforcement learning across extended interaction trajectories, the training objective employs a multi-tiered composite reward function:
The process reward \(R_{\text{process}}\) quantifies grounding quality: an emitted box is considered valid if its intersection-over-union (\(\text{IoU}\)) with any ground-truth region exceeds 0.8, and a positive reward is awarded whenever the agent identifies \(\ge 50\%\) of all task-relevant regions, incentivizing thorough visual search while tolerating parsimonious reasoning paths. The feedback reward \(R_{\text{feedback}}\) evaluates whether multi-step actions (such as sequential icon selections or directional candidate transitions) align with valid progress in the simulator's state machine. The outcome reward \(R_{\text{outcome}}\) confers a binary bonus upon full puzzle completion. Importantly, \(R_{\text{process}}\) is difficulty-adaptive: it is selectively enabled only on challenging tasks where SFT accuracy falls below 50% (connect_icon and image_select), preventing reward competition and policy over-regularization on tasks where baseline reasoning is already robust.
Loss & Training¶
The framework is instantiated on Qwen2.5-VL-7B (selected after scaling explorations showed that 3B suffered from sparse rewards with only 25.8% SFT accuracy, while 32B attained 89.6% leaving little headroom for RL at prohibitive compute costs). Optimization is performed across 16 NVIDIA A100 (80GB) GPUs: - SFT Stage: Conducted on 8,000 samples (1,000 per task type) over 3 epochs with a learning rate of \(5\times 10^{-6}\), incorporating rule-generated CoT traces for counting and rotation tasks to instantiate robust think-then-act formatting. - RL Stage: Conducted on the remaining 8,000 samples over 1 epoch using the GRPO algorithm with batch size 64, sampling 8 rollout responses per prompt, with a learning rate of \(5\times 10^{-6}\) (requiring approximately 20 hours total training wall-time).
Key Experimental Results¶
Main Results¶
On the 1,600-sample CaptchaBench test split, CaptchaMind was evaluated against top proprietary commercial VLMs, specialized agentic frameworks, GUI agents, and training baselines. Success rates across all eight task categories are summarized below:
| Method Category | Model / Pipeline | Avg. (%) | connect_icon | coordinates | dart_count | dice_count | rotation_match | image_select | click_order | slide_puzzle |
|---|---|---|---|---|---|---|---|---|---|---|
| Closed-source VLMs | Claude-4-Sonnet | 47.4 | 7.5 | 93.5 | 68.5 | 51.0 | 20.0 | 43.0 | 71.5 | 24.5 |
| Claude-3.7-Sonnet | 43.9 | 19.0 | 83.5 | 51.0 | 75.5 | 10.0 | 38.5 | 30.0 | 44.0 | |
| Gemini-3-Pro | 45.8 | 47.5 | 62.0 | 88.5 | 63.0 | 50.0 | 52.0 | 0.0 | 3.0 | |
| OpenAI-o3 | 46.9 | 37.0 | 69.0 | 51.0 | 72.0 | 4.0 | 46.0 | 62.0 | 34.0 | |
| GPT-5 | 33.6 | 15.0 | 35.5 | 37.0 | 42.0 | 3.5 | 32.5 | 67.0 | 36.0 | |
| Qwen-VL-Max | 38.6 | 14.5 | 55.5 | 40.0 | 42.5 | 5.5 | 39.5 | 72.5 | 39.0 | |
| Agentic Frameworks | Oedipus (CCS 2025) | 51.9 | 35.0 | 68.0 | 76.0 | 44.5 | 31.5 | 53.0 | 68.5 | 38.5 |
| Halligan (USENIX 2025) | 54.7 | 34.0 | 76.5 | 69.5 | 43.0 | 36.0 | 66.0 | 71.0 | 41.5 | |
| GUI Agent | GUI_R1_7b | 6.5 | 0.0 | 13.5 | 14.0 | 0.5 | 3.5 | 18.0 | 2.0 | 0.0 |
| Training Baseline | SFT-only | 68.1 | 49.0 | 79.5 | 77.0 | 76.0 | 88.0 | 14.0 | 70.5 | 91.0 |
| Our Approach | CaptchaMind (Ours) | 82.9 | 71.0 | 91.0 | 93.0 | 72.5 | 86.0 | 71.0 | 87.5 | 91.5 |
Ablation Study¶
To isolate the empirical contribution of reasoning process supervision, intermediate step feedback, and SFT initialization, ablation runs were benchmarked under identical test protocols:
| Training Configuration | Avg. (%) | connect_icon | coordinates | dart_count | dice_count | rotation_match | image_select | click_order | slide_puzzle |
|---|---|---|---|---|---|---|---|---|---|
| Full (Complete Method) | 82.9 | 71.0 | 91.0 | 93.0 | 72.5 | 86.0 | 71.0 | 87.5 | 91.5 |
| SFT-only (No RL optimization) | 68.1 | 49.0 | 79.5 | 77.0 | 76.0 | 88.0 | 14.0 | 70.5 | 91.0 |
| w/o RPS (RL without process supervision) | 78.9 | 56.0 | 83.5 | 89.5 | 76.5 | 89.5 | 54.5 | 90.0 | 92.0 |
| w/o FBR (RL without feedback rewards) | 80.0 | 70.5 | 90.0 | 92.5 | 73.0 | 86.5 | 58.0 | 79.0 | 91.0 |
| RL-only (RL from scratch without SFT) | 25.6 | 31.0 | 26.0 | 65.0 | 12.5 | 16.0 | 11.5 | 37.0 | 6.0 |
In cross-dataset zero-shot generalization on OpenCaptchaWorld (151 challenging instances across overlapping tasks), CaptchaMind reached 66.2% accuracy (100/151), substantially outpacing SFT-only (29.8%, 45/151) and commercial Claude-4-Sonnet (51.0%, 77/151). On 100 live commercial CAPTCHAs scraped from web services (GeeTest, hCaptcha), CaptchaMind sustained a 71.0% real-world success rate.
Key Findings¶
- Reasoning process supervision resolves the fine-grained visual bottleneck: On tasks demanding subtle pixel-level verification (
connect_iconandimage_select), removing \(R_{\text{process}}\) triggers severe degradation of 15.0% (71.0% \(\to\) 56.0%) and 16.5% (71.0% \(\to\) 54.5%), confirming that explicit region grounding is essential where models otherwise rely on blurry global heuristics. - Grounding coverage correlates strongly with task success: Diagnostic tracking across all 1,600 evaluation runs reveals that episodes with \(<20\%\) ground-truth region grounding achieved only a 16% solve rate; performance jumped to 51% for coverage in \([0.2, 0.4)\), 82% for \([0.4, 0.6)\), and plateaued at 90%โ91% once grounding exceeded 60%.
- SFT warm-up is indispensable for policy convergence: Skipping SFT warm-up and directly optimizing RL from scratch led to a collapsed policy achieving only 25.6% average accuracy due to near-total reward sparsity during early exploration.
- GUI agent pre-training does not transfer to CAPTCHA environments: Generic GUI agents (such as GUI_R1_7b) scored only 6.5%, demonstrating that standard screen-element grounding lacks the multi-image mental rotation, spatial composition, and arithmetic reasoning needed for CAPTCHAs.
Highlights & Insights¶
- Dual-role operationalization of visual tools: Rather than treating tool invocation purely as an actuator for manipulating user interfaces, the framework repurposed the bounding box tool as an explicit verification interface that exposes internal reasoning traces for direct programmatic reward computation.
- Selective difficulty-adaptive process supervision: Instead of indiscriminately forcing dense process rewards across all scenarios, the authors applied process supervision exclusively to tasks with SFT baseline accuracy below 50%, preventing policy interference and optimizing compute efficiency.
- Zero-human-annotation process-level flywheel: By pairing procedural rendering with real-time semantic coordinate caching, the work created 16,000 highly authentic training trajectories complete with step-by-step Ground Truth reasoning, establishing a repeatable data blueprint for training visual interactive agents.
Limitations & Future Work¶
- Static screenshot evaluation vs. end-to-end browser automation: Due to legal and ethical constraints regarding live website scraping, real-world generalization was assessed on static captured screens rather than live browser sessions that execute dynamic JavaScript challenges, WebGL canvas checks, and behavioral biometrics (e.g., mouse velocity entropy).
- Architecture breadth restricted to single model family: All core experiments and RL optimizations were conducted on Qwen2.5-VL-7B; evaluating cross-family portability to other architectures (such as InternVL, Llama-Vision, or Gemini Nano) remains an open empirical frontier.
- Adversarial dynamics and defense adaptation: As CAPTCHA services integrate adversarial spatial noise or dynamic temporal distortions, exploring the robustness limits of explicit region grounding policies will be critical.
Related Work & Insights¶
- vs. Prompt-based & Agentic Solvers (Oedipus, Halligan): Prior approaches rely heavily on prompting proprietary closed models via complex DSL search loops, incurring significant API latency and costs; CaptchaMind demonstrates that an open 7B model trained with explicit process supervision outperforms Oedipus by 31.0% and Halligan by 28.2% in average accuracy with zero inference API costs.
- vs. Outcome-only RL for Multimodal Reasoning (DeepSeek-R1, UI-R1): Conventional outcome-only RL forces models to discover visual evidence through unguided trial and error, which fails in visually dense, multi-candidate verification settings; CaptchaMind establishes that adding intermediate spatial process rewards dramatically improves sample efficiency and final accuracy on fine-grained visual tasks.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering the use of tool-based explicit grounding for intermediate process-level reinforcement learning in visual agents]
- Experimental Thoroughness: โญโญโญโญโญ [16k dataset synthesis, human indistinguishability validation, 8-task comprehensive benchmarking, detailed ablations, and real-world out-of-distribution testing]
- Writing Quality: โญโญโญโญโญ [Exceptionally clear motivation, rigorous methodology, coherent narrative structure, and compelling empirical exposition]
- Value: โญโญโญโญโญ [Fills a long-standing void in training-based open-source CAPTCHA solving while offering a generalizable paradigm for process-supervised multimodal agent learning]