Learning to Zoom Efficiently with a Contrastive Curriculum¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://www.ukp.tu-darmstadt.de
Area: Multimodal Efficiency
Keywords: Vision-Language Models, Adaptive Zoom-in, Reinforcement Learning, Contrastive Learning, Curriculum Learning
TL;DR¶
Addressing the heavy reliance of multimodal large language models on expensive human annotations and trajectory-level warm-start supervised fine-tuning (SFT) for acquiring zoom-in tool capabilities, this work presents a self-supervised intrinsic exploration reward formulated as an InfoNCE-style difference over subsequent reasoning logits coupled with an IoU-guided negative bounding box curriculum, achieving Pareto-optimal accuracy on high-resolution VQA benchmarks with just a single tool call.
Background & Motivation¶
Modern visual agents confronting ultra-high-resolution images or dense visual scenes must rely on interactive tools such as zoom-in operations to inspect fine-grained regions of interest (RoIs). Existing mainstream pipelines (including Pixel-Reasoner, DeepEyes, and Mini-o3) almost universally adhere to a two-stage regime: they first synthesize full tool-use trajectories using massive compute or proprietary black-box LLMs to impart tool syntax and coarse spatial bounding box grounding via supervised fine-tuning (SFT), followed by reinforcement learning (RL) optimization for downstream task alignment. However, acquiring high-quality multimodal tool trajectories is prohibitively expensive, and the warm-start SFT stage frequently induces distribution shifts or catastrophic overfitting, robbing models of autonomous visual exploration capabilities.
Attempting to teach visual tool use in an RL-only setting without warm-start SFT inevitably stumbles upon a severe sequential mismatch in skill acquisition: during online trajectory generation, the policy must first decide whether to deploy the tool before determining how to generate continuous spatial coordinates; yet from a developmental perspective, an agent should master the mechanics of using the tool before determining when it is economically sound to trigger it. If guided solely by task-level sparse accuracy rewards, models quickly suffer negative returns due to poor initial tool execution and degenerate into abandoning tool calls entirely; conversely, introducing unconstrained tool-use rewards causes severe reward hacking, where agents inflate sequence lengths or blindly invoke tools to harvest superficial points.
This work addresses this dilemma by reframing the zoom-in operation as an active information-seeking action whose utility should be measured by whether the returned crop increases the predictive confidence of subsequent reasoning. Core Idea: Treating the tool call from a correctly answered trajectory as a silver-label positive sample, the method synthesizes hard negative crops with controlled Intersection-over-Union (IoU) overlap and formulates an InfoNCE-style intrinsic exploration reward by contrasting the logits of subsequent reasoning conditioned on positive versus negative crops; paired with a curriculum that gradually increases negative hardness, it provides an SFT-free drop-in replacement that attains Pareto-optimal performance with just a single zoom-in.
Method¶
Overall Architecture¶
The framework is established upon Group Relative Policy Optimization (GRPO) and comprises three integrated stages: trajectory rollouts with silver-label positive identification, curriculum-guided hard negative bounding box synthesis, and conditional logit preference scoring. When the multimodal policy generates an initial reasoning prefix \(M_{R1}\) and emits a bounding box \(M_{T1}\) to receive cropped visual patch \(T_E\), trajectories leading to correct answers are confirmed as silver-label positives. The curriculum generator then constructs an adversarial negative box \(M_{T1}'\) and corresponding patch \(T_E'\) with an IoU overlap strictly constrained by curriculum parameter \(\tau\). Finally, a single forward pass re-scores the existing second-stage reasoning tokens \(M_2\) under both prefixes to quantify information gain, which is injected as an intrinsic exploration reward.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Original Image + Query Q"] --> B["Forward Generation & Silver Filtering<br/>Turn 1 reasoning MR1 + Tool call MT1 + Crop execution TE"]
B --> C["Conditional Branch: Is final answer correct?"]
C -->|Yes: Confirm positive sample| D["Curriculum Negative Sampling<br/>Enforce overlap IoU(TE, TE') = ฯ over training progress"]
C -->|No: Zero exploration reward| E["Task reward racc = 0"]
D --> F["Dual-Prefix Forward Rescoring<br/>Single forward pass evaluates logit preference of M2"]
F --> G["Contrastive Intrinsic Reward<br/>InfoNCE alignment + Token clipping + Tanh bound"]
G --> H["GRPO Advantage Update<br/>Single zoom achieves Pareto-optimal accuracy and efficiency"]
Key Designs¶
1. Forward Generation & Silver Filtering: Converting Sparse Task Rewards into Dense Supervision In pure RL exploration without ground-truth coordinate annotations, assessing whether an unprompted model tool call lands on the true region of interest is ill-posed. The method solves this by adopting a posterior verification paradigm: whenever the agent uses the tool and subsequently produces a correct final answer, it is inferred with high confidence that the bounding box \(M_{T1}\) and cropped area \(T_E\) provided sufficient semantic information, allowing it to be treated as a "silver-label positive sample". This design bypasses manual RoI annotation entirely, bootstrapping the learning process from ordinary question-answering benchmarks in a self-supervised manner.
2. Curriculum Negative Sampling: Progressive Adversarial Control from Distant to Overlapping Patches
If negative samples are chosen randomly from disjoint image regions (an easy negative setting where \(\tau = 0\)), models quickly discover degenerate reward-hacking shortcuts: repeating lexical tokens across turns, prolonging second-stage reasoning sequences, or emitting stereotypical coordinate patterns (such as [100, 100, 400, 400]). To eliminate these shortcuts, an IoU curriculum is introduced: during the first 30% of training steps, easy negatives with \(\tau = 0\) are maintained to teach basic crop syntax, after which \(\tau\) is linearly increased to \(\tau_{\max} = 0.175\). By sampling perturbed areas and aspect ratios from Lognormal distributions and adjusting offsets via Bernoulli trials, candidate boxes \(M_{T1}'\) are sampled such that \(\text{IoU}(T_E, T_E') = \tau\), forcing the model to distinguish subtle visual variations rather than relying on superficial formatting cues.
3. Contrastive Intrinsic Reward: InfoNCE Surrogate via Logit Differences Rather than auto-regressively decoding a counterfactual response from the negative prefixโwhich incurs prohibitive compute and creates mismatched sequence lengths that destabilize policy gradientsโthe method re-scores the existing second-generation tokens \(M_2\) in a single forward pass. Denoting the positive prefix as \(S_{\text{pre}} = (Q, M_{R1}, M_{T1}, T_E)\) and the negative prefix as \(S'_{\text{pre}} = (Q, M_{R1}, M_{T1}', T_E')\), token-level Bradley-Terry preference odds are calculated, clipped, and bounded using the hyperbolic tangent function: $$ \tilde{r}{\text{tool}, i} = \log \frac{2 P(M \mid S'} \mid S_{\text{pre}}, M_{2, <i})}{P(M_{2, i} \mid S_{\text{pre}}, M_{2, <i}) + P(M_{2, i{\text{pre}}, M $$ $$ r_{\text{tool}} = \tanh \left( \sum_{i=1}^N \text{clip}(\tilde{r}}){\text{tool}, i}, \pm\gamma) \right) \in [-1, 1] $$ This objective provably implements the negative InfoNCE loss shifted by \(\ln(2)\), measuring whether the model can discriminate between valid and invalid tool calls based purely on subsequent reasoning. The overall RL composite reward is formulated as: $$ r = r $$ where }} + \alpha \cdot \mathbb{I}{\text{answer correct}} \cdot r_{\text{tool}\(\alpha = 0.1\) and \(\gamma = 1.54\). Conditioning the intrinsic reward on \(\mathbb{I}\{\text{answer correct}\}\) ensures exploration incentives remain strictly anchored to solving the primary objective.
Loss & Training¶
The approach is implemented on Qwen2.5-VL 7B using GRPO with selective sample replay to counteract vanishing advantages across multi-turn trajectories. Models are trained on the PR\video dataset (6,698 single-image QA pairs derived from InfographicVQA and Llava-CoT) using 8 sampled generations per query at temperature 1.0 with KL divergence weight \(\beta = 0.04\). The replay buffer is flushed every two gradient steps to ensure near on-policy dynamics. Following intrinsic exploration, the checkpoint can be directly used as an SFT-free drop-in replacement and continually trained on the Visual Probe training set with downstream accuracy rewards (denoted as Ours+).
Key Experimental Results¶
Main Results¶
Evaluations across four representative high-resolution VQA benchmarks (V*-Bench, HRBench 4k/8k, and MME-RealWorld) demonstrate that the proposed SFT-free intrinsic reward outperforms prior baselines, with continual RL yielding state-of-the-art results within its model class.
| Method / Evaluation Setting | Training Pipeline | V*-Bench (tool) | HRBench 4k (tool) | HRBench 8k (tool) | MME-RealWorld (tool) | Overall Mean (tool) |
|---|---|---|---|---|---|---|
| Qwen 2.5 VL 7B (Base) | Zero-shot Base | 15.71 | 10.38 | 9.75 | 16.77 | 13.15 |
| Pixel-Reasoner | Trajectory SFT + Curiosity RL | 84.82 | 73.62 | 67.00 | 63.80 | 72.31 |
| Mini o3 | Full SFT + Multi-turn RL | 86.91 | 77.73 | 73.39 | 65.63 | 75.92 |
| DeepEyes v2 | SFT + RL | 81.80 | 77.90 | 73.80 | 64.90 | 74.60 |
| Baseline: no tool | Pure RL (Single-turn, no tool) | - | - | - | - | 72.56 (tool-free) |
| Baseline: Curiosity | Pure RL (Absolute bbox, no SFT) | 86.91 | 76.62 | 70.25 | 63.58 | 74.34 |
| Baseline: Conditional | Pure RL (Constant tool reward) | 84.29 | 78.00 | 72.12 | 64.89 | 74.83 |
| Ours | Contrastive Curriculum (No SFT) | 85.34 | 78.50 | 73.38 | 64.38 | 75.40 |
| Mini o3 (Max 32 turns) | Trajectory SFT + Multi-turn Search | 88.20 | 77.50 | 73.30 | 65.50 | 76.13 |
| Ours+ | Exploration + Continual RL | 89.01 | 79.12 | 73.00 | 67.10 | 77.06 |
Ablation Study¶
Ablations on negative sampling strategies, curriculum target overlap \(\tau_{\max}\), and scoring granularity conducted on the PR\video dataset:
| Configuration / Variant | Architectural Modification | V*-Bench (tool) | HRBench 4k (tool) | HRBench 8k (tool) | MME-RW (tool) | Overall Mean |
|---|---|---|---|---|---|---|
| Ours (Full Model) | Token-level contrast + Curriculum (\(\tau_{\max}=0.175\)) | 85.34 | 78.50 | 73.38 | 64.38 | 75.40 |
| Sequence-level scoring (per seq) | Sequence probability ratio without token-level clipping | 83.77 | 77.62 | 72.12 | 62.56 | 74.02 |
| Easy negatives only (\(\tau=0\)) | No overlapping negatives; triggers reward hacking | 84.82 | 74.62 | 66.75 | 61.66 | 71.96 |
| Two negatives | Contrasting two negative bounding boxes simultaneously | 84.82 | 76.12 | 71.25 | 61.73 | 73.48 |
| Relaxed curriculum (\(\tau_{\max}=0.15\)) | Weaker adversarial overlap between positive and negative | 80.63 | 76.50 | 72.75 | 63.50 | 73.34 |
| Overly strict curriculum (\(\tau_{\max}=0.20\)) | High overlap destabilizes policy optimization | 70.16 | 71.50 | 63.88 | 56.57 | 65.53 |
Key Findings¶
- Hard Negative Curricula Prevent Degenerate Reward Hacking: Omitting hard negatives (\(\tau = 0\)) causes overall accuracy to drop sharply from 75.40 to 71.96. Training dynamics reveal that after step 300, models artificially inflate sequence lengths and repeat vocabulary from earlier turns to exploit \(P(M_2 \mid S_{\text{pre}})\). Introducing a progressive curriculum peaking at \(\tau_{\max} = 0.175\) stabilizes output length and directs policy attention to genuine visual discrimination.
- Single Zoom-in Reaches Pareto Optimality: Mini o3 requires 16 to 32 iterative tool turns during inference to marginally exceed the accuracy of Ours (75.40 vs 76.13), which calls the tool only once. Under continual training on identical data, Ours+ achieves 77.06 with a single zoom call, demonstrating dramatic efficiency and latency advantages.
- RoI Recall Dominates Downstream Success: Across 3,600 annotated splits in the synthetic Muffin&Chihuahua benchmark, cross-model Pearson correlation between bounding box overlap metrics and task accuracy is highest for Recall (0.93), compared to 0.43 for Precision and 0.53 for IoU. This demonstrates that when deploying vision tools, capturing a slightly larger context that reliably encompasses the target RoI is vastly superior to narrow, high-precision crops that risk target truncation.
Highlights & Insights¶
- Efficient Mutual Information Estimation: Converting the utility of physical visual grounding into conditional mutual information differences over subsequent reasoning enables seamless InfoNCE alignment, executed cheaply via a single forward pass with under 4% runtime overhead.
- Implicit Distillation into Parametric Weights: On Qwen2.5-VL, models trained with zoom-in tools significantly outperform pure no-tool baselines even when evaluated in tool-free mode (73.73 vs 72.56), indicating that spatial reasoning trajectories are partially distilled directly into model weights during RL.
- Self-Sufficient Visual Tool Discovery: By abolishing the prerequisite for proprietary LLM-curated tool trajectories, this work presents a practical, reproducible paradigm for training visual agents to interact with tools from raw QA data.
Limitations & Future Work¶
- Reflexive Tool Invocations on Full-Frame Targets: When evaluated on \(1 \times 1\) grids where the full image constitutes the target, the policy exhibits a habitual reflex to zoom into irrelevant corner numbers, highlighting the ongoing challenge of learning adaptive tool pruning.
- Sensitivity to Coordinate Representations: Cross-model evaluations on InternVL3.5 and Gemma 4 revealed tendencies to discriminate prefixes based solely on coordinate digit patterns (e.g., rounded integers), necessitating static prompt formatting with dynamic underlying crops.
- Subpar Multi-Turn Scaling: Directly extending the contrastive reward across multi-step zooming chains degraded empirical stability, indicating that multi-step visual navigation remains fragile in medium-sized open models without initial warm-start scaffolding.
Related Work & Insights¶
- vs Pixel-Reasoner: Pixel-Reasoner relies on trajectory SFT and static curiosity ratios; this work demonstrates that SFT is entirely dispensable when pairing intrinsic contrastive rewards with negative curricula.
- vs Mini o3 / DeepEyes: While prior agents emphasize deep multi-turn search (6 to 32 turns), this paper shows that current open weights frequently lose coherence over long chains; a single, well-placed, high-recall zoom achieves superior accuracy at a fraction of the inference cost.
Rating¶
- Novelty: โญโญโญโญโญ An elegant unsupervised intrinsic exploration reward grounding tool utility in subsequent reasoning tokens via InfoNCE.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 4 general benchmarks, a dedicated synthetic diagnostic benchmark (M&C), extensive parameter sweeps, and 3 distinct foundation model families.
- Writing Quality: โญโญโญโญโญ Rigorous mathematical formulation and insightful analysis of reward-hacking modes.
- Value: โญโญโญโญโญ Provides a foundational blueprint for removing proprietary data dependencies in agentic visual tool learning.