Vero: Open Reinforcement Learning Recipes for Visual Reasoning¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM / LLM Reasoning / Reinforcement Learning
Keywords: visual reasoning, reinforcement learning, vision-language model, GSPO, task-routed reward
TL;DR¶
Addressing the lack of open data and reward designs in leading vision-language models, Vero introduces the first fully open single-stage multi-task reinforcement learning recipe across six visual reasoning categories, building Vero-600K and task-routed rewards that outperform distillation-heavy open models across 30 benchmarks.
Background & Motivation¶
Vision-language models (VLMs) are increasingly expected to perform complex reasoning across charts, scientific diagrams, spatial perception, and embodied environments. Frontier models such as GPT-5, Qwen3-VL, and Kimi K2.5 demonstrate that reinforcement learning (RL) using policy gradient frameworks like PPO and GRPO serves as a pivotal driver for eliciting multimodal chain-of-thought (CoT) reasoning. However, the top-performing visual reasoners are developed behind closed doors, using proprietary post-training data mixtures and undisclosed reward schemes, leaving their empirical successes difficult to reproduce, analyze, or extend in the academic community.
In contrast, existing open-source visual RL recipes (such as OpenMMReasoner and VL-Rethinker) concentrate predominantly on narrow domains like visual mathematics and geometry. Applying RL to heterogeneous multi-task visual distributions presents severe optimization obstacles: heterogeneous answer formats and distinct solution paths induce catastrophic negative transfer across domains. For example, policies reinforced purely on mathematical reasoning suffer dramatic performance drops of 15 to 35 points on image captioning and instruction following. This raises a fundamental open question: what does it take to train a broadly capable visual reasoning model using a fully open RL recipe?
This paper demonstrates that neither complex multi-stage warm-ups nor synthetic distillation from proprietary teacher models is necessary; rather, a well-curated, uniformly balanced multi-task dataset paired with task-routed reward verifiers suffices for robust single-stage online RL. Core idea: curate a 600K-sample multi-task dataset (Vero-600K) spanning six core visual capability dimensions, adopt a uniform category mixture to eliminate cross-task negative transfer, and optimize via GSPO with ten task-routed reward verifiers in a single-stage online RL pipeline.
Method¶
Overall Architecture¶
Vero aims to optimize a vision-language policy \(\pi_\theta\) via single-stage online RL. Given an image \(v\) and a natural language question \(q\), the model generates a structured CoT response \(y = (z, a)\), where \(z\) represents the reasoning trace and \(a\) denotes the final verifiable answer. The training pipeline consists of four major stages: candidate dataset sourcing and manual quality audit, LLM-guided multi-criteria question/answer filtering, uniform 6-way category data mixing, and sequence-level policy optimization via GSPO coupled with task-routed reward functions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["250+ Candidate Multimodal Datasets<br/>Charts / STEM / Spatial / Knowledge / Grounding / IF"] --> B["Multi-Stage Data Selection & Filtering<br/>Heuristics + Manual Audit + LLM Verification"]
B --> C["Vero-600K Balanced Mixture<br/>Uniform 100K Samples per Category"]
C --> D["Task-Routed Reward System<br/>Binary Matches + Graded IoU + Rule Engine + LLM Judge"]
D --> E["GSPO Policy Optimization<br/>Sequence Ratios + Format Constraints + Length Penalty"]
Key Designs¶
1. Multi-Stage Data Selection & Filtering: Eliminating Noise and Ambiguity Across Tasks
To curate high-quality samples from over 250 candidate datasets, Vero applies a two-tiered selection pipeline. At the dataset level, datasets with fewer than 1K samples, average image resolution below 200K pixels (retaining five high-quality exceptions), or binary yes/no questions are pruned to prevent the policy from gaming rewards via blind guessing. Each remaining dataset is audited by inspecting ~50 samples against rigorous criteria: annotation error rate under 5%, clear unambiguity, and verifiable target formats. Out of ~100 datasets passing heuristic checks, 59 are retained. At the sample level, Qwen3-VL-235B evaluates individual samples across five criteria (relevance, ambiguity, English language validity, visible verifiability, and numerical precision), filtering out any prompt failing a criterion. Finally, text-only Qwen3-235B normalizes ground-truth answers into standardized canonical forms to guarantee stable reward computation.
2. Vero-600K Balanced Mixture: Eliminating Cross-Task Negative Transfer
To overcome the cross-task degradation inherent in single-domain training, Vero establishes a comprehensive taxonomy covering six distinct capabilities: Chart & OCR (9 datasets), STEM (13 datasets), Spatial & Action (8 datasets), Knowledge & Recognition (12 datasets), Grounding, Counting & Search (11 datasets), and Captioning & Instruction Following (6 datasets). While intuitive heuristics such as weighting by inverse accuracy, reasoning trace length, or input image area seem plausible, empirical results show they introduce performance trade-offs across categories. Vero adopts a strictly uniform mixture where each category contributes 100K samples (600K in total). This equal-ratio mixture achieves the highest overall benchmark gain (+5.8 points over base) and reliably avoids negative transfer across all evaluated visual skills.
3. Task-Routed Reward System: Accurate Verification Across Heterogeneous Answers
Standard symbolic solvers (such as math-verify) fail on non-mathematical visual tasks like document understanding, spatial coordinate extraction, or open-ended instruction following. Vero addresses this by formulating a composite reward combining accuracy, format constraints, and soft overlong penalties:
$\(R(y, y^*) = (1 - \alpha) R_{\text{acc}}(y, y^*) + \alpha R_{\text{fmt}}(y) + R_{\text{overlong}}(y)\)$
with \(\alpha = 0.2\). The accuracy reward \(R_{\text{acc}}\) dynamically routes to one of ten specialized verifiers: binary matches for exact strings, multiple choice, numeric values (via math-verify), and string sets; permutation scoring for visual ordering; field-level weighted match for web actions; Hungarian matching with IoU/F1 at threshold 0.5 for bounding box grounding; point-in-box verification for clicking; rule-based constraint checks for instruction following; and an OLMo3-style LLM-as-judge prompt for open-ended captions. The format reward \(R_{\text{fmt}}\) mandates <think>...</think><answer>...</answer> structures with boxed answers for discrete symbolic queries.
4. GSPO Policy Optimization: Stabilizing Entropy and Policy Updates
Standard GRPO utilizes per-token importance weight ratios, which exhibit significant variance accumulation and risk sudden policy entropy collapse during extended multimodal CoT generation. Vero implements Group Sequence Policy Optimization (GSPO), replacing independent token ratios with sequence-level log-probability differences \(\bar{\Delta}_i = \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} (\log \pi_\theta(y_{i,t}) - \log \pi_{\theta_{\text{old}}}(y_{i,t}))\) to derive token-level importance ratios \(s_{i,t}(\theta) = \exp(\text{sg}(\bar{\Delta}_i) + \log \pi_\theta(y_{i,t}) - \text{sg}(\log \pi_\theta(y_{i,t})))\). Coupled with asymmetric clipping (\(\epsilon_{\text{high}} > \epsilon_{\text{low}}\)), complete removal of the KL penalty, and a linear overlong penalty ramp \(R_{\text{overlong}}(y)\), GSPO maintains an exploration entropy of \(0.58 \pm 0.11\) throughout training, substantially outperforming DAPO (\(0.22 \pm 0.15\)) and GRPO (\(0.50 \pm 0.11\)).
Loss & Training¶
The policy parameters \(\theta\) are trained on groups of \(G\) rollout responses by maximizing the clipped sequence-level objective: $\(J(\theta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min \left( s_{i,t}(\theta) A_i, \, \text{clip}(s_{i,t}(\theta), 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}) A_i \right)\)$ where \(A_i = (r_i - \mu_g) / (\sigma_g + \epsilon)\) represents the sequence-constant advantage normalized across the group. Within the buffer region \([L_{\max} - B, L_{\max}]\) prior to the context limit (\(B=2048, \lambda=1.0\)), a soft linear penalty \(R_{\text{overlong}}(y) = \min(-\frac{|y| - (L_{\max} - B)}{B}\lambda, 0)\) is applied to penalize overlong CoT sequences without inducing abrupt truncation artifacts.
Key Experimental Results¶
Main Results¶
Evaluation is conducted on VeroEval, an evaluation suite consisting of 30 benchmarks spanning all six categories. The table below presents Avg@5 across runs for Vero variants versus open-weight and open-recipe baselines:
| Model | Setup | Chart & OCR | STEM | Spatial & Action | Knowl. & Recog. | Ground. & Search | Cap. & IF | Overall Avg (30 Evals) |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B-Instruct | Open Base | 57.6 | 41.0 | 55.1 | 49.5 | 51.1 | 64.8 | 52.9 |
| OpenMMReasoner-7B | Open RL (874k Distill) | 61.1 | 52.0 | 56.3 | 49.4 | 46.0 | 47.8 | 52.2 |
| Vero-Qwen25-7B | Ours (No Distill) | 61.2 | 46.4 | 59.5 | 52.8 | 57.1 | 72.1 | 57.8 (+4.9) |
| Qwen3-VL-8B-Instruct | Open Base | 61.2 | 57.3 | 62.6 | 52.3 | 58.5 | 78.2 | 60.7 |
| Qwen3-VL-8B-Thinking | Open CoT Base | 67.3 | 64.5 | 63.7 | 54.3 | 53.3 | 81.4 | 62.3 |
| OneThinker-8B | Open RL (340k Distill) | 61.5 | 57.0 | 60.3 | 50.1 | 47.3 | 65.1 | 55.7 |
| Vero-Qwen3I-8B | Ours (No Distill) | 69.5 | 64.0 | 67.2 | 53.8 | 63.4 | 83.5 | 66.1 (+5.4) |
| Vero-Qwen3T-8B | Ours (No Distill) | 71.5 | 65.3 | 67.3 | 54.9 | 60.7 | 80.7 | 65.8 (+3.5) |
| Qwen3.5-9B | Open Base | 78.1 | 73.3 | 71.0 | 58.2 | 68.2 | 83.3 | 71.5 |
| Vero-Qwen35-9B | Ours (No Distill) | 80.5 | 76.7 | 75.0 | 60.7 | 70.5 | 87.4 | 74.4 (+2.9) |
| Vero-Qwen35-9B-Base | Ours (Pure RL on Base) | 79.2 | 73.6 | 72.4 | 60.7 | 69.1 | 87.8 | 73.0 (+12.9) |
Ablation Study¶
Systematic ablations on training paradigms, reward formulation, and RL algorithms were conducted on Qwen2.5-VL-7B-Instruct (absolute benchmark averages reported):
| Ablation Dimension | Configuration | Overall Avg | Policy Entropy | Key Findings & Notes |
|---|---|---|---|---|
| Paradigm | Base Model | 52.4 | - | Initial base instruction model performance |
| Paradigm | FineVision SFT | 46.2 | - | Supervised fine-tuning leads to cross-task overfitting (-6.2) |
| Paradigm | Vero SFT | 52.8 | - | SFT on Vero-600K provides minimal improvement (+0.4) |
| Paradigm | Vero RL (GSPO) | 57.2 | 0.58 | Online RL unlocks substantial general capability gains (+4.8) |
| Reward Design | Math-Verify Only | 51.8 | - | Fails to parse non-math outputs; Captioning & IF collapses to 34.3 |
| Reward Design | Task-Routed Reward | 57.2 | - | 10 specialized verifiers ensure balanced gains across all domains |
| RL Algorithm | DAPO | 54.3 | 0.22 | Suffers rapid entropy decay and premature policy collapse |
| RL Algorithm | GRPO | 54.3 | 0.50 | Token-level ratios yield higher variance and sub-optimal convergence |
| RL Algorithm | GSPO | 54.7 | 0.58 | Sequence-averaged importance ratio stabilizes gradient updates |
Key Findings¶
- Data diversity eliminates negative cross-task transfer: Single-task RL leads to severe degradation on non-target domains (e.g., training only on STEM drops Captioning & IF by -21.2 points; training on non-grounding tasks drops Grounding & Search by 3 to 4 points). In contrast, uniformly mixed training across all six categories yields consistent positive transfer across all benchmarks.
- Base models unlocked via pure RL: Directly training Qwen3.5-9B-Base with Vero RL without any intermediate SFT yields an extraordinary +12.9 point leap (from 60.1 to 73.0), outperforming many tuned post-trained models and validating the standalone power of multi-task RL.
- Distinct cognitive behaviors across task domains: Trace analysis using cognitive frameworks reveals distinct operational modes: Spatial tasks trigger the longest reasoning traces (averaging 1983 words) with perception-then-reasoning ordering; STEM tasks substantially increase backtracking behaviors (from 0.27 to 0.48); Grounding tasks actively suppress introspective self-awareness (dropping from 0.73 to 0.49), channeling compute toward directed spatial search.
Highlights & Insights¶
- Uniform mixing outperforms complex reweighting: Balancing task ratios equally (100K per category) consistently outperforms schemes tuned for difficulty, response length, or image resolution, establishing a remarkably clean baseline for multimodal RL data engineering.
- Beating distillation warm starts with open RL: Without utilizing any teacher-generated synthetic reasoning traces, Vero-Qwen25-7B (57.8) beats OpenMMReasoner-7B (52.2), which relied on 874K samples of proprietary teacher distillation, demonstrating that online multi-task RL is inherently self-sufficient.
- Expressive routed verifiers prevent modality collapse: Reward routing across ten distinct verification mechanisms is essential for multi-task RL, preventing format collapse on open-ended and spatial grounding queries.
Limitations & Future Work¶
- Computational overhead of LLM judges: Relying on an LLM-as-judge for open-ended descriptive tasks introduces non-trivial inference latency and memory footprints during large-scale RL rollouts.
- Dense small-object localization: On extreme high-resolution aerial datasets like AerialVG, fine-grained object grounding performance shows remaining headroom, highlighting the need for dynamic multi-crop and adaptive zooming policies.
- Future directions: Investigating compact learned reward discriminators to replace LLM judges, and extending the multi-task RL formulation to long-context video causal reasoning and interactive embodied robotics.
Related Work & Insights¶
- vs OpenMMReasoner / VL-Rethinker: Prior works focus narrowly on visual math or perception and rely on teacher distillation warm-ups; Vero proves that multi-task data diversity and routed rewards enable broader generalisation via pure online RL.
- vs Qwen3-VL-8B-Thinking / MiMo-VL-7B-RL: Proprietary recipes conceal data distributions and reward designs; Vero matches or surpasses these models under identical architectures while releasing all data, code, and model weights.
Rating¶
- Novelty: ⭐⭐⭐⭐ [First fully open single-stage multi-task RL recipe spanning six visual capabilities]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 6 initial models, 30 VeroEval benchmarks, and fine-grained cognitive ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, structured presentation, and insightful cognitive behavioral analysis]
- Value: ⭐⭐⭐⭐⭐ [High-impact open-source milestone democratizing post-training visual RL for the community]