Skip to content

Self-Evolving Vision Language Models for Structured Visual Understanding From Zero Data

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Vision-Language Models, Self-Evolution, Reinforcement Learning, GRPO, Structured Visual Reasoning, Zero Data

TL;DR

MM-Zero is the first reinforcement learning framework to accomplish zero-data self-evolution for vision-language models, decomposing a single base model into a collaborative triad—Proposer (abstract design & QA), Coder (executable SVG rendering), and Solver (multimodal reasoning)—to achieve continuous reasoning gains without any human annotations or seed images.

Background & Motivation

Self-evolving systems driven by self-play and synthetic task generation have achieved remarkable breakthroughs in large language models (LLMs), allowing reasoning capabilities to blossom from scratch without human intervention. However, extending this self-improvement frontier to Vision-Language Models (VLMs) is fundamentally bottlenecked by cross-modal asymmetry: unlike pure language agents that can freely postulate and answer text queries, multimodal models require perceptual visual anchors to ground visual reasoning. Existing multimodal self-evolution paradigms (such as VisPlay, Evolmm, and V-Zero) adapt classic dual-role (Proposer–Solver) setups, yet their iterative generation loops remain rigidly conditioned on massive collections of static, pre-collected real-world images.

This reliance merely shifts the curation bottleneck from human-annotated QA labels to image harvesting and data cleaning: the model's evolutionary trajectory remains strictly bounded by the distribution, resolution, diversity, and noise ceiling of the static image repository. Collecting and filtering open-domain web images introduces heavy engineering overhead and fails to synthesize dynamically controllable, targeted visual scenarios for complex geometric construction, dense numeric charts, and multi-step symbolic deduction. In contrast, programmatically rendered visual data offers deterministic precision, infinite combinatorial diversity, and intrinsic rule-based verifiability, presenting a transformative pathway toward bootstrapping multimodal reasoning.

The core insight of this work is that multimodal self-evolution should not be restricted to solving questions over fixed images, but must encompass the autonomous synthesis of visual environments within the evolutionary loop. Core idea: build a multi-role zero-data self-evolving framework (MM-Zero) where a single base model is decoupled into three specialized roles—a Proposer generating structured concepts and tiered questions, a Coder compiling specifications into executable SVG graphics, and a Solver executing multimodal reasoning—interacting in a closed loop guided by verifiable execution feedback, difficulty-balancing rewards, and GRPO.

Method

Overall Architecture

MM-Zero abandons the conventional two-player setup in favor of an interacting triad instantiated from the identical base model checkpoint: a Proposer (\(\pi_P\)), a Coder (\(\pi_D\)), and a Solver (\(\pi_S\)). The entire system operates in an automated, closed-loop lifecycle: 1. Abstract Conception (Proposer): The Proposer outputs a structured quadruple \(x = (c, q_{\text{easy}}, a_{\text{easy}}, q_{\text{hard}})\), containing a fine-grained visual scene description \(c\), a simple verification question and its deterministic answer \((q_{\text{easy}}, a_{\text{easy}})\) readable upon successful image rendering, and a challenging multi-step question \(q_{\text{hard}}\). 2. Visual Synthesis (Coder): The Coder translates the specification \(c\) into raw, executable SVG markup \(C\). A sandboxed CairoSVG engine renders the code into high-resolution PNG images while enforcing strict geometric formatting checks. 3. Multimodal Reasoning & RLVR Feedback (Solver): The Solver inspects the rendered visual image and first answers \(q_{\text{easy}}\) to confirm semantic rendering fidelity, providing verifiable correctness signals to update the Coder. Subsequently, the Solver performs multi-path sampling on the challenging question \(q_{\text{hard}}\), establishing silver consensus labels via majority voting (TTRL) and computing difficulty rewards that optimize the Proposer while updating its own reasoning policy under Group Relative Policy Optimization (GRPO).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single Base Model Checkpoint<br/>Qwen3-VL / MiMo-VL"] --> B["1. Triadic Collaborative Architecture<br/>Proposer + Coder + Solver"]
    B --> C["2. Executable Vector Visual Synthesis<br/>Raw SVG Code Generation & CairoSVG"]
    C --> D["3. Difficulty Balancing & Multi-Faceted Rewards<br/>Goldilocks Curricula + Capped Solvability + Type Diversity"]
    D --> E["4. Test-Time Silver Labels & Bi-Level Filtering<br/>TTRL Majority Consensus + Non-Degenerate Pruning"]
    E -->|Iterative Multi-Role GRPO Updates| B

Key Designs

1. Triadic Collaborative Architecture: Breaking the Static Image Bottleneck

Addressing the fundamental limitation of prior dual-role (Proposer–Solver) systems that depend on pre-existing static image corpora, MM-Zero expands the agent ecosystem by incorporating a dedicated Coder role to establish a closed loop: "Concept Proposal \(\to\) Code Synthesis \(\to\) Multimodal Solving \(\to\) Reward Verification". All three roles originate from the exact same pre-trained VLM backbone. During training, roles are optimized in an alternating sequential schedule: while the active agent undergoes policy improvement via GRPO, the other two agents remain frozen, acting as deterministic environment generators and objective reward critics. The Proposer expands conceptual scenarios, the Coder sharpens programmatic spatial rendering, and the Solver refines long-chain multimodal reasoning—mutually co-evolving without a single external seed image.

2. Executable Vector Visual Synthesis: Grounding Semantics with Rule-Based Precision

Rather than relying on diffusion models or raster image generators—which suffer from hallucinated artifacts, imprecise coordinates, deformed text, and unverifiable semantics—MM-Zero leverages executable vector graphics (SVG markup and Python/Matplotlib code) as the synthesis medium. The Coder generates self-contained SVG code specifying explicit coordinate spaces, color codes, bounding geometries, and data series. The rendering pipeline uses a high-throughput sandboxed cairosvg engine with strict topology filters (aspect ratio \(\le 100\), dimensions \(\le 16384\)). Because SVG is mathematically transparent and programmatically verifiable, it guarantees exact geometric alignment and zero visual ambiguity, establishing a deterministic physical foundation for Reinforcement Learning with Verifiable Rewards (RLVR).

3. Difficulty Balancing & Multi-Faceted Rewards: Preventing Shortcut Exploitation and Collapse

In unconstrained self-improving multi-agent environments, policies tend to exploit degenerate shortcuts: the Proposer may craft trivially easy questions or bake answers directly into the visual canvas (Reward Hacking), or generate intractable gibberish. MM-Zero introduces a balanced, multi-faceted reward function to regularize optimization. For the Proposer, the total reward is formalized as: $\(R_P(x) = \begin{cases} -1, & \text{if format invalid} \\ \frac{1}{N}\sum_{i=1}^N \mathbf{1}_{\text{exec}}(C_i) \cdot \left(\min(R_{\text{solv}}(I_i), 0.5) + R_{\text{diff}}(I_i)\right) + r_{\text{eh}} + r_{\text{ct}} + r_{\text{div}}, & \text{otherwise} \end{cases}\)$ The solvability score \(R_{\text{solv}}\) is capped strictly at 0.5, preventing the model from maximizing reward solely by generating simplistic images; the difficulty reward \(R_{\text{diff}} = \min(c_i, 1 - c_i) \in [0, 0.5]\) enforces the Goldilocks principle, peaking when the Solver exhibits maximal uncertainty on the hard question. Furthermore, \(r_{\text{eh}}\) penalizes trivial hard queries, \(r_{\text{ct}}\) penalizes category over-representation across seven content classes (data charts, diagrams, geometry, timelines, maps, tables, others), and \(r_{\text{div}}\) employs agglomerative clustering over BLEU distances to penalize lexical repetition. For the Coder, rewards dynamically combine rendering execution \(R_{\text{render}}\), solvability \(R_{\text{solv}}\), difficulty \(R_{\text{diff}}\), and explicit syntax error penalties \(\lambda_{\text{err}}\).

4. Test-Time Silver Labels & Bi-Level Filtering: Robust Self-Supervision without Gold Truth

To train the Solver on complex hard questions \(q_{\text{hard}}\) where human ground-truth labels do not exist, MM-Zero adopts Test-Time Reinforcement Learning (TTRL) via self-consistency. The Solver produces \(K\) independent reasoning trajectories, deriving an unbiased consensus silver label \(\bar{y} = \text{Mode}(\{y_1, \dots, y_K\})\). To prevent degenerate training data from corrupting the optimization loop, a bi-level filtering protocol is established: - Coder Data Filtering: Only captions yielding execution success rates between \([0.25, 0.75]\) across 4 rollouts are retained, excluding trivially simple or non-renderable prompts. - Solver Data Filtering: Samples are kept only if the easy question accuracy exceeds \(0.5\) (verifying visual faithfulness) while the hard question consensus accuracy falls within the informative, non-trivial window \([0.27, 0.75]\). This filtering protocol guarantees that downstream training operates exclusively within the zone of proximal development, making policy instability self-limiting.

Loss & Training

All roles are optimized via Group Relative Policy Optimization (GRPO). For a given prompt \(p\), the policy samples a group of \(N\) candidate responses \(\{x_1, \dots, x_N\}\) with corresponding rewards \(\{r_1, \dots, r_N\}\). The response-level advantage is normalized across the group: $\(\hat{A}_i = \frac{r_i - \text{mean}(r_1, \dots, r_N)}{\text{std}(r_1, \dots, r_N) + \varepsilon_{\text{norm}}}\)$ The clipped surrogate objective with a reference policy KL divergence penalty is formulated as: $\(\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{N}\sum_{i=1}^N \min\left( \frac{\pi_\theta(x_i)}{\pi_{\theta_{\text{old}}}(x_i)}\hat{A}_i, \text{clip}\left(\frac{\pi_\theta(x_i)}{\pi_{\theta_{\text{old}}}(x_i)}, 1-\epsilon, 1+\epsilon\right)\hat{A}_i \right) + \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\theta_{\text{old}}})\)$ Training is conducted across 8 \(\times\) 96GB NVIDIA RTX 6000 Ada/Pro GPUs over 3 sequential iterations (each iteration comprises 20 gradient steps per role, totaling 60 steps). The learning rate is fixed at \(1 \times 10^{-6}\) with weight decay \(1 \times 10^{-2}\).

Key Experimental Results

Main Results

Evaluation spans 9 established benchmarks divided into Natural Visual Reasoning (MMMU, MMMU-Pro, MM-Vet, HallusionBench) and Structured Visual Reasoning (MathVerse, MathVision, MathVista, VisNumBench, ChartQA). All evaluations are conducted via VLMEvalKit with Qwen3-32B serving as the judge model:

Method / Backbone MMMU MMMU-Pro MM-Vet HallusionBench ChartQA MathVerse MathVision MathVista VisNumBench Average
Prior Self-Evolution (Qwen2.5-VL-7B)
VisPlay (requires static images) 38.3 - 46.3 92.3 - 39.1 31.2 - 32.6 -
Vision-Zero (gamified self-play) - - - - 86.3 52.2 28.0 72.4 - -
V-Zero (requires static images) 58.6 - - - - 42.6 27.0 69.2 - -
Same-Backbone Control (Qwen2.5-VL-7B)
Base Model (published) 58.6 - 67.1 52.9 87.3 43.1 25.1 68.2 43.8 55.8
MM-Zero (Solver v3, Zero Data) 60.8 - 69.6 55.2 88.6 45.7 27.6 71.2 46.2 58.1 (+2.3)
Qwen3-VL-4B-Instruct
Base Model 53.4 36.5 71.1 51.4 79.1 52.1 31.6 61.3 45.3 53.5
MM-Zero (Step 20, Iter 1) 55.9 44.6 71.5 67.8 79.8 49.8 36.9 66.0 49.6 58.0
MM-Zero (Step 40, Iter 2) 56.8 45.1 71.8 69.1 79.6 49.4 37.1 64.7 49.1 58.1
MM-Zero (Step 60, Iter 3) 57.7 45.3 72.1 69.1 80.3 50.0 36.7 65.4 49.9 58.5 (+5.0)
Qwen3-VL-8B-Instruct
Base Model 69.6 55.9 76.1 61.1 89.6 62.1 53.9 77.2 47.7 65.9
MM-Zero (Step 20, Iter 1) 70.4 59.3 76.0 71.9 90.0 62.8 54.6 81.7 53.2 68.9
MM-Zero (Step 40, Iter 2) 71.8 61.3 75.4 71.3 90.4 64.2 55.1 83.1 54.5 69.7
MM-Zero (Step 60, Iter 3) 72.0 63.0 77.2 71.7 90.4 65.1 55.2 84.0 55.0 70.4 (+4.5)
MiMo-VL-7B-SFT
Base Model 64.6 45.2 75.6 70.8 92.9 67.1 57.9 71.8 44.7 65.6
MM-Zero (Step 60, Iter 3) 68.9 48.7 76.9 75.0 95.0 56.0 62.7 78.9 48.5 67.8 (+2.2)

Ablation Study

Ablations on Qwen3-VL-8B-Instruct dissect the influence of reward design choices and agent co-evolution components:

Configuration / Ablation MMMU MMMU-Pro MM-Vet HallusionBench ChartQA MathVerse MathVision MathVista VisNumBench Average
MM-Zero Full Model (Iter 3) 72.0 63.0 77.2 71.7 90.4 65.1 55.2 84.0 55.0 70.4
w/o Solvability & Difficulty Balance 70.4 56.1 78.8 58.9 91.1 64.9 59.4 77.6 54.8 68.0 (-2.4)
w/o Content Diversity (\(r_{\text{ct}}\)) 69.1 54.3 76.1 60.5 81.8 63.6 57.5 75.4 51.5 65.5 (-4.9)
w/o Easy-Hard Penalty (\(r_{\text{eh}}\)) 71.0 60.2 76.5 70.4 89.9 63.5 54.6 82.0 53.8 69.1 (-1.3)
Frozen Coder (Two-Role, Qwen3-4B) 54.2 37.4 72.6 52.7 78.8 51.2 31.5 61.1 47.1 54.1 (vs 58.5 full)

Key Findings

  • Coder Evolution Drives Systematic Gains: Freezing the Coder to isolate a two-role setup results in an average score gain of only +0.6 (from 53.5 to 54.1) across three iterations on Qwen3-4B, whereas the three-role MM-Zero improves by +5.0 points. Sustained code evolution is indispensable for expanding visual complexity and unlocking harder tasks.
  • Diversity Regularization Guards Against Mode Collapse: Removing the content diversity reward causes performance to initially peak at 67.4% before collapsing to 65.5% at iteration 3—falling below the base model (65.9%). Inspection reveals that without diversity penalties, the Proposer progressively collapses into generating trivial, easy-to-render histograms.
  • Transfer from Synthetic Vectors to Real-World Images: Despite training solely on programmatic SVG visuals, the Solver exhibits pronounced gains on natural image benchmarks, improving MMMU by +2.4 points and MMMU-Pro by +7.1 points. Precise symbolic layout understanding transfers effectively to real-world visual comprehension.

Highlights & Insights

  • Zero-Data Multimodal Closed Loop: Replaces the conventional dual-role paradigm with an autonomous Proposer-Coder-Solver triad, eliminating reliance on curated seed images and human labels.
  • Code Execution as Verifiable Reality: Utilizes SVG markup and Python rendering as an objective, programmatic bridge between language thought and visual geometry, providing deterministic RLVR feedback without neural perceptual hallucination.
  • Capped Solvability & Goldilocks Principle: The \(\min(R_{\text{solv}}, 0.5) + \min(c_i, 1 - c_i)\) formulation effectively eliminates reward hacking (e.g., burning answers into the image canvas) while preventing generation of intractable noise.

Limitations & Future Work

  • Domain Confinement of Vector Graphics: While SVG is ideal for quantitative plots, geometric figures, and schematics, it cannot naturally represent organic textures, fine-grained photorealistic noise, and 3D scenes.
  • Homogeneous Model Backbone: Initializing all three roles from the same base model may induce shared blind spots in later iterations. Future directions include exploring multi-backbone ecosystems and incorporating 3D graphics engines or diffusion generative models into the closed-loop system.
  • vs VisPlay / Evolmm / V-Zero: Prior VLM self-evolution works rely on pre-collected static image datasets, merely automating QA labeling. MM-Zero integrates generative Coder execution to achieve genuine zero-data self-evolution from scratch.
  • vs R-Zero / Absolute Zero: While LLM self-play leverages text-only self-consistency, MM-Zero overcomes the modal gap by grounding language in verifiable programmatic code execution.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering multi-role framework that overcomes the multimodal zero-data barrier via executable visual code generation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 3 base models, 9 standard benchmarks, iterative trajectories, decoding controls, and granular ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, crisp mathematical formulations, and compelling causal analyses.
  • Value: ⭐⭐⭐⭐⭐ Offers a transformative and sustainable paradigm for post-training vision-language models facing web data exhaustion.