Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Multimodal Self-Evolution, Majority Voting Bias, Retracing Re-inference, Softened Frequency Reward, Geometric Reasoning
TL;DR¶
To alleviate model collapse and sampling sparsity caused by discrete binary majority voting in unsupervised self-evolution of Multimodal LLMs, this paper proposes Continuous Softened Retracing reSampling (CSRS), which performs local retracing re-inference from anchor points, provides continuous softened frequency rewards calibrated by variance, and injects visual semantic perturbation to enforce invariant mathematical logic.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) have demonstrated substantial proficiency in multimodal perception and scientific reasoning. However, unlocking further advanced reasoning capabilities is heavily bottlenecked by the scarcity and high cost of human-annotated chain-of-thought demonstrations. To circumvent this annotation bottleneck, unsupervised self-evolution post-training paradigmsโsuch as MM-UPTโhave emerged. In these setups, reinforcement learning frameworks like Group Relative Policy Optimization (GRPO) enable the model to generate multiple candidate reasoning paths for unlabeled inputs, using the most frequent output from majority voting as a pseudo-golden label to compute relative intra-group advantages for autonomous policy optimization.
Despite its promise, generating internal reward signals via majority voting inherently triggers a hazardous confirmation bias cycle. Because the candidate reasoning space grows exponentially (\(O(b^L)\)) with chain length, objectively correct reasoning paths often reside in the low-probability long-tail distribution. Meanwhile, an uncalibrated base model inevitably possesses intrinsic inductive biases and hallucinated heuristics. When majority voting assigns discrete binary (0/1) rewards based exclusively on the final answer, gradient updates become excessively concentrated on high-frequency erroneous modes. This positive feedback loop rapidly contracts the model's exploratory entropy, causing the contrastive factor between dominant and tail paths to expand exponentially until the policy degenerates into a deterministic mappingโa catastrophic phenomenon termed "model collapse."
The fundamental tension lies in the fact that unsupervised self-evolution requires self-consistency feedback to bootstrap capabilities, yet coarse-grained majority voting exponentially amplifies early-stage distributional errors instead of discovering valid long-tail paths. This paper attacks the problem by recognizing that global stochastic exploration is inefficient at critical reasoning forks, and that reward signals must transition from discrete binary indicators to continuous, variance-sensitive dampeners. Core idea: Continuous Softened Retracing reSampling (CSRS) truncates initial maternal trajectories at critical anchor points to perform dense local re-inference, calibrates continuous softened frequency rewards using answer frequency variance across global and local sets, and applies visual semantic perturbation to compel the model to prioritize invariant mathematical logic.
Method¶
Overall Architecture¶
CSRS establishes an unsupervised self-evolution pipeline for multimodal geometric reasoning. Given an input question paired with a geometric diagram, the framework first generates a batch of maternal reasoning trajectories, identifies intermediate retracing anchor points, performs local re-inference under visual Gaussian perturbation, computes continuous softened rewards based on answer frequencies across pooled sets, and finally updates the base MLLM via GRPO.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Geometric Image and Problem Prompt"] --> B["Maternal Trajectory Sampling<br/>Generate n initial reasoning paths"]
B --> C["Retracing Re-inference Mechanism<br/>Truncate at anchor point and sample m local paths"]
A --> D["Visual Semantic Perturbation<br/>Inject Gaussian noise into image"]
D --> C
C --> E["Pooled Answer Set Aggregation<br/>Construct unified pool combining maternal and local sets"]
E --> F["Softened Frequency Reward<br/>Calibrate continuous reward via frequency variance and tanh"]
F --> G["GRPO Policy Optimization<br/>Compute group relative advantage and update model weights"]
Key Designs¶
1. Retracing Re-inference Mechanism: Overcoming Sampling Sparsity in Exponential Spaces
In multi-step mathematical reasoning, minor early deviations amplify exponentially over long horizons, making globally sampled correct paths exceedingly rare under practical compute budgets. Drawing upon the assumption of local stabilityโthat valid reasoning must exhibit semantic consistency and invariance when restarted from key intermediate pivotsโthe Retracing Re-inference Mechanism (RRM) transforms sparse global search into dense local exploration. The model initially generates \(n=8\) maternal trajectories \(A_n = \{R_{p1}, \dots, R_{pn}\}\). For each trajectory of length \(\text{len}(R_p)\), an anchor point is established at a fixed retracing rate \(\omega=0.7\), truncating the trajectory at token \(\lfloor \omega \cdot \text{len}(R_p) \rfloor\) and concatenating it with the prompt: $\(X'_p = \text{Concat}(X_p, (y_1, y_2, \dots, y_{\lfloor \omega \cdot \text{len}(R_p) \rfloor}))\)$ From this conditioned prompt, the model samples \(m=5\) local re-inference rollouts, producing \(mn=40\) re-inference trajectories \(A_m\). This dense local branching at critical reasoning forks forces the model to re-explore alternative deductions, systematically recovering high-value, long-tail paths that would otherwise be extinguished by the dominant initial distribution.
2. Softened Frequency Reward: Mitigating Model Collapse via Continuous Variance Calibration
Binary 0/1 rewards under majority voting drive the policy contrastive factor \(G_n = \exp(\eta \Delta r_n)\) to its theoretical maximum \(e^\eta\), causing distribution collapse toward majority modes. The Softened Frequency Reward (SFR) introduces continuous probability damping. By constructing the unified set \(A_{\text{all}} = A_n \cup A_m\) containing all \((m+1)n=48\) trajectory answers, the base reward of any candidate answer \(a\) is computed as its statistical frequency in the whole pool: $\(R_{\text{base}}(a) = \frac{\text{Count}(a, A_{\text{all}})}{(m + 1)n}\)$ To evaluate whether a path demonstrates latent robustness, SFR measures the frequency variance between the local re-inference set \(A_m\) (frequency \(f_r(a) = \frac{\text{Count}(a, A_m)}{mn}\)) and the pooled set \(A_{\text{all}}\). A frequency increase (\(f_r > f_{\text{base}}\)) signifies that the path possesses superior conditional stability at decision pivots despite being a long-tail candidate globally. The final reward is formulated as: $\(R_{\text{final}} = \left( \gamma \tanh\left( \beta \left( \frac{f_r}{f_{\text{base}} + \epsilon} - 1 \right) \right) + 1 \right) \times R_{\text{base}}\)$ with hyperparameters \(\gamma = 0.2\) and \(\beta = 5.0\). The parameter \(\beta\) sensitizes the reward to correct low-frequency paths, while the saturation property of \(\tanh\) caps the reward for high-frequency samples, ensuring \(G_{\text{SR}} = \exp(\eta(\rho - \epsilon)) < \exp(\eta)\) and preventing overfitting to common false modes.
3. Visual Semantic Perturbation: Suppressing Superficial Image Heuristics
MLLMs often exploit superficial visual patterns or background artifacts in geometric figures (e.g., stroke styles, canvas textures) rather than executing genuine theorem-based deductions. To enforce reliance on invariant mathematical logic, CSRS injects zero-mean Gaussian noise \(\epsilon \sim \mathcal{N}(0, \sigma^2)\) into the input image during the re-inference phase: $\(I' = I + \epsilon\)$ By disrupting pixel-level visual shortcuts, the perturbation forces the model to focus strictly on structural geometric invariants and formal symbolic relations, shielding the self-evolution loop from visual hallucinations and spurious correlations.
A Worked Example¶
Consider a representative geometric problem from the testbed: Inscribed quadrilateral \(ABCD\) in circle \(Z\) with central angle \(m\angle BZA = 104^\circ\), arc \(m\widehat{CB} = 94^\circ\), and \(AB \parallel DC\); find \(m\angle BDA\) (ground truth: \(52^\circ\)). 1. Initial Maternal Rollouts: The model generates 8 initial paths. Distracted by the parallel lines, 6 out of 8 trajectories inappropriately apply alternate interior angle properties and yield the incorrect answer \(47^\circ\) (a 75% dominant majority), while only 1 path correctly uses the inscribed angle theorem (\(\angle BDA = \frac{1}{2}\angle BZA = 52^\circ\), frequency 12.5%). Under conventional majority voting, \(47^\circ\) receives reward 1 while \(52^\circ\) receives reward 0, reinforcing the error. 2. Retracing Re-inference: CSRS establishes an anchor point at 70% of the reasoning trajectory, adds Gaussian visual noise, and resamples 5 rollouts per anchor. Unburdened by the superficial distraction, the model discovers the direct relationship between arc \(BA\) and central angle \(BZA\), causing the frequency of \(52^\circ\) in the re-inference set to rise sharply to 20.0% or higher. 3. Continuous Reward Calibration: Because \(\frac{f_r}{f_{\text{base}}} > 1\), the variance ratio stimulates the \(\tanh\) scaling, lifting the reward of the correct \(52^\circ\) response (e.g., from 0 to 0.237+), while damping the high-frequency reward of \(47^\circ\), guiding policy updates toward genuine mathematical deductions.
Loss & Training¶
The framework is implemented in the veRL framework using GRPO, training a Qwen2.5-VL-7B-Instruct backbone over 15 epochs on 8 NVIDIA A800-80G GPUs (taking ~36 hours). It uses the AdamW optimizer with an initial learning rate of \(1 \times 10^{-6}\), weight decay of 0.01, train batch size of 256, micro-batch size of 8, and a low-variance KL coefficient \(\beta = 0.01\). The retracing rate is fixed at \(\omega = 0.7\) with \(m = 5\) local rollouts. Compared to stochastic anchor selection \(\omega \sim \mathcal{U}(0, 1)\), the fixed-anchor strategy prevents redundant prompt re-generation and avoids distributed sync bottlenecks caused by high response length variance, cutting relative training runtime by 13% (1.0ร vs. 1.13ร).
Key Experimental Results¶
Main Results¶
CSRS was evaluated across four challenging multimodal mathematical and geometric reasoning benchmarks (MathVision, MathVerse, MathVista, and We-Math), benchmarking against the un-tuned base models and the leading unsupervised baseline MM-UPT.
| Model & Method | Unsupervised? | Training Dataset | MathVision | MathVerse | MathVista | WeMath | Avg |
|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B (Base) | - | - | 22.47 | 34.54 | 62.30 | 57.53 | 44.21 |
| Qwen2.5-VL-7B (Base) | - | - | 25.40 | 44.24 | 66.42 | 67.65 | 50.93 |
| MM-UPT (Baseline) | Yes | Geometry3K | 26.95 | 44.53 | 66.47 | 68.49 | 51.61 |
| MM-UPT (Baseline) | Yes | GeoQA | 26.61 | 44.15 | 65.84 | 68.25 | 51.21 |
| MM-UPT (Baseline) | Yes | MMR1 | 25.98 | 45.12 | 66.27 | 69.14 | 51.63 |
| CSRS (Ours) | Yes | Geometry3K | 27.97 | 46.01 | 67.81 | 71.77 | 53.39 |
| CSRS (Ours) | Yes | GeoQA | 28.95 | 45.82 | 68.25 | 69.32 | 53.09 |
| CSRS (Ours) | Yes | MMR1 | 27.86 | 45.89 | 67.81 | 70.53 | 53.05 |
Generalization across model families was also confirmed: on Geometry3K, InternVL3-8B improved from 54.82% to 56.49% average accuracy under CSRS, and ThinkLite-VL-7B rose from 51.79% to 53.58%.
Ablation Study¶
Ablation experiments on Qwen2.5-VL-7B trained on Geometry3K systematically dissect the contributions of each individual component.
| Configuration | MathVision | MathVerse | MathVista | WeMath | Note |
|---|---|---|---|---|---|
| MajorityVote (Baseline) | 25.40 | 44.24 | 66.42 | 67.65 | Binary hard rewards, suffers from collapse |
| + SFR | 26.52 | 45.10 | 67.15 | 69.42 | Continuous reward curtails distributional polarization |
| + RRM | 26.85 | 45.35 | 67.38 | 69.85 | Local resampling uncovers long-tail candidates |
| + VSP | 25.55 | 44.35 | 66.55 | 67.82 | Suppresses superficial visual shortcuts |
| + SFR + RRM | 27.68 | 45.85 | 67.62 | 71.30 | Strong synergy between exploration and calibration |
| + SFR + VSP | 26.70 | 45.22 | 67.28 | 69.65 | Lacks dense local exploration at decision forks |
| + RRM + VSP | 27.02 | 45.48 | 67.50 | 70.12 | Lacks continuous reward damping |
| CSRS (Full: SFR+RRM+VSP) | 27.97 | 46.01 | 67.81 | 71.77 | Full system achieves top performance across all tasks |
Hyperparameter analyses demonstrate that: 1. Re-inference Rollouts \(m\): Accuracy peaks at \(m=5\) (68.25% on MathVista); smaller values fail to explore sufficient local diversity, while larger values (\(m=8\), 64.98%) introduce noise and error accumulation. 2. Retracing Rate \(\omega\): Accuracy peaks at \(\omega=0.7\) (68.25%); smaller rates (\(\omega=0.1\), 66.31%) revert the search to an intractable global space, while larger rates (\(\omega=0.9\), 66.43%) leave virtually no remaining path for error correction. 3. Reward Sensitivity: Optimal robustness is achieved at \(\gamma=0.2\) and \(\beta=5.0\), with minor perturbations exhibiting negligible performance variance.
Key Findings¶
- RRM and SFR act as a dual engine of exploration and calibration: Adding either SFR (+1.12% to +1.77%) or RRM (+1.11% to +2.20%) independently improves results, but combining them (+SFR+RRM) yields a dramatic jump on WeMath to 71.30% (+3.65% over baseline). RRM retrieves dormant long-tail solutions, while SFR accurately rewards them.
- Entropy decay and certainty distributions verify collapse prevention: Visualization of policy entropy during training indicates that MM-UPT suffers from steep entropy collapse and an unnatural surge of high-certainty samples (\(f \in [0.8, 1.0]\)). In contrast, CSRS maintains a steady, gradual entropy decay, preserving necessary exploration diversity.
- Fixed anchor points outperform stochastic retracing in training throughput: Random anchor selection \(\omega \sim \mathcal{U}(0, 1)\) slightly degrades accuracy (MathVision 27.63% vs. 27.97%) and incurs a 1.13ร compute time penalty due to redundant re-generation of short prompts and distributed straggler latency under uneven rollout lengths.
Highlights & Insights¶
- Reformulating discrete pseudo-label voting into continuous frequency variance: Instead of forcing a hard 0/1 partition, CSRS uses frequency variance across global and conditioned local sets mapped through a saturating hyperbolic tangent, elegantly rewarding promising long-tail paths while bounding high-frequency overconfidence.
- Point-conditioned local exploration over brute-force scaling: Rather than inflating batch sizes across the entire reasoning horizon, establishing retracing anchor points at critical intermediate forks provides dense search precisely where branching decisions matter most.
- Noise injection as a regularizer against visual shortcuts: Applying Gaussian perturbation during re-inference successfully severs superficial visual dependencies, ensuring the model evolves invariant geometric reasoning rules.
Limitations & Future Work¶
- Restricted scope to closed-form mathematical and geometric domains: The experimental evaluation focuses exclusively on problems with unique, deterministic symbolic solutions. Extending the continuous retracing paradigm to open-ended visual question answering and subjective dialogue remains an open challenge.
- Heuristic uniform anchor positioning: The retracing rate \(\omega\) is fixed at 70% across all problems. An adaptive anchor selector driven by token-level perplexity spikes or attention entropy drops could pinpoint semantic decision forks more precisely.
- Single-turn retracing without deep tree search: The current framework relies on a single maternal rollout followed by one retracing step. Integrating CSRS with multi-turn Monte Carlo Tree Search (MCTS) or recursive self-correction could further enhance complex competition-level reasoning.
Related Work & Insights¶
- vs MM-UPT (Wei et al., 2025): MM-UPT established the unsupervised post-training framework combining majority voting with GRPO, but its binary rewards render it prone to early model collapse; CSRS introduces local re-inference and continuous softened rewards, outperforming MM-UPT by up to 3.28% across benchmarks.
- vs EvolRL & VisPlay (Zhou et al., 2025; He et al., 2025): EvolRL employs novelty rewards, and VisPlay adopts questioner-solver self-play; CSRS uniquely tackles intermediate trajectory sparsity and visual surface shortcuts via anchor retracing and semantic noise injection.
Rating¶
- Novelty: โญโญโญโญโ [Pioneers continuous softened frequency variance rewards and anchor-based retracing re-inference to resolve majority voting collapse in MLLM self-evolution]
- Experimental Thoroughness: โญโญโญโญโญ [Evaluated across four major benchmarks, three MLLM families, with extensive ablations, variance sensitivity, and case studies]
- Writing Quality: โญโญโญโญโญ [Clear mathematical derivations, tightly linked motivation and methodology, and highly structured empirical reporting]
- Value: โญโญโญโญโญ [Provides a practical, label-free reinforcement learning post-training recipe for multimodal reasoning models]