Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/hqhQAQ/Syn-GRPO
Area: Multimodal VLM
Keywords: multimodal LLM, GRPO, online data synthesis, perception reasoning, exploration & diversity

TL;DR

Syn-GRPO attaches an online image synthesis service to GRPO training: the MLLM must additionally predict a diversity score and a text description for a new image, a data server keeps the foreground of the original image and re-paints the background from that description to produce replacements whose box labels stay valid, and a diversity reward teaches the model which samples are worth regenerating — lifting both the exploration space and the final accuracy on three visual perception tasks.

Background & Motivation

MLLM perception (referring expression comprehension, open-vocabulary detection, scene understanding) is the prerequisite for every downstream capability, and perception tasks fit reinforcement learning unusually well — box annotations make correctness programmatically verifiable, so GRPO, which needs no value network, has become the dominant training paradigm. VLM-R1, Visual-RFT, VisionReasoner and others have pushed perception metrics up with carefully designed task rewards. But this line of work silently assumes that the training data stays fixed throughout training. Analysing the GRPO training process on REC, the authors observe that both the entropy of the model's predictions (the uncertainty of its token distribution) and its diversity (the variance of correctness rewards across multiple responses to the same sample) start to fall off very early — the paper calls this entropy collapse and diversity collapse and traces it to data quality: visual perception samples are too uniformly formatted to elicit varied answers per sample, so within-group rewards barely differ, the advantage estimate degenerates into noise, the exploration space is squeezed flat, and training efficiency is extremely low.

The two existing lines of attack do not address the root. One constrains the algorithm: DAPO / Clip-Higher widen the sampling-ratio clipping threshold, Clip-Cov and KL-Cov clip or penalise high-covariance tokens, and Entropy Adv. writes entropy into the advantage function. These do slow entropy collapse down, but since the data itself cannot produce diverse responses, the gains saturate quickly — most visibly on visual perception tasks whose formats are more uniform. The other line adds data: PromptCoT, Genetic-Instruct, MetaSynth and TaskCraft target text domains such as math, code and tool use, while Absolute Zero and R-Zero do hook data synthesis into the GRPO loop but still synthesise text, and cannot adapt to visual perception tasks. The real tension is that a perception sample is an image plus a box: a language model can rewrite the question but cannot touch the image, and generating a brand-new image from scratch invalidates the box annotation, turning the RL reward into noise.

This paper's angle is to let image generation touch only the part that is allowed to change: as long as the target object (the foreground) stays intact, the box label remains correct no matter what the background becomes. On that basis the authors propose Syn-GRPO, an online data synthesis framework with two parts — a data server and a GRPO workflow. The data server keeps the foreground of the old image and re-paints the background via foreground segmentation plus outpainting, producing new samples whose labels are still usable; the GRPO workflow asks the MLLM to predict, besides its reasoning trace and final answer, two extra things — a \([0,1]\) diversity score and a sentence describing the new image — and supervises both with a diversity reward, so the model itself knows which samples are worth regenerating and what they should become. Core idea: define a sample's exploration value as the variance of its within-group correctness rewards, use that definition as a reward to supervise the MLLM's predicted image descriptions, and let a decoupled asynchronous data server render those descriptions into new foreground-preserving images, so that the training data co-evolves with the model.

Method

Overall Architecture

Syn-GRPO is a single loop with a feedback edge: the input is labelled visual perception data (image plus ground-truth boxes), the output is the trained MLLM, and in between two asynchronously running modules cooperate — the GRPO workflow trains the model and decides what the next batch of data should look like, while the data server turns that decision into an actual image. At the start of each epoch the data pool already holds images synthesised by the previous round; the workflow samples \(G\) responses per sample, each response carrying a reasoning trace, a final answer, a predicted diversity and a new image description. It then computes rewards including the diversity term, picks the description from the response with the highest diversity reward, and sends it to the data server over a unified API; the server extracts the foreground, re-paints the background according to the description, and returns a new image that carries the original box labels into the next round of rollouts. The image generation models stay frozen throughout — only the MLLM is optimised.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["original image + box labels"] --> B["self-evolving loop driven by the diversity reward<br/>sample responses, predict diversity and description"]
    B -->|ground-truth diversity| C["diversity smoothing<br/>EMA calibration of the declining diversity"]
    C -->|smoothed reward target| B
    B -->|asynchronous request via unified API| D["decoupled asynchronous data server"]
    D --> E["foreground-consistent data synthesis<br/>cut the foreground, repaint the background"]
    E -->|new images replace the old data| B

Key Designs

1. A self-evolving loop driven by the diversity reward: the MLLM decides which samples are worth regenerating

Picking a sample at random and regenerating it may well produce another image that elicits no diverse answers, wasting an entire generation. The authors' fix is to turn "which sample has exploration value" into a prediction task that RL itself can learn. A sample's ground-truth diversity is defined as the variance of the correctness rewards over its \(G\) responses:

\[V(q)=\mathrm{var}\big(\{R_{\rm acc}(o_i)\}_{i=1}^{G}\big)\]

A larger variance means the model's answers to this sample are more scattered and the sample has more exploration value; following the theoretical upper bound \(1/4\) of the variance, the paper normalises it to \([0,1]\) so it can be compared with the model's output. The MLLM is required to emit a predicted diversity \(v_i\in[0,1]\), and the diversity reward compares it against the (smoothed) ground-truth diversity — the closer the prediction, the higher the reward, again confined to \([0,1]\) (⚠️ in the cached full text, Eq. 5 only preserves the comparison form between \(R_{\rm diversity}(o_i)\), \(v_i\) and \(\mathcal{V}(q)\); the comparison operator could not be recovered, so this follows the paper's verbal description "compare the predicted diversity with the ground-truth diversity, result in \([0,1]\)" — refer to the original paper for the exact form). The final reward of a response sums three terms:

\[r_i = R_{\rm acc}(o_i) + R_{\rm format}(o_i) + R_{\rm diversity}(o_i)\]

where the accuracy reward follows each task's definition and the format reward enforces the four-part structure of reasoning trace, predicted diversity, new image description and final answer. The workflow then sends only the description of the highest-diversity-reward response for synthesis (\(i^{*}=\arg\max_i R_{\rm diversity}(o_i)\)), because that response judged the sample's potential for diverse answers most accurately, so its description is the most likely to yield an image that really does trigger diverse responses.

What makes this step pivotal is that data selection stops being a hand-written rule and becomes part of the model's capability: the more accurate the predicted diversity, the better the chosen description, the higher the quality of the synthesised data, and the stronger the model in the next round. Data synthesis is therefore not a one-off offline artefact but a process that iterates together with training.

2. Diversity smoothing: anchoring a reward target that keeps sliding downwards

There is a trap specific to self-bootstrapped systems here: the ground-truth diversity itself declines during training (diversity collapse), yet it serves as the supervision target of the diversity reward, so the current model is effectively learning the diversity level of the previous model and the target drifts further away the longer training runs — the paper calls this diversity drift. Left alone, the model chases a steadily sinking target and the reward signal is neither stable nor accurate.

The fix adds a layer of exponential-moving-average calibration to the reward target. Let the mean ground-truth diversity inside the \(k\)-th batch \(B_k\) be the batch average, and maintain a cross-batch moving average \(\mathcal{V}_k^{\rm global\_avg}=\beta\,\mathcal{V}_{k-1}^{\rm global\_avg}+(1-\beta)\,\mathcal{V}_k^{\rm batch\_avg}\) (initialised to the first batch average). The smoothed ground-truth diversity of sample \(q\) at that step is

\[\tilde{\mathcal{V}}(q)=\mathrm{clip}\Big(\mathcal{V}(q)\cdot \frac{\mathcal{V}_k^{\rm global\_avg}}{\mathcal{V}_k^{\rm batch\_avg}}\Big)\]

When the current batch sits below the historical average — i.e. collapse is happening — the ratio exceeds 1 and scales the batch back up to the historical level, keeping the yardstick of the reward from drifting with training; clip confines the result to \([0,1]\). β controls how much inertia the calibration has: β=0.7 is best for both 3B and 7B, while β=0.9 loses points because too much inertia freezes the smoothed value and amounts to no calibration at all.

3. A decoupled asynchronous data server: training and generation run in parallel

A synchronous implementation would destroy the economics of the method: if every batch had to wait for image generation before training could start, training time would nearly double (Table 5: Qwen2.5-VL-3B goes from 6.10 to 13.16 hours, 7B from 13.66 to 28.47 hours). The authors therefore make the data server a standalone module fully decoupled from the training loop: it runs as a Python HTTPServer, the GRPO workflow submits generation parameters (the original image plus the selected description) through a unified communication API, and the server returns results once generation finishes, so the two sides advance asynchronously. In practice the asynchronous version costs only about 0.35 extra hours over vanilla GRPO (6.45 hours for 3B, 13.93 for 7B), essentially hiding the generation cost inside the training time. Decoupling has a second benefit: the workflow depends on the API rather than the implementation, so swapping outpainting for another synthesis route (controllable editing, video generation) would not require touching the training code.

4. Foreground-consistent data synthesis: change the background, keep the labels

This is the precondition for the whole pipeline to work with RL at all. Generating a brand-new image from scratch immediately invalidates the box labels and turns \(R_{\rm acc}\) into noise; even if the boxes could be re-annotated, a large distribution gap between new and old data would destabilise training. The authors instead extract the target object with the foreground segmentation model BEN2, mask out everything outside the ground-truth boxes, and then feed the MLLM's predicted image description as a condition to a ControlNet outpainting model built on SDXL to re-paint the background and the extended canvas. Because the foreground pixels and the object's position are preserved verbatim, the box labels remain correct on the new image and the accuracy reward can be computed as usual; painting over an existing image rather than generating one from scratch also keeps new samples near the original data distribution. The segmentation and outpainting models are frozen for the entire run and act purely as data generators, adding no learnable parameters and receiving no gradients.

A Worked Example

Take one REC sample: the original image shows a dog lying on a living-room floor with a ground-truth box around the dog. The GRPO workflow samples \(G=6\) responses, each emitting four parts — a reasoning trace, a predicted diversity (say 0.32 / 0.55 / 0.78 …), a new image description (for instance "A dog in a messy living room with knit blanket, warm and soft floor lamps.") and the final answer box. From the six correctness rewards the authors compute the ground-truth diversity \(V(q)\), calibrate it through the moving average to \(\tilde{\mathcal{V}}(q)\), and compare each predicted diversity against it to obtain the diversity reward; the three reward terms are summed into the total reward used for the GRPO update, and the description of the highest-diversity-reward response is selected. That sentence travels over the unified API to the data server, which cuts out the dog and replaces the living room with the warm, cluttered space described — knit blanket, floor lamps — returning a new image. The new image carries the original box into the next epoch's rollouts. As epochs progress (the paper shows up to epoch 4), the scene generated from the same original image grows steadily more complex and crowded with objects, which is the visible evidence that self-evolution is actually happening.

Loss & Training

The training objective is GRPO under the sum of the three rewards above: the advantage is the group-normalised reward, \(\hat{A}_{i,t} = \big(r_i-\mathrm{mean}(\mathbf{r})\big)/\mathrm{std}(\mathbf{r})\), with a KL regularisation term against the reference model to keep the policy from drifting. Task rewards are defined per task: REC uses the IoU between the predicted and ground-truth boxes; OVD uses \(s_{\rm ovd}\cdot \mathrm{mAP}\) with \(s_{\rm ovd}=\min(N^{\rm gt}/N^{\rm pred},1)\) to penalise redundant predictions; ISR is decomposed into perception and refinement stages, the first sharing OVD's reward, and Syn-GRPO is applied to that first stage. The implementation builds on the verl framework with vLLM for rollout acceleration and FSDP for distributed training; base models are Qwen2.5-VL-3B / 7B, optimised with AdamW at a learning rate of \(1\times10^{-6}\), batch size 20, rollout number \(G=6\), 5 training epochs and β=0.7; four GPUs run the GRPO workflow and one GPU runs the data server. For data, REC follows VLM-R1 and uses RefCOCO / RefCOCO+ / RefCOCOg, randomly sampling 2,000 training examples for 5 epochs (10,000 sample updates, comparable to VLM-R1's 9,600); OVD uses 2,000 samples from D3; ISR uses 2,000 samples from 3D-FRONT.

Key Experimental Results

Main Results

Table 1 reports the main REC results, with both base models evaluated out of domain (LISA-Grounding) and in domain (RefCOCO / RefCOCO+ / RefCOCOg); Table 2 covers OVD. Syn-GRPO beats both vanilla GRPO at the same budget and the other training methods on both models and both tasks.

Model	Method	LISA	RefCOCO	RefCOCO+	RefCOCOg
Qwen2.5-VL-3B	Original	56.51	88.70	81.95	86.05
Qwen2.5-VL-3B	SFT	54.82	88.70	82.25	85.95
Qwen2.5-VL-3B	VLM-R1	63.14	90.55	84.30	87.10
Qwen2.5-VL-3B	Visionary-R1	60.80	86.85	82.65	86.50
Qwen2.5-VL-3B	GRPO	62.24	89.40	84.10	86.60
Qwen2.5-VL-3B	GRPO + Entropy Loss	64.23	90.20	82.55	85.60
Qwen2.5-VL-3B	GRPO + Entropy Adv.	63.69	90.90	83.85	86.40
Qwen2.5-VL-3B	GRPO + Offline Generation	64.23	91.10	83.40	86.30
Qwen2.5-VL-3B	Syn-GRPO (w/o \(R_{\rm diversity}\))	64.60	91.35	83.85	85.85
Qwen2.5-VL-3B	Syn-GRPO	68.28	92.15	85.30	87.45
Qwen2.5-VL-7B	Original	61.34	90.00	84.20	87.20
Qwen2.5-VL-7B	SFT	63.27	89.10	85.95	86.65
Qwen2.5-VL-7B	VLM-R1	66.71	92.60	89.40	89.00
Qwen2.5-VL-7B	Rex-Thinker	67.49	91.20	86.35	87.80
Qwen2.5-VL-7B	GRPO	65.26	91.90	89.85	87.80
Qwen2.5-VL-7B	GRPO + Entropy Loss	66.59	91.40	87.65	87.95
Qwen2.5-VL-7B	GRPO + Entropy Adv.	67.25	92.05	88.30	87.30
Qwen2.5-VL-7B	GRPO + Offline Generation	66.95	91.80	89.85	88.55
Qwen2.5-VL-7B	Syn-GRPO (w/o \(R_{\rm diversity}\))	67.67	92.75	89.50	88.75
Qwen2.5-VL-7B	Syn-GRPO	70.14	93.55	90.65	89.25

Method (OVD, Qwen2.5-VL-3B)	mAP	GP (IoU=0.5)	GR (IoU=0.5)
Original	14.20	56.06	33.79
SFT	18.50	53.15	39.40
VLM-R1	21.10	67.34	43.84
GRPO	18.66	63.83	36.95
Syn-GRPO	23.74	71.42	46.44

Ablation Study

Ablation	Setting	LISA accuracy	Note
Diversity reward	w/o \(R_{\rm diversity}\) (3B)	64.60	still above GRPO's 62.24 and offline generation's 64.23, but 3.68 below the full model
Diversity reward	w/o \(R_{\rm diversity}\) (7B)	67.67	2.47 below the full model's 70.14, so choosing which description to use matters a lot
Smoothing weight β	0.1 / 0.3 / 0.5 / 0.7 / 0.9 (3B)	67.43 / 67.31 / 67.85 / 68.28 / 67.55	rises then falls; too large a β limits how much the smoothed value updates
Smoothing weight β	0.1 / 0.3 / 0.5 / 0.7 / 0.9 (7B)	68.88 / 69.60 / 69.84 / 70.14 / 69.24	same trend as 3B
Data size	REC training samples 400 → 2,000	see Fig. 6	accuracy rises monotonically with data size, a data-scaling trend
Sync vs async	training time (3B / 7B)	6.10h vs 13.16h vs 6.45h / 13.66h vs 28.47h vs 13.93h	vanilla GRPO, synchronous synthesis, asynchronous synthesis

Key Findings

• The largest gains appear out of domain, not in domain: 3B goes from GRPO's 62.24 to 68.28 on LISA-Grounding (+6.04) and 7B from 65.26 to 70.14 (+4.88), while in-domain RefCOCO improves by only about 2.75 / 0.95. The authors attribute this to the broader distribution covered by synthesised data, which yields more robust and comprehensive perception — suggesting the value of the method lies more in generalisation than in chasing in-domain points.
• Entropy-constraint methods only treat the symptom: Entropy Loss / Entropy Adv. improve only marginally over GRPO (the 63–64 range for 3B), and on some in-domain columns (RefCOCO+, RefCOCOg) they even fall below GRPO, supporting the claim that nothing is fixed without changing the data.
• The diversity reward is the core contribution: removing it (descriptions chosen at random) still benefits from having new images, but lands 2.5–3.7 lower on LISA than the full model, showing that teaching the model to judge which sample deserves regeneration matters more than synthesis itself.
• Offline synthesis (descriptions from GPT-4o) loses to the online self-evolving loop: GRPO + Offline Generation reaches 64.23 on LISA for 3B — close to but below the w/o \(R_{\rm diversity}\) variant at 64.60, and clearly below the online version — indicating that a synthesis capability which evolves with training is something offline data cannot provide.
• Fair-comparison caveat: GRPO trails VLM-R1 on 3B (62.24 vs 63.14) because this paper trains on only 2,000 samples while VLM-R1 uses 9,600; the comparison is not budget-matched, and Syn-GRPO's gains are obtained under the smaller budget.
• Synthesised images grow more complex with each epoch: the scene generated from one original image becomes steadily more complex across epochs 1–4 (Fig. 8), the most direct evidence for the "self-evolving" claim; the paper also shows a few "surreal" generated images (Fig. 9) that the authors argue can induce novel reasoning paths.
• No capability regression is observed on the same tasks: the full model beats every ablated variant on all metrics; however, the paper does not report regression tests on general capabilities (e.g. generic VQA), so whether general perception is harmed cannot be judged from the current experiments.

Highlights & Insights

• Data synthesis moves from a one-off offline batch to a learnable capability inside the training loop: a sample's exploration value is defined as the variance of its within-group correctness rewards, and that same definition supervises the model's prediction — the better the model judges, the better the sample chosen, the stronger the next round, forming a positive feedback loop. This template of "use a statistic as supervision, let the model select its own data" transfers almost directly to any RL setting that needs active data selection (agent task filtering, curriculum learning).
• Foreground consistency is a very practical trick: the image can change its background while the box labels need no re-annotation, so the RL reward carries over untouched. Any task whose labels are bound to a local region (segmentation masks, keypoints, counting regions) can adopt the same recipe of "cut out the labelled region and repaint the surroundings", paying only the cost of masking that region as a condition.
• The decoupled async service design hides the generation cost: the synchronous version doubles training time while the asynchronous version adds roughly 0.35 hours — taking the heavy module out of the training main loop and talking to it over a unified API is an engineering decision that applies to any "training plus online generation/retrieval" pipeline.
• The discovery of diversity drift deserves its own note: in bootstrapped training, supervising the current model with historical statistics inevitably drifts, and the authors' re-calibration via the ratio of batch mean to global EMA is a general non-stationary-target correction (transferable to reward-model self-training, curriculum difficulty estimation, and similar settings).

Limitations & Future Work

• Dependence on generation quality and label consistency is never quantified: the whole pipeline assumes the foreground pixels survive outpainting and the box labels stay valid, but the paper offers only qualitative visualisations — no quantitative measure of foreground fidelity, seam artefacts, or annotation bias. For fine-grained localisation, the visual discontinuity at the boundary between repainted background and preserved foreground may itself become noise.
• "Long-horizon scalability" is still a trend-level claim: experiments run for 5 epochs, visualisations stop at epoch 4, and data size tops out at 2,000 samples. Fig. 6 shows an upward trend with data size, but real long-term self-evolution (behaviour after dozens of epochs, whether the data eventually collapses back to homogeneity) is untested.
• The definition of ground-truth diversity has edge cases: \(V(q)\) is the variance of correctness rewards over responses, so when all \(G=6\) responses are right or all are wrong the variance is 0 and the sample is treated as having no diversity — yet the hardest samples, where everything fails, are often exactly the ones with the highest exploration value. The definition lumps them together with samples that are simply too easy to discriminate. The paper does not discuss replacing it with entropy or another response-difference measure.
• Extra resource overhead: training needs one more GPU for the data server plus an SDXL-based outpainting model and the BEN2 segmentation model; the paper does not report per-sample generation latency as a function of dataset size (say tens of thousands of samples), so the throughput bottleneck at scale remains unclear.
• Task applicability is limited by the label format: the method needs box annotations that can be localised and a foreground that can be segmented, so it does not directly apply to pure VQA, counting or OCR tasks with no localisation labels; even the ISR task is only partially adapted (its perception stage).
• Improvement directions: replacing "take the single description with the highest \(R_{\rm diversity}\)" with combining or de-duplicating several descriptions could further lift the diversity of synthesised data; difficulty-aware scheduling in place of the fixed per-epoch full replacement is also worth trying.

Related Work & Insights

• vs VLM-R1 / Visual-RFT / Rex-Thinker: these design better task rewards and output formats on top of GRPO while keeping the training data fixed; this paper leaves the reward backbone alone and makes the data itself evolve during training, so the two are orthogonal and composable (the accuracy reward here is inherited directly from VLM-R1).
• vs DAPO / Clip-Higher / Clip-Cov / KL-Cov / Entropy Adv.: these constrain entropy from the optimisation side (wider clipping, clipping high-covariance tokens, KL penalties, entropy in the advantage), i.e. they make the model braver on passive data. This paper argues that the root cause of entropy and diversity collapse is homogeneous data, and instead injects samples that support diverse answers; experiments show the purely algorithmic route yields clearly smaller gains than changing the data.
• vs Absolute Zero / R-Zero: these are the closest in spirit, also hooking data synthesis into the GRPO loop; the difference is that they synthesise textual problems (for math/reasoning), cannot handle visual perception tasks whose labels are bound to images, and lack both the foreground-preserving synthesis mechanism and the diversity-reward design used here.
• vs PromptCoT / Genetic-Instruct / MetaSynth / TaskCraft: these are offline or multi-agent data synthesis pipelines for text, code and tool tasks, whose output is fixed before training begins; synthesis here happens online during training and is driven by the model's current ability, and the paper directly compares against an offline variant (descriptions from GPT-4o).

Rating

• Novelty: ⭐⭐⭐⭐ Hooking an image-synthesis loop into GRPO online, and defining sample exploration value by within-group reward variance to supervise the model's own "question writing", is a new combination
• Experimental Thoroughness: ⭐⭐⭐ Three perception tasks and two model sizes with a complete ablation, but the main experiments use a small data budget (2,000 samples, 5 epochs) and lack general-capability regression tests and quantitative evaluation of generation quality
• Writing Quality: ⭐⭐⭐⭐ The problem–mechanism–experiment chain is clear and the motivation is argued with entropy/diversity curves; formula typesetting in the cache is corrupted but the structure is legible
• Value: ⭐⭐⭐⭐ Offers a reusable paradigm of training data co-evolving with the model, and its two engineering decisions — asynchronous service decoupling and foreground consistency — are directly transferable

Related Papers

• [ICLR 2026] Composition-Grounded Data Synthesis for Visual Reasoning
• [CVPR 2026] Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoning
• [ECCV 2026] Learning Active Perception for Pixel-Space Reasoning via Visual-Intent Stratified GRPO
• [CVPR 2026] CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning
• [ICML 2026] Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners