ECCV 2026 Multimodal VLM Object Detection (text spotting) Text Spotting GRPO Matching-based Online SFT Multimodal Large Language Model Artistic Text

SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/Psycho-9/SupGRPO
Area: Multimodal VLM / Object Detection (text spotting)
Keywords: Text Spotting, GRPO, Matching-based Online SFT, Multimodal Large Language Model, Artistic Text

TL;DR¶

This paper jointly fine-tunes a multimodal large language model with GRPO and a matching-based online SFT that acts only on coordinate tokens: the SFT branch supplies the token-level localization supervision GRPO lacks, while the matching step removes the instance-order prior that standard SFT imposes on an unordered set of text instances, improving both detection and recognition on Total-Text, ICDAR 2015, CTW1500 and the newly built artistic-text dataset ATS.

Background & Motivation¶

Text spotting asks a single model to produce, for every text instance in an image, both a tight bounding box and its transcript. Specialized models are already strong here: segmentation-based Mask TextSpotter v3 uses instance and character mask branches for arbitrary shapes, regression-based ABCNet v2 parameterizes text shape with differentiable Bezier curves sampled via BezierAlign, and DETR-style TESTR and DeepSolo cast both detection and recognition as point/query sequence prediction. These methods, however, struggle badly on artistic text — stylized fonts, unconventional layouts, intricate textures, and text visually fused with design patterns demand visual understanding and reasoning beyond what models built for ordinary scene text can offer. Conversely, multimodal large language models (TextMonkey, Vary, InternVL3.5, Qwen2.5-VL and others) have made rapid progress in recognition and document understanding and recognize far better than specialized spotters, yet they localize poorly at fine granularity: in the paper's Tables 3/4, an untuned Qwen2.5-VL-7B reaches only 27.4 detection on ATS, against 86.7 for DeepSolo.

The authors therefore try fine-tuning along two separate routes and reach an interesting complementary conclusion: SFT is more effective than GRPO at improving detection (localization), while GRPO is more effective at improving recognition (Table 6, same data: SFT 63.3/77.1 versus GRPO 62.0/80.0). The reasons are clear. GRPO optimizes non-differentiable evaluation metrics (IoU, word-level F1) directly through rule-based rewards and lets the model explore freely in ambiguous, reasoning-heavy artistic scenes, but every box in an image shares one sparse scalar reward — nothing tells the model that this particular edge should move two pixels left. SFT supplies exactly that token-level supervision, but it supervises the entire output sequence, which forces the model to emit instances in the writing order of the ground truth. Text instances actually form a set; which one comes first is an annotation convention unrelated to localization and content, and that order prior drags in learning objectives that actively interfere with detection and recognition.

The two can be stitched together because SFT and GRPO already admit a common view: both are policy-gradient updates of the form "data source × reward signal source × gradient coefficient," where SFT uses ground-truth data with a coefficient of 1 and GRPO uses data sampled from the policy with the coefficient given by group-relative advantages. Since both optimize the same policy, one can hang a controlled SFT branch off the GRPO rollout and supervise only those predictions that actually correspond to a ground-truth box. Core idea: use dual matching on text content and IoU to assign the policy's own sampled boxes to ground-truth boxes, then add a cross-entropy term only on the coordinate tokens of matched instances and optimize it jointly with the GRPO objective — restoring dense coordinate supervision without imposing the ground truth's instance ordering on the model.

Method¶

Overall Architecture¶

The input is an image plus a task prompt (spot every text instance, give its coordinates and content); the policy model (Qwen2.5-VL-3B/7B and Qwen3-VL-8B with LoRA, denoted TS-VL) emits a structured list of {"bbox": [x1, y1, x2, y2], "text": "..."} entries. Each training step runs two parallel gradient paths. One is standard GRPO: sample G outputs for the same input, compute group-relative advantages from four rule-based rewards, and take a KL-regularized policy-gradient step. The other is matching-based online SFT: parse the predicted boxes out of each rollout, assign them to the image's ground-truth boxes under "identical text content and IoU > 0," and compute cross-entropy only on the coordinate tokens of successfully matched instances. The two gradients are summed within the same batch to form SupGRPO's joint objective; unmatched predictions receive no coordinate loss but are still judged by the precision/recall rewards on the GRPO side.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image + task prompt + GT annotations"] --> B["Policy samples G outputs<br/>parse boxes and transcripts"]
    B --> C["Four rule-based rewards<br/>format / text F1 / IoU precision and recall"]
    B --> D["Text + IoU dual matching<br/>predicted boxes ↔ GT boxes"]
    D --> E["Matching-based online SFT<br/>cross-entropy on coordinate tokens only"]
    C --> F["Joint loss<br/>GRPO objective + λ · coordinate SFT loss"]
    E --> F

Key Designs¶

1. Four rule-based rewards: turn non-differentiable detection and recognition metrics directly into an optimization signal

GRPO needs a scalar that scores a whole output, and this paper assembles that scalar from four rules. The format reward is a 0/1 indicator (can the output be parsed into the required list-of-dict structure), which guarantees the remaining rewards are computable. The text reward measures recognition and is computed by splitting predicted and ground-truth strings into words and treating the two as bags of words. A detail that is easy to miss but matters: the intersection is counted over multisets, so the intersection for a word w is the smaller of its counts on the prediction and ground-truth sides. If "Sale" appears twice in the image, the model must also predict it twice to earn full recall, while hallucinating "Together" as two copies when it occurs once adds only to the denominator and not the numerator, correctly penalizing precision.

\[R_{\text{text}}(y, GT) = \frac{2\cdot|P_{\text{words}} \cap GT_{\text{words}}|}{|P_{\text{words}}| + |GT_{\text{words}}|}\]

Detection is split into two rewards: a predicted box whose IoU against a ground-truth box exceeds a threshold τ (0.5 by default) counts as a true positive, precision is the fraction of predictions that are true positives, and recall is the fraction of ground-truth boxes that are matched (i.e., "boxes are accurate" and "nothing is missed" are scored separately). Soft rewards such as ANLS or edit distance are deliberately avoided because these benchmarks evaluate by exact match: a soft reward encourages the model to learn near-miss spellings like "Applo" for partial credit, whereas the test protocol counts any character error as a complete failure, so reward and evaluation would be misaligned. Precision and recall are likewise not merged into a single F1 reward, because the harmonic mean makes the reward function considerably more complex, hurts convergence and lowers sample efficiency; two independent rewards work better (Table 9: Text & F 69.3/83.5 versus Text & P & R 71.6/84.4).

2. Text + IoU dual matching: decide which predicted box the supervision signal attaches to

Online SFT first has to answer "which predicted box corresponds to which ground-truth box." Matching by IoU alone (take the highest-IoU ground truth per prediction, threshold 0.3) misses many valid boxes and mis-assigns others to a neighboring instance in the dense, heavily overlapping artistic-text scenes. Matching by text alone depends entirely on recognition correctness — MLLM recognition is strong and the match rate is high, but it breaks down when an image contains several instances with identical content. This paper requires both conditions simultaneously: matching succeeds only when the text content is identical and the IoU is greater than 0, giving the set of matched pairs \(M = \{(pb_i, gtb_j)\}\). The IoU condition blocks the case where the text is right but the box landed on another same-text instance, while the text condition keeps boxes whose content was misread out of the supervision set so that wrong content never reinforces the coordinates. The ablation in Table 8 confirms the complementarity: IoU alone 66.5/81.7, text alone 68.6/82.8, and 71.6/84.4 only when combined.

3. Matching-based online SFT: supervise coordinate tokens only, taking instance order out of the optimization entirely

This is the core of the paper. Standard SFT's problem is not that its supervision is fine-grained but that its scope is too wide: it computes cross-entropy over every token of the sequence, so the writing order of ground-truth instances and even the positions of JSON commas become learning targets. SupGRPO shrinks the supervised surface to the minimum — only the coordinate tokens of matched instances (the four numbers of each bbox), scored by negative log-likelihood:

\[L_{\text{SFT-coord}}(y, GT(x)) = -\sum_{(pb_i,\,gtb_j) \in M}\ \sum_{t \in GT_{j,\text{coord}}} \log \pi_\theta(t \mid x,\, y_{<t})\]

"Online" means these supervised samples are not an offline dataset but sequences sampled by the current policy in this very rollout: the context \(y_{<t}\) of a coordinate token is the model's own generated prefix, so the SFT branch shares one forward pass with GRPO, requires no extra sampling, and suffers no distribution drift between offline data and the current policy. "Matching" is what makes the supervision an assignment over a set rather than an alignment along a sequence — it does not matter whether the model writes "Sugar" first or second, as long as the box matches a ground-truth box its coordinates are supervised; unmatched predictions produce no coordinate gradient at all and are never penalized for "getting the order wrong." The token ablation in Table 7 draws this boundary precisely: online SFT over all tokens only reaches 64.0/81.2, supervising text tokens alone pushes recognition to 84.1 but drops detection to 60.8 (below the 62.0 of pure GRPO), and only coordinate-token supervision raises detection and recognition together to 71.6/84.4.

A Worked Example¶

Take the image in Figure 1 of the paper: it contains exactly two instances, and the ground-truth writing order is [{"bbox":[50,303,522,648],"text":"Rush"}, {"bbox":[30,25,557,321],"text":"Sugar"}] — Rush first, Sugar second. Suppose a rollout from the current policy outputs [{"bbox":[32,25,552,329],"text":"Sugar"}, {"bbox":[46,303,515,644],"text":"Rush"}]: everything is correct, but the order is reversed relative to the ground truth. Standard SFT aligns positionally, treats the first output as the coordinates of "Rush," and pushes both boxes' coordinates — and even content tokens — toward the wrong targets, producing a whole stretch of useless gradient about "which order instances should be emitted in." SupGRPO first matches by text (Sugar↔Sugar, Rush↔Rush), then confirms via IoU > 0 that these two boxes really sit on their own instances rather than on a same-named neighbor, and finally computes cross-entropy over just those eight coordinate tokens, with order playing no role. Conversely, if the model hallucinates an extra "Sugar," no ground-truth box can be paired with it, so it contributes no coordinate loss — it is still penalized through the GRPO precision reward. The SFT branch is responsible only for sharpening accuracy; punishing wrong predictions stays with the rewards.

Loss & Training¶

SupGRPO writes the GRPO policy-gradient objective and the coordinate SFT loss as a single minimization target, where λ balances the two and defaults to 1e-4:

\[\mathcal{L}_{\text{SupGRPO}}(\theta) = -\mathcal{J}_{\text{GRPO}}(\theta) + \lambda \cdot \mathbb{E}_{x \sim \mathcal{D}_{\text{GRPO}},\ y \sim \pi_\theta(\cdot|x)}\big[L_{\text{SFT-coord}}(y, GT(x))\big]\]

Here the GRPO term is the standard group-relative form: sample G outputs per input, normalize rewards within the group to obtain per-token advantages, multiply by the importance ratio, and add a KL regularizer (⚠️ Equation 1 is garbled in the cached text; it is restated here in the standard GRPO form and details should be checked against the original paper). The order of magnitude of λ indicates that coordinate SFT is an auxiliary term: it provides strongly directional, dense gradients for localization while the overall optimization direction is still reward-driven, so the policy is not dragged back toward the offline data distribution.

Implementation: the open-source Open-R1 framework and its multimodal counterpart VLM-R1 are used for LoRA fine-tuning of Qwen2.5-VL-3B/7B and Qwen3-VL-8B for one epoch on 4 NVIDIA A100 GPUs with a batch size of 2 per device. GRPO samples 8 outputs per input with a maximum output length of 1024 tokens; the initial learning rate is 1e-6 and the KL coefficient β is 0.04. Training data mixes the training splits of ATS, Total-Text, ICDAR 2015, CTW1500 and ReCTS, reformatted with a task description per entry, roughly 30,000 samples in total.

Key Experimental Results¶

Four benchmarks are used: the standard scene-text sets Total-Text, ICDAR 2015 and CTW1500, plus ATS, an artistic-text dataset built for this paper (artistic samples manually curated from the TextSeg and WAS text-segmentation datasets, with erroneous OCR labels corrected and re-annotated; word-level quadrilateral boxes and transcripts; 6500 training and 2500 test images). The paper highlights three specific difficulties in ATS: severe overlap between text instances in greeting cards and advertisements; irregular character arrangement and variable reading order, which make it hard to tell which characters form one word; and artistic text often fused with design patterns, producing complex inter-instance relationships that are hard to separate from decorative elements. Results are reported separately by method family: specialized spotters and conventional-protocol results use the end-to-end protocol (a prediction is correct only if both localization and transcript are right), MLLM baselines that emit only text strings without reliable word-level boxes are reported as recognition-only word F1, and detection is evaluated under the standard detection protocol.

Main Results¶

End-to-end text spotting (for ICDAR 2015, S/W/G/None are the four lexicon settings; ⚠️ the header-to-column alignment is incomplete in the cached text, so columns are ordered here by the ordering that holds in every row, S > W > G > None — refer to the original table for exact column assignment):

Method	TT None	TT Full	IC15 S	IC15 W	IC15 G	IC15 None	CTW1500	ATS
Mask TextSpotter v3	71.2	78.4	83.3	78.1	74.2	−	−	56.9
ABCNet v2	70.4	78.1	82.7	78.5	73.0	57.5	77.2	53.4
TESTR	73.3	83.9	85.2	79.4	73.6	56.0	81.5	55.0
SPTS	74.2	82.4	77.5	70.2	65.8	63.6	83.8	65.3
DeepSolo	79.7	87.0	86.8	81.9	76.9	64.2	81.4	64.8
Bridge	83.3	88.3	89.1	84.2	80.4	69.8	83.9	67.2
TS-VL-8B (SupGRPO)	85.6	88.7	89.8	85.6	82.1	72.6	84.9	83.7

Recognition-only word-level F1 of MLLM-based methods (some baselines provide no reliable word-level boxes, so only recognition is compared):

Method	TT None	TT Full	IC15 S	IC15 W	IC15 G	IC15 None	CTW1500	ATS
Qwen2.5-VL-7B	78.1	87.0	89.0	83.5	79.2	73.2	80.4	72.7
InternVL3.5-8B	81.7	86.7	87.8	84.7	81.5	71.5	78.5	72.1
Gemini 3.1-Pro ⚠️	84.0	90.0	91.2	86.6	83.2	78.4	85.1	81.6
HunyuanOCR	83.8	89.5	91.6	85.8	82.6	79.0	84.3	80.8
Qwen2.5-VL-7B (SFT)	82.6	86.4	86.8	83.6	81.4	79.2	85.6	86.2
Qwen2.5-VL-7B (GRPO)	84.2	88.3	86.9	85.5	84.4	82.4	87.7	87.9
Two-Stage (SFT → GRPO)	84.5	86.2	87.1	85.9	82.5	83.0	85.7	87.2
TS-VL-7B (Qwen2.5-VL-7B)	88.2	92.3	94.4	89.9	86.7	86.2	91.7	89.6
TS-VL-8B (Qwen3-VL-8B)	91.0	93.4	95.1	90.2	89.1	89.9	92.5	92.4

⚠️ Tables 2 and 3/4 of the original paper label the same citation [30] as Gemini 3.1-Pro and Gemini 1.5-Pro respectively; refer to the original paper.

Detection-only results (ATS, plus Total-Text / IC15):

Method	ATS	Total-Text	IC15
DeepSolo	86.7	87.3	90.0
Bridge	−	89.2	90.5
Qwen2.5-VL-7B	27.4	23.2	19.2
InternVL3.5-8B	21.2	17.8	16.5
TextMonkey	17.5	11.7	10.2
Qwen2.5-VL-7B (SFT)	72.5	65.2	68.6
Qwen2.5-VL-7B (GRPO)	69.6	60.5	63.6
Two-Stage (SFT → GRPO)	70.6	66.1	67.3
TS-VL-3B	71.6	68.4	71.6
TS-VL-7B	77.1	70.1	73.8
TS-VL-8B	88.8	90.1	90.5

Note: the "ABINet++" entry in the ATS detection table of the original is cited to reference [45] (i.e., TESTR); ABINet++ is normally a recognition/correction model, so this looks like a labeling artifact in the original or in the cached extraction — ⚠️ refer to the original; it does not affect the conclusions.

Ablation Study¶

All ablations are on ATS with Qwen2.5-VL-3B as the base, reporting detection / recognition:

Config	Det.	Rec.	Note
SFT	63.3	77.1	SFT alone: good detection, weak recognition
SFT + augmentation	65.1	77.4	small gain; still far below joint training
GRPO	62.0	80.0	GRPO alone: good recognition, weak detection
SupGRPO	71.6	84.4	joint training lifts both substantially
GRPO + all-token online SFT	64.0	81.2	order prior interferes; detection barely moves
GRPO + text-token online SFT	60.8	84.1	recognition up, detection below pure GRPO
GRPO + coordinate-token online SFT	71.6	84.4	this paper's setting
Matching: IoU only	66.5	81.7	IoU > 0.3; misses or mis-assigns boxes
Matching: text only	68.6	82.8	cannot disambiguate identical text
Rewards: text only	61.1	83.2	detection drops to 61.1
Rewards: text + F1	69.3	83.5	merged F1 reward converges worse
Rewards: text + precision + recall	71.6	84.4	two independent rewards are best
Training data without ATS (Total / IC15 / ATS / CTW)	65.8/67.2/65.3/77.9	83.9/82.8/75.7	detection/recognition drop across benchmarks
Training data with ATS (same order)	68.4/71.6/71.6/81.8	86.0/84.7/84.4	ATS data helps generalization

Key Findings¶

The complementarity has direct evidence: with identical data and base model, SFT alone reaches 63.3 detection versus 62.0 for GRPO alone, while GRPO alone reaches 80.0 recognition versus 77.1 for SFT alone; joint training gives 71.6/84.4, far above either single route (+8.3 detection, +4.4 recognition). Two-stage SFT→GRPO reaches only 70.6 detection / 87.2 recognition on ATS (Tables 3 and 2), showing the complementarity is not obtainable by "doing the two things in sequence" — sequential training merely optimizes each capability in isolation.
Which tokens get supervised is the decisive boundary: supervising text tokens alone raises recognition to 84.1 but pushes detection down to 60.8 (below pure GRPO), while coordinate-token supervision is what yields simultaneous gains of +9.6 detection and +4.4 recognition — evidence that the instance-order prior mainly interferes with localization, not content.
Both matching conditions are necessary: IoU only 66.5, text only 68.6, combined 71.6, and their failure modes are complementary (missed/cross-instance assignments versus ambiguity among identical texts).
Splitting rewards beats merging them: combining precision and recall into an F1 reward (69.3) is worse than two independent rewards (71.6), which the authors attribute to the merged form substantially increasing reward complexity and hurting convergence and sample efficiency.
Training is more stable: in Figure 4 both the individual reward curves and the total reward of SupGRPO sit consistently above vanilla GRPO, and its total loss converges lower with markedly smaller fluctuation.
A stronger base model can overtake specialized models: with Qwen3-VL-8B, SupGRPO detects at 88.8 on ATS (DeepSolo 86.7) and 90.1 on Total-Text (Bridge 89.2), with end-to-end ATS at 83.7 versus Bridge's 67.2.

Highlights & Insights¶

Shrinking SFT's supervised surface from the whole sequence to matched coordinate tokens removes the order dependency rather than mitigating it, because matching is an assignment over a set and output order never enters the loss. The move is clean: instead of designing ordering rules or order augmentation, change what is supervised.
"Online" means reusing the GRPO rollout: the SFT branch's samples and contexts come from the current policy's own sampling step, so it costs no extra sampling and avoids distribution drift from offline data; the price is that one must solve "how do the rollout's boxes line up with the ground truth," which is exactly what the matching stage does.
Deliberate alignment between reward and evaluation protocol: the paper explicitly rejects soft rewards such as edit distance or ANLS because the test protocol is exact match and soft rewards invite near-miss spellings, and likewise splits precision/recall instead of merging into F1 — concrete trade-offs between "the reward must be simple and optimizable" and "the reward must match the metric," rather than generic claims about "well-designed rewards."
Transferability: any task whose output is a set (object detection, referring expression, multi-target grounding, multi-region captioning) suffers simultaneously from annotation-order priors and sparse rewards, and can adopt the same "match first, then supervise only the localization tokens" recipe for auxiliary dense supervision; the training idea also applies to RLVR settings where rewards are sparse and dense auxiliary gradients are needed.
The dataset itself is a contribution: ATS turns scattered artistic-text segmentation data into a detection benchmark with word-level quadrilateral boxes and transcripts, and provides a controlled comparison of specialized models against MLLMs, filling a gap where visually demanding text scenarios had no standard evaluation.

Limitations & Future Work¶

Admitted by the authors: the matching heuristic relies on IoU together with exact text agreement and will still fail in extremely dense scenes where several overlapping instances share the exact same text; the method also depends on the base model's pre-existing spatial grounding ability, so applying it to a model with no spatial awareness would leave too few valid matches and stall coordinate optimization in a cold-start failure.
Observed independently: λ is given only as a default of 1e-4 with no sensitivity analysis, so how brittle the joint training is to λ is unknown; the matching criterion IoU > 0 is quite loose (in practice it almost reduces to text matching plus a tiny overlap), no ablation over the IoU threshold is reported, and the IoU-only comparison in Table 8 uses 0.3, so the two thresholds are not directly comparable.
Experimental scope: every ablation uses the 3B base model and the ATS dataset only, so whether the training-strategy conclusions (which metric moves and by how much) transfer to 7B/8B and to natural-scene benchmarks is untested; training is LoRA for a single epoch, leaving open whether results are limited by LoRA capacity or epoch count; training cost is not reported in detail (wall-clock for one epoch on 4×A100, actual overhead of 8 rollouts of up to 1024 tokens per step), so the extra cost relative to pure GRPO cannot be assessed.
Possible improvements: replace the heuristic text + IoU matching with differentiable bipartite matching (e.g., Hungarian) to handle multiple instances with identical text; make λ decay over training or adapt to the match rate, which may stabilize the cold-start phase; add harder cases beyond ATS (handwritten artistic text, low-resolution artistic text) to further test generalization.

vs specialized spotters (DeepSolo / TESTR / ABCNet v2 / Bridge): they push localization to the limit (Total-Text detection 87–89) but their recognition is bounded by the character distribution of the training data. This paper builds on an MLLM, whose recognition is inherently stronger (ATS recognition 92.4 versus DeepSolo's 64.8 end-to-end), and lifts localization to 88.8–90.5 with GRPO plus matching-based SFT, overtaking specialized models on ATS and Total-Text detection. The cost is far higher model size and inference expense than a specialized spotter.
vs pure SFT / pure GRPO fine-tuning of MLLMs: controlled comparisons on identical data and base model (Tables 2/3/6) show the two have exactly complementary weaknesses; this paper does not "train again" but adds one class of supervision inside the GRPO framework, and the failure of the two-stage SFT→GRPO pipeline further shows sequential composition cannot produce the same complementarity.
vs standard sequence-level SFT: standard SFT forces an alignment between the unordered instance set and a written sequence; this paper aligns coordinates only within matched instances, making supervision instance-level and order-agnostic, which is the direct reason it beats all-token online SFT by 7.6 detection points.
vs RLVR work such as VLM-R1 / Open-R1: those works generalize the RLVR paradigm to vision-language tasks, mostly on reasoning and localization question answering with well-defined answers; this paper brings RLVR to text spotting, where the output is a dense structured set, and identifies the shared shortcoming of sparse rewards on dense localization, offering a fix (matched coordinate-token online SFT) that applies to other dense structured prediction tasks.

Rating¶

Novelty: ⭐⭐⭐⭐ Joint SFT + GRPO is not brand new, but "match first, then supervise coordinate tokens only" is a well-aimed cut that removes the order dependency at its root, and this is the first systematic study of artistic text spotting.
Experimental Thoroughness: ⭐⭐⭐⭐ Four benchmarks and four ablation families (training strategy / token choice / matching mechanism / reward design) are well covered, but ablations are limited to the 3B model and ATS, with no sensitivity analysis for λ or the IoU threshold and no training-cost report.
Writing Quality: ⭐⭐⭐⭐ The motivation–finding–method chain is clear and the complementarity claim is backed by controlled experiments; failure modes of rewards and matching are described concretely and the derivations are largely self-consistent.
Value: ⭐⭐⭐⭐ Provides a reusable paradigm — patching pure RL's dense-supervision gap with matching-based online SFT — plus the ATS artistic-text benchmark and detection results that surpass specialized models.