TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Project: https://seokhunchoi.github.io/TMI
Area: Image Generation
Keywords: generative data synthesis, diffusion models, long-tailed instance segmentation, text-to-image, image-to-image editing

TL;DR¶

TMI introduces a complementary data synthesis framework coupling global text-to-image (T2I) generation for scene diversity with context-aware image-to-image (I2I) editing via VRAIN for verified rare-class augmentation, achieving up to +4.0 overall AP and +9.5 rare-class AP gains on the LVIS benchmark.

Background & Motivation¶

In large-vocabulary instance segmentation benchmarks such as LVIS (spanning 1,203 categories), performance is severely bounded by extreme category imbalance and fine-grained inter-class ambiguity. While algorithmic mitigations—such as loss re-weighting, balanced data sampling, and classifier calibration—help balance prediction distributions, they cannot overcome the fundamental absence of physical supervision signals for scarce tail classes. Consequently, generative data synthesis leveraging advanced diffusion models has emerged as an attractive, scalable alternative to expensive manual data collection.

However, existing generative data synthesis paradigms suffer from notable individual limitations and complementary failure modes. Text-to-image (T2I) approaches (e.g., MosaicFusion) provide rich scene diversity and broad category combinations, but they inevitably inherit severe label noise when annotated with offline perception models due to the substantial domain gap between synthetic and real images. This limitation is particularly detrimental for rare categories where pseudo-labelers fail to produce accurate masks. Conversely, image-to-image (I2I) methods based on copy-paste (e.g., X-Paste, DiverGen) preserve clean instance masks, but pasting objects onto arbitrary backgrounds introduces jarring lighting, shadow, and perspective inconsistencies. In fact, prior studies demonstrate that explicitly targeting rare classes with naive copy-paste causes catastrophic performance degradation on those very tail classes, as segmentation networks overfit to synthetic boundary artifacts rather than genuine visual representations. Inpainting alternatives also struggle with the challenge of finding plausible mask placement locations in unconstrained complex scenes.

This paper identifies that T2I excels in broad semantic diversity while I2I excels in targeted, instance-level annotation accuracy. Core idea: unite T2I generation and context-aware I2I editing into a complementary hybrid pipeline, eliminating T2I label noise via prompt-consistent filtering and teacher-student adaptation, while introducing a two-stage "place-and-verify" I2I editor (VRAIN) to inject high-fidelity rare-class instances.

Method¶

Overall Architecture¶

The TMI framework consists of three integrated stages: (1) diverse all-class text-to-image generation accompanied by prompt-consistent offline filtering; (2) the VRAIN (Verified Rare-class Augmentation via INstructed editing) pipeline for contextually coherent, verified rare-class insertion; and (3) a joint training framework governed by an EMA teacher that dynamically adapts pseudo-labels on the synthetic domain. The end-to-end generation and training pipeline is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    subgraph S1["Data Synthesis Stage"]
        direction TB
        A["Real Dataset D_real<br/>and Full Category Vocabulary C_all"] --> B["Prompt-Consistent Diverse T2I Synthesis<br/>GPT-4o Prompt Sampling + Flux.1 + Category Filtering"]
        A --> C["VRAIN Place-and-Verify I2I Editing<br/>VLM Instruction + Flux.1 Kontext + Red-Box VLM Verification + SAM"]
    end
    B --> D["Weakly Labeled T2I Dataset D_T2I"]
    C --> E["High-Fidelity Verified I2I Dataset D_I2I"]
    D --> F["EMA Teacher Pseudo-Label Adaptation & Joint Training<br/>Dynamic Confidence Thresholding + Unlabeled Mining + Cross-Domain Adaptation"]
    E --> F
    A --> F
    F --> G["Enhanced Student Segmentation Model M_student"]

During the synthesis stage, the pipeline generates a weakly-labeled, scene-diverse dataset $\mathcal{D}_{\text{T2I}}$ alongside a precisely annotated rare-class targeted dataset $\mathcal{D}_{\text{I2I}}$. During training, the student model ingests real images, I2I edited images, and T2I images simultaneously. The EMA teacher model continuously adapts to the synthetic image domain and leverages the high-confidence tail-class features from $\mathcal{D}_{\text{I2I}}$ to progressively refine and uncover previously missed rare objects in $\mathcal{D}_{\text{T2I}}$.

Key Designs¶

1. Prompt-Consistent Filtering and Diverse T2I Synthesis: Eliminating Hallucinated Pseudo-Labels via Text Priors

T2I generation is deployed to provide broad contextual variety across all target categories $C_{\text{all}}$. To construct diverse scenes, subsets of categories $C_m$ of size $l \in [5, 10]$ are uniformly sampled from $C_{\text{all}}$, and GPT-4o synthesizes detailed photographic scene descriptions $t_m$. High-resolution images $I_m$ are then synthesized using Flux.1-dev. Rather than mining pseudo-labels using a global high-confidence threshold—which inevitably discards valid rare-class instances or accepts out-of-context background false positives—TMI introduces a prompt-consistent filter $\mathcal{F}_{\text{text}}$.

Given the raw predictions $\hat{A}_m^{\mathcal{P}}$ from a pre-trained offline instance segmentation model $M_P$, the filter strictly retains instances whose predicted classes belong to the prompt-specified set $C_m$: $$A_m^{\mathcal{P}} = \mathcal{F}_{\text{text}}(\hat{A}_m^{\mathcal{P}}, C_m) = \{ a_k^{\mathcal{P}} \mid a_k^{\mathcal{P}} \in \hat{A}_m^{\mathcal{P}}, \, \text{class}(a_k^{\mathcal{P}}) \in C_m \}$$ By leveraging the explicit text prior that $I_m$ was conditioned on $C_m$, this strategy safely captures low-confidence true positive objects (common for rare classes under domain shift) while pruning hallucinations, establishing a robust offline supervision baseline.

2. VRAIN Place-and-Verify I2I Editing: Context-Aware Rare-Class Insertion and Spatial Grounding

To overcome the contextual inconsistencies of copy-paste and the placement ambiguities of inpainting, VRAIN implements a two-stage "Place-and-Verify" paradigm: - Context-Aware Proposal & Synthesis (Place): To maintain balanced coverage, target rare categories $C_{\text{I2I}}$ are sampled using a softmax distribution inversely weighted by their cumulative generation counts. For each real image $I_{\text{real}}$, a candidate set $C_q$ of $Q=5$ underrepresented classes is evaluated by InternVL3-14B alongside $I_{\text{real}}$. The VLM selects the most plausible class $c^*$ and generates an instruction $inst^*$ specifying realistic placement and physical attributes (e.g., "Add a black headset to the skier's head"). The instruction-based editor Flux.1 Kontext then synthesizes the edited image $I_{\text{edit}} = \Phi_{\text{I2I}}(I_{\text{real}}, inst^*)$. - Verification & Annotation Update (Verify): To eliminate rendering artifacts and semantic drift, VRAIN calculates the structural similarity (SSIM) difference map between $I_{\text{real}}$ and $I_{\text{edit}}$ at threshold $\tau_{\text{edit}}=5$ to isolate the modified bounding area. An open-vocabulary detector scans this area for class $c^*$, producing candidate detections $A_{\text{det}}$. Each candidate $a_k$ is then visually verified: the bounding box is drawn in red over the full edited image to preserve scene context ($I^{\text{box}}_{\text{edit}}$), and the VLM answers a binary query ("Is the red bounding box a $c^*$?"): $$\mathcal{F}_{\text{VLM}}(A_{\text{det}}, c^*) = \{ a_k \mid a_k \in A_{\text{det}}, \, \mathrm{VLM}(I^{\text{box}}_{\text{edit}}, t^{c^*}; a_k) = \text{Yes} \}$$ Accepted instances are segmented via SAM, and any spatial overlap with existing ground-truth masks in $A_{\text{real}}$ is resolved via mask subtraction to produce the clean annotation $A_{\text{edit}}$.

3. EMA Teacher Pseudo-Label Adaptation: Dynamic In-Domain Calibration and Localization Mining

Training an instance segmentation model on fully synthetic images causes performance drops due to domain shifts. TMI incorporates an EMA teacher $M_{\text{teacher}}$ updated via $\Theta_{\text{teacher}} \leftarrow \gamma \Theta_{\text{teacher}} + (1-\gamma) \Theta_{\text{student}}$ ($\gamma = 0.999$). Because the student simultaneously learns from real images and verified rare instances in $\mathcal{D}_{\text{I2I}}$, the EMA teacher dynamically adapts to the T2I domain while acquiring enhanced discriminative power on tail categories.

At each step, the teacher generates raw predictions $\hat{A}_m^{\text{teacher}}$ on T2I images, processed through a dual-mechanism filter: - Adaptive In-Prompt Thresholding: A running memory of the recent 100 confidence scores per class is maintained. Predictions matching the prompt ($\hat{c} \in C_m$) must exceed $\tau_{\text{label}} = 0.7$ times the running mean confidence, allowing thresholds to adapt as the model improves. - Unlabeled Background Mining: High-confidence predictions outside the prompt ($\hat{c} \notin C_m$, exceeding $\tau_{\text{unlabel}} = 1.2$ times running mean confidence) are designated as "unlabeled." They are excluded from classification loss but included in bounding box and mask regression losses, providing rich localization supervision for unprompted foreground objects. - IoU-Based Label Fusion: The adapted online annotations $\tilde{A}_m^{\text{teacher}}$ are merged with static offline annotations $A_m^{\mathcal{P}}$ via IoU matching, uniting stable common-class baselines with progressive rare-class discoveries.

Loss & Training¶

The framework trains CenterNet2 detectors over batches sampled jointly from $\{\mathcal{D}_{\text{real}}, \mathcal{D}_{\text{I2I}}, \mathcal{D}_{\text{T2I}}\}$. Standard classification, box regression, and mask segmentation losses apply to $\mathcal{D}_{\text{real}}$ and $\mathcal{D}_{\text{I2I}}$, whereas for $\mathcal{D}_{\text{T2I}}$, "unlabeled" instances contribute zero classification loss while supervising mask and box regression heads. Training spans 180k iterations with resolution 640 and batch size 32 on ResNet-50, and resolution 896 and batch size 16 on Swin-L.

Key Experimental Results¶

Main Results¶

The main comparison on the LVIS validation set across ResNet-50 and Swin-L backbones evaluates TMI against Real-only baselines, T2I methods (MosaicFusion), and copy-paste I2I methods (DiverGen, X-Paste).

Backbone	Method	T2I Data	I2I Data	$\text{AP}^{\text{box}}$	$\text{AP}^{\text{mask}}$	$\text{AP}^{\text{box}}_r$ (Rare)	$\text{AP}^{\text{mask}}_r$ (Rare)
ResNet-50	Real-only [53]	-	-	34.5	30.8	24.0	21.6
ResNet-50	MosaicFusion [45]	✓	-	34.1	30.4	24.4	22.5
ResNet-50	DiverGen [6]	-	✓	35.1	31.2	25.6	23.8
ResNet-50	X-Paste [51]	-	✓	36.7	33.0	29.6	27.8
ResNet-50	Ours (TMI)	✓	✓	38.1	34.0	33.9	31.7
Swin-L	Real-only [53]	-	-	47.5	42.3	41.4	36.8
Swin-L	MosaicFusion [45]	✓	-	47.7	42.8	41.3	37.5
Swin-L	DiverGen [6]	-	✓	49.6	44.2	44.5	39.8
Swin-L	X-Paste [51]	-	✓	50.1	44.4	48.2	43.3
Swin-L	Ours (TMI)	✓	✓	50.7	45.2	49.1	44.0

Note: Data from Table 2 of the original paper. On ResNet-50, TMI achieves +3.6 $\text{AP}^{\text{box}}$ and +9.9 $\text{AP}^{\text{box}}_r$ improvements over Real-only; consistent state-of-the-art gains are demonstrated on Swin-L.

Ablation Study¶

The ablation on supervision sources for T2I images (Table 4 of the original paper) validates the complementary interaction between offline predictions ($M_P$) and online EMA-teacher predictions ($M_{\text{teacher}}$) on Swin-L.

Supervision Source	$\text{AP}^{\text{box}}$	$\text{AP}^{\text{mask}}$	$\text{AP}^{\text{box}}_r$	$\text{AP}^{\text{mask}}_r$	Note
(a) Real-only	47.5	42.3	41.4	36.8	Lower bound trained purely on real annotations
(b) $M_{\text{teacher}}$ only	49.4	43.9	47.9	43.0	Adapts dynamically to synthetic domain; strong rare-class gains (+6.5)
(c) $M_P$ only	49.9	44.9	45.3	41.7	High precision on common classes, but tail coverage remains capped
(d) $M_{\text{teacher}} \oplus M_P$	50.3	45.1	47.5	43.8	Best balanced trade-off combining stability and progressive adaptation

Note: In the ablation evaluating the role of $\mathcal{D}_{\text{I2I}}$ (Table 7), removing $\mathcal{D}_{\text{I2I}}$ from the full model causes $\text{AP}^{\text{box}}$ to drop from 50.7 to 50.3, while rare-class $\text{AP}^{\text{box}}_r$ falls sharply from 49.1 to 47.5, confirming the necessity of high-fidelity I2I instances.

Key Findings¶

The Failure Mode of Naive Rare-Class Copy-Pasting: In Table 3 of the original paper, when the baseline copy-paste method DiverGen is restricted to synthesizing rare categories only, its rare-class $\text{AP}^{\text{box}}_r$ crashes from 41.4 down to 34.5 (-6.9 points), demonstrating that unnatural compositions cause detector overfitting. In contrast, VRAIN alone improves $\text{AP}^{\text{box}}_r$ to 42.9 (+1.5 points) thanks to context-aware placement and verified natural blending.
Spatially Grounded VLM Verification: Table 5 evaluates verifiers on a perturbed LVIS validation benchmark (244k instances). Standard CLIP exhibits a high false positive rate of 18.7%, while cropped-image VLM evaluation suffers from a 47.8% false negative rate due to loss of scale context. TMI's full-image red-box VLM prompting drops the FP rate to 3.3% while slashing FN rate to 15.4%, achieving an optimal balance.

Highlights & Insights¶

Orthogonal Synergy Between T2I and I2I: Rather than viewing T2I and I2I as competing generation paradigms, TMI treats T2I as an engine for macro scene/category diversity and I2I as a precision tool for micro instance injection, resolving the core bottlenecks of each.
Spatial Red-Box Prompting for VLM Auditing: Overlaying a red bounding box onto the full-resolution edited image allows the VLM to perform grounded instance classification while retaining surrounding scene context and relative object scales, vastly outperforming isolated image crops.
Adaptive Memory-Driven Dynamic Thresholding: Tracking per-class confidence percentiles in running memory smoothly lowers thresholds for emerging rare classes while filtering unprompted objects into an "unlabeled" regression-only track, preventing noisy classifications from corrupting representation learning.

Limitations & Future Work¶

Instruction Adherence Bottlenecks: The system relies on off-the-shelf generative models (Flux.1 family), which occasionally fail to render fine-grained rare objects requested in complex prompts or ignore detailed placement constraints (e.g., placing an object in the wrong part of a room).
Multi-Stage Pipeline Overhead: The pipeline chains several large vision-language models and diffusion systems (GPT-4o, Flux.1, Flux.1 Kontext, InternVL3-14B, open-vocabulary detectors, and SAM), entailing considerable offline computational costs.
Future Directions: Consolidating the place-and-verify steps into an end-to-end generative pipeline with internal consistency rewards could streamline execution and improve synthesis speed.

vs MosaicFusion: MosaicFusion stitches text prompts into mosaic canvases to generate multi-object images via T2I, but suffers from severe pseudo-label noise on long-tailed classes. TMI mitigates this with text-consistent filtering and online EMA-teacher domain adaptation.
vs DiverGen & X-Paste: DiverGen and X-Paste harvest isolated objects and paste them onto real images, causing visual artifacts and contextual collapse on rare categories. TMI's VRAIN uses instruction-guided diffusion to naturally integrate instances and validates them with a spatial VLM and SAM, achieving realistic annotations.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Elegantly orchestrates the complementary strengths of T2I and I2I generation with a rigorous place-and-verify verification protocol.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning ResNet-50 and Swin-L backbones, perturbation benchmarks, pseudo-label source ablations, and mask fidelity audits.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual structure, articulate problem formulation, and crisp diagrammatic explanations.
Value: ⭐⭐⭐⭐☆ Establishes a practical, scalable blueprint for synthetic data generation in complex, large-vocabulary long-tailed visual recognition tasks.