Free-Lunch Augmentation by Revisiting Diffusion-Based Data Generation for Cross-Domain Few-Shot Object Detection¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/zzzzj311-droid/Free-Lunch-SITN
Area: Object Detection
Keywords: Cross-Domain Few-Shot Object Detection, Diffusion Data Augmentation, Tailored Weak Noise, Selective Inpainting, Training-Free Augmentation

TL;DR¶

Addressing the failure of diffusion models in Cross-Domain Few-Shot Object Detection (CDFSOD) caused by severe domain shifts, this paper decouples domain gaps into visual and semantic gaps, introducing a training-free framework termed Selective Inpainting with Tailored Noise (SITN) that achieves new state-of-the-art results across 6 CDFSOD and 4 CDFSS benchmarks.

Background & Motivation¶

Cross-Domain Few-Shot Object Detection (CDFSOD) seeks to transfer detection knowledge learned from data-abundant generic source domains (such as COCO) to specialized downstream expert domains (e.g., medical analysis, underwater exploration, satellite imagery, and industrial surface defect inspection) where only scarce annotated samples (e.g., 1-shot, 5-shot, 10-shot) are accessible. Due to the formidable annotation costs and severe privacy constraints inherent in expert domains, training data remains exceptionally sparse. To date, mainstream efforts predominantly concentrate on transfer learning, parameter-efficient fine-tuning, cross-domain prompt optimization, and feature prototype alignment. Yet, an intuitive and fundamental direction in modern generative AI—directly synthesizing scarce training samples with pre-trained diffusion models—has remained largely unsuccessful in CDFSOD.

Empirical investigations reveal that directly applying standard off-the-shelf text-to-image (T2I) or image-to-image (I2I) generative models (such as Stable Diffusion, Playground, or Kandinsky) to expert domains leads to severe degradation; fine-tuning downstream detectors on the resulting synthetic data often performs even worse than training exclusively on the scarce original samples. Quantitative evaluations using Centered Kernel Alignment (CKA) similarity and Maximum Mean Discrepancy (MMD) demonstrate that while conventional geometric augmentations (e.g., horizontal/vertical flipping) exhibit steady cross-domain feature similarity, diffusion-generated samples suffer a dramatic collapse in semantic fidelity on expert domains. The fundamental root cause is that pre-trained diffusion models have virtually never encountered niche expert distributions, rendering them incapable of separating random Gaussian noise from meaningful domain-specific visual cues and reducing the reverse generation to blind speculation guided purely by generic source priors.

To conquer this generation bottleneck, the paper systematically decomposes the overarching domain discrepancy into a visual gap (e.g., natural photography vs. grayscale X-ray or degraded underwater imagery) and a semantic gap (e.g., common everyday objects vs. specialized unobserved categories). For the visual gap, the authors find that standard full-scale diffusion destroys subtle target structures, whereas injecting tailored weak noise effectively preserves vital low-level visual and structural topology. For the semantic gap, their key insight is that unobserved semantic classes are predominantly confined to foreground objects, whereas background contexts exhibit substantially higher transferability and domain-sharing properties across disparate environments. Core idea: tackle cross-domain generation by decoupling visual and semantic gaps, proposing a training-free framework termed Selective Inpainting with Tailored Noise (SITN) that couples tailored weak noise injection with dynamic foreground/background selective inpainting—preserving bounding-box annotations without re-labeling and filtering out low-quality artifacts via RPN proposal consensus to supply a genuine "free-lunch" data augmentation.

Method¶

Overall Architecture¶

SITN operates exclusively on the scarce support set of the target domain without updating or fine-tuning any parameters of the diffusion backbone. The end-to-end pipeline consists of three sequential stages: prompt extraction and mask localization, dual-branch tailored noise inpainting, and consensus-driven sample selection followed by downstream detector fine-tuning.

Initially, a large language model (LLM) processes the original support image to produce a contextual textual prompt, while ground-truth bounding boxes are mapped into binary masks separating foreground targets from background regions. In the Generation Module, controlled weak Gaussian noise is injected into the support images, and the frozen diffusion model performs background inpainting (primary branch) and constrained foreground inpainting (exploratory branch), yielding two sets of augmented candidates that strictly retain original spatial coordinates. Finally, the Selection Module leverages a pre-trained Region Proposal Network (RPN) from the base detector to compute Intersection over Union (IoU) scores against ground-truth annotations, dynamically picking high-confidence samples into the augmented support set for robust target-domain adaptation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Scarce Support Images and Bounding Boxes"] --> B["Multimodal Prompting & Binary Mask Localization<br/>LLM textual description + bounding box to 0/1 masks"]
    B --> C["Dual-Branch Tailored Noise Inpainting<br/>Controlled noise injection + background-first reverse denoising"]
    C --> D["RPN Proposal Consensus Sample Selection<br/>IoU evaluation between predicted proposals and GT boxes"]
    D --> E["Output: High-Quality Augmented Support Set & Fine-Tuned Detector"]

Key Designs¶

1. Dual-Branch Tailored Noise Inpainting: Decoupling Visual and Semantic Gaps

Standard diffusion forward passes progressively corrupt images up to \(T_{\max} = 1000\), degrading structural features into pure Gaussian noise and forcing the reverse network to hallucinate content based entirely on generic source-domain priors. To bridge the visual gap, SITN introduces a tailored noise scaling coefficient \(\epsilon_{\text{low}} < 1\) to clamp the forward noising schedule to a modest timestep:

\[x_{\text{sup}}^{\text{low}} = \text{AddNoise}(x_{\text{sup}}, \epsilon_{\text{low}})\]

where \(T = \epsilon_{\text{low}} T_{\max}\), reliably preserving underlying contours, contrast, and anatomical/textural foundations of the target domain. To bridge the semantic gap without synthesizing unknown foreground classes from scratch, SITN partitions images into foreground and background masks using ground-truth bounding boxes. Because background features yield substantially higher cross-domain CKA similarity than foreground features, background inpainting serves as the primary synthesis engine—freezing foreground objects and inpainting exclusively the background under tailored noise:

\[x_{\text{sup}}^{\text{back}} = \mathcal{M}_{\text{diff}}(x_{\text{sup}}^{\text{low}}, \text{mask}_{\text{sup}})\]

Concurrently, within an empirically validated noise interval, the LLM prompt \(prompt = \mathcal{M}_{\text{LLM}}(x_{\text{sup}})\) guides the foreground inpainting branch \(x_{\text{sup}}^{\text{fore}} = \mathcal{M}_{\text{diff}}(x_{\text{sup}}^{\text{low}}, prompt + (1 - \text{mask}_{\text{sup}}))\) to introduce controlled foreground variations. Because inpainting respects the original bounding box footprint, all generated samples automatically inherit exact spatial annotations without costly manual re-labeling.

2. RPN Proposal Consensus Sample Selection: Dynamically Filtering Hallucinated Artifacts

Although tailored noise and background-centric inpainting dramatically elevate visual plausibility, suboptimal LLM descriptions or imperfect text-to-image grounding can still produce synthetic candidates with blurred contours, warped boundaries, or anomalous artifacts. Introducing such corrupted samples indiscriminately into scarce few-shot support sets distorts detector classification boundaries and impairs bounding box regression.

To safeguard downstream fine-tuning, SITN introduces an automated quality-filtering mechanism leveraging the innate perception of a pre-trained detector. The frozen Region Proposal Network (RPN) pre-trained on generic datasets generates candidate proposals over both background-augmented images \(x_{\text{sup}}^{\text{back}}\) and foreground-augmented images \(x_{\text{sup}}^{\text{fore}}\). The average Intersection over Union (IoU) between predicted proposals and ground-truth boxes \(Y\) is measured, selecting only top-performing samples:

\[x_{\text{select}}^k = \text{Top-k}_{\text{IoU}}\left(\mathcal{M}_{\text{RPN}}(x_{\text{sup}}^{\text{back}}, x_{\text{sup}}^{\text{fore}}), Y\right)\]

This design operates on an elegant principle: if a generated image suffers from structural degradation, phantom occlusions, or boundary warping, the RPN fails to propose boxes that align tightly with the ground-truth annotation. Consequently, only candidates that preserve object clarity, natural context harmony, and geometric integrity achieve high IoU scores and earn admission into the augmented training pool.

Loss & Training¶

SITN operates completely training-free during data generation, eliminating diffusion gradient back-propagation. Employing a DDIM sampler, the reverse denoising schedule is compressed to fewer than 50 steps, driving single-image generation time down to 0.02s–1.5s (yielding massive computational savings compared to DDPM's 460s and SDEdit's 300s–450s).

During downstream fine-tuning, the framework follows established CDFSOD protocols (e.g., CD-ViTO), freezing the backbone visual encoder (such as ViT-B, ViT-L, Swin-B, or DETR-R101) while training only detection heads, classification heads, and lightweight adaptation modules on the unified dataset \(S \cup S_{\text{select}}\) via standard cross-entropy classification loss and smooth L1 bounding box regression loss.

Key Experimental Results¶

Main Results¶

SITN is rigorously evaluated on the CDFSOD benchmark encompassing 6 distinct datasets: ArTaxOr (arthropod taxonomy), Clipart1k (artistic illustrations), DIOR (optical remote sensing), DeepFish (underwater marine habitats), NEU-DET (steel surface defects), and UODD (underwater object detection), spanning multiple backbones across 1-shot, 5-shot, and 10-shot configurations.

The table below summarizes performance comparisons in mAP across 1-shot and 10-shot settings using Swin-B (GroundDINO / DomainRAG lines) and ViT-L/14 (CD-ViTO line):

Setting / Method	Backbone	ArTaxOr	Clipart1k	DIOR	DeepFish	NEU-DET	UODD	Average mAP
1-shot CD-ViTO	ViT-L/14	21.0	17.7	17.8	20.3	3.6	3.1	13.9
1-shot CD-ViTO + Ours	ViT-L/14	26.7	22.3	19.8	22.1	5.4	5.6	17.0 (+3.1)
1-shot GroundDINO	Swin-B	20.0	57.6	8.0	34.3	7.2	17.1	24.0
1-shot DomainRAG	Swin-B	57.2	56.1	18.0	38.0	12.1	20.2	33.6
1-shot GroundDINO + Ours	Swin-B	52.3	59.0	16.5	41.3	13.6	22.8	34.3 (+0.7)
10-shot CD-ViTO	ViT-L/14	60.5	44.3	30.8	22.3	12.8	7.0	29.6
10-shot CD-ViTO + Ours	ViT-L/14	61.4	46.5	31.9	23.6	15.2	9.3	31.3 (+1.7)
10-shot DomainRAG	Swin-B	73.4	61.1	39.0	41.3	26.3	31.2	45.4
10-shot GroundDINO + Ours	Swin-B	69.4	65.2	32.6	53.5	24.6	34.8	46.9 (+1.5)

Ablation Study¶

To isolate the distinct contributions of foreground inpainting, background inpainting, and the Selection Module, fine-grained ablations under the 1-shot CD-ViTO setting are detailed below:

Configuration	ArTaxOr	Clipart1k	DIOR	DeepFish	NEU-DET	UODD	Average mAP	Note
Baseline (CD-ViTO)	21.0	17.7	17.8	20.3	3.6	3.1	13.9	Fine-tuning purely on original scarce samples
Foreground Inpainting Only	22.3	19.3	18.8	18.3	3.6	3.1	14.2	Minor foreground variations; prone to drift without selection
Background Inpainting Only	24.9	18.6	18.7	22.1	3.7	3.9	15.3	High background transferability; brings universal improvements
Unfiltered Dual-Branch (Fore.+Back.)	22.5	19.3	16.6	13.5	3.2	2.1	12.9	Noise artifacts corrupt training set (-1.0 drop)
Full Model (Fore.+Back.+Sele.)	26.7	22.3	19.8	22.1	5.4	5.6	17.0	RPN IoU filtering achieves peak performance (+3.1)

In benchmarking against other generative augmentation pipelines (Diffmix 4.8, IMBABM 3.4, Da-fusion 12.0, SDEdit 13.2, IP-Adapter 5.1, ControlNet 11.4), SITN achieves a dominant 34.3 mAP. Furthermore, conventional augmentations (Mixup 8.7, Flip 13.9, Bright+ 13.4, Copy-Paste 12.9) frequently underperform the baseline (13.9), accentuating the severe difficulty of cross-domain data synthesis.

When extended to Cross-Domain Few-Shot Segmentation (CDFSS), combining SITN with FPTrans yields consistent gains across FSS-1000 (84.5), Deepglobe (43.8), ISIC (53.4), and Chest X-ray (84.9), boosting average mIoU from 62.2 to 66.7 and demonstrating cross-task universality.

Key Findings¶

Unfiltered synthetic data causes negative transfer: Ingesting raw synthetic images without RPN-based quality filtering causes average mAP to plummet from 13.9 to 12.9, with severe drops on DeepFish and UODD, demonstrating that hallucinated artifacts have historically been the primary culprit behind generative augmentation failures.
Background inpainting provides intrinsic domain transferability: Applying background inpainting alone reliably increases mAP from 13.9 to 15.3 across all 6 datasets. This empirically confirms that background contexts exhibit lower semantic discrepancy than novel foreground classes, serving as the most dependable entry point for cross-domain generation.
Excessive noise compounds the visual gap: As noise level \(\epsilon\) scales up, CKA similarity plunges while MMD distance surges. Restricting noise within a conservative tailored window is essential to inject contextual diversity while maintaining structural detectability.

Highlights & Insights¶

Insightful domain gap factorization: Rather than viewing cross-domain failure as an amorphous "distribution shift," the authors cleanly separate the challenge into visual gaps (addressed by tailored weak noise) and semantic gaps (addressed by background inpainting), tackling each with dedicated mechanisms.
Zero-reannotation bounding box reuse: By exploiting existing ground-truth bounding boxes directly as inpainting masks, the framework generates rich background and object variations while maintaining exact spatial alignment, eliminating the prohibitive cost of secondary annotation.
Exceptional computational efficiency: Completely training-free and decoupled from external retrieval databases (unlike DomainRAG which requires scanning COCO), SITN synthesizes images via DDIM in just 0.02s–1.5s per image, making it highly practical for large-scale workflows.

Limitations & Future Work¶

Coarse rectangular bounding-box masks: Using rectangular bounding boxes as inpainting masks means that non-target background pixels inside the box are treated as foreground, which can introduce boundary artifacts in complex scenes with irregular object contours.
Dependence on initial detector RPN capability: The selection module assumes the pre-trained RPN can produce meaningful candidate proposals. In extreme domains where standard RPNs fail completely to localize candidate regions, the IoU filtering mechanism may degrade.
Future avenues: Integrating open-world promptable segmentation backbones (e.g., SAM-2) to extract instance-contour masks and introducing closed-loop prompt optimization could further sharpen foreground fidelity.

vs DomainRAG: DomainRAG relies on an external retrieval-augmented pipeline, necessitating extensive feature extraction and background matching across large source datasets (COCO) with high latency (5s–8s per sample); SITN uses a pure generative inpainting formulation, executing in 0.02s–1.5s per image without external databases.
vs SDEdit: SDEdit applies stochastic differential equations to the entire image canvas, degrading fine-grained target details on niche medical or satellite images and requiring over 300s per image; SITN utilizes localized selective masks with tailored weak noise, preserving structural fidelity while boosting throughput.
vs Conventional Augmentations (Mixup / Copy-Paste): Traditional pixel-level augmentations merely interpolate within scarce existing data without creating new contextual semantics, often causing negative transfer in CDFSOD; SITN leverages deep generative priors to synthesize novel, physically coherent environmental contexts.

Rating¶

Novelty: ⭐⭐⭐⭐ [Decoupling domain gaps into visual and semantic components with tailored weak noise and selective inpainting is clear, insightful, and well-formulated]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 6 CDFSOD and 4 CDFSS benchmarks, featuring multiple detector backbones and rich generative baselines]
Writing Quality: ⭐⭐⭐⭐⭐ [Smooth narrative flow, rigorous motivation, crisp problem formulation, and complete self-consistency between findings and empirical data]
Value: ⭐⭐⭐⭐⭐ [Training-free, extremely fast, and plug-and-play, providing a practical blueprint for synthetic data augmentation in few-shot specialized domains]