title: >- [Paper Note] TOPA: Mitigating Concept Dominance in Diffusion Personalization via Target-Oriented Perturbation Augmentation description: >- [ECCV 2026][Image Generation][Diffusion Personalization] An offline, reference-image-only augmentation framework that mitigates concept-dominant failures by rebalancing token-region cross-attention with optimized perturbations and diversifying backgrounds via subject-isolation compositing. tags: - ECCV 2026 - Image Generation - Diffusion Model - Concept Personalization - Data Augmentation date: 2026-09-19 content_hash: 434809f7a572434c

TOPA: Mitigating Concept Dominance in Diffusion Personalization via Target-Oriented Perturbation Augmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Concept Personalization, Diffusion Model, Data Augmentation, Cross-Attention, Concept Dominance

TL;DR¶

Addressing the concept-dominant failure in few-shot diffusion personalization where the customized concept suppresses prompt-specified context, TOPA presents an offline, reference-image-only data augmentation framework that enriches backgrounds via subject-isolation compositing and optimizes additive spatial perturbations to rebalance token-region cross-attention, restoring contextual controllability without altering downstream personalization pipelines.

Background & Motivation¶

Diffusion-based few-shot concept personalization methods (such as Textual Inversion, DreamBooth, and Custom Diffusion) have become the primary paradigm for customized image generation. By presenting merely a handful of subject images (typically 3–15), users can bind a unique physical entity to a dedicated pseudo-token or a specialized parameter subspace, enabling the generative model to synthesize the target subject within arbitrary novel scenes. However, existing personalization approaches frequently suffer from a severe structural issue termed concept-dominant failure: while the model reproduces the customized subject with high fidelity, prompt-specified contexts (including background scenery, object attributes, and interaction dynamics) are heavily neglected or completely suppressed. For instance, when prompting the model to depict a personalized toy against a city backdrop, the output persistently renders the mountain or indoor scene present in the original reference photos, crippling prompt controllability and compositional generation.

This failure stems from two interconnected structural biases during low-shot personalization. First, cross-token interference occurs within the cross-attention mechanism: context tokens undesirably allocate substantial attention weights to the subject region rather than the background, forfeiting their ability to govern background semantics. Second, spurious statistical correlations arise between the subject and co-occurring backgrounds due to limited background variability across reference images. Because standard denoising objectives lack explicit pixel-level subject supervision, the model inadvertently entangles background textures with the subject concept. Previous remedies predominantly rely on training-time objective modifications (e.g., anchor-based regularization in Compositional Inversion or CoRe) or inference-time latent guidance (e.g., Attend-and-Excite). These strategies are tightly coupled to specific personalization algorithms, require invasive modifications to training or inference pipelines, and lack broad compatibility across varied personalization architectures.

This paper tackles the challenge from a data-centric perspective: since attention imbalance and statistical entanglement originate from the training inputs, can we resolve them purely at the data level without altering model architectures or personalization objectives? Core idea: formulate personalization debiasing as an offline, reference-image-only augmentation framework (TOPA) that breaks subject-background spurious correlation via Subject-Isolation Compositing (SIC) and aligns token-region cross-attention via learned additive perturbations (CARM), achieving seamless, plug-and-play concept-context disentanglement for downstream personalization algorithms.

Method¶

Overall Architecture¶

TOPA is an entirely offline data augmentation pipeline. Taking a small set of reference images as input, it outputs an augmented reference dataset enriched with diverse synthesized backgrounds and imperceptible spatial perturbations. This augmented dataset is directly supplied to any standard downstream diffusion personalization pipeline without modifying downstream training objectives or sampling procedures.

The framework consists of two complementary components: 1. Subject-Isolation Compositing (SIC): Segments the target subject from reference images and composites it onto a diverse set of diffusion-generated backgrounds, statistically isolating the subject as the only persistent feature across the dataset; 2. Cross-Attention Rebalancing Module (CARM): Optimizes imperceptible additive perturbations $\delta$ on the composited images by penalizing context-token attention leakage into the subject region and simultaneously anchoring the pseudo-token attention to the subject mask. An alternating optimization schedule and region-based perturbation sharing are introduced to guarantee convergence and scalability.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Few-shot reference image set<br/>{x_i} with subject masks M_i"] --> B["Stage 1: Subject-Isolation Compositing SIC<br/>Segment subject and composite onto diverse backgrounds"]
    B --> C["Stage 2: Cross-Attention Rebalancing CARM<br/>Optimize perturbations δ to disentangle attention"]
    C --> D["Alternating Optimization Schedule<br/>Jointly update perturbation δ and token embedding e*"]
    D --> E["Augmented reference image set<br/>{x_tilde} with balanced cross-attention"]
    E --> F["Downstream Standard Personalization<br/>TI / DreamBooth / Custom Diffusion / etc."]

Key Designs¶

1. Subject-Isolation Compositing: Eliminating Spurious Statistical Correlations In few-shot personalization setups, reference images typically share identical capture environments. Without explicit semantic localization, the diffusion model binds recurring background features into the concept representation. SIC breaks this correlation by explicitly amplifying background variance while keeping subject identity invariant. Given $h$ reference images, SIC uses an automated segmentation model to extract the subject mask $M$ and generates $q$ diverse backgrounds $\{b_j\}$ using a pretrained Stable Diffusion model guided by diverse background prompts ($q=34$ in experiments). Each segmented subject $s_i$ is composited onto each background: $$\tilde{x}_{i,j} = \mathcal{C}(s_i, b_j)$$ Because the subject is the single invariant feature across all $h \times q$ composited images, the trainable concept representation is constrained to capture subject-specific identity rather than co-occurring background cues.

2. Cross-Attention Rebalancing Module: Dual-Target Interference Suppression and Anchoring Even when backgrounds are diversified, cross-attention distribution across tokens can remain suboptimal. CARM introduces an additive, imperceptible spatial perturbation $\delta$ onto the input image to actively guide cross-attention allocation during personalization. Given prompt $y^*$ containing pseudo-token $R^*$ with embedding $e^*$, let $\mathcal{C}$ denote the set of all context tokens (excluding $R^*$ and stopwords). CARM computes the aggregated context attention map $A_{\text{ctx}}(\tilde{x}, y^*)$ and penalizes its activation within the subject mask $M$: $$\mathcal{L}_{\text{ctx}} = \left\| M \odot A_{\text{ctx}}(x + \delta, y^*) \right\|_1$$ Because optimizing $\mathcal{L}_{\text{ctx}}$ in isolation can cause the pseudo-token's attention to diffuse spatially across the image, CARM adds a subject-token anchoring loss to align the pseudo-token cross-attention map $A_{R^*}$ with the subject mask: $$\mathcal{L}_{R^*} = \mathrm{MSE}\Big(M, A_{R^*}(x + \delta, y^*)\Big)$$ The overall rebalancing objective is formulated as: $$\mathcal{L}_{\text{CARM}} = \mathcal{L}_{\text{ctx}} + \lambda \mathcal{L}_{R^*}$$ with $\lambda=1.0$. This joint formulation suppresses context-to-subject attention intrusion while firmly anchoring the concept representation to the subject geometry.

3. Region-Based Perturbation Sharing and Alternating Optimization Optimizing independent perturbations across $h \times q$ composited images introduces prohibitive computational overhead, and static perturbations struggle to adapt as the pseudo-token embedding $e^*$ evolves during training. TOPA addresses these challenges via two strategies: - Region-based perturbation sharing: Leveraging the structural decomposition of composited images into subject $s_i$ and background $b_j$, TOPA learns a universal perturbation restricted to the background region $\delta^{(b)}_j$ shared across all subjects (or alternatively a subject perturbation $\delta^{(s)}_i$ shared across all backgrounds). Defaulting to background sharing reduces the number of perturbation optimization targets from $h \times q$ to $q$; - Alternating optimization schedule: Over $S=5000$ perturbation training iterations, standard denoising loss and $\mathcal{L}_{\text{CARM}}$ are optimized alternately. Every $K=2$ steps, the perturbation $\delta$ is updated using $\nabla \mathcal{L}_{\text{CARM}}$ (learning rate 0.02), while intermediate steps optimize the embedding $e^*$ with the standard diffusion loss (learning rate $1 \times 10^{-4}$). After convergence, the intermediate embedding is discarded, leaving the augmented images for standard downstream personalization.

Key Experimental Results¶

Main Results¶

Evaluations are conducted on 10 concepts randomly sampled from CustomConcept101, tested across 24 context prompts from DreamBench (10 images per prompt, totaling 2400 evaluated images per method). Experiments cover five representative personalization approaches: Textual Inversion (TI), Custom Diffusion, DreamBooth-LoRA, ELITE, and P+, comparing original baselines, naive augmentations (random cropping, flipping, color jittering, blur), and TOPA.

Main results from Table 1:

Method Config	BRISQUE ↓	CLIP-IQA ↑	CLIP-I† ↑	CLIP-T ↑	FID ↓	Flan-VQA ↑	BLIP-VQA ↑	LLaMA-VQA ↑
TI (Original baseline)	16.634	0.847	0.667	0.446	247	0.608	0.577	0.693
TI + naive augment	16.941	0.765	0.623	0.425	213	0.688	0.521	0.688
TI + TOPA (Ours)	14.574	0.871	0.671	0.472	220	0.654	0.683	0.773
Custom Diffusion	15.147	0.826	0.627	0.506	294	0.711	0.715	0.764
Custom + naive augment	16.131	0.815	0.608	0.525	279	0.715	0.750	0.806
Custom + TOPA (Ours)	15.740	0.855	0.613	0.534	259	0.717	0.810	0.844
DreamBooth LoRA	19.937	0.768	0.717	0.311	181	0.367	0.165	0.273
DreamBooth LoRA + TOPA	15.496	0.908	0.727	0.352	179	0.410	0.325	0.427
ELITE	15.711	0.832	0.691	0.427	219	0.752	0.622	0.915
ELITE + TOPA (Ours)	14.244	0.858	0.674	0.453	176	0.807	0.681	0.944

Note: CLIP-I† denotes the background-neutralized concept similarity introduced in the paper, which eliminates false-positive score inflation caused by background leakage. VQA scores evaluate background attribute presence via three vision-language models.

Ablation Study¶

Ablations on Textual Inversion demonstrate the impact of background replacement and individual loss formulations (Table 3):

Configuration	BRISQUE ↓	CLIP-IQA ↑	CLIP-I† ↑	CLIP-T ↑	FID ↓	BLIP-VQA ↑	LLaMA-VQA ↑	Note
Original TI	16.634	0.847	0.667	0.446	247	0.577	0.693	Unaugmented baseline
Background replaced only	16.899	0.871	0.661	0.484	231	0.673	0.731	Background metrics gain, but subject fidelity dips
With $\mathcal{L}_{\text{ctx}}$ only	15.346	0.855	0.669	0.453	251	0.608	0.727	Reduces interference, but lacks anchor stability
With $\mathcal{L}_{R^*}$ only	17.166	0.846	0.650	0.470	269	0.640	0.740	Anchoring alone degrades perceptual quality
Full TOPA (SIC + both losses)	14.574	0.871	0.671	0.472	220	0.683	0.773	Optimal balance of fidelity, quality, and control

Key Findings¶

Naive data augmentations fail on personalization: Standard vision augmentations (crop, flip, color jitter) degrade concept identity severely (TI's CLIP-I† drops from 0.667 to 0.623) while deteriorating perceptual quality (BRISQUE degrades on Custom Diffusion from 15.147 to 16.131);
Dual loss constraints are essential: Optimizing $\mathcal{L}_{\text{ctx}}$ without $\mathcal{L}_{R^*}$ allows the pseudo-token attention to spread unpredictably; conversely, optimizing $\mathcal{L}_{R^*}$ alone fails to prevent context tokens from intruding onto the subject. Combining both yields an 18.4% improvement in BLIP-VQA (0.577 → 0.683) and a 10.9% improvement in FID (247 → 220);
Orthogonal compatibility: When applied alongside methods specifically engineered for contextual consistency (Compositional Inversion, Attend-and-Excite, and Pivotal Tuning Inversion LoRA), TOPA provides consistent orthogonal improvements (e.g., boosting Attend-and-Excite's BLIP-VQA from 0.567 to 0.703 and reducing FID from 296 to 256).

Highlights & Insights¶

Model-agnostic, data-level paradigm: Instead of engineering intricate loss regularizers or modifying inference guidance steps, TOPA resolves concept dominance purely at the data boundary, rendering it instantly compatible with any downstream personalization method;
Constructive utilization of adversarial perturbations: It ingeniously flips the role of imperceptible perturbations—traditionally used for model poisoning or concept erasure—into a constructive regularizer that steers cross-attention geometry during fine-tuning;
Background-neutralized evaluation (CLIP-I†): Identifies the systematic bias in standard CLIP-I metrics caused by background leakage, offering a sounder evaluation standard by matching synthesized backgrounds before computing concept similarity.

Limitations & Future Work¶

Dependency on automated segmentation: Both SIC compositing and CARM masking require precise foreground masks; boundary artifacts or imperfect segmentation in intricate subjects (e.g., fine hair, fur, transparent surfaces) can introduce seam artifacts and corrupt perturbation optimization;
Offline preprocessing overhead: Generating 34 synthesized backgrounds and running alternating optimization over 5000 iterations introduces non-trivial upfront latency before personalization fine-tuning begins;
Future trajectories: Developing feed-forward perturbation generators that bypass explicit iterative optimization, and extending data-level attention rebalancing to video and 3D personalization workflows.

vs Textual Inversion [Gal et al., 2022]: TI optimizes only a pseudo-token embedding and frequently suffers from concept dominance. TOPA enhances the input data distribution directly, lifting TI's BLIP-VQA score from 0.577 to 0.683 without any modifications to TI's codebase;
vs Compositional Inversion [30] & CoRe [26]: These approaches impose regularization within the textual embedding space or cross-attention maps during training. TOPA operates at the data level and exhibits clear orthogonality, achieving cumulative gains when paired together;
vs Attend-and-Excite [Chefer et al., 2023]: Attend-and-Excite performs iterative latent optimization during each sampling step, introducing significant inference slowdown. TOPA pre-balances the training data, retaining standard, high-speed inference sampling.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Novel application of learned input perturbations to cross-attention rebalancing in personalization]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 5 personalization baselines and 3 context-improving methods with custom neutralized metric]
Writing Quality: ⭐⭐⭐⭐⭐ [Clean narrative, well-structured formulations, and thorough ablation insights]
Value: ⭐⭐⭐⭐☆ [Highly practical plug-and-play data preparation paradigm for customized content generation]