RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Rare Concept Synthesis / Compositional Generation / IP-Adapter / Closed-Loop Feedback / Training-Free Inference

TL;DR

Addressing attribute omission and semantic drift caused by dominant base-object priors in rare concept generation, RADIANCE introduces a training-free closed-loop feedback framework that monitors latent semantic emergence via CLIP and dynamically applies bidirectional IP-Adapter "restoring forces" to maintain compositional equilibrium.

Background & Motivation

Modern text-to-image (T2I) diffusion models have made remarkable progress in synthesizing photorealistic images aligned with complex textual prompts. With the rise of flow-matching backbones and Multimodal Diffusion Transformers (MM-DiT, such as SD 3.5 and FLUX), cross-modal reasoning capabilities have expanded substantially. However, when prompted with rare concepts involving unusual attribute-object pairings—such as "a thorny dolphin," "a butterfly shaped bowl," or "a zebra striped palm tree"—even state-of-the-art models frequently fail. Because web-scale training data follow a long-tailed distribution, common pairings dominate while rare co-occurrences have joint probabilities approaching zero. Consequently, generative sampling inevitably gravitates toward frequent co-occurrences.

This compositional failure stems from a fundamental tension during the denoising trajectory: the high-frequency prior of the base object dominates the feature space early in the trajectory, effectively suppressing the lower-frequency rare attribute. Even when models attempt to manifest the rare attribute, they often trigger severe failure modes—either decoupling into two disjoint entities (e.g., generating a regular bowl beside a separate fluttering butterfly) or completely distorting the base object's geometry (e.g., the dolphin collapsing into an unrecognizable spiky cluster). Existing prompt-switching approaches like Rare-to-Frequent (R2F) substitute rare tokens with frequent proxies before switching back, but they lack online monitoring and cannot rectify step-wise drift once a component dominates. Static guidance scales and attention-map interventions also remain too rigid to handle varying prompt dynamics.

The critical insight of this work is that diffusion sampling should not be treated as an open-loop, one-way injection, but as a closed-loop dynamical feedback system. Core idea: reframe diffusion inference as a training-free closed-loop feedback controller that tracks intermediate latent similarity in real time and applies bidirectional positive-negative IP-Adapter scaling as an adaptive "restoring force" to preserve compositional equilibrium between rare attributes and base objects.

Method

Overall Architecture

RADIANCE augments pretrained T2I diffusion backbones through a modular closed-loop pipeline operating strictly at test time without gradient computation. At each denoising step \(t\), the system first derives a single-step flow-based estimate of the target latent, decodes it into an estimated image, and extracts visual similarity signals for both the base object and rare attributes using a frozen CLIP encoder. The scheduler evaluates these similarity trends against the previous step to detect emerging bias. In response, a bidirectional controller assigns signed scaling factors to the IP-Adapter cross-attention pathways, applying an active restoring force that reinforces lagging concepts or suppresses over-dominant entities. Furthermore, for complex multi-object prompts, delayed activation and layer-wise alternating guidance prevent premature concept fusion.

To maintain practical throughput, the framework pipelines latent decoding and CLIP similarity calculation at step \(t\) concurrently with diffusion backbone forward passes at step \(t+1\), minimizing wall-clock overhead while boosting per-sample success rates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input text prompt and noisy latent"] --> B["Compositional Similarity Monitor<br/>Single-step flow decode and CLIP similarity"]
    B --> C["Feedback Guidance Scheduler<br/>Temporal stage split and trend detection"]
    C --> D["Bidirectional Scale Controller<br/>Signed IP-Adapter restoring force"]
    D --> E["Multiple Rare Concepts Merging<br/>Delayed activation and layer-wise cycling"]
    E --> F["Balanced denoising state and synthesized image"]

Key Designs

1. Compositional Similarity Monitor: Step-wise Flow Estimation and CLIP Tracking

To capture compositional drift without expensive VQA evaluators or iterative gradient backpropagation, the Compositional Similarity Monitor (CSM) provides lightweight, step-wise semantic tracking. Prior to sampling, reference images for the base object \(I_o\) and each attribute \(I_{a_i}\) are briefly synthesized using the base model. At timestep \(t\), CSM computes a one-step flow prediction \(\hat{z}_0\) from the intermediate latent \(\hat{z}_t\), decodes it via the VAE into an estimated image \(\hat{x}_t\), and computes cosine similarities against reference embeddings using CLIP encoder \(f_c\): $\(\text{sim}^{a_i}_t = \cos\big(f_c(\hat{\mathbf{x}}_t), f_c(I_{a_i})\big), \quad \text{sim}^o_t = \cos\big(f_c(\hat{\mathbf{x}}_t), f_c(I_o)\big)\)$ Crucially, empirical analysis reveals that successful composition does not require maximizing both similarities to extreme values (which tends to split the concepts into two separate visual items); rather, it requires keeping both similarity trajectories within a moderate, balanced band. CSM delivers this vital diagnostic signal at minimal computational cost.

2. Bidirectional Scale Controller: Negative IP-Adapter Scaling as an Active Restoring Force

Standard adapter guidance uses positive scale factors to inject reference features, but when a dominant prior takes over, increasing positive guidance often causes oversaturation or artifacts. The Bidirectional Scale Controller (BSC) exploits a key empirical property: negative scaling in IP-Adapter decoupled cross-attention layers actively suppresses reference concepts within the latent space. Within layer \(l\), the feature update is formulated as: $\(\mathbf{h}_t^{l+1} = F_A^l\Big(\mathbf{h}_t^l, f_\mathcal{E}(c)\Big) + \sum_{i=1}^n s_i F_A^l\Big(\mathbf{h}_t^l, f_\phi(I_{a_i})\Big)\)$ When the monitor detects that a rare attribute is being overwhelmed by the base object, BSC assigns a positive scale \(s_a > 0\) to reinforce the attribute and a negative scale \(s_o < 0\) to actively suppress the base object. Conversely, if the rare attribute begins distorting the base object's identity, the signs flip. This paired push-and-pull mechanism acts as a reactive physical "restoring force," pulling the latent trajectory back to equilibrium.

3. Feedback Guidance Scheduler: Temporal Partitioning and Adaptive Dynamic Policy

Applying arbitrary corrections at arbitrary timesteps disrupts the diffusion model's natural trajectory from global structure to fine-grained textures. The Feedback Guidance Scheduler (FGS) resolves this by coordinating dynamic updates across time. It establishes a threshold \(T_{\text{early}}\) dividing the trajectory into early emergence and late consolidation phases. For \(t \ge T_{\text{early}}\), if the attribute similarity decreases (\(\text{sim}^a_t < \text{sim}^a_{t+1}\)), the scheduler applies \((s^a, s^o) = (1, -1)\) to penalize object dominance; if the object similarity exhibits a downward trend, it switches to \((-1, 1)\). When both similarities evolve harmoniously, scales remain neutral at \((0, 0)\). In the late phase (\(t < T_{\text{early}}\)), the policy shifts priority toward preserving structural identity and visual fidelity, ensuring seamless convergence.

4. Multiple Rare Concepts Merging: Delayed Activation and Layer-Wise Alternating Guidance

When extended to prompts describing multiple rare concepts (e.g., "a donut shaped watermelon and a hamburger made of glass"), feeding multiple reference images into IP-Adapter in parallel causes severe semantic bleeding and premature concept fusion. RADIANCE prevents this through two complementary strategies. First, Delayed Adapter Activation (DAA) disables adapter guidance for the initial \(T_{\text{delay}}\) steps, allowing the base model to construct a clean multi-object spatial layout. Second, Layer-wise Alternating Guidance (LAG) cyclically assigns \(n\) reference images across Transformer layers \(l\): $\(\mathbf{h}_t^{l+1} = F_A^l\Big(\mathbf{h}_t^l, f_\mathcal{E}(c)\Big) + F_A^l\Big(\mathbf{h}_t^l, f_\phi(I_{(l \bmod n) + 1})\Big)\)$ Allocating individual concepts to distinct functional network layers preserves separate identity representations while enabling natural interactions in global self-attention blocks.

Loss & Training

RADIANCE is an entirely training-free, test-time framework that requires zero parameter updates, fine-tuning, or offline adapter training. In all experiments using SD 3.5, sampling is conducted across 24 denoising steps. Default scheduling hyperparameters are set to \(T_{\text{early}} = 15\) and \(T_{\text{late}} = 20\) (with \(T_{\text{early}} = 18\) for relation-heavy prompts). By overlapping single-step VAE decoding and CLIP feature encoding with Transformer denoising passes via pipelined execution, per-image latency on SD 3.5 remains competitive at 9.34 seconds (compared to 7.36 seconds for vanilla SD 3.5 and 16.48 seconds without pipelining).

Key Experimental Results

Main Results

Quantitative evaluations were conducted on RareBench (covering rare property, shape, texture, action, and multi-object subsets evaluated via GPT-4o consistency scoring) and T2I-CompBench (evaluated via BLIP).

Table 1: Quantitative Results on RareBench (Transcribed from Table 1 of the paper)

Method	Backbone	Venue	Single Property	Single Shape	Single Texture	Multi Concat	Multi Complex	Overall
SD 1.5	SD 1.5	CVPR'22	55.0	38.8	33.8	23.1	36.3	33.9
SDXL	SDXL	ICLR'24	60.0	56.9	71.3	39.4	47.5	52.0
SD 3.0	SD 3.0	ICML'24	49.4	76.3	53.1	55.0	70.0	61.5
FLUX	FLUX	-	69.4	78.1	52.5	63.1	81.3	67.3
SD 3.5	SD 3.5	ICML'24	77.5	80.6	75.0	71.9	80.6	75.9
SynGen	SD 1.4	NeurIPS'23	61.3	54.4	50.6	32.5	40.0	46.9
InitNO	SD 3.5	CVPR'24	56.9	62.5	51.2	56.3	75.6	63.0
R2F	SD 3.5	ICLR'25	90.0	82.5	89.4	78.1	81.3	81.7
RADIANCE (Ours)	SD 1.5	-	63.1	55.0	47.5	33.1	43.1	44.0
RADIANCE (Ours)	SD 3.5	-	97.5	89.4	89.4	80.0	85.0	84.7

Table 2: Quantitative Results on T2I-CompBench (Transcribed from Table 2 of the paper)

Method	Single Color	Single Shape	Single Texture	Multi Color	Multi Shape	Multi Texture	Overall
SD 1.5	81.8	75.2	74.0	34.6	32.8	38.4	56.1
FLUX	82.4	82.6	80.3	75.1	58.7	70.4	74.9
SD 3.5	87.5	77.6	73.0	76.2	60.0	71.1	74.2
R2F	89.1	76.9	73.2	75.1	51.6	68.3	72.4
RADIANCE (Ours)	88.6	79.6	77.0	77.2	61.8	71.3	75.9

Ablation Study

Extensive ablations on RareBench isolate the impact of bidirectional scale control, scheduling timesteps, and multi-concept decoupling mechanisms.

Table 3: Ablation Study on Single-Object Components and Feedback Timing (Transcribed from Table 4)

Method / Variant	Property	Shape	Texture	Action	Complex	Overall
Fixed \(s_o\)	46.3	65.6	71.9	35.0	63.1	56.4
Fixed \(s_a\)	66.3	84.4	57.5	55.0	71.9	67.0
Fixed both	83.8	51.2	82.5	78.8	68.8	73.0
w/o BSC	93.8	83.8	88.8	86.3	81.3	86.8
Negative prompt	87.5	84.4	87.5	83.1	74.4	83.4
\(T_{\text{early}}=13\)	94.4	88.1	88.1	88.1	86.9	89.1
\(T_{\text{early}}=15^*\) (Default)	97.5	89.4	89.4	87.5	85.6	89.9
\(T_{\text{early}}=17\)	95.6	87.5	86.3	83.8	87.5	88.1

Table 4: Multi-Object Disentanglement Ablation (Transcribed from Table 5)

Variant	Concat	Relation	Complex	Overall
w/o DAA	65.6	40.6	81.9	62.7
w/o LAG	76.2	58.1	78.8	71.0
RADIANCE (Full Model)	80.0	63.1	85.0	76.0

Key Findings

• Bidirectional restoring force is essential: Fixing either attribute or object scales causes severe collapse in complementary categories (e.g., fixed \(s_o\) drops Property to 46.3; fixed both drops Shape to 51.2). Removing BSC entirely reduces overall single-concept accuracy from 89.9 to 86.8, highlighting that negative suppression is vital to counter prior dominance.
• Feedback timing aligns with generative dynamics: Peak performance occurs at \(T_{\text{early}}=15\), which coincides with the transition from coarse global layout to fine textural synthesis. Stepping earlier (\(T=13\)) or later (\(T=17\)) slightly degrades performance.
• DAA and LAG decouple distinct stages of multi-object synthesis: DAA is critical for preventing early semantic confusion in basic spatial configurations (without DAA, Relation score plummets by 22.5 points), whereas LAG prevents feature blending across complex scenes by dedicating alternating Transformer layers to individual entities.

Highlights & Insights

• Active suppression via negative guidance: Unlike conventional adapter techniques that treat external conditioning purely as an additive signal, RADIANCE demonstrates that negative adapter scales serve as a targeted semantic brake, suppressing over-dominant base concepts without destabilizing the latent manifold.
• Efficient closed-loop test-time control: By pairing flow-based single-step projection with lightweight CLIP cosine scoring, RADIANCE achieves effective trajectory correction without relying on heavy VQA models or gradient optimization loops.
• Layer-wise orthogonal capacity allocation: The LAG mechanism allocates distinct concepts to alternating network depths, offering an elegant architectural blueprint for multi-subject and multi-reference diffusion synthesis.

Limitations & Future Work

• Reliance on backbone spatial priors: While RADIANCE excels at semantic attribute-object binding, it does not explicitly manipulate 2D spatial coordinate maps; hence, high-order topological relationships (e.g., "sitting strictly to the left of") remain constrained by the underlying model's layout biases.
• Sensitivity to coarse latent decoding: Single-step flow approximations in high-noise regimes are inevitably blurry, occasionally introducing momentary noise into early CLIP similarity estimates.
• Future directions: Integrating closed-loop semantic feedback with explicit bounding-box spatial conditioning and cross-object interaction modeling could provide unified control over both semantic binding and spatial geometry.

Related Work & Insights

• vs R2F (Rare-to-Frequent): R2F relies on open-loop text token substitution, which lacks dynamic monitoring and frequently swings between attribute omission and base-object destruction; RADIANCE monitors latent trajectories in real time, surpassing R2F by 3.0 points on RareBench.
• vs SynGen / InitNO: SynGen optimizes cross-attention maps via auxiliary loss functions and InitNO optimizes initial Gaussian noise. Both depend heavily on internal attention map structures and transfer poorly across diverse architectures. In contrast, RADIANCE operates via external image-level latent feedback, demonstrating robust generalization across both SD 1.5 and SD 3.5 backbones.

Rating

• Novelty: ⭐⭐⭐⭐☆ Creative formulation of negative IP-Adapter scales as a closed-loop restoring force during diffusion sampling.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across RareBench and T2I-CompBench, complemented by extensive user studies and systematic component ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, rigorous mathematical grounding, and clear narrative progression.
• Value: ⭐⭐⭐⭐☆ Highly practical, training-free plug-and-play solution for rare concept synthesis and compositional generation.

Related Papers

• [ICLR 2026] Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter
• [ACL 2025] Synthia: Novel Concept Design with Affordance Composition
• [ECCV 2026] To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion
• [ICLR 2026] SDErasure: Concept-Specific Trajectory Shifting for Concept Erasure via Adaptive Diffusion Classifier
• [ECCV 2026] Training-Free Multi-Concept Image Editing