Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generation¶
Conference: ECCV 2026
Paper: ECCV 2026
Area: Image Generation
Keywords: Personalized T2I, Diffusion Transformers, Decoupling Learning, Mixture of Experts, Multi-Subject Personalization
TL;DR¶
Addressing the core challenges of concept preservation versus prompt following trade-off, loss of fine-grained details, and limited multi-subject scalability in adapter-based zero-shot personalized T2I generation, DynaIP leverages the inherent decoupling learning behavior of MM-DiT dual branches and introduces a Hierarchical Mixture-of-Experts feature fusion module, achieving state-of-the-art single- and multi-subject personalization trained solely on single-subject datasets.
Background & Motivation¶
Personalized Text-to-Image (PT2I) generation aims to synthesize customized images adhering to target reference subjects while faithfully following complex textual instructions. While early approaches like DreamBooth and Textual Inversion relied on test-time model or embedding fine-tuning with prohibitive computational costs, adapter-based paradigms—represented by IP-Adapter—emerged as the dominant tuning-free alternative due to their plug-and-play flexibility and low inference overhead. However, when transitioning to cutting-edge Multimodal Diffusion Transformers (MM-DiT, e.g., FLUX.1), existing adapter-based solutions suffer from three fundamental limitations: an irreconcilable trade-off between Concept Preservation (CP) and Prompt Following (PF) caused by the entanglement of concept-specific attributes (identity, geometry, textures) with concept-agnostic attributes (pose, perspective, lighting); a severe loss of fine-grained visual details stemming from the sole reliance on deep CLIP features; and poor scalability when extending single-subject personalization (SS-PT2I) to multi-subject composition (MS-PT2I) via spatial mask injection, which often triggers jarring visual disharmony and identity leakage.
The core tension lies in the holistic injection mechanism of conventional adapters: injecting reference visual tokens without decoupling forces the generative backbone to overfit to reference context attributes, creating copy-paste artifacts and suppressing prompt control. Prior attempts to address attribute entanglement, such as DisEnvisioner, attempt explicit token separation for U-Net architectures but remain constrained by deep CLIP representations and lack compatibility with dual-stream MM-DiT backbones. Meanwhile, current multi-subject methods typically resort to curating massive, expensive multi-subject paired datasets to retrain the foundation model, sacrificing the lightweight nature of prompt adapters.
This paper's angle of attack stems from a key discovery regarding the MM-DiT architecture: within its multimodal attention mechanism, injecting reference visual features naturally triggers an emergent decoupling behavior across its dual branches—the noisy image branch selectively encodes concept-specific visual details (identity, core morphology), whereas the text branch predominantly learns concept-agnostic global context (pose, view angle, illumination). Core idea: exploit the intrinsic decoupling learning behavior of MM-DiT via dual-branch cross-attention training while dynamically pruning the text-branch injection during inference, paired with a Hierarchical Mixture-of-Experts Feature Fusion Module (HMoE-FFM) to dynamically integrate multi-granularity CLIP features for scalable, high-fidelity single- and multi-subject generation.
Method¶
Overall Architecture¶
DynaIP is built upon the state-of-the-art MM-DiT backbone (FLUX.1-Dev). Given reference images and input prompts, the reference visual features are first extracted by CLIP and processed through a Hierarchical Mixture-of-Experts Feature Fusion Module (HMoE-FFM), dynamically blending low-, mid-, and high-level representations to retain intricate textures alongside global semantics. In the diffusion transformer backbone, each of the 57 MM-DiT blocks is augmented with decoupled cross-attention layers. During training, the fused reference tokens attend to both the noisy image tokens and text tokens, allowing the dual branches to naturally partition concept-specific and concept-agnostic information. During inference, the Dynamic Decoupling Strategy (DDS) activates: reference features are exclusively injected into the noisy image branch, completely discarding the concept-agnostic context learned by the text branch. For multi-subject generation, region-level masks guide the isolated injection of multiple reference subjects, enabling harmonious multi-subject synthesis without retraining.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Reference Images"] --> B["Hierarchical Mixture-of-Experts Feature Fusion<br/>Multi-layer CLIP token processing"]
B --> C["Gated Routing Aggregation<br/>Dynamic coefficient calibration"]
C --> D["Dual-Branch Decoupled Training<br/>Text branch learns context / Image branch learns ID"]
D --> E["Dynamic Decoupling Strategy<br/>Inference-time context pruning"]
E -->|Region Masks| F["Mask-Guided Regional Injection<br/>Harmonious multi-subject composition"]
Key Designs¶
1. Dynamic Decoupling Strategy: Eliminating Concept-Agnostic Interference for Optimal CP·PF Balance Standard IP-Adapter configurations inject reference features directly into image cross-attentions, causing illumination, viewpoint, and camera angle to be rigidly replicated, which cripples prompt-guided edits. DynaIP uncovers that when reference visual features interact with both the text and image branches of MM-DiT during training: $\(DCA'(T, X, C) = [T^{MMA}, X^{MMA}] + \lambda \cdot \text{CA}([T, X], C)\)$ the text branch inherently absorbs concept-agnostic information (pose, perspective, scene lighting), leaving the noisy image branch to focus purely on invariant concept-specific characteristics (identity, fine textures). DynaIP capitalizes on this phenomenon by introducing the Dynamic Decoupling Strategy (DDS): during inference, the cross-attention with the text branch is completely disabled, retaining only the noisy image cross-attention. This selective pruning frees the generator to follow textual instructions flawlessly while maintaining pristine concept fidelity, completely preventing copy-paste collapse.
2. Hierarchical Mixture-of-Experts Feature Fusion Module: Bridging Multi-Granularity Visual Details Relying solely on deep CLIP representations (penultimate or final layer) discards vital low-level geometric and mid-level textural details. Empirical probing shows that CLIP layer 10 captures low-level contours and legible text; layer 17 captures fine facial and pattern structures; and layer 24 captures semantic abstraction. To harness this hierarchy, HMoE-FFM deploys layer-specific expert networks \(Expert_l\) (\(l \in \{Low, Mid, High\}\)), each composed of a Linear layer, GELU activation, and Layer Normalization, to process full feature tokens \(\Phi^{Full}_l\): $\(e_l = Expert_l(\Phi^{Full}_l)\)$ Simultaneously, a routing module comprising a two-layer MLP with Tanh activation predicts input-conditioned fusion coefficients \(w_l\) from the class tokens \(\Phi^{CLS}_l\): $\(\Phi_{Fused} = \sum_{l \in \{Low, Mid, High\}} w_l \cdot e_l\)$ This structure dynamically calibrates feature granularity per reference image. Furthermore, it unlocks runtime granularity control: users can manually modulate expert coefficients to dial between abstract semantic transfer and pixel-level texture preservation.
3. Mask-Guided Regional Injection: Zero-Shot Multi-Subject Personalization Without Retraining Conventional multi-subject generation requires complex joint-training on multi-subject datasets. Because DynaIP isolates pure concept-specific attributes without scene-specific illumination or pose biases, reference features can be injected regionally into distinct spatial locations without cross-subject conflict. Given \(N\) reference images with corresponding binary masks \(M_i\) (obtained via automated detection/segmentation tools like Grounding DINO and SAM or manual input): $\(DCA''(T, X, C) = X^{MMA} + \sum_{i=1}^N \lambda_i \cdot M_i \cdot \text{CA}(X, C_i)\)$ Each reference subject is injected into its corresponding region, while global composition, natural physical interaction, and uniform lighting are harmonized end-to-end by the backbone MM-DiT, enabling seamless multi-subject generation using only single-subject training weights.
Loss & Training¶
The underlying FLUX.1-Dev base model (57 MM-DiT blocks) is entirely frozen; only the newly added image cross-attention layers and the HMoE-FFM module are optimized, encompassing ~1.4B trainable parameters. Optimization employs standard flow-matching loss across two sequential stages: an intra-pair training stage (20,000 iterations) to establish robust subject identity binding, followed by a cross-pair training stage (80,000 iterations) to penalize background copy-paste tendencies. Models are trained on 8 Ascend 910B NPUs using AdamW (initial learning rate \(2 \times 10^{-5}\) with cosine schedule, weight decay 0.0001, batch size 16, resolution \(1024 \times 1024\)). Classifier-free guidance is facilitated by a 5% independent text dropout, 5% image dropout, and 5% joint dropout; expert features in HMoE-FFM undergo independent 5% dropout during training.
Key Experimental Results¶
Main Results¶
Evaluations follow the DreamBench++ benchmark protocol, leveraging two SOTA open-source MLLMs (InternVL3-78B and Qwen3-VL-32B) to assess Concept Preservation (CP) and Prompt Following (PF), with their geometric product (Nash utility, \(CP \cdot PF\)) serving as the primary metric. Single-subject evaluation uses DreamBench++ (1,350 samples); multi-subject evaluation uses DynaIP-Bench (888 two- and three-subject samples).
| Scenario / Dataset | Method | Concept Pres. (CP) | Prompt Foll. (PF) | Nash Utility (CP·PF) |
|---|---|---|---|---|
| Single-Subject (DreamBench++) | DreamBooth | 0.458 | 0.721 | 0.330 |
| Single-Subject (DreamBench++) | DreamBooth LoRA | 0.594 | 0.840 | 0.499 |
| Single-Subject (DreamBench++) | Textual Inversion | 0.348 | 0.633 | 0.220 |
| Single-Subject (DreamBench++) | IP-Adapter-Plus | 0.738 | 0.668 | 0.493 |
| Single-Subject (DreamBench++) | DisEnvisioner | 0.559 | 0.664 | 0.371 |
| Single-Subject (DreamBench++) | FLUX.1 IP-Adapter | 0.681 | 0.600 | 0.408 |
| Single-Subject (DreamBench++) | Diptych Prompting | 0.616 | 0.839 | 0.517 |
| Single-Subject (DreamBench++) | OminiControl | 0.596 | 0.895 | 0.534 |
| Single-Subject (DreamBench++) | FLUX.1 Kontext Dev | 0.718 | 0.893 | 0.641 |
| Single-Subject (DreamBench++) | DynaIP (Ours) | 0.696 | 0.934 | 0.650 |
| Multi-Subject (DynaIP-Bench) | Qwen-Image-Edit | 0.612 | 0.984 | 0.602 |
| Multi-Subject (DynaIP-Bench) | BAGEL | 0.566 | 0.885 | 0.500 |
| Multi-Subject (DynaIP-Bench) | OmniGen2 | 0.552 | 0.952 | 0.526 |
| Multi-Subject (DynaIP-Bench) | MS-Diffusion | 0.584 | 0.850 | 0.496 |
| Multi-Subject (DynaIP-Bench) | MIP-Adapter | 0.388 | 0.713 | 0.276 |
| Multi-Subject (DynaIP-Bench) | UNO | 0.509 | 0.857 | 0.436 |
| Multi-Subject (DynaIP-Bench) | XVerse | 0.548 | 0.890 | 0.488 |
| Multi-Subject (DynaIP-Bench) | DynaIP (Ours) | 0.617 | 0.997 | 0.615 |
Ablation Study¶
Ablation experiments isolate the impact of the Dynamic Decoupling Strategy (DDS), multi-layer fusion designs, individual hierarchical features, and token concatenation.
| Index | Configuration | SS CP | SS PF | SS CP·PF | MS CP | MS PF | MS CP·PF | Note |
|---|---|---|---|---|---|---|---|---|
| (1) | Full Model | 0.696 | 0.934 | 0.650 | 0.617 | 0.997 | 0.615 | Full DDS + HMoE-FFM |
| (2) | w/o DDS | 0.785 | 0.799 | 0.627 | 0.499 | 0.545 | 0.272 | Dual-branch inference retains context; MS collapses |
| (3) | Add Fusion | 0.691 | 0.916 | 0.633 | 0.609 | 0.995 | 0.606 | Simple element-wise addition without routing |
| (4) | Concat Fusion | 0.692 | 0.909 | 0.629 | 0.607 | 0.992 | 0.602 | Feature channel concatenation |
| (5) | Only Shallow | 0.627 | 0.924 | 0.579 | 0.464 | 0.991 | 0.460 | CLIP layer 10 only; lacks semantic coherence |
| (6) | Only Mid | 0.670 | 0.928 | 0.622 | 0.603 | 0.993 | 0.599 | CLIP layer 17 only; balanced local textures |
| (7) | Only Deep | 0.480 | 0.950 | 0.456 | 0.474 | 0.995 | 0.471 | CLIP layer 24 only; severe loss of fine details |
| (8) | Token-Concat | 0.709 | 0.886 | 0.628 | - | - | - | Direct token concatenation with LoRA tuning |
Key Findings¶
- DDS is the linchpin of multi-subject scalability: Disabling DDS (row 2) forces concept-agnostic attributes into the generation, causing prompt following to plummet and multi-subject Nash utility to drop precipitously by 55.8% (from 0.615 to 0.272) due to conflicting lighting and pose artifacts.
- Hierarchical features are indispensable for detail preservation: Utilizing only deep features (row 7) severely penalizes CP (0.480 in SS), whereas incorporating shallow and mid-level layers recovers critical high-frequency textures, elevating CP to 0.696. The adaptive MoE routing distinctly outperforms static addition and concatenation.
- Single-subject training surpasses multi-subject specialized models: By virtue of uncorrupted concept-specific features, DynaIP outperforms models explicitly trained on multi-subject datasets (e.g., Qwen-Image-Edit, MS-Diffusion), achieving top-ranked MS CP (0.617) and near-perfect MS PF (0.997).
Highlights & Insights¶
- Harnessing emergent decoupling in MM-DiT dual streams: Identifying that the text branch naturally absorbs context-agnostic semantics while the image branch retains concept identity allows attribute disentanglement purely through training-time routing and inference-time pruning, bypassing complex regularization losses.
- Restoring multi-granularity utility from standard CLIP: Systematically uncovering that intermediate CLIP layers encode fine-grained lines and textures, and structuring them via an MoE router, offers both improved fidelity and real-time visual granularity control.
- Native composability with generative extensions: As a decoupled adapter module, DynaIP integrates seamlessly with downstream community plugins such as ControlNet, LoRA, and regional prompt controllers without conflicting parameters.
Limitations & Future Work¶
- Severe overlap in crowded multi-subject scenes: When generating scenes with four or more subjects undergoing intricate physical interactions, spatial mask bounding may encounter boundary bleed or occlusion ambiguities.
- Dependency on upstream localization tools: Automated inference relies on the accuracy of detection and segmentation backbones (Grounding DINO + SAM); segmentation failures directly degrade regional injection accuracy.
- Future directions: Incorporating soft attention mutual exclusion masks directly within self-attention maps could enable mask-free, fully autonomous multi-subject layout reasoning.
Related Work & Insights¶
- vs IP-Adapter / FLUX.1 IP-Adapter: Prior adapters inject entangled visual features into generation, causing severe context copying and prompt suppression. DynaIP isolates concept-specific information via MM-DiT dual-branch decoupling, dramatically boosting prompt responsiveness.
- vs DisEnvisioner: While DisEnvisioner attempts token-based concept separation on U-Nets, it remains restricted to deep CLIP features and loses fine-grained textures. DynaIP leverages native MM-DiT dual branches and multi-layer MoE fusion, recovering fine-grained details.
- vs MS-Diffusion / UNO: Previous multi-subject methods rely on multi-subject training corpora or full attention joint training. DynaIP proves that clean, decoupled single-subject representations can generalize zero-shot to multi-subject composition with superior visual harmony.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant discovery of MM-DiT's intrinsic dual-branch decoupling behavior combined with an adaptive hierarchical MoE fusion mechanism.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across DreamBench++ and DynaIP-Bench using two SOTA MLLMs, backed by comprehensive ablation and granularity analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid formulation, highly systematic problem framing, and intuitive visual explanations.
- Value: ⭐⭐⭐⭐⭐ Delivers a highly practical, plug-and-play adapter that sets a new benchmark for scalable zero-shot personalization on modern diffusion transformers.