Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/HKUST-LongGroup/MoKus
Area: Knowledge Editing
Keywords: Concept Customization / Knowledge Editing / Cross-Modal Knowledge Transfer / Diffusion Transformer / Text-to-Image Generation

TL;DR

Addressing the instability and lack of semantic reasoning in conventional rare-token concept customization, this paper introduces the knowledge-aware concept customization task alongside MoKus, a two-stage framework that leverages cross-modal knowledge transfer to project diverse natural language knowledge onto a visual anchor via closed-form model editing, achieving efficient and high-fidelity customized generation.

Background & Motivation

Subject-driven image customization methods, spearheaded by techniques such as DreamBooth, CustomDiffusion, and Textual Inversion, have made substantial strides in personalizing generative visual content. These established paradigms typically designate a rare token (such as <sks>) to represent a user-specified concept, fitting the low-rank weights or text embeddings to reconstruct reference visual latents. Nevertheless, relying on artificial rare tokens incurs two severe bottlenecks: unstable generation performance and knowledge unawareness. Because rare tokens virtually never appear in pretraining corpora, combining them with arbitrary narrative prompts disrupts the text encoder's linguistic manifolds, frequently resulting in catastrophic distortion or identity loss. Concurrently, rare tokens act merely as ungrounded visual placeholders stripped of all world knowledge. For instance, while a tuned model may reconstruct the Little Mermaid statue using <sks> sculpture, it completely fails to recognize natural language descriptions such as "the bronze sculpture in Copenhagen harbour" or "Little Mermaid Statue Denmark".

Knowledge-aware concept customization introduces challenging requirements. During generation, the model must accurately spot nuanced concept-identifying knowledge phrases within prompts and synthesize them seamlessly with contextual descriptions depicting scenes, lighting, and actions. Furthermore, real-world visual concepts naturally link to multiple knowledge items spanning objective facts (e.g., origin, physical composition) and personal associations (e.g., "my favorite sculpture"). Naively extending existing methods is fundamentally impractical: rare-token fine-tuning requires repeating the complete diffusion optimization for every newly added piece of knowledge, taking dozens of minutes, whereas feed-forward image encoders cannot bind open-domain textual knowledge without collecting vast paired corpora and undergoing prohibitive retraining.

Overcoming these limitations requires a modular and lightweight transfer mechanism. When investigating multimodal diffusion architectures built with Large Language Model (LLM) text encoders and Diffusion Transformer (DiT) backbones, the authors uncovered a critical phenomenon: cross-modal knowledge transfer. Specifically, updating answers to factual queries inside the LLM text encoder via parameter editing directly propagates to the downstream visual modality, causing the diffusion model to generate images aligned with the edited text answers without modifying the generative backbone. Core idea: decouple knowledge-aware concept customization into visual concept learning and textual knowledge updating, where visual appearance is first mapped to a rare token acting as an anchor representation via light LoRA, after which diverse natural language knowledge entries are projected onto this anchor through closed-form least-squares LLM editing in seconds.

Method

Overall Architecture

MoKus is built upon a text-to-image architecture comprising an LLM text encoder \(\phi\) and a Multi-Modal Diffusion Transformer (MMDiT) generation backbone. The operational pipeline consists of two decoupled stages: visual concept learning and textual knowledge updating. In the visual concept learning stage, given a handful of reference images for a concept, the framework freezes the base generative model and optimizes only lightweight Low-Rank Adaptation (LoRA) parameters injected into the MMDiT self-attention layers. This stage binds the visual identity to a rare token prompt, turning its textual embedding into an "anchor representation." In the textual knowledge updating stage, multiple natural language knowledge phrases associated with the concept are converted into question-answer queries. By solving a regularized closed-form least-squares objective, MoKus updates only selected MLP projection layers in the LLM text encoder, aligning the queries' representations directly with the pre-computed visual anchor. At inference time, users can freely compose any bound natural language knowledge phrase with arbitrary scene prompts to generate high-fidelity, customized images.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Concept Reference Images + Multiple Textual Knowledge Entries"] --> B["Stage 1: Visual Concept Learning<br/>MMDiT LoRA fitting to produce anchor representation"]
    B --> C["Stage 2: Textual Knowledge Updating<br/>Query formulation and closed-form least-squares parameter shift"]
    C --> D["Inference: Knowledge prompt combined with scene descriptions"]

Key Designs

1. Visual Concept Learning: Constructing Decoupled Visual Anchor Representations

Traditional customization frameworks expose rare tokens directly to downstream prompt synthesis, leading to severe linguistic brittleness. MoKus resolves this by treating the rare token (e.g., <sks> dog) strictly as an intermediate "anchor representation" whose sole objective is to memorize the high-frequency geometric and textural features of the target concept rather than serving as the end-user prompt. Given concept images \(X = \{x_i\}_{i=1}^M\), a pretrained Variational Autoencoder (VAE) extracts visual latents \(z_0 = \mathcal{E}(x_i)\). Following Rectified Flow dynamics, the latent at sampled timestep \(t \sim \text{Logit-Normal}(0, 1)\) is formulated as \(z_t = t z_1 + (1 - t) z_0\) with standard Gaussian noise \(z_1 \sim \mathcal{N}(0, I)\). Meanwhile, the LLM encoder \(\phi\) extracts textual latents \(h = \phi(P)\) from the rare token template \(P\). To preserve the generation prior of the foundation model, MoKus freezes the VAE and MMDiT weights, updating only trainable LoRA parameters \(\theta_v\) in the MMDiT self-attention blocks by minimizing velocity prediction error:

\[\mathcal{L}(\theta_v) = \mathbb{E}_{x_i, z_1, t} \left\| v_{\theta_v}(z_t, t, h) - (z_1 - z_0) \right\|^2\]

This stage equips the rare token representation with high visual reconstruction fidelity, establishing a grounded target for subsequent cross-modal knowledge projection.

2. Textual Knowledge Updating: Sub-Minute Multi-Knowledge Binding via Closed-Form Least-Squares

Once the visual anchor is obtained, the core challenge is enabling the model to comprehend diverse natural language expressions—such as "my friend's dog" or "the dog belonging to my neighbor"—and associate them with this anchor without distorting linguistic priors. Standard gradient backpropagation through the text encoder risks representation collapse and catastrophic forgetting. MoKus reframes this challenge as model editing: given knowledge entries \(K = \{k_i\}_{i=1}^N\), each phrase is transformed into an interrogative query \(q_i\) (e.g., "What is my friend's dog?"), paired with the shared visual anchor representation \(y = \phi(P)\) as the expected target. To update parameters \(\theta_t\) in designated editable layers (specifically Gate Projection and Up Projection matrices in intermediate MLP blocks), MoKus extracts forward hidden states \(h_i\) and computes the update direction:

\[v_i = -\eta \cdot \|h_i\|^2 \cdot \nabla_{\theta_t} y_i\]

where \(\eta\) denotes a scaling factor. To assimilate new knowledge associations while minimizing disruption to general linguistic capabilities, MoKus optimizes a regularized least-squares objective penalizing parameter drift:

\[\min_{\Delta \theta_t} \|H \Delta \theta_t - V\|^2 + \lambda \|\Delta \theta_t\|^2\]

Benefiting from convexity, this formulation yields a deterministic closed-form solution:

\[\Delta \theta_t = (H^\top H + \lambda I)^{-1} H^\top V\]

The updated parameters are directly assigned via \(\hat{\theta}_t = \theta_t + \Delta \theta_t\). Bypassing iterative backpropagation, updating each piece of knowledge requires only forward passes and matrix inversion, completing in approximately 7 seconds. Because the bound knowledge resides natively within natural language manifolds, it generalizes robustly when combined with complex scene prompts.

Loss & Training

MoKus experiments were implemented on 8 NVIDIA H800 GPUs built upon Qwen-Image. In Stage 1 (Visual Concept Learning), LoRA parameters inserted into the MMDiT self-attention blocks are optimized using AdamW with a learning rate of \(2 \times 10^{-4}\) under standard diffusion hyperparameters. In Stage 2 (Textual Knowledge Updating), iterative gradient steps are eliminated entirely by adapting the UltraEdit protocol to edit the Gate Projection and Up Projection weights across layers 18 to 26 of the LLM text encoder (16 parameter matrices total). Knowledge updating uses a batch size of \(m=1\) and scaling factor \(\eta = 1 \times 10^{-6}\). The minimal parameter shift guarantees preservation of pretraining distributions while executing rapid cross-modal transfer.

Key Experimental Results

Main Results

Evaluation was conducted on KnowCusBench, the first dedicated benchmark for knowledge-aware concept customization covering 35 diverse concepts spanning toys, plushies, pets, and scenes. The benchmark measures both direct knowledge reconstruction and compositional generation across complex prompts. MoKus is compared against Naive-DB (DreamBooth repeated for each knowledge item) and Enc-FT (fine-tuning the LLM text encoder directly). Visual fidelity is evaluated via standard CLIP-I alongside CLIP-I-Seg, which uses SAM3 to isolate the customized concept and remove background bias. Prompt fidelity and human preference are measured using CLIP-T and Pick Score, respectively.

Method	Recon: CLIP-I ↑	Recon: CLIP-I-Seg ↑	Gen: CLIP-I ↑	Gen: CLIP-I-Seg ↑	Gen: CLIP-T ↑	Gen: Pick Score ↑	Training Time ↓
Naive-DB	0.874	0.758	0.789	0.717	0.291	20.80	~27 min
Enc-FT	0.582	0.553	0.591	0.562	0.197	18.34	~10 min
MoKus (Ours)	0.867	0.764	0.761	0.718	0.305	21.30	~6 min

Ablation Study

1. Scalability Across Knowledge Count

Evaluating reconstruction quality, generation fidelity, and cumulative training time as the number of bound textual knowledge items scales from 1 to 5.

Knowledge Number	Recon: CLIP-I ↑	Recon: CLIP-I-Seg ↑	Gen: CLIP-I ↑	Gen: CLIP-I-Seg ↑	Gen: CLIP-T ↑	Gen: Pick Score ↑	Training Time (s) ↓
1	0.868	0.761	0.767	0.722	0.304	21.29	331.3 s
2	0.868	0.762	0.762	0.718	0.305	21.30	338.3 s
3	0.867	0.761	0.762	0.719	0.305	21.30	345.1 s
4	0.868	0.761	0.763	0.718	0.305	21.30	352.1 s
5	0.867	0.764	0.762	0.718	0.305	21.30	360.0 s

2. Sensitivity of Scaling Factor \(\eta\)

Analyzing the impact of scaling factor \(\eta\) in controlling the magnitude of parameter updates within the closed-form formulation.

Scaling Factor \(\eta\)	Recon: CLIP-I ↑	Recon: CLIP-I-Seg ↑	Gen: CLIP-I ↑	Gen: CLIP-I-Seg ↑	Gen: CLIP-T ↑	Gen: Pick Score ↑
\(1 \times 10^{-4}\)	0.557	0.535	0.588	0.564	0.218	18.40
\(1 \times 10^{-5}\)	0.866	0.763	0.759	0.717	0.304	21.29
\(1 \times 10^{-6}\) (Default)	0.867	0.764	0.761	0.718	0.305	21.30
\(1 \times 10^{-7}\)	0.867	0.761	0.760	0.717	0.304	21.29
\(1 \times 10^{-8}\)	0.867	0.764	0.760	0.718	0.304	21.29

Key Findings

• Superior Subject Fidelity and Text Alignment: On CLIP-I-Seg, which isolates subject pixels using SAM3, MoKus surpasses Naive-DB in both reconstruction (0.764 vs 0.758) and scene generation (0.718 vs 0.717). MoKus also delivers markedly higher text prompt compliance (CLIP-T 0.305 vs 0.291) and human preference (Pick Score 21.30 vs 20.80) while cutting total training time by more than 75%.
• Text Encoder Fine-Tuning Triggers Latent Collapse: Enc-FT experiences severe degradation across all metrics (reconstruction CLIP-I plummets to 0.582, generation CLIP-T to 0.197). Gradient fine-tuning alters the encoder's global latent distribution, rendering the diffusion model incapable of text-guided generation.
• Linear Scaling with Minimal Overhead: Adding an additional knowledge item incurs only ~7 seconds of compute time (scaling from 331.3 s for 1 item to 360.0 s for 5 items). Performance metrics remain virtually constant, demonstrating that closed-form editing prevents interference across multiple knowledge updates.
• Versatile Extension to Downstream Tasks: On the world knowledge benchmark WISE, MoKus boosts WiScore from 0.81 to 1.33 without retraining the diffusion generator. It also successfully supports virtual concept creation and targeted concept erasure.

Highlights & Insights

• Empirical Confirmation of Cross-Modal Knowledge Transfer: This work provides compelling evidence that localized parameter edits in an LLM text encoder transfer directly to the visual generation outputs of downstream diffusion models, overturning earlier negative findings from GapEval and UniSandbox.
• Decoupling Rare Tokens into Latent Anchors: Treating rare tokens purely as intermediate mathematical anchors sidesteps linguistic incompatibilities, pairing the precision of localized LoRA adaptation with the semantic fluency of natural language knowledge.
• Closed-Form Editing Guarantees Efficiency and Stability: Closed-form least-squares updates execute in seconds, avoiding gradient descent iterations and eliminating the risk of catastrophic forgetting in the text encoder.

Limitations & Future Work

• Author-Acknowledged Limitations: The framework currently targets static 2D image customization and has not been validated on dynamic video or 3D generation. Moreover, the pipeline operates in two distinct stages rather than a unified end-to-end framework.
• Benchmark Evaluation Constraints: KnowCusBench primarily evaluates visual similarity via CLIP-I and SAM3 segmentation; richer semantic benchmarks are needed to evaluate abstract functional and emotional knowledge bindings.
• Future Directions: Exploring end-to-end joint optimization via reinforcement learning or interleaved token processing, as well as integrating dynamic external retrieval to support open-ended lifelong customization.

Related Work & Insights

• vs DreamBooth / CustomDiffusion: Standard personalization embeds rare tokens directly into user prompts, suffering from poor semantic composition and failing to support natural language knowledge queries. MoKus treats rare tokens as internal visual anchors and exposes natural language phrases to users.
• vs Enc-FT / GapEval: Prior attempts to update knowledge via text encoder fine-tuning disrupted latent distributions. MoKus demonstrates that localized, closed-form MLP editing preserves linguistic structure while achieving reliable cross-modal transfer.
• vs ROME / MEMIT: Traditional model editing is limited to factual QA in text-only LLMs. MoKus demonstrates that edited representations can serve as conditioning vectors that steer multimodal generative diffusion models.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates the knowledge-aware concept customization task and establishes a sound visual-anchor framework via cross-modal knowledge transfer.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces KnowCusBench, rigorous SAM3 segmentation metrics, extensive multi-knowledge ablations, and world knowledge validation.
• Writing Quality: ⭐⭐⭐⭐⭐ Logical flow, clear formalization, and transparent experimental analyses.
• Value: ⭐⭐⭐⭐⭐ Offers a practical, highly efficient blueprint for personalized, knowledge-grounded content creation in foundation models.