ECCV 2026 Image Generation Video Generation Diffusion Model One-shot concept disentanglement Flow matching Velocity-field probing Stage-aware guidance Compositional generation

ConceptWeaver: Weaving Disentangled Concepts with Flow¶

Conference: ECCV 2026
Paper: ECCV page
Code: https://github.com/JasperChennn/ConceptWeaver
Area: Image Generation / Video Generation / Diffusion Model
Keywords: One-shot concept disentanglement, Flow matching, Velocity-field probing, Stage-aware guidance, Compositional generation

TL;DR¶

The paper first introduces a differential probing technique showing that concept formation in flow models follows a three-stage "Blueprint–Instantiation–Refinement" schedule, in which content concepts peak in both intensity and spatial localization and become naturally disentangled at mid-stage; guided by this, ConceptWeaver learns low-rank semantic offsets from a single reference image and optimizes them only within each concept's peak stage, then injects them stage-awarely via ConceptWeaver Guidance (CWG), outperforming TokenVerse, MS-Diffusion and DreamBooth on single- and multi-concept image generation (multi-concept S5.5GPT 4.14 vs 3.08/2.98/2.47).

Background & Motivation¶

Flow matching backbones (the Flux / Wan2.2 generation) already synthesize richly detailed scenes from text prompts, and this compositional ability in itself implies that concepts are encoded in some decomposable form inside the model. Yet "take one concept out of a reference image and drop it into a completely new scene" remains hard, and the three existing routes each pay a price. In textual space, Textual Inversion compresses a concept into a pseudo-word, trading subject fidelity for prompt controllability. In parameter space, DreamBooth / LoRA modify the model weights directly and tend to overfit the reference, producing the so-called "copy-paste phenomenon" — background and composition are pinned by the reference no matter what the prompt says. For multi-concept work, TokenVerse found the per-token text modulation space of Diffusion Transformers and learned offsets along Flux's channel-modulation pathway, achieving impressive mask-free multi-concept personalization for the first time.

The problem is that TokenVerse treats "where to modify" as the only handle, which leaves two structural gaps. First, it relies on the text-conditioned modulation pathway, an architectural feature absent from most DiTs, whose modulation typically encodes only non-semantic signals such as the timestep; the method is therefore locked to a narrow set of models. Second and more importantly, it never considers when to modify a concept — when a concept takes shape along the generative trajectory, when it is most malleable, and when touching it damages something else. That temporal dimension is ignored entirely. Prior flow-editing work (FlowEdit, SplitFlow, and others) does care about entanglement, but its angle is "how to decompose the editing trajectory or prompt"; it likewise never answers when a reference concept emerges in the velocity field.

This paper's position is that a concept's temporal behaviour is not a heuristic to be tuned by feel but a physical quantity that can be measured. The velocity field is the entirety of the generative force at each step, so removing a concept token from the prompt and observing how much the velocity field changes yields a curve of that concept's "existence contribution" over time. Using this differential probe — contrasting against a partially masked prompt rather than a null one — the authors measure concept dynamics systematically in both image and video generation and find that the standard CFG prompt-level shift is dense and entangled from start to finish with no visible structure, whereas the concept-specific shift is highly dynamic: weak and diffuse early, peaking simultaneously in intensity and spatial locality at mid-stage, then decaying into sparse high-frequency noise late. Core idea: concepts only take shape and disentangle within a specific temporal stage, so "learn a concept in its own peak stage and inject it in that same stage" is itself a disentanglement mechanism that needs no explicit disentanglement objective.

Method¶

Overall Architecture¶

The paper tackles one-shot concept disentanglement and recomposition: given one reference image and one concept word, learn a representation of that concept, then inject it into a new scene at inference time so that it can be freely combined with other concepts without cross-contamination. The method proceeds in two moves — diagnose first, then operate. The diagnosis stage uses differential probing to measure each concept's contribution to the velocity field over time, yielding a three-stage schedule of which concept peaks in which stage. The operation stage then trains a lightweight Semantic Offset Module that attaches a low-rank learnable offset to the key/value representations of a concept token in the cross-attention layers, back-propagating the flow matching loss only inside that concept's own peak stage. At inference, the learned offsets are handed to ConceptWeaver Guidance, which extends standard CFG by decomposing the holistic prompt guidance into per-concept shift terms, injected in their respective stages and weighted per concept. The whole pipeline depends on no particular modulation pathway, and the backbone stays frozen.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["reference image + concept prompt"] --> B["differential probing & three-stage schedule<br/>measure per-concept velocity shift → locate peak stage"]
    B -->|peak stage per concept| C["semantic offset module & stage-aware optimization<br/>low-rank offset into K/V, loss only in peak stage"]
    C -->|a set of disentangled concept offsets| D["ConceptWeaver Guidance<br/>stage-aware injection + per-concept weights"]
    D --> E["compositional synthesis / motion transfer / image editing"]

Key Designs¶

1. Differential probing and the three-stage schedule: turning "when to modify a concept" from a heuristic into a measurable quantity

The pain point is concrete. The holistic semantic shift computed by CFG is "velocity prediction with the prompt minus velocity prediction with a null prompt"; it superposes the effects of every concept in the prompt into one dense guidance force that exists throughout generation, so it cannot tell when any single concept is active. Yet applying guidance uniformly across the whole trajectory is precisely what produces structural artifacts and concept bleeding.

The paper reuses the contrastive structure of CFG but swaps the contrast target for a partially masked prompt: only the i-th concept token is dropped while the rest is kept, and the velocity difference is that concept's own shift at time t — a leave-one-out "existence shift."

\[\Delta \boldsymbol{v}_{\mathrm{concept}_i}(t) = \boldsymbol{v}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c}) - \boldsymbol{v}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c}_{-i})\]

Plotting this curve along t together with its spatial locality yields the concept's temporal lifecycle. ⚠️ The masking form of this equation is corrupted in the cached text; whether the partially masked prompt removes that token or keeps only that token should be checked against the original paper.

Measurements (probing on Wan2.2, for both image and video generation) show the curve is non-monotonic: weak and diffuse early, peaking simultaneously in intensity and spatial locality at mid-stage, then degrading into sparse high-frequency noise late. More importantly, structural and content concepts have differently shaped curves — structural concepts ("riding a bike", motion, layout) are strongest early-to-mid, while content concepts ("panda", "glasses") have a more concentrated lifecycle. This yields the three-stage framework: the Blueprint Stage (early, where the model lays down a low-frequency structural scaffold and structural concepts are strongest), the Instantiation Stage (mid, where a content concept's semantic identity fully materializes with peak intensity and strong spatial localization, and where concepts are naturally disentangled — the optimal manipulation window), and the Refinement Stage (late, where concept signals fade and the model only synthesizes textures and global coherence). The value of this discovery is that it turns "when to intervene" into a lookup table, and it directly explains why uniform guidance over the whole trajectory drops S5.5GPT to 3.34 in the ablation.

2. Semantic offset module and stage-aware optimization: learn a concept only in its peak stage so disentanglement happens on its own

The "existence shift" in the probe is measured by applying a negative offset to the prompt (removing the concept word). To synthesize, one needs the dual operation — a "customization shift" (generic → specific) that is learned from the reference image and collapses the model's conditional velocity field toward the reference distribution. The difficulty is that if one optimizes uniformly over all timesteps, as ordinary LoRA training does, the offset absorbs structure and detail at once, and the learned representation is neither pure nor disentangled: Figure 7(a) shows that this naive training simply fails to learn a high-fidelity concept.

Concretely, a very lightweight semantic offset module learns an additive offset Δe_i for each selected concept token and injects it into the key/value representations of the cross-attention layers. This choice matters: putting the offset on K/V means the method does not need TokenVerse's text-conditioned modulation pathway, so any flow backbone with cross-attention can host it. The training objective is an ordinary flow matching regression, only with the offset-augmented prompt as the condition:

\[\mathcal{L}_{\mathrm{flow}} = \mathbb{E}\left[\left\| \boldsymbol{v}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c} + \Delta \boldsymbol{c}_{\mathrm{ref}}) - \boldsymbol{v}_t \right\|^2\right]\]

Stage awareness lives in the loss window: content concepts (object identity, e.g. glasses, necklace) compute the loss only in the Instantiation Stage, while structural concepts (motion, layout) compute it only in the Blueprint Stage. The module itself is low-rank (rank 16, alpha 32), and each reference image is trained for 2000 steps.

Crucially, the paper stresses that there is no explicit disentanglement objective here — disentanglement is a by-product of aligning with the model's own attention window. In a concept's peak stage its signal in the velocity field is already the strongest and spatially the most localized, so back-propagating gradients then is equivalent to writing information along that single direction, and the model does not have to separate it out. Conversely, forcing the offset to learn "the style of the glasses" during the Blueprint Stage, or forcing it to learn layout during Refinement, mixes the gradients of other concepts into it. This is also why dropping stage-aware optimization drops S5.5GPT from 4.14 to 3.61 in the ablation.

3. ConceptWeaver Guidance: decomposing holistic prompt guidance into per-concept, stage-aware injection

Standard CFG is "one holistic semantic direction plus one global strength w", which cannot modulate concepts independently when several coexist. To make "the dog wears the glasses from the reference image", one can only brute-force w, and the result is usually one concept overshooting while the other never takes effect.

CWG splits the guidance term: it keeps the holistic prompt term and adds N reference-concept terms, each with its own strength, each active only in its own stage.

\[\boldsymbol{v}_{\mathrm{CWG}} = \boldsymbol{v}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c}_\emptyset) + w\cdot\Delta \boldsymbol{v}_{\mathrm{prompt}} + \sum_{i=1}^{N} w_{\mathrm{c},i}\cdot\Delta \boldsymbol{v}_{\mathrm{ref},i}, \qquad \Delta \boldsymbol{v}_{\mathrm{ref},i} = \boldsymbol{v}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c} + \Delta \boldsymbol{c}_{\mathrm{ref},i}) - \boldsymbol{v}_\theta(\boldsymbol{x}_t, t, \boldsymbol{c})\]

Each injected offset costs one extra "offset-augmented model minus original model" velocity difference, which is then stacked with weight w_{c,i}. These terms are not applied throughout: structural concept shifts are added only in the Blueprint Stage and content concept shifts only in the Instantiation Stage, strictly aligned with the schedule measured by design 1.

This buys two degrees of freedom — how strong each concept is, and how concepts are balanced against each other. Figure 8 shows that raising w_c progressively aligns generated attributes (the style of the glasses, the style of the necklace, the identity of the dog) with the reference, a continuously usable personalization strength knob. Stage-aware injection, meanwhile, avoids pushing a concept during Refinement, where its signal has already decayed and only high-frequency noise remains — pushing there turns directly into structural artifacts and concept bleeding. In the ablation, removing stage-aware guidance hurts more than removing stage-aware optimization (3.34 vs 3.61), confirming that "when to inject" matters as much as "what to learn".

A Worked Example¶

Take the compositional task from §3.3 of the paper: reference image I₁ describes "cat" and "glasses", reference image I₂ describes "dog" and "a beach", and the target is "a dog wearing glasses".

Probing: run one leave-one-out contrast each for "dog", "glasses" and "beach" in the prompt to obtain three Δv curves. Content concepts such as "dog" and "glasses" peak in the Instantiation Stage with strong spatial locality, whereas "beach" — a scene/structural concept — peaks earlier, in the Blueprint Stage.
Training: attach one low-rank offset module to each of the three concepts (rank 16, alpha 32). The flow matching loss for "dog"/"glasses" is computed only inside the Instantiation window and that for "beach" only inside the Blueprint window; each case trains for 2000 steps with the backbone frozen.
Inference: sample from noise under the standard 10/10/10 split. The first 10 steps add only the "beach" structural offset to fix the scene layout; the middle 10 steps add only the "dog" and "glasses" content offsets, where each step's guidance term equals "holistic prompt term + w_c(dog)·Δv_dog + w_c(glasses)·Δv_glasses"; the last 10 steps add no concept offset at all and leave texture refinement to the backbone. If the dog's identity drowns out the glasses, one simply raises w_c(glasses) alone instead of touching the global prompt strength.
Result: the dog appears in a new scene wearing the glasses from the reference image, while the reference's background and composition are not carried over — unlike DreamBooth's copy-paste behaviour.

Loss & Training¶

The only objective is the flow matching regression loss above, conditioned on c + Δc_ref; there is no explicit disentanglement regularizer, no adversarial term, no mask supervision.
Stage awareness lives in the loss window: the loss is computed only inside the target concept's peak stage — content concepts in Instantiation, structural concepts in Blueprint. The standard split is 10/10/10; the ablation shows 8/12/10 and 12/8/10 come close but do not beat it.
Module and hyper-parameters: semantic offset module rank = 16, rank alpha = 32; learning rate 1e-4; 2000 steps per case; training and inference resolution 704×704 for images (704×1280 for inference with many concepts); video training resolution 704×1280, 49 frames (the cache writes "49 frames per second (fps) at 16", presumably 49 frames at 16 fps, ⚠️ refer to the original paper); data augmentation uses random cropping and mirror flipping.
Only the offset module is trained; the backbone is frozen — this is what makes one reference image and 2000 steps sufficient.

Key Experimental Results¶

The evaluation protocol needs stating first, otherwise the numbers are unreadable: textual alignment uses CLIP similarity S_TCLIP between generated images and prompts; concept fidelity uses CLIP / DINO similarity S_ICLIP / S_DINO between generated and reference images (the mean over subject–reference pairs for multi-subject cases); the MLLM score feeds the concept name(s), reference image(s) and generated image(s) to GPT-4o (single-concept, S_4oGPT) or GPT-5.5 (multi-concept, S_5.5GPT) for a 0–4 score, higher being better aligned. On the data side there are 20 single-concept reference images (11 object/content concepts for appearance personalization plus 9 layout/structural concepts for spatial control, plus 10 motion concepts), 15 multi-concept reference samples, and 50 composition prompts generated with GPT-5.5 at 5 images each, giving 250 generated images for quantitative evaluation; the user study has 10 participants on a 5-point Likert scale. The backbone is Wan2.2-TI2V-5B throughout; MS-Diffusion, a reproduced TokenVerse* and DreamBooth are all run on the same backbone in the image setting, with VACE-14B as the video-side comparison.

Main Results¶

Method	Single-concept S_TCLIP↑	S_ICLIP↑	S_DINO↑	S_4oGPT↑	Multi-concept S_TCLIP↑	S_ICLIP↑	S_DINO↑	S_5.5GPT↑
TokenVerse*	35.20	61.75	26.31	3.20	33.06	54.89	26.33	2.47
MS-Diffusion	35.02	75.58	70.69	3.60	33.98	68.90	66.59	3.08
DreamBooth	34.50	73.14	68.96	3.70	33.42	66.80	65.79	2.98
Ours	35.22	76.13	69.57	3.80	35.36	77.06	69.02	4.14

User study (10 participants, 1–5 scale):

Metric	ConceptWeaver	MS Diffusion	Dreambooth
Concept Consistency	4.06	3.75	3.02
Text Adherence	3.78	3.14	2.88
Aesthetic Quality	4.12	3.75	3.05

Ablation Study¶

Config	S_TCLIP↑	S_ICLIP↑	S_DINO↑	S_5.5GPT↑
w/o Stage-aware Opt.	34.87	73.99	67.83	3.61
w/o Stage-aware Guid.	34.41	74.05	66.46	3.34
Stage split 5/10/15	35.03	75.67	68.54	3.79
Stage split 10/5/15	34.74	75.93	68.47	3.67
Stage split 8/12/10	35.11	77.07	68.68	4.01
Stage split 12/8/10	35.09	76.83	68.70	3.93
Ours (10/10/10)	35.36	77.06	69.02	4.14

(In the stage-split rows, T1/T2/T3 denote the timesteps assigned to Blueprint / Instantiation / Refinement; the standard 10/10/10 setting gives each stage 10 steps.)

Key Findings¶

"When to inject" is slightly more critical than "what to learn": removing stage-aware guidance costs 0.80 S_5.5GPT (4.14 → 3.34), more than removing stage-aware optimization (4.14 → 3.61, −0.53). Qualitatively, Figure 7 shows the former introduces structural artifacts and concept bleeding, while the latter merely fails to learn a high-fidelity concept. The two mechanisms are complementary: optimization learns a faithful semantic offset from the reference, guidance preserves text–concept correspondence at inference.
Stage partitions cannot be cut short: 5/10/15 and 10/5/15 both drop clearly (3.79 / 3.67), because ending the Blueprint or Instantiation Stage too early causes concept drift. The 8/12/10 and 12/8/10 splits come very close without beating the standard one, indicating the method is robust around balanced partitions but genuinely benefits from a stage-aware schedule.
The advantage comes mainly from compositional settings: on single-concept generation, S_DINO is actually slightly below MS-Diffusion (69.57 vs 70.69), and the lead is limited to S_ICLIP (76.13 vs 75.58), S_4oGPT (3.80 vs 3.60) and a roughly tied S_TCLIP. In the multi-concept setting, S_5.5GPT opens up to 4.14 vs 3.08 and S_ICLIP to 77.06 vs 68.90. The gain therefore comes not from stronger single-concept fitting but from concepts not contaminating one another.
TokenVerse*'s low scores reflect the reproduction setting: it was ported to Wan2.2-TI2V-5B, which lacks the text-conditioned modulation pathway, yielding S_DINO of only 26.31; this is not directly comparable to the original paper's results in Flux's modulation space. The gap itself supports the paper's argument about architectural generality.
w_c is a continuous knob: a small w_c yields a generic, prompt-driven concept, and increasing it progressively aligns attributes with the reference with no discontinuity, allowing per-concept strength tuning.
Disentanglement lacks a direct metric: concept composition is measured mainly by the multi-concept MLLM score and per-reference similarity, while "do concepts truly leave each other alone" has no dedicated quantitative measure (e.g. whether attributes follow after swapping concepts, or whether editing one concept moves unrelated regions). This currently rests on the qualitative comparisons in Figures 5 and 6.

Highlights & Insights¶

Reframing concept disentanglement as a measurable physical quantity: differential probing reuses CFG's contrastive structure and only swaps the contrast target for a partially masked prompt, yielding concept-level temporal curves at almost no extra cost. The recipe transfers wholesale — any prompt-conditioned generative model (video, 3D, audio) can first build such a schedule and only then decide its intervention windows.
Disentanglement as a by-product of aligning with the model's own temporal structure: with no extra regularizer, adversarial term or mask supervision, back-propagating only inside a concept's peak stage makes the offset encode that concept alone. This is cheaper than "adding a disentanglement loss" and closer to the model's real dynamics.
The training window and the injection window share one schedule: the same probing discovery decides both where the loss is computed and where guidance is added. This avoids the common mismatch between training and inference strategy and is the source of the paper's internal consistency.
Per-concept guidance weights turn CFG's scalar w into a vector: "raise A, lower B" in multi-concept composition becomes a direct control, at trivial cost (one extra forward pass per concept), and can be dropped into any guidance-based editing method.
Putting offsets on cross-attention K/V buys architectural generality: no Flux-style text-conditioned modulation pathway is required, which is both a practical advantage over TokenVerse and the precondition for moving the method onto a video backbone (Wan2.2).
"Analysis-driven control" is a research paradigm worth borrowing: explain the model's internal temporal structure first, then align the control mechanism to it; this is often cheaper and more stable than bolting constraints on from outside.

Limitations & Future Work¶

Concept categorization is manual: the authors admit that a human currently decides whether a concept counts as structural or content (11 content and 9 structural concepts in the paper were categorized by hand), which is both a hyper-parameter and an engineering burden. The stated future direction — learning these temporal profiles automatically — is only an outlook and is not implemented.
No dedicated metric for the degree of disentanglement: S_5.5GPT measures "how good the composition is" and S_ICLIP / S_DINO measure "how much it resembles the reference"; neither directly demonstrates that concepts do not interfere. There is no quantitative test of concept swapping, or of whether a single-concept local edit changes unrelated regions. This is the largest gap in the evaluation.
Incomplete comparison against concept-editing methods: Concept Sliders and Textual Inversion, which this note was asked to check, appear only in related work with no experimental comparison; the actual baselines are DreamBooth (parameter space), TokenVerse (modulation space) and MS-Diffusion (multi-subject). Whether stage awareness works equally well for concept representations in textual space therefore remains open.
Cost scales linearly with the number of concepts: one module trained for 2000 steps per concept, with no reported training time or memory footprint, and no cost comparison against training-free alternatives.
Stage boundaries are a fixed timestep partition: the standard 10/10/10 split was calibrated under a specific sampler and step count; whether it needs recalibration under a different sampler or number of steps is not discussed, and all concepts of a class share one window, ignoring per-concept variation.
Small evaluation scale: the multi-concept quantitative evaluation uses 250 generated images, 15 multi-concept references and 50 composition prompts, with only 10 user-study participants; robustness at larger scale remains to be verified.
Concrete improvement directions: automate both concept categorization and peak-stage localization (infer the stage window directly from the probing curves); introduce causal disentanglement metrics (concept-swap consistency, the spatial footprint of a local edit); and combine the learned per-concept offset with training-free attention manipulation to cut the 2000-step-per-concept cost.

vs TokenVerse: both travel the "learn a concept offset" road, but TokenVerse depends on the per-token text-conditioned modulation space of DiTs (Flux) and puts its control point in the spatial dimension (which parameters to modify), whereas this paper attaches offsets to cross-attention K/V and puts its control point in the temporal dimension (when to inject). The advantages here are architectural generality and alignment with the model's own temporal dynamics; the disadvantage is that each concept still needs 2000 training steps, while multi-concept expansion inside TokenVerse's modulation space is more direct.
vs DreamBooth / LoRA (parameter space): they fine-tune model parameters to internalize a concept and naturally overfit the reference, producing copy-paste; this paper learns one low-rank offset optimized only in the peak stage, achieving multi-concept S_5.5GPT 4.14 vs 2.98, a gap that comes mainly from resistance to copy-paste and from controlling inter-concept interference.
vs Textual Inversion (textual space): it compresses a concept into a learnable pseudo-word, where fidelity and prompt controllability cannot both be maximized; this paper leaves the vocabulary untouched and instead applies additive offsets on K/V, preserving the semantic structure of the original prompt — hence better textual alignment (S_TCLIP 35.36 vs 33.42).
vs MS-Diffusion: a zero-shot multi-subject personalization method that improves subject fidelity via layout guidance; its single-concept S_DINO of 70.69 slightly exceeds this paper's 69.57, showing strong single-concept fitting, but its multi-concept MLLM score is only 3.08, indicating that layout-level constraints help little with keeping concepts from interfering — precisely where temporal control applies leverage.
vs flow-editing methods such as FlowEdit / SplitFlow: they reduce entanglement by decomposing the editing trajectory or prompt; this paper does not decompose trajectories but measures when a reference concept emerges in the velocity field and uses the same schedule to time its injection — a complementary temporal view (the paper also wires its learned offsets into FlowEdit for real-image editing, where the edited image keeps the source structure and only changes appearance).
vs Break-a-Scene / ConceptExpress / BiCo: the mask-based route relies on manual spatial annotation and the unsupervised multi-token route gives users little control over which token governs which semantics; this paper replaces "which region governs which concept" with "which stage governs which concept", moving the control point from space to time.

Rating¶

Novelty: ⭐⭐⭐⭐ The three-stage temporal structure revealed by differential probing is a genuinely new and verifiable observation, and "learn in the peak stage, inject in the peak stage" is a clean idea; the underlying mechanism (offset learning plus stacked guidance) is nevertheless a combinatorial advance over TokenVerse / DreamBooth.
Experimental Thoroughness: ⭐⭐⭐ Coverage is decent (single/multi-concept images, video R2V, a user study, stage-split and weight ablations), but the quantitative scale is small (250 images, 10 participants), and it lacks a direct disentanglement metric, training cost figures, and comparisons against textual-space concept-editing methods.
Writing Quality: ⭐⭐⭐⭐ The probing → discovery → method → validation chain is clearly told, and Figures 2 and 4 visualize the temporal-window concept well; minor flaws are somewhat cramped formula typesetting and the fact that manual concept categorization is only made explicit in the conclusion.
Value: ⭐⭐⭐⭐ The methodology of "probe the model's internal temporal structure first, then decide the intervention window" transfers cheaply to personalization and editing on any flow / diffusion backbone, and is more valuable in the long run than the specific implementation.