ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation¶

Conference: ECCV 2026
arXiv: 2505.20935
Code: https://shjo-april.github.io/ISAC
Area: Image Generation
Keywords: Multi-Instance Generation, Attention Control, Self-Attention, Diffusion Models, Training-Free Guidance

TL;DR¶

ISAC proposes a hierarchical, instance-first, training-free attention control method. It first leverages self-attention to stabilize instance layouts in early denoising steps, and then binds additional semantics within the established instance regions, significantly improving count accuracy and semantic separation in multi-instance generation with diffusion models.

Background & Motivation¶

Text-to-image diffusion models achieve remarkable results on single objects or simple composing prompts, but they frequently fail in multi-instance scenarios: when "two horses" are requested, they often generate only one; when "a dog and a sheep" are requested, the identities of the two animals are merged (attribute bleeding) or cross-instance attribute leakage occurs. These failures are not occasional long-tail cases; when prompts contain multiple instances of the same category (e.g., cats and dogs under the same supercategory, or different numbers of horses), existing models struggle to achieve accurate counts and semantic separation simultaneously. Existing training-free guidance methods (Attend-and-Excite, SynGen, InitNO, Self-Cross, etc.) mainly improve generation by correcting semantic signals in cross-attention (e.g., boosting attention peaks of overlooked tokens or using contrastive objectives to separate semantic spaces of different tokens). These methods assume that instance regions are already formed and work only at the semantic level. However, they cannot fundamentally guarantee that each requested instance is partitioned into an independent image region. When multiple instances of the same category share a semantic token, this "semantic-driven structure" paradigm fails because the semantic token itself cannot distinguish between multiple instances of the same category.

The key challenge lies in the out-of-sync development speed of "instance structure" and "semantic information" during the denoising process of diffusion models. Self-attention can already expose category-agnostic instance layouts at early denoising stages (at timesteps close to \(T\))—determining which pixels belong to the same object and which to different objects. This structural signal is relatively clear in approximately the first 20-30% of the denoising steps. In contrast, the semantic signals in cross-attention at this point are still in a coarse, entangled state, far from being able to provide reliable instance boundaries. Current methods all use semantic signals from cross-attention to drive instance structures, which is equivalent to "using unreliable semantics to guess unformed structures," naturally leading to failure in multi-instance scenarios of the same category. To systematically verify this diagnosis, the authors define a Dice overlap metric to quantify the degree of semantic signal confusion between different category pairs, finding that semantic overlap within the same supercategory is significantly higher than that across different category pairs—the root cause of the failure of semantic-driven methods.

This paper starts from this diagnosis and proposes a hierarchical decoupling scheme: splitting the generation process into "instance structure formation" and "semantic assignment" stages, leveraging the temporal advantages of the two attention mechanisms within the diffusion model. Core Idea: Decouple multi-instance generation into two stages: "instance layout formation" and "semantic assignment". First, leverage self-attention in early denoising steps to extract \(N\) category-agnostic instance masks and suppress overlap (Phase 1). Next, inject these stable structural signals into cross-attention to bind additional semantics within the partitioned instance regions (Phase 2), achieving a smooth transition between the two stages via a simple timestep-adaptive weight.

Method¶

Overall Architecture¶

ISAC is a model-agnostic, training-free inference-time guidance objective. At each denoising step \(t\), it reads the internal self-attention map \(SA_t\) and cross-attention map \(CA_t\) through a hook mechanism, computes the guidance loss \(\mathcal{L}_{\text{ISAC}}\) in two stages, and finally optimizes the generation by updating the latent variables via gradients (ISAC_LO variant) or choosing the best candidate as a scoring function (ISAC_LS variant). Phase 1 builds a foreground gate based on self-attention and clusters \(N\) instance masks, using a maximum pixel-wise overlap repulsion loss to force these \(N\) regions to separate. Phase 2 propagates structural cues from self-attention into cross-attention to obtain instance-aware semantic masks, and applies cross-instance repulsion-binding losses to token pairs. The two phases are smoothly connected by a timestep-adaptive weight \(\lambda(t)\)—focusing on forming instance structures early on and transitioning to semantic refinement later.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt +<br/>Random Latent X_T"] --> B["Denoising Step t<br/>(t=T→1)"]
    B --> C["Hook Attention Layers<br/>Read SA_t, CA_t"]
    C --> D["Foreground Gating:<br/>CA Binarization + Combine Class Tokens<br/>→ Foreground Region M_fg"]
    D --> E["SA Clustering:<br/>K-means inside Foreground<br/>→ N Instance Masks"]
    E --> F["ℒ_ins: MPO Repulsion<br/>Penalize Max Instance Overlap"]
    F --> G["SA_t Structure Propagation to CA<br/>→ Instance-Aware Semantic Mask<br/>CA_t^{ins}=SA_t·CA_t"]
    G --> H["ℒ_sem: Cross-Instance Repulsion<br/>+ Intra-Instance Binding"]
    H --> I["ℒ_ISAC = λ_ins(t)·ℒ_ins<br/>+ λ_sem(t)·ℒ_sem<br/>λ_ins=t/T, λ_sem=1-t/T"]
    I -->|ISAC_LO: Gradient Update X_t| B
    I -->|ISAC_LS: Cumulative Score| J["Select Lowest-Score Candidate"]
    J --> K["Generated Image"]

Key Designs¶

1. Self-Attention Clustering for Instance Layout: Finding \(N\) Regions within the Foreground

Core premise: Attention weights between pixels belonging to the same instance in the self-attention map \(SA_t\) are significantly higher than those belonging to different instances. Therefore, instance boundaries can be discovered by clustering the row vectors of \(SA\). However, clustering directly on the entire image space is prone to background interference, so ISAC first constructs a foreground gate. It computes the instance-aware semantic mask \(CA_t^{ins} = SA_t \cdot CA_t\), and binarizes it before taking the union of all class tokens as the foreground \(M_{\text{fg}}\). The intuition of this product is: \(SA\) provides "pixel-to-pixel" instance structural relations, and \(CA_t\) provides "pixel-to-token" semantic responses; multiplying them yields the "semantic response constrained by instance structure"—the foreground region is the structurally continuous region with semantic responses. Within the foreground pixel set \(\mathcal{I}\), K-means clustering (\(K=N\), total instances) is applied to \(SA\) row vectors augmented with normalized spatial coordinates, yielding a one-hot assignment \(K \in \{0, 1\}^{F \times N}\), which is then weighted with \(SA\) row vectors to obtain soft instance masks \(M = SA_t[\mathcal{I}, \mathcal{I}] \cdot \text{stopgrad}(K)\). The stopgrad ensures that gradients only update \(SA_t\) rather than the cluster assignments, allowing gradient optimization to reinforce inner-instance attention and suppress cross-instance attention, making the boundaries increasingly clear over iterations. This process requires no category labels, discovering "how many things there should be" and "where their boundaries lie" purely from structural correlations between pixels.

2. Maximum Pixel-wise Overlap Repulsion Loss: Pushing Instances Apart using the Worst Case

Once \(N\) soft masks are obtained, mutual overlap must be avoided. ISAC designs the Maximum Pixel-wise Overlap (MPO) metric—taking the maximum value of the pixel-wise product of two masks. The elegance of MPO lies in its focus on the most severe conflicting pixel: as long as a single pixel is contested by two instances, MPO outputs a high value, forcing the optimization to resolve this conflict, rather than tolerating low overlap on average while suffering local splitting. The instance separation loss is defined as the maximum MPO among all mask pairs:

\[\mathcal{L}_{\text{ins}}(X_t) = \max_{1 \leq i < j \leq N} \text{MPO}\big(M[i], M[j]\big)\]

This loss ensures that instance masks are not only separated on average, but also have clear ownership at each pixel level, preventing "merging" failures at the source.

3. Instance-Aware Semantic Separation: Binding Semantics within Layouts

After Phase 1, \(SA_t\) has encoded stable instance boundaries. ISAC utilizes this structural signal for two purposes. First, it propagates instance boundaries to the semantic space: \(CA_t^{ins} = SA_t \cdot CA_t\)—this multiplication is equivalent to filtering the semantic response of cross-attention using the instance structural information of self-attention, making each column of \(CA_t^{ins}\) highlight "the region corresponding to that token and within the instance boundary". Next, a repulsion loss \(\mathcal{L}_{\text{repel}}\) (maximizing their MPO—making semantic masks non-overlapping) is applied to token pairs belonging to different instances, and a binding loss \(\mathcal{L}_{\text{bind}}\) (minimizing \(1-\text{MPO}\)—activating them in the same region) is applied to token pairs belonging to the same instance (e.g., a class token and its attribute token). The combined loss is \(\mathcal{L}_{\text{sem}} = \mathcal{L}_{\text{repel}} + \mathcal{L}_{\text{bind}}\). Although prior works like SynGen and CONFORM also perform semantic repulsion/binding, their repulsion is free-form in the semantic space itself, without the concept of "instance regions." ISAC restricts repulsion and binding to the partitioned instance regions from Phase 1, preventing semantics from spilling over to neighboring instances, thus working reliably in intra-supercategory multi-instance scenarios.

4. Progressive Instance-to-Semantic Schedule

The two phases are linked by elegant and simple scheduling parameters: \(\lambda_{\text{ins}}(t) = t/T\), \(\lambda_{\text{sem}}(t) = 1 - t/T\), and \(\mathcal{L}_{\text{ISAC}} = \lambda_{\text{ins}} \cdot \mathcal{L}_{\text{ins}} + \lambda_{\text{sem}} \cdot \mathcal{L}_{\text{sem}}\). In the early denoising steps (\(t \approx T\)), \(\lambda_{\text{ins}} \approx 1\), and the model focuses almost entirely on instance layout formation—semantic signals are too weak and unreliable to be used. As denoising progresses toward \(t \approx 0\), \(\lambda_{\text{sem}} \approx 1\), the instance layout is stabilized, and the model transitions to semantic refinement. Ablation studies show that using only the instance loss (65% multi-instance accuracy but only 10% multi-class accuracy), only the semantic loss (54% and 28%), fixed weights (60% and 25%), or reversed scheduling (55% and 21%) are all inferior to the progressive schedule (69% and 36%), demonstrating that the temporal order of "structure first, semantics second" is indispensable.

A Complete Example: Two Horses¶

Taking the prompt "two horses" as an example. The LLM parser extracts the class token {horse}, instance count \(n_1=2\), and total instances \(N=2\). In the early denoising steps (\(t \approx T\)), \(SA_t\) implicitly encodes two object regions but does not yet know they are horses. ISAC localizes the foreground region in the image through foreground gating, performs K-means clustering with \(K=2\) within it to find two sets of pixels roughly corresponding to the two horses, and calculates MPO repulsion to prevent them from overlapping. In the middle denoising steps, the two instance boundaries are stabilized by gradient updates, and the clustering structure of \(SA_t\) becomes clearer. In the later denoising steps (\(t \to 0\)), \(CA_t^{ins} = SA_t \cdot CA_t\) restricts the semantic response of the "horse" token to the two instance regions, and \(\mathcal{L}_{\text{bind}}\) ensures that the attributes of each horse (e.g., "brown" vs. "white") are bound to their respective regions without bleeding into the other. In the final output, the two horses form independently with correct attributes. Quantitative results show that on SD1.5, ISAC_LO improves the instance accuracy of the 2-horse category from the baseline's 55% to 100%, and 3-horse from 2% to 31%, indicating highly significant effectiveness in intra-supercategory multi-instance tasks.

Loss & Training¶

ISAC is a training-free, inference-time guidance method that requires no training process. The total guidance loss is \(\mathcal{L}_{\text{ISAC}}(X_t, t) = \lambda_{\text{ins}}(t) \cdot \mathcal{L}_{\text{ins}}(X_t) + \lambda_{\text{sem}}(t) \cdot \mathcal{L}_{\text{sem}}(X_t)\). Both variants share the same \(\mathcal{L}_{\text{ISAC}}\) formulation and scheduling weights: ISAC_LO (Latent Optimization) performs standard denoising prediction at each denoising step, computes the gradient of \(\mathcal{L}_{\text{ISAC}}\) with respect to \(X_t\), updates the latent variable with a globally uniform learning rate \(\eta=0.01\), and then proceeds to the next denoising step. ISAC_LS (Latent Selection) requires no backpropagation; it generates \(B=10\) candidates in parallel, accumulates the \(\mathcal{L}_{\text{ISAC}}\) scores at each step for each candidate, and decodes the candidate with the lowest total score to the final image.

Key Experimental Results¶

Main Results¶

Benchmark	Metric	SD1.5 Baseline	+ISAC_LO	SD3.5-M Baseline	+ISAC_LO
HRS-Bench Color	Score	0.136	0.318	0.425	0.473
HRS-Bench Spatial	Score	0.094	0.263	0.264	0.350
HRS-Bench Size	Score	0.091	0.252	0.209	0.258
T2I-CompBench Color	Score	0.356	0.683	0.796	0.838
T2I-CompBench Texture	Score	0.406	0.631	0.726	0.739
IntraCompBench #2	Multi-Class Accuracy	28%	65%	62%	98%
IntraCompBench #3	Multi-Class Accuracy	2%	31%	23%	51%
IntraCompBench #4	Multi-Class Accuracy	1%	29%	12%	40%
IntraCompBench #5	Multi-Class Accuracy	0%	18%	3%	20%
IntraCompBench Average	Multi-Class Accuracy	8%	36%	25%	52%

Ablation Study¶

Configuration	Multi-Class Accuracy	Multi-Instance Accuracy	Description
Instance loss only (A)	10%	65%	Can form correct number of instances but fails to assign semantics
Semantic loss only (B)	28%	54%	Semantic separation without layout constraint, unstable
Fixed weights (C)	25%	60%	Temporal mismatch, loses the optimal window
Semantic-to-Instance schedule (D)	21%	55%	Reverse schedule: early semantics are invalid, late structures cannot be repaired
Instance-to-Semantic schedule (E)	36%	69%	Structure first, semantics second; balances both

Key Findings¶

ISAC achieved the greatest improvement in intra-supercategory multi-instance scenarios: this is the common weak spot of all prior methods. ISAC fundamentally solves the problem of distinguishing multiple instances under the same token through self-attention structural clustering.
ISAC_LO improves the HRS-Bench Spatial score on SD1.5 by approximately 2.8 times, and increases the average class accuracy of IntraCompBench on SD3.5-M from 25% to 52%, demonstrating consistent improvements across different backbone models.
ISAC can enhance layout-to-image controllers (GLIGEN + ISAC improves HRS-Bench from 0.666 to 0.713, and counting F1 from 0.307 to 0.452), showing that it can automatically refine coarse, overlapping bounding boxes into dense instance masks.
ISAC remains effective on few-step models (Z-Image-Turbo with 8 steps, Flux.2-klein-4B with 4 steps) (Z-Image-Turbo multi-class accuracy 48% -> 68%), validating its applicability in low-latency scenarios.

Highlights & Insights¶

Instance-First Hierarchical Design: Discovers and exploits the temporal asymmetry between the "structural advantage" of self-attention in early denoising steps and the "semantic advantage" of cross-attention in later steps—an elegant design choice that aligns naturally with the internal dynamics of diffusion models, rather than forcing external constraints.
MPO Metric Driving Optimization via the Worst Case: Replaces average overlap with the maximum value of pixel-wise products, enabling the loss function to directly target "the most severe merging conflicts" rather than overall statistics, leading to more efficient convergence.
Comprehensive Variant Coverage: Provides both Latent Optimization (ISAC_LO, high quality) and Latent Selection (ISAC_LS, gradient-free, highly scalable) variants, leaving ample design space for quality-computation trade-offs and backbone compatibility.
Model-Agnostic and Zero-Training: Requires no fine-tuning, no external detection models, and no changes to any backbone parameters; it is plug-and-play and directly transferable across models of different scales (SD1.5 to Flux.2 to Qwen-Image) and tasks (T2I and L2I/Layout-to-Image).

Limitations & Future Work¶

Dependence on instance count parsing: It requires automatically parsing the count of each category from the prompt, which is out of scope for ambiguous expressions (e.g., "some dogs"). In practice, an LLM-based parser is needed.
Computational overhead: The gradient computation in ISAC_LO increases inference latency and VRAM (from 8s / 4.9GB to 21s / 9.7GB on SD1.5), comparable to similar attention guidance methods, but optimization may still be needed for large-scale deployment.
Upper bound of multi-instance generation: As shown in IntraCompBench, when the number of instances increases to #5, the accuracy drops significantly (from #2 = 98% to #5 = 20% on SD3.5-M). Extremely crowded scenes remain an open challenge.
The paper does not explore extending instance separation to video generation or multi-view generation, though the authors point to this direction in the conclusion.

vs Attend-and-Excite (A&E): A&E only boosts the attention peaks of neglected tokens and does not resolve the separation of multiple instances of the same category; ISAC separates instances structurally.
vs InitNO / Self-Cross: Although these methods also utilize self-attention, they use cross-attention semantics to select and aggregate self-attention maps, meaning the structure remains semantically driven; ISAC starts purely from self-attention structures.
vs TEBOpt / DOS: Text embedding optimization methods lack spatial awareness and rely heavily on the CLIP architecture, making them incompatible with modern LLM-based T2I models (like Flux or Qwen-Image); ISAC operates solely on attention maps and is model-agnostic.
vs CountGen / Counting Guidance: Counting guidance methods require additional training or external vision models, and only take effect in later denoising stages when semantics are clear enough; ISAC is training-free and works starting from the early layout formation stage.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Decoupling instance formation and semantic assignment with an "instance-first" hierarchy is a clean and powerful insight.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 3 benchmarks × multiple backbone networks × 2 tasks × comparison with 7 methods × comprehensive ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, well-explained two-stage method, and extensive ablation studies supporting design choices.
Value: ⭐⭐⭐⭐⭐ Training-free, plug-and-play, and significantly improves multi-instance generation, which is a highly practical and high-frequency real-world demand.