How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions¶

Conference: AAAI 2026 arXiv: 2511.07091
Code: None
Area: Image Generation Keywords: Text-to-image generation, bias control, semantic binding, compositional generation, fairness

TL;DR¶

This work is the first to investigate compositional semantic binding bias in text-to-image generation. It proposes the Bias Adherence Score (BA-Score) to quantify how object–attribute binding activates bias, and introduces a training-free Context-Bias Control (CBC) framework that achieves over 10% debiasing improvement in compositional generation via token embedding decoupling and residual injection.

Background & Motivation¶

Text-to-image (T2I) diffusion models (e.g., Stable Diffusion) frequently capture spurious correlations from training data, causing generated outputs to exhibit gender, racial, and other biases. A critical blind spot in existing debiasing research:

Focus on single-object prompts only: e.g., "a headshot of an assistant" — existing debiasing methods are effective for such simple prompts.

Neglect of compositional semantic binding: When the prompt becomes "a headshot of an assistant wearing a pink hat," the context "pink hat" itself carries a female-leaning bias tendency that existing methods cannot handle effectively.

The authors' core findings are particularly compelling:

Analysis of embedding similarities via the CLIP text encoder reveals that "pink" is highly correlated with the "woman" prototype embedding, while "hat" is more correlated with "man."
Compositional effect: When "pink" and "hat" are combined into "pink hat," the bias is amplified, pushing generation toward female characteristics.
Applying state-of-the-art debiasing methods (SelfDisc, DGDebias, FairQueue) to compositional prompts causes severe visual quality degradation: unrealistic styles, missing occupational features, and even safety checker violations.

This reveals a fundamental challenge: bias reduction must not destroy necessary semantic relationships.

Method¶

Overall Architecture¶

The CBC framework consists of three core steps: 1. Token Semantic Bias Decoupling — separating sensitive attribute components 2. BA-Score Computation — quantifying bias tendency for initialization 3. Token Residual Injection — dynamically balancing bias throughout the denoising process

Key Designs¶

1. Token Semantic Bias Decoupling¶

Core Idea: Project context token embeddings onto directions orthogonal to sensitive attributes via Schmidt orthogonalization.

For token embedding $c$ and sensitive attribute embedding $s_k$: $$c^* = c - r_k = c - \frac{\langle c, s_k \rangle}{\langle s_k, s_k \rangle} s_k$$

$c^*$ is the attribute-orthogonal embedding: dependency on the sensitive attribute is removed.
$r_k$ is the attribute-specific residual vector: retains attribute information for subsequent injection.

Unlike prior methods that directly manipulate the latent space or employ loss-based guidance, CBC decouples bias before generation begins, allowing the model to form clean attention relationships from attribute-orthogonal input embeddings.

2. Bias Adherence Score (BA-Score)¶

Quantifies the percentage contribution of context tokens and the subject toward bias.

Given subject embedding $c_m$, context token set $C = \{c_i\}$, and prototype embedding $p_k$: $$B_{m,k} = \frac{\sum_{i=1}^{M} \mathbf{I}_{i \neq m} \exp((\cos(c_m, c_i) + \cos(p_k, c_i))/\tau)}{\sum_{i=1}^M \exp((\cos(c_m, c_i) + \cos(p_k, c_i))/\tau)}$$

The BA-Score is defined as the maximum deviation: $B_m = \max_k |\pi - B_{k,m}|$, where $\pi = 0.5$ represents the balance target.

A high BA-Score indicates severely imbalanced contributions from the subject and context toward a particular attribute group, suggesting the presence of strong spurious correlations and predicting greater difficulty in maintaining quality after debiasing.

3. Token Residual Injection¶

At each step of diffusion denoising: - The current bias tendency is measured using a latent-space BA-Score (based on distances between latent vectors and attribute cluster centers). - When a bias toward attribute group $s_k$ is detected, the average residual embedding of the remaining attributes is injected: $$c_t^* = \delta_r \cdot \bar{r} + (1-\delta_r) \cdot c_{t-1}^*$$

where $\bar{r} = \frac{1}{K-1}\sum_{j \neq k} r_j$ and $\delta_r = 0.2$ controls injection strength.

Attention Rescaling Mechanism: To prevent residual injection from affecting the attention computations of other compositional attributes and objects, the attention vectors of injected tokens are rescaled: $$\mathcal{M}_i^* = w(t) \delta_c \mathcal{M}_i$$

where $w(t) = 1 - t/T$ is a time decay function and $\delta_c = 2$ is the scaling factor.

Loss & Training¶

CBC is a completely training-free framework. It uses pretrained Stable Diffusion v1.5 to generate 512×512 images with guidance scale 7.5 and 50 sampling steps. The BA-Score initializes bias measurement, and subsequent steps dynamically adjust via latent-space cluster centers.

Key Experimental Results¶

Main Results¶

Baseline prompts (non-compositional, e.g., "a headshot of a [profession]")

Method	FD↓	VQA↑	AFS↑	Note
SD-1.5	0.69	0.74	0.44	Original model exhibits clear bias
SelfDisc	0.62	0.61	0.47	Limited effectiveness of concept editing
DGDebias	0.68	0.64	0.43	Limited effectiveness of distribution guidance
FairQueue	0.03	0.33	0.49	Low bias but severe quality degradation
CBC (Ours)	0.04	0.68	0.80	Low bias with high quality

Compositional prompts (e.g., "a headshot of a [profession] wearing a [color] [object]")

Method	FD↓	VQA↑	AFS↑	Note
SD-1.5	0.69	0.63	0.41	Composition exacerbates bias
SelfDisc	0.42	0.56	0.58	Fails in compositional settings
DGDebias	0.73	0.62	0.35	Severe overcorrection
FairQueue	0.04	0.45	0.60	Low bias but very low VQA
CBC (Ours)	0.04	0.62	0.75	Best overall performance

Ablation Study¶

Configuration	FD↓	VQA↑	AFS↑	Note
Full CBC ($\delta_c$=2, $\delta_r$=0.2)	0.04	0.62	0.75	Optimal configuration
w/o BA-Score initialization	0.16	0.58	0.68	AFS drops 7%; BA-Score initialization is critical
Semantic similarity initialization	0.19	0.53	0.64	AFS drops 11%; simple similarity is insufficient
$\delta_c=1$	0.08	0.61	0.73	Insufficient attention rescaling
$\delta_c=5$	0.11	0.59	0.71	Excessive rescaling
$\delta_r=0.5$	0.12	0.58	0.70	Excessive residual injection

Key Findings¶

Compositional bias amplification: Adding "wearing a scarf" can cause the bias score for "assistant" to surge from 0.6 to above 0.9.
Color-driven bias: Changes in hat color alone can shift the bias direction — a green hat significantly increases bias across multiple white-collar professions.
Side effects of decoupling: Simultaneously decoupling too many tokens (e.g., five) causes semantic loss — the model may cease to generate humans altogether.
Key finding: Preserving the subject (assistant) while decoupling only the associated attributes and objects yields the best debiasing results.
Cascading spurious correlations: Other spurious correlations may emerge after successful debiasing (e.g., "pink hat" consistently co-occurring with outdoor green scenery).

Highlights & Insights¶

Highly innovative problem formulation: This is the first systematic study of bias propagation through compositional semantic binding, which represents a fundamental blind spot for existing debiasing methods.
BA-Score design cleverly quantifies the contribution of context to bias while jointly considering each token's correlation with both the subject and bias prototypes.
Experimental analysis of token decoupling reveals profound semantic entanglement: due to token information leakage in cross-attention, decoupling a single word in isolation may be ineffective, necessitating consideration of the joint effect of neighboring tokens.
The observation that "debiasing may compromise reliability" is particularly insightful: when a model is forced to generate combinations that are extremely rare in its training distribution, quality degradation is inevitable.

Limitations & Future Work¶

Currently limited to gender bias; other sensitive attributes such as race and age are not addressed.
Validated only on SD-1.5; whether larger models (SDXL, SD3, Flux) exhibit the same issues remains unknown.
BA-Score relies on the embedding quality of the CLIP text encoder and may inherit its inherent biases.
Lacks a large-scale benchmark: current experiments are based on a limited set of occupations and attribute combinations.
The injection strategy is relatively simple (linear mixing) and may lack sufficient granularity.

Serves as an important complement to debiasing methods such as SelfDisc and FairQueue, identifying their fundamental shortcomings in compositional settings.
BA-Score design is inspired by the softmax attention mechanism and contrastive learning.
Token decoupling is conceptually related to FreeCustom's multi-reference self-attention and token merging approaches.
The "bias cascading" phenomenon (removing one bias may expose another) has broad implications for the AI fairness community.
Future direction: leverage associations among non-sensitive attributes to improve realism, rather than blindly eliminating all correlations.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ — Highly original problem formulation; BA-Score and compositional bias analysis fill an important gap.
Experimental Thoroughness: ⭐⭐⭐⭐ — In-depth analysis of compositional settings, though dataset and base model coverage is limited.
Writing Quality: ⭐⭐⭐⭐ — Problem articulation is clear; figure design aids comprehension.
Value: ⭐⭐⭐⭐⭐ — Reveals fundamental challenges in T2I debiasing with important implications for the field.

Configuration	FD↓	VQA↑	AFS↑	Note
Full CBC (\(\delta_c\)=2, \(\delta_r\)=0.2)	0.04	0.62	0.75	Optimal configuration
w/o BA-Score initialization	0.16	0.58	0.68	AFS drops 7%; BA-Score initialization is critical
Semantic similarity initialization	0.19	0.53	0.64	AFS drops 11%; simple similarity is insufficient
\(\delta_c=1\)	0.08	0.61	0.73	Insufficient attention rescaling
\(\delta_c=5\)	0.11	0.59	0.71	Excessive rescaling
\(\delta_r=0.5\)	0.12	0.58	0.70	Excessive residual injection