FairSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/Atmyre/EquiSteer
Area: Image Generation
Keywords: Text-to-Image Generation, Diffusion Models, Fairness Debiasing, Cross-Attention Steering, Training-Free Inference Intervention

TL;DR

EquiSteer (also referred to as FairSteer) is a training-free, per-sample inference-time debiasing framework that employs early-step cross-attention dot-product gating to selectively skip attribute-specific prompts while orthogonalizing and adaptively injecting target attribute steering vectors for neutral prompts, reducing demographic disparity by up to 87% across SD-1.5, SD-2.1, SDXL, and SANA without sacrificing visual quality or alignment.

Background & Motivation

Text-to-image (T2I) diffusion models have emerged as foundational tools in digital art and creative workflows. However, because these systems are trained on web-scale datasets saturated with societal biases, they reliably reproduce and amplify acute demographic skews. For instance, querying state-of-the-art models with neutral occupational prompts such as "a photo of a nurse" produces almost exclusively female subjects, whereas "a photo of a CEO" skews overwhelmingly male. Rather than isolated stochastic anomalies, these represent deep-seated distributional artifacts that risk propagating stereotypes and raising serious ethical challenges regarding algorithmic fairness and social inclusion.

Existing mitigation paradigms face significant operational limitations. Finetuning-based approaches necessitate additional curated balanced datasets and costly gradient updates, rendering them impractical for end-users and non-transferable across different model architectures. Distributional guidance approaches are training-free but operate exclusively at the batch level, relying on external attribute classifiers and suffering from adverse quality-fairness trade-offs. Prompt- and text-embedding intervention techniques operate purely in the text conditioning space; while lightweight, they are notoriously sensitive to phrasing variations and lack spatial control during diffusion denoising. Fundamentally, a practical debiasing method must be selective: it should balance marginal attribute distributions on attribute-neutral prompts (e.g., "a doctor") while strictly respecting user intent on attribute-specific prompts (e.g., "a male doctor").

This work looks directly into the inner mechanics of diffusion models—specifically cross-attention (CA) layers where prompt tokens modulate spatial visual features. Prior attribution work confirms that demographic identities are actively encoded in CA activations. Core idea: without retraining or modifying weights, exploit early-step cross-attention dot products as a lightweight prompt-aware gate to bypass explicit prompts, while using attribute subspace orthogonalization and calibrated steering injection on neutral prompts to achieve robust, sample-level fair image synthesis.

Method

Overall Architecture

EquiSteer formulates training-free, per-sample debiasing for a concept \(X\) with discrete attribute set \(\{a_i\}_{i=1}^n\) (e.g., gender with attributes male and female), targeting a uniform marginal distribution \(p(a_i) = 1/n\) across neutral generations while leaving attribute-specific queries unchanged.

The framework is organized into an offline calibration phase and an online inference pipeline. Offline, steering vectors \(s_{lt}^{a_i}\) are precomputed across all cross-attention layers \(l \in [1, L]\) and timesteps \(t \in [1, T]\) using pairs of contrastive prompts, alongside gating thresholds and empirical target response magnitudes. Online, for any input prompt, the model samples an initial step \(t=0\) at a designated gating layer \(l_{\text{gate}}\) to evaluate the maximal dot-product score against attribute directions. If an explicit attribute signal is detected, the pipeline proceeds with standard diffusion generation. Otherwise, a target attribute is uniformly sampled, and during each denoising step, cross-attention activations are projected onto the orthogonal complement of the attribute subspace to eliminate pre-existing bias before injecting the calibrated steering vector with norm preservation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Prompt + Initial Latent x_T"] --> B["Prompt-Aware Gate Detection<br/>t=0, evaluate max dot-product dp at l_gate"]
    B -->|Explicit Attribute: dp > thr| C["Vanilla Generation<br/>Skip intervention, preserve user intent"]
    B -->|Neutral Prompt: dp ≤ thr| D["Uniform Target Attribute Sampling<br/>a ~ Uniform({a_i})"]
    D --> E["Subspace Orthogonalization<br/>Project out pre-existing bias span({s})"]
    E --> F["Adaptive Steering Vector Injection<br/>Inject calibrated s_a and re-normalize"]
    F --> G["Output Debiased Image<br/>Balanced marginal fairness + high fidelity"]
    C --> G

Key Designs

1. Prompt-Aware Gate Detection Unconditionally intervening on all incoming prompts catastrophically corrupts generations where the user explicitly stipulated an attribute. The authors observe that early-step cross-attention outputs inherently encode whether an attribute is explicitly specified in the text prompt. For token position \(k\) at layer \(l\) and timestep \(t\), the dot product \(\langle ca_{ltk}^{\text{out}}, s_{lt}^{X} \rangle\) quantifies local attribute expression against steering direction \(s_{lt}^{X}\). To capture localized semantic cues, the maximal token response is formulated as: $\(dp_{lt} = \max_{0 \le k \le K_l} \langle ca_{ltk}^{\text{out}}, s_{lt}^{X} \rangle\)$ At timestep \(t=0\) across intermediate layers (e.g., layer 4 in SD-1.5, layer 17 in SDXL, layer 5 in SANA), attribute-specific prompts exhibit substantially higher \(dp\) values than neutral ones. EquiSteer precomputes an attribute-specific threshold as the empirical midpoint: $\(thr^{a} = \frac{dp_{\text{specific}}^{a} + dp_{\text{neutral}}^{a}}{2}\)$ During inference, if \(dp_{l_{\text{gate}}0}^{a} > thr^{a}\) for any attribute \(a \in \{a_i\}\), the prompt is classified as attribute-specific and EquiSteer abstains from steering. Because this check is evaluated only once at \((l_{\text{gate}}, t=0)\), its computational overhead is negligible.

2. Subspace Orthogonalization Simply adding a steering vector to neutral prompt activations frequently yields blended or ambiguous features (e.g., mixed gender characteristics) because the baseline activations already harbor latent bias signals inherited from biased training priors. To eliminate pre-existing bias, EquiSteer purges all components lying within the subspace spanned by all attribute directions \(\text{span}(\{s_{lt}^{a_i}\}_{i=1}^n)\) prior to injection. Using Gram-Schmidt orthonormalization on the precomputed steering vectors, an orthonormal basis \(\{u_{lt}^j\}_{j=1}^m\) is formed and assembled into matrix \(U_{lt} \in \mathbb{R}^{d \times m}\). The projection removal is defined as: $\(ca_{lt}^{\text{out\_tmp}} = (I - U_{lt}U_{lt}^\top) ca_{lt}^{\text{out}}\)$ Since the projection matrix \((I - U_{lt}U_{lt}^\top)\) is static and precalculated offline, this requires only a single matrix multiplication per cross-attention layer during sampling, providing a clean slate free of confounding attribute components.

3. Adaptive Steering Vector Injection Once confounding attribute signals are removed, the target attribute direction \(s_{lt}^{a}\) must be injected at an appropriate strength. A constant steering coefficient \(\alpha\) either under-steers strongly biased professions or distorts background context. EquiSteer calibrates the steering strength to match the natural activation level observed on attribute-specific prompts at layer \(l\) and timestep \(t\): $\(\alpha = dp_{\text{mean}}^{a}(l, t) = \mathbb{E}_{\text{attr-spec prompts}}[dp_{lt}^{a}]\)$ The updated activation is then computed as: $\(ca_{lt}^{\text{out\_new}} = ca_{lt}^{\text{out\_tmp}} + \alpha s_{lt}^{a}\)$ To prevent numerical drift and maintain distribution scale across denoising steps, the updated activation is re-normalized to match the original \(L_2\) norm: $\(ca_{lt}^{\text{out\_new\_renorm}} = \|ca_{lt}^{\text{out}}\|_2 \cdot \frac{ca_{lt}^{\text{out\_new}}}{\|ca_{lt}^{\text{out\_new}}\|_2}\)$ This guarantees that the target demographic attribute is rendered crisply while maintaining structural visual integrity and prompt fidelity.

Loss & Training

EquiSteer is entirely training-free and operates purely at inference time; all diffusion backbone parameters remain completely frozen without gradient updates or fine-tuning loss functions. Offline setup hyperparameters include: - Steering Vector Construction: Derived following CASteer by feeding paired contrastive prompts (positive vs. neutral prompts) through the frozen diffusion model, capturing average activation differences across layers and timesteps. - Gating Layer Selection: Determined empirically on a small validation set by sweeping for the earliest intermediate layer exhibiting distinct AUROC separation (\(\ge 0.988\)) between specific and neutral prompts (\(l_{\text{gate}}=4\) for SD-1.5/2.1, \(l_{\text{gate}}=17\) for SDXL, \(l_{\text{gate}}=5\) for SANA).

Key Experimental Results

Main Results

Evaluation follows the TEI benchmark across 8 strongly biased occupational prompts ("CEO", "doctor", "pilot", "technician", "fashion designer", "librarian", "teacher", "nurse"), generating 1,000 images per neutral profession and 300 per attribute-specific prompt. Fairness is measured via the average absolute deviation from uniform target parity: \(\text{Avg. } \Delta = |p - 1/n|\) (where \(1/n = 0.5\) for binary gender; lower is better).

Table 1 reports the ratio of minor demographic attributes across 1,000 images on SD-1.5 for neutral prompts:

Minor Attribute	Profession	Vanilla SD	FairDiff	UCE	FTDiff	SelfDisc	TEI	EquiSteer (Ours)
Female	CEO	0.030	0.452	0.027	0.190	0.445	0.389	0.483
Female	Doctor	0.081	0.502	0.049	0.198	0.502	0.334	0.500
Female	Pilot	0.150	0.739	0.244	0.260	0.568	0.408	0.416
Female	Technician	0.007	0.553	0.005	0.168	0.347	0.164	0.375
Male	Fashion designer	0.078	0.333	0.018	0.167	0.067	0.451	0.504
Male	Librarian	0.194	0.300	0.297	0.538	0.174	0.421	0.473
Male	Teacher	0.222	0.205	0.155	0.231	0.081	0.492	0.421
Male	Nurse	0.007	0.162	0.003	0.208	0.004	0.039	0.432
Aggregate	Avg. \(\Delta\) (↓)	0.403	0.167	0.400	0.264	0.244	0.167	0.051

Cross-model generalization and image fidelity are evaluated across multiple diffusion backbones. Table 2 summarizes parity gap \(\Delta\) on SDXL and SANA, while Table 3 presents image-text alignment (CLIP Score) and fidelity (CMMD) over 30,000 MS-COCO validation prompts:

Backbone	Method / Setting	Gender Avg. \(\Delta\) (↓)	CLIP Score (↑)	CMMD (↓)
SD 1.5	Vanilla SD	0.403	26.42	0.532
SD 1.5	TEI Baseline	0.167	26.56	0.509
SD 1.5	EquiSteer (Ours)	0.051	26.63	0.519
SDXL	Vanilla SDXL	0.381	26.51	0.794
SDXL	TEI Baseline	0.242	26.58	0.757
SDXL	EquiSteer (Ours)	0.075	26.75	0.796
SANA	Vanilla SANA	0.473	26.93	0.890
SANA	EquiSteer (Ours)	0.097	26.77	0.905

For multi-attribute debiasing across race (5 classes), age (3 classes), body type (3 classes), and eyeglasses, EquiSteer consistently suppresses disparity. On SDXL, race \(\Delta\) decreases from 0.172 to 0.042 (76% reduction) and on SANA from 0.178 to 0.039 (78% reduction). In Joint-4 debiasing (gender + race + age + body type simultaneously), SDXL reduces all four disparity metrics in a single pass: gender \(\Delta\) from 0.381 to 0.012, race from 0.192 to 0.047, age from 0.276 to 0.158, and body type from 0.317 to 0.162.

Ablation Study

A component ablation on SD-1.5 gender debiasing evaluates the impact of adaptive steering injection (add), subspace orthogonalization (erase), and the prompt-aware gate (gate):

Configuration	Neutral Prompt Avg. \(\Delta\) (↓)	Female-Specific \(\Delta\) (↑)	Male-Specific \(\Delta\) (↑)	Note
add (steering only)	0.165	0.481	0.379	Moderate parity gain; lacks complete bias removal
add & erase	0.095	0.394	0.230	Tighter parity, but severely degrades attribute preservation
add & erase & gate (full model)	0.051	0.492	0.495	Restores perfect attribute preservation and achieves lowest parity gap

Key Findings

• Necessity of Tripartite Coordination: Ablating any single component causes failure: steering injection alone cannot overcome strong prior skews (\(\Delta=0.165\)); orthogonalization alone erases intentional user prompts, dropping male-specific preservation to 0.230; the prompt gate successfully isolates explicit prompts, restoring fidelity to 0.495 while driving neutral parity gap to 0.051.
• Efficacy on Severely Skewed Occupations: For stubborn edge cases like "nurse" (vanilla male ratio 0.007) and "technician" (vanilla female ratio 0.007), EquiSteer boosts minority representation to 0.432 and 0.375, outperforming all prior baselines.
• Zero Quality-Fairness Trade-off: Across 30,000 COCO generations, EquiSteer slightly improves CLIP Score (26.63 vs 26.42 on SD-1.5; 26.75 vs 26.51 on SDXL) and retains CMMD, debunking the assumption that debiasing inevitably degrades image aesthetics.

Highlights & Insights

• Activation-Level Intervention Efficiency: Leveraging cross-attention dot products at the very first denoising step provides an elegant, zero-overhead classifier for prompt intent without requiring external vision-language or LLM judges.
• Subspace Orthogonalization Before Injection: Treating concept debiasing as an explicit nullspace projection followed by calibrated target vector addition prevents mixed-attribute artifacts, providing a clean mathematical formulation for inference-time steering.
• Seamless Multi-Concept Scalability: The pipeline naturally accommodates sequential execution per layer across multiple concepts (e.g., gender, race, age, body type), achieving joint multi-attribute fairness in a single forward generation pass.

Limitations & Future Work

• Global Intervention on Multi-Subject Compositions: The current gating and steering mechanisms operate at the whole-image activation level. In complex compositional prompts involving multiple distinct subjects (e.g., "an elderly male doctor and a young female nurse"), a global gate cannot decouple spatial instances independently.
• Contrastive Set Dependency: Offline steering vectors depend on the quality and linguistic coverage of the calibration prompt templates; rare or highly contextual prompt distributions might show reduced steering precision.
• Future Directions: Integrating localized spatial cross-attention masks to enable instance-level fairness steering, and extending the methodology to video diffusion models and Diffusion Transformers (DiT).

Related Work & Insights

• vs FTDiff / FairDiff (Finetuning Approaches): Finetuning methods require model weight modifications and balanced training data, which scales poorly across diverse architectures; EquiSteer is plug-and-play, training-free, and model-agnostic.
• vs Balancing Act (Batch Guidance): Balancing Act requires coupled multi-sample batches and external attribute predictors; EquiSteer operates strictly per sample and runs independently of batch size.
• vs TEI / FairImagen (Text Embedding Projection): TEI and FairImagen project static text embeddings before diffusion, lacking spatial control over visual feature synthesis; EquiSteer modulates internal cross-attention layers directly, yielding substantially better parity (\(\Delta=0.051\) vs 0.167 on SD-1.5).

Rating

• Novelty: ⭐⭐⭐⭐☆ (Extends cross-attention steering from negative concept erasure to selective fairness debiasing via early dot-product gating and subspace orthogonalization)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across SD-1.5, SD-2.1, SDXL, and SANA backbones, covering single-concept, multi-attribute, joint debiasing, prompt transferability, and human validation)
• Writing Quality: ⭐⭐⭐⭐⭐ (Extremely clear presentation, rigorous mathematical formulations, and sound empirical ablations)
• Value: ⭐⭐⭐⭐⭐ (Provides an immediate, production-ready, training-free fairness mitigation tool for text-to-image foundation models)