EMAG: Self-Rectifying Diffusion Sampling with Exponential Moving Average Guidance

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/drkkgy/EMAG
Area: Image Generation
Keywords: Diffusion Models, Self-Rectifying Sampling, Classifier-Free Guidance, Exponential Moving Average, Hard Negatives, Human Preference Score

TL;DR

EMAG is a training-free guidance mechanism for diffusion transformers that temporally smooths self-attention maps via exponential moving average (EMA) alongside statistics-driven adaptive layer selection to construct semantically faithful hard negatives, elevating Human Preference Score (HPS) by +0.54 over CFG on SD3 and composably augmenting orthogonal guidance methods like APG and CADS.

Background & Motivation

Denoising diffusion models and rectified flow transformers have redefined modern visual generation across modalities. In conditional synthesis, Classifier-Free Guidance (CFG) has long served as the de facto driving force, steering generative trajectories toward prompt fidelity by extrapolating between conditional and unconditional score estimates. Nevertheless, aggressively scaling CFG incurs well-documented pathologies, including diversity collapse, visual over-saturation, and high-frequency noise amplification. In response, auto-guidance frameworks have sought to eliminate such failure modes by contrasting a strong primary model against a deliberately weakened surrogate. Existing works instantiate this degraded negative prediction through reduced parameter capacities, self-attention masking (SAG), smoothed energy curvatures (SEG), identity map replacement (PAG), or stochastic layer dropping (S2-Guidance).

Crucially, current negative-guidance formulations suffer from a foundational design limitation: they focus on recipes to obtain an inferior model but lack direct, fine-grained control over the difficulty and granularity of the generated negative samples. Most existing perturbation schemes induce coarse, catastrophic corruptions that shatter the global semantic coherence and structural geometry of the scene. Such obvious failure modes constitute "easy negatives" that modern, over-parameterized diffusion backbones can effortlessly sidestep during sampling. Consequently, the denoiser receives little informative contrastive signal to rectify subtle, localized micro-imperfections such as warped geometries, unnatural button alignments, or blurred fine textures. Furthermore, existing layer perturbation schedules rely on rigid, time-invariant heuristics, ignoring the fact that different transformer blocks dynamically modulate high-frequency content at distinct sampling timesteps.

The core premise of this work is that optimal guidance requires negative samples that lie strictly on the boundary of subtle degradation: preserving global semantic fidelity while selectively damping high-frequency refinements. Because diffusion sampling inherently resolves semantics from coarse layouts to fine details over time, lagging the attention updates via an exponential moving average naturally attenuates high-frequency synthesis without destabilizing structural layout. Core idea: propose Exponential Moving Average Guidance (EMAG), a training-free inference technique that maintains temporal EMA buffers over self-attention maps and adaptively targets the layer exhibiting the largest instantaneous statistical divergence, yielding semantically faithful hard negatives that enable the denoiser to self-rectify fine-grained visual artifacts.

Method

Overall Architecture

EMAG operates purely at inference time across the reverse sampling trajectory without modifying pretrained model weights. At each reverse timestep, the system maintains a running exponential moving average of self-attention maps across candidate middle layers; an adaptive layer selection module calculates the mean absolute error between current attention activations and their EMA representations, dynamically picking the single layer with the most intense high-frequency activity; the attention tensor of the chosen layer is replaced with its EMA blend to generate a degraded conditional score prediction; and a two-stage guidance update extrapolates away from this hard negative before performing standard CFG composition. For joint image-text attention in MMDiT backbones, EMAG isolates the image-to-image attention block (EMAG-I) and further introduces a query-space smoothing variant (EMAG-Q) that natively preserves fused scaled dot-product attention (SDPA) kernel acceleration.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Latent state $x_t$ and condition $c$"] --> B["Attention Temporal Smoothing<br/>Maintain attention EMA buffer $E_t$"]
    B --> C["Adaptive Layer Selection<br/>Greedily pick maximum-divergence layer $l_n^{*t}$"]
    C --> D["Two-Stage Guidance Update<br/>Extrapolate from hard negative and compose CFG"]
    D --> E["Efficient Variants EMAG-Q / EMAG-I<br/>SDPA kernel fusion and cross-attention isolation"]
    E --> F["Rectified score prediction $\bar{z}_t$"]

Key Designs

1. Attention Temporal Smoothing: Damping High-Frequency Updates to Construct Hard Negatives

Conventional negative-guidance approaches induce severe, non-local corruptions (such as zeroing attention maps or injecting isotropic noise) that cause structural collapse in the negative sample, yielding easy negatives that fail to challenge the denoiser. EMAG remedies this by exploiting the temporal continuity of diffusion trajectories, tracking an exponential moving average of the softmax-normalized attention maps \(A_t \in [0, 1]^{B \times H \times Q \times K}\) across middle transformer layers. Starting from \(E_1 := A_1\), the EMA buffer updates recursively at step \(t\) via: $$ E_t = \beta E_{t-1} + (1 - \beta) A_t $$ where the decay factor \(\beta \in [0, 1)\) acts as a precise dial for negative hardness and degradation granularity. In diffusion trajectories, early-to-mid steps crystallize global semantics while late steps inject fine-grained details; temporal smoothing acts as a low-pass filter over attention updates. By substituting \(A_t\) with the convex combination \(\widetilde{A}_t = (1-\lambda) A_t + \lambda E_t\) (using \(\lambda=1\) as the default hard swap), the degraded prediction \(\epsilon'_\theta(x_t, c)\) preserves global structure while suppressing incremental detail refinements. Quantified by CLIP-I and DINOv2 cosine similarities against the clean target output, EMAG achieves substantially higher semantic similarity than SAG, PAG, or ERG, confirming that it generates the hardest and most semantically faithful negative samples in the literature.

2. Adaptive Layer Selection: Dynamically Pinpointing High-Frequency Insertion

In deep diffusion transformers, individual layers contribute unequally to feature synthesis across time; fixed single-layer perturbations (e.g., permanently perturbing layer 7) risk targeting a dormant block during certain steps while missing active high-frequency updates. EMAG addresses this with a statistics-driven selection policy over a candidate middle-layer band \([l_{\min}, l_{\max}]\) (set to layers 12–15 in DiT-XL/2 and layers 6–8 in SD3-Medium). At each reverse timestep \(t\), the system computes the mean absolute error (MAE) between the instantaneous attention map and its smoothed EMA buffer for each candidate layer \(n\): $$ \Delta_t^{(n)} = \operatorname{MAE}(E_t^{(n)}, A_t^{(n)}) = \frac{1}{|A_t|} \sum_{i} \left| E_t^{(i)} - A_t^{(i)} \right| $$ The framework then greedily perturbs the layer maximizing this divergence: \(l_n^{*t} = \arg\max_{n \in \{l_{\min}, \dots, l_{\max}\}} \Delta_t^{(n)}\). Because a large MAE signals that the layer is actively altering attention patterns to introduce new features, targeting it induces maximal local degradation without requiring perturbations across the entire network, avoiding catastrophic quality collapse.

3. Two-Stage Guidance Update and Efficient Variants (EMAG-Q / EMAG-I)

Once the degraded score \(\epsilon'_\theta(x_t, c)\) is computed, EMAG applies a two-step update formulation. In the first step, it calculates an intermediate condition vector \(\bar{z}_{c,t}\) by steering the standard conditional prediction \(z_{c,t} = \epsilon_\theta(x_t, c)\) away from the hard negative \(\hat{z}_{c,t} = \epsilon'_\theta(x_t, c)\) scaled by the EMAG factor \(w_e\): $$ \bar{z}{c,t} = \hat{z}} + w_e \cdot (z_{c,t} - \hat{z{c,t}) $$ In the second step, this rectified conditional score is fed into the standard CFG formulation against the unconditional score \(z_t = \epsilon_\theta(x_t, \varnothing)\) scaled by \(w_{\text{cfg}}\): $$ \bar{z}_t = z_t + w - z_t) $$ In multimodal architectures (MMDiT), where cross-modal interaction occurs within joint self-attention, EMAG-I isolates perturbations strictly to the Image}} \cdot (\bar{z}_{c,t\(\to\)Image attention block to avoid corrupting text grounding. Furthermore, because explicitly unrolling large post-softmax attention maps precludes hardware-optimized kernels, the authors formulate EMAG-Q: tracking EMA directly over query representations \(Q_t\). This enables full compatibility with fused scaled dot-product attention (SDPA), reducing inference latency overhead from +213.2% to +94.5% and trimming peak VRAM usage from 29.58 GB to 16.96 GB on an A100 GPU.

Key Experimental Results

Main Results

Quantitative evaluations are conducted using the Stable Diffusion 3 Medium checkpoint (1024×1024 resolution, 28 sampling steps) across the MS-COCO 2014 validation split (40,000 samples, fixed random seed 8). Perceptual quality is assessed via Fréchet Inception Distance (FID) and Human Preference Score v2 (HPS, scaled by 100), benchmarked against CFG, attention perturbation baselines, and orthogonal guidance strategies (Table 2 of the original paper).

Guidance Method	FID ↓	HPS v2 ↑	HPS Gain vs. CFG
No Guidance (No CFG)	23.859	21.61 ± 0.11	-7.61
CFG (Baseline)	22.877	29.22 ± 0.05	Baseline
SAG + CFG	23.039	29.28 ± 0.04	+0.06
PAG + CFG	24.066	28.73 ± 0.04	-0.49
SEG + CFG	23.453	29.25 ± 0.04	+0.03
S2-Guidance	20.821	29.15 ± 0.00	-0.07
ERG	23.989	29.35 ± 0.03	+0.13
APG	21.816	29.45 ± 0.02	+0.23
CADS	18.320	28.36	-0.86
EMAG-I (Ours)	22.154	29.56	+0.34
EMAG (Ours)	22.890	29.76 ± 0.03	+0.54
EMAG-Q (Ours)	23.530	29.64	+0.42
EMAG + APG	21.819	29.79 ± 0.04	+0.57
EMAG + CADS	19.950	28.86 ± 0.02	-0.36 (High Diversity)
EMAG-Q + APG	22.480	29.77	+0.55

Ablation Study

Ablation on the adaptive layer selection policy evaluated on SD3 with 5,000 samples over MS-COCO 2014 validation prompts (Table 5 of the original paper):

Layer Selection Setting	FID ↓	HPS v2 ↑	Note
Fixed Layer 6	27.47	29.54	Stronger FID but limited preference gain
Fixed Layer 7	28.94	29.51	Sub-optimal across both metrics
Fixed Layer 8	28.23	29.66	Higher preference but noticeably degraded FID
Perturb All Middle Layers	35.60	28.80	Severe negative collapse degrades image fidelity
EMAG Adaptive Selection	28.52	29.60	Optimal Pareto trade-off between fidelity and preference

Inference efficiency benchmarking on a single NVIDIA A100-SXM4-40GB GPU (SD3 Medium at 1024×1024, batch size 1, Table 4 of the original paper): - Vanilla CFG: 4.04s per image, 16.89 GB peak memory, FID 22.88 / HPS 29.22. - Full EMAG: 12.66s per image (+213.2% overhead), 29.58 GB peak memory, FID 22.89 / HPS 29.76. - EMAG-Q (Query EMA): 7.86s per image (+94.5% overhead), 16.96 GB peak memory (+0.07 GB increase), FID 23.53 / HPS 29.64.

Key Findings

• Hard Negatives Maximize Human Preference: Standard EMAG attains the highest standalone HPS score of 29.76, an impressive +0.54 increase over vanilla CFG. This demonstrates that contrastive guidance against structurally aligned, temporally smoothed negative samples directly targets fine-grained artifacts favored by human visual assessment.
• Catastrophic Failure of All-Layer Degradation: Applying EMA perturbations across all candidate layers simultaneously causes severe quality collapse (FID ballooning to 35.60 and HPS tumbling to 28.80), demonstrating that overly degraded surrogates regress to unhelpful "easy negatives."
• Adaptive Selection Dominates Fixed Baselines: Dynamically routing perturbations to the layer with the largest attention divergence effectively avoids the metric polarization seen in fixed single-layer ablations, achieving the superior balance on the FID-HPS Pareto frontier.
• Orthogonal Synergy with APG and CADS: Because EMAG specifically refines negative sample fidelity, it cleanly composes with APG (which curbs saturation via projection) to reach 29.79 HPS and 21.819 FID, and with CADS to achieve substantial diversity gains without sacrificing structural control.

Highlights & Insights

• Inverting EMA into a Negative Sample Synthesizer: While exponential moving averages are conventionally leveraged as parameter stabilizers to improve convergence, EMAG repurposes temporal EMA on attention maps as a high-frequency suppression filter, yielding structurally identical, detail-attenuated negative samples.
• Formal Quantification of Negative Sample Hardness: The authors rigorously define negative hardness via frozen CLIP-I and DINOv2 cosine similarities, validating empirically that contrastive guidance is most effective when negative samples closely mirror positive samples in semantic representation space.
• Hardware-Aware Approximation via EMAG-Q: Recognizing that full attention matrix tracking breaks fused FlashAttention / SDPA acceleration, the authors introduce query-level EMA, slashing inference latency by more than half while sacrificing only 0.12 HPS points.

Limitations & Future Work

• Computational Overhead: Due to the auxiliary forward evaluation through the perturbed attention pathway, standard EMAG incurs a +213% latency overhead; even the lightweight EMAG-Q requires +94.5% extra runtime compared to baseline CFG.
• Hyperparameter Sensitivity: The decay rate \(\beta\), guidance scale \(w_e\), and active timestep boundaries \(\tau_s, \tau_e\) require empirical calibration across different backbone architectures and sampler configurations.
• Future Directions: Exploring lightweight predictive heads to estimate optimal dynamic decay rates \(\beta_t\) on the fly; extending temporal attention smoothing to the temporal self-attention layers of video diffusion architectures to rectify inter-frame flickering and dynamic boundary tearing.

Related Work & Insights

• vs. CFG [Ho & Salimans, 2022]: CFG extrapolates against an unconditional null-text branch, which often induces color over-saturation; EMAG synthesizes an in-distribution, subtly degraded conditional negative, guiding localized self-rectification.
• vs. SAG [Hong et al., ICCV 2023] & PAG [Ahn et al., ECCV 2024]: SAG blurs input latent regions while PAG replaces attention with identity matrices, both inducing discontinuous spatial corruptions. EMAG utilizes temporal smoothing to preserve attention continuity and topological coherence.
• vs. SEG [Hong, NeurIPS 2024] & ERG [Ifriqi et al., 2025]: SEG applies spatial Gaussian filtering to energy maps, and ERG perturbs attention entropy and prompt tokens; EMAG filters temporal trajectory dynamics and incorporates adaptive, cross-layer statistical dispatching rather than relying on manual, fixed-layer selection.

Rating

• Novelty: ⭐⭐⭐⭐ [Innovative conceptual framing of temporal attention smoothing for hard-negative generation, combined with adaptive MAE layer selection]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across DiT and MMDiT backbones, comprehensive Pareto frontier analysis, and detailed hardware profiling]
• Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear narrative structure, and compelling empirical visual insights]
• Value: ⭐⭐⭐⭐ [Fully training-free, plug-and-play enhancement that provides substantial visual quality gains for diffusion transformers at inference]

Related Papers

• [ECCV 2026] Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?
• [ECCV 2026] Don't Settle at the Mode! Mitigating Diversity Collapse in Pretrained Flow Models via Feature Self-Guidance
• [ECCV 2026] Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold
• [ECCV 2026] Accelerated Likelihood Maximization for Diffusion-based Versatile Content Generation
• [ECCV 2026] Filterless Snapshot Hyperspectral Imaging: Guided Patch Diffusion Models