AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generation

Conference: ECCV2026
Paper: ECCV official page / Paper PDF
Code: https://github.com/xuanhuayin/AccelAes
Area: Image Generation
Keywords: Diffusion Transformers, aesthetic semantic masks, spatial sparsity, prediction caching, spatial CFG

TL;DR

AccelAes uses aesthetic prompt semantics and cross-attention to identify regions deserving precise updates, combines sparse computation with spatial CFG and step-level prediction reuse, and reduces Lumina-Next latency from 12.37 to 5.86 seconds while increasing ImageReward from 0.7518 to 0.8410 without training.

Background & Motivation

High-resolution text-to-image generation is expensive because a DiT repeatedly processes a large spatial token grid across dozens of denoising steps. Self-attention establishes pairwise token interactions, so both resolution and repeated model evaluations matter. Training-free acceleration methods address this cost through token merging, feature reuse, or predictions of future outputs. However, numerical similarity and uniform temporal schedules do not necessarily identify which local errors will most affect the final image's perceived quality.

AccelAes connects this budgeting problem to aesthetic language in the prompt. Words such as "intricate" and "cinematic" influence textures, materials, and lighting through text-image attention rather than specifying additional objects. The authors observe concentrated attention and stronger temporal variation in regions associated with such descriptors, whereas lower-affinity regions evolve more smoothly. Uniformly skipping updates can therefore approximate the hardest regions too aggressively; preserving every region equally leaves considerable redundancy untouched. Importantly, attention to aesthetic language is an internal proxy for perceptual importance, not a ground-truth local aesthetic annotation.

Instead of training a new reward-guided generator, the paper extracts a spatial priority map from frozen semantic representations and the generator's own attention, then uses temporal caching for further savings. Core idea: give aesthetic-relevant regions more computation and stronger guidance, while reusing low-priority spatial activations and predictable step outputs, so inference budgeting can improve preference quality rather than merely tolerate its degradation.

Method

Overall Architecture

The inputs remain a text prompt and an initial noisy latent, and the output remains an image decoded by the VAE. After initial full inference, AesMask constructs a binary spatial mask at step 5 by default. SkipSparse uses that mask for selective query and FFN updates and for spatially varying CFG. StepCache stores final model predictions and replaces some subsequent Transformer evaluations with linear extrapolation. The pretrained backbone weights are unchanged.

The two savings mechanisms operate at different levels: spatial sparsity reduces work inside an actual model evaluation, while temporal reuse reduces the number of evaluations. The output cache is applicable even when the architecture does not support selective internal updates; the spatial path explicitly requires localized computation to be feasible. In the diagram, returning to the spatial path means a prediction refresh, whereas a reuse step returns directly to the cache. It does not mean a full Transformer evaluation is performed before every cached output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Prompt and noise<br/>Full-forward warmup"] --> Mask["AesMask<br/>Aesthetic semantic localization"]
    Mask --> Spatial["SkipSparse<br/>Local computation and guidance"]
    Spatial --> Cache["StepCache<br/>Prediction caching and extrapolation"]
    Cache --> Update["Sampler updates latent"]
    Update -->|Prediction refresh| Spatial
    Update -->|Reuse step| Cache
    Update -->|Sampling complete| Output["VAE decoding"]

Key Designs

1. AesMask: identify aesthetic text semantics before locating their image regions

AesMask does not simply preserve every location receiving strong attention. It first asks which text tokens should define aesthetic relevance. A fixed vocabulary supplies reference descriptors, with 34 anchors used in the experiments. A frozen CLIP text encoder provides representations for cosine similarity between prompt tokens and anchors. Each prompt token receives the maximum similarity over the anchors, and the top-r tokens are selected. This makes the selection semantic rather than an exact string match against a list of adjectives. Even a descriptor-free prompt can be ranked, although ranking alone does not guarantee that its highest-scoring tokens are reliable aesthetic cues.

For each image token, the method sums cross-attention weights directed at the selected text tokens and averages the result across the chosen layers. This produces a spatial affinity map indicating how strongly each location attends to aesthetic-related prompt content. Thresholding at the map's p-th percentile creates the binary focus mask. A percentile accommodates variation in attention scale across prompts; increasing p generally reduces the active region. Although the formulation allows time-indexed masks, the default implementation constructs the mask once at step 5 and keeps it fixed. That avoids repeated construction and region jitter, but assumes that the early focus estimate remains useful as the image develops.

2. SkipSparse: preserve global context while jointly allocating updates and guidance

The mask partitions block hidden states into focus and background tokens without removing the background from the attention context. Self-attention updates only focus queries, but keys and values still cover the entire image. Consequently, an active location can read distant objects and background context even though fewer query outputs are computed. This is not equivalent to cropping the focus region and generating it independently. The FFN is also recomputed only for active tokens; background activations come from the most recent full update. Updated tokens are scattered back into their original positions so the block output retains the dense tensor shape.

The same mask controls classifier-free guidance. Focus locations receive a higher guidance scale and background locations a milder one, with both scales at least 1. The conditional-unconditional prediction gap is therefore amplified differently across space:

\[ \hat\epsilon_{\mathrm{cfg}}[i] =\hat\epsilon_{\mathrm{uncond}}[i] +\bigl(g_{\mathrm{bg}}+(g_{\mathrm{aes}}-g_{\mathrm{bg}})M[i]\bigr) \bigl(\hat\epsilon_{\mathrm{cond}}[i]-\hat\epsilon_{\mathrm{uncond}}[i]\bigr). \]

Here M is AesMask, and the two guidance scales correspond to background and focus locations. Sparse execution explains the computational saving; spatial CFG provides an explicit way to change the generated image's local emphasis. The quality gain should therefore not be attributed solely to removing redundant operations: the full method changes both computation and guidance allocation. When two-pass CFG is enabled, the reported latency includes conditional and unconditional forwards.

3. StepCache: reuse final predictions without requiring every internal feature to be reusable

StepCache stores final noise or velocity predictions from actual model evaluations and uses their recent trend for linear extrapolation at intermediate steps. This cache is distinct from the background FFN cache. The latter replaces part of a block's work, whereas the former can avoid the entire Transformer evaluation at a step. The sampler still advances its latent using the supplied prediction, so temporal reuse replaces a model evaluation rather than simply deleting a sampling step.

The default schedule begins with 5 full-inference warmup steps and refreshes the prediction cache every 2 steps afterward. Periodic real predictions limit drift from repeated approximations. Protecting difficult spatial regions and exploiting correlated outputs are complementary, but the source does not fully specify every backbone's internal cache-refresh interaction. Equations (12) and (13) are also damaged in the local PDF text extraction, and their timestep direction is insufficiently clear to reconstruct a trustworthy implementation-level extrapolation formula. This note therefore describes the mechanism without inventing coefficients or update ordering; reproduction requires checking the original PDF and implementation.

A Worked Example

Consider the paper's astronaut portrait prompt, which asks for intricate spacesuit details and a photorealistic, cinematic appearance. Initial full forwards establish a rough layout. At the default fifth step, anchor matching identifies prompt tokens associated with those aesthetic attributes, and cross-attention maps their relevance to image positions. Percentile thresholding selects the focus region. This explanation follows the paper's example without inventing a retained-token count or an exact segmentation result.

During actual model evaluations, selected portrait and suit-detail locations receive fresh attention-query and FFN computation and stronger guidance. They can still read the whole image through global keys and values, while background activations are reused where the spatial path permits it. At a temporal reuse step, an extrapolated final prediction advances the sampler; a later refresh corrects the trajectory, and the VAE decodes the completed latent. The failure mode is equally concrete: a small accessory that becomes visible only after the early mask is fixed may remain outside the active region and receive insufficient refinement.

Loss & Training

There is no new training loss, parameter fine-tuning, or reward backpropagation. The text encoder and diffusion backbone remain frozen. ImageReward, HPSv2, and the LAION aesthetic predictor are evaluation tools, not repeated gradient-based objectives inside sampling. The default configuration uses a one-shot mask, 34 anchors, 5 warmup steps, and a prediction refresh interval of 2.

The evaluation uses backbone-default sampling settings, with 30 steps for Lumina-Next and 28 for SD3-Medium and FLUX.1-dev, all at 1024 by 1024 resolution. The cached main text does not adequately specify the numerical top-r setting, percentile p, complete layer-selection policy, or both spatial guidance scales for each backbone. These values should not be inferred from the reported speedups.

Key Experimental Results

Main Results

The protocol uses 10,000 Pick-a-Pic prompts and three fixed seeds, 1, 2, and 3, producing 30,000 images per configuration. Methods within a table share prompts and seeds. Time is average end-to-end per-image latency from sampling start through final decoding, including the VAE. Speedup is dense-baseline time divided by method time. The following values are from Table 1; absolute latency across different backbones is not a controlled comparison of interchangeable architectures.

Backbone and configuration	Time / seconds	Speedup	CLIP	ImageReward	HPSv2	Aesthetic score
Lumina-Next baseline	12.37	1.00x	0.2531	0.7518	0.2710	5.9407
Lumina-Next + AccelAes	5.86	2.11x	0.2640	0.8410	0.2740	6.0414
SD3-Medium baseline	3.52	1.00x	0.2662	0.8788	0.2895	5.7330
SD3-Medium + AccelAes	2.34	1.50x	0.2744	0.9042	0.3014	5.9898
FLUX.1-dev baseline	12.66	1.00x	0.2753	1.233	0.3200	6.243
FLUX.1-dev + AccelAes	7.31	1.73x	0.2772	1.317	0.3214	6.372

CLIP measures text-image alignment; ImageReward and HPSv2 are preference proxies; the LAION aesthetic predictor provides an image-level aesthetic score. They are related but not interchangeable. Lumina-Next gains 0.0892 absolute ImageReward, approximately 11.9% relative to its baseline, not 11.9 percentage points of a preference win rate. The paper also reports Edge Density as a sharpness/detail proxy, but the main text does not sufficiently specify the edge-extraction threshold, and a higher edge count is not equivalent to better detail fidelity.

In the same-backbone comparisons of Table 2, TaylorSeer takes 6.02 seconds at 2.05x speedup with IR 0.604, while RAS takes 8.38 seconds at 1.47x with IR 0.788. AccelAes takes 5.86 seconds at 2.11x with IR 0.841. The useful conclusion is stronger preference quality at a comparable speed regime, not dominance on every metric: TaylorSeer's Edge value is 0.668, exceeding AccelAes at 0.629.

Ablation Study

These values come from Figure 5 and the accompanying Section 4.4 discussion, retaining the ablation's three-decimal precision. Some numerical plot labels in Figure 4 are corrupted in the local extraction and are not transcribed here.

Configuration	Speedup	ImageReward	Interpretation
Dense baseline	1.00x	0.752	Original inference
StepCache only	1.57x	0.743	Temporal reuse saves time but slightly reduces preference quality
Spatial path only	1.43x	0.818	AesMask and spatial allocation improve quality
Without AesMask, retaining spatial and temporal paths	2.11x	0.822	Same speed as the full method, without the aesthetic region prior
Full AccelAes	2.11x	0.841	Joint spatial and temporal paths with AesMask

The most discriminating comparison is between the final two rows: AesMask adds 0.019 IR at the same reported speedup. This supports a contribution from region selection rather than merely additional computation. Spatial allocation alone increases IR, while temporal reuse alone slightly decreases it, explaining why arbitrary caching should not be expected to improve aesthetics by itself.

Key Findings

• A blind human study uses 15 annotators, 60 pairs, and questions about overall preference, alignment, appeal, and fewer artifacts. Excluding ties, AccelAes wins 67% on overall preference and 68% on appeal against the dense baseline. These are study-specific non-tie win rates, not population-wide rates over all generated images.
• With descriptor-free prompts, IR rises from 1.109 to 1.144; with aesthetic descriptors it rises from 1.168 to 1.224. These robustness results have a different evaluation context from the main table and should not be mixed into its relative improvement calculation.
• Frequent dynamic mask updates perform worse than the default static mask, with region jitter offered as an explanation. On mined difficult failures, late-mask and entropy fallback rules recover 92.7%-95.4% of dense ImageReward. This is a recovery ratio, not an additional improvement; the cached text does not fully specify the fallback implementation.

Highlights & Insights

• Aesthetic semantics becomes a computational priority signal rather than only a post-generation score. Frozen text matching identifies relevant language, and internal attention supplies spatial localization without training a separate region-level aesthetic predictor.
• Restricting queries while retaining global keys and values separates update cost from contextual visibility. This explains how selective refinement can preserve long-range interactions rather than behaving like independent crop generation.
• The paper distinguishes sources of quality and speed improvements. Spatial CFG can actively change local preferences, while temporal prediction reuse provides substantial computational savings; the component ablation shows why their combination matters.

Limitations & Future Work

• The authors identify diffuse early attention and shifting salient regions as central failure modes. A narrow fixed mask can freeze small details outside its active area, with failures mined from crowds, cluttered interiors, abstract art, and landscapes. The mask is not a universally reliable perceptual segmentation.
• Reproducibility is limited by missing mask and guidance hyperparameters, incomplete fallback details, and no explicit hardware model in the cached main text. The damaged extrapolation equations add an extraction-specific verification limitation. Reported speedups are measurements from the evaluated implementation, not a performance guarantee derivable from the prose alone.
• Local computation requires architectural support, and global keys/values, mask construction, and token scattering still incur costs. A more portable output cache does not make the entire system equally plug-and-play across DiTs. The additional 4-step FLUX.1-schnell result supports quality compatibility, but no corresponding speedup is reported in the main text.
• Automatic preference scores and a relatively small human study offer complementary evidence but do not settle distributional generalization or annotation dependence. Confidence intervals, failure rates by prompt category, and compute-only controls with unchanged CFG would sharpen the interpretation of the gains.

Related Work & Insights

• Compared with RAS and SDiT: these methods also allocate budgets non-uniformly across regions, so prior work should not all be characterized as spatially uniform. AccelAes distinguishes itself through an aesthetic anchor-conditioned attention prior shared by sparse computation and spatial guidance.
• Compared with FORA, TeaCache, and TaylorSeer: all exploit temporal redundancy, but AccelAes supplements output-level reuse with an explicit spatial priority mechanism. Its objective is not only approximating the dense trajectory; spatial guidance can reinforce regions that influence preference quality.
• Compared with aesthetic post-training and spatial CFG: post-training changes weights, while spatial CFG changes local guidance without necessarily reducing computation. AccelAes keeps weights frozen and ties guidance to actual compute allocation through the same mask. Reusing task-relevant semantics as a budget signal is a transferable principle, not evidence that it already works in other tasks.

Rating

These are subjective reading assessments on a five-point scale, not conference review scores.

• Novelty: 4/5. The coupling of aesthetic semantics, local computation, and guidance is distinctive, while sparse execution and extrapolation have established precedents.
• Experimental Thoroughness: 4/5. Three backbones, paired seeds, component ablations, and human preferences provide useful coverage; uncertainty estimates and implementation details remain incomplete.
• Writing Quality: 3/5. The motivation and component roles are clear, but several hyperparameters, fallback rules, and timestep conventions need more detail; local equation extraction also limits verification.
• Value: 4/5. The work shows that training-free acceleration can improve preference quality rather than only trading it away, making it relevant to latency-sensitive image generation.