AViTS: Adaptive Spatiotemporal Token Selection for Efficient Dynamic-Resolution Generation

Conference: ECCV2026
Paper: ECCV Official Page · PDF
Code: https://github.com/QHR69/AViTS
Area: Diffusion Acceleration / Dynamic-Resolution Image Generation and Editing
Keywords: spatiotemporal importance, text attention, cross-step variance, selective upsampling, training-free acceleration

TL;DR

AViTS jointly asks which latent tokens are relevant to the text and which are still changing during denoising to decide where to upsample early, reducing FLUX.1-dev latency from 25.78 s to 3.65 s in its 14-step configuration while ImageReward changes from 0.9719 to 0.9723, although more aggressive acceleration still sacrifices quality.

Background & Motivation

High-resolution DiT inference is expensive not only because of the number of sampling steps, but also because each step processes many spatial tokens. Dynamic-resolution sampling establishes composition at low resolution before progressively recovering details. Yet if every token is upsampled simultaneously at a resolution transition, background regions receive the same high-resolution budget as important objects. Partial upsampling permits some regions to become detailed earlier, leaving a more consequential question: which regions should receive that limited budget first?

RALU relies on edges, while Fresco uses inter-channel variance. These cues can locate structural complexity without necessarily identifying the content requested by the prompt. When an editing instruction asks to add a lounge chair, strong edges already present in the image may be less relevant than the intended insertion area. Text attention alone has a different blind spot: a semantically relevant region may already be stable, while another region with lower attention continues to change and needs more spatial detail.

On FLUX.1-dev and Qwen-Image-Edit, the authors observe long-tailed distributions for both text attention and cross-step variance, with complementary spatial heatmaps. Here, temporal refers to diffusion denoising steps, not video frames. Core idea: treat early upsampling as budget-constrained token ranking, combining text relevance with cross-step evolution instead of selecting regions solely from local structure in a single state.

Method

Overall Architecture

The inputs are text conditioning and generation noise; editing backbones also retain their native reference-image conditioning. The output remains a target-resolution image. AViTS leaves backbone weights unchanged and organizes inference into low-resolution, mixed-resolution, and full-resolution stages: establish global structure cheaply, collect signals near the end of low-resolution denoising, expand important tokens early, and finally expand the remainder for refinement.

For a target latent sequence of \(M\) tokens, the spatial downsampling factor is \(f=2\), giving \(M'=M/4\) low-resolution tokens. The first stage performs \(N_1\) denoising steps followed by \(N_T\) signal-collection steps at the same resolution. The second stage performs \(N_2\) mixed-resolution steps, and the third performs \(N_3\) full-resolution refinement steps. Importance is estimated from a multi-step collection window; the main text does not describe reranking tokens at every subsequent step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    input["Text and model conditions<br/>Downsample target noise"] --> collect["Low-resolution signal collection"]
    collect --> spatial["Text-aligned spatial scoring"]
    collect --> temporal["Cross-step temporal scoring"]
    spatial --> select["Joint ranking and staged upsampling"]
    temporal --> select
    select --> mixed["Mixed-resolution denoising"]
    mixed --> output["Expand remaining tokens and refine<br/>VAE image decoding"]

Spatial alignment and temporal dynamics are parallel scoring branches for the same low-resolution tokens. Mixed resolution does not mean discarding low-scoring tokens: it postpones their spatial refinement. The new contribution centers on importance estimation and selection; orthogonal upsampling and coordinate-bound noise reinjection follow prior practice.

Key Designs

1. Low-resolution signal collection: observe consecutive states after coarse structure emerges

The first stage downsamples target-resolution noise and runs the velocity network on fewer tokens. During the later collection window, denoising continues while latent snapshots and cross-modal attention are recorded at each step. Ranking can therefore use the trajectory of a spatial location rather than a single moment of local texture. Because every token still lies on the same low-resolution grid, the two signals can be aligned position by position.

Multi-step observation should not be confused with computing many channel statistics from one snapshot: the latter cannot show whether a token is becoming stable over time. AViTS records cross-step states, so the subsequent variance actually describes the denoising trajectory. The main text does not provide the complete \(N_1,N_T,N_2,N_3\) allocation for each experiment; total NFE alone cannot recover the collection-window length.

2. Text-aligned spatial scoring: measure conditional relevance through image-to-text attention

The spatial score sums an image token's attention to all text tokens, averages over attention heads, and further aggregates across selected modules and collection steps. It asks how much attention mass a location assigns to the text condition, rather than whether an edge exists there. With \(\alpha_{i,j}^{(h)}\) denoting attention from image token \(i\) to text token \(j\) in head \(h\), the score for one observation is:

\[ A_i=\frac{1}{|\mathcal H|}\sum_{h\in\mathcal H}\sum_{j=1}^{L}\alpha_{i,j}^{(h)}. \]

The paper describes two extraction routes. For MLLM-based models such as Qwen-Image-Edit, it takes the image-to-text submatrix from a selected layer's unified image-text attention. For FLUX-family models, it uses joint attention in double-stream blocks. Fused attention kernels do not expose weights, so forward hooks collect queries and keys, apply the backbone's QK normalization, reconstruct the joint softmax, and then select image rows and text columns. Renormalizing over text columns alone would destroy the intended distinction in total image-to-text attention mass. This reconstruction also means collection is not free.

3. Cross-step temporal scoring: identify unsettled tokens through temporal variance

Spatial relevance does not reveal whether a location is still being constructed. The temporal score computes the unbiased sample variance over the \(N_T\) collected steps for each token channel, then averages across channels. A token with larger cross-step changes receives priority for high-resolution processing. This differs from Fresco's inter-channel variation within a single state. With \(\bar z_{i,d}\) denoting the temporal mean for channel \(d\):

\[ V_i=\frac{1}{D}\sum_{d=1}^{D}\frac{1}{N_T-1}\sum_{k=1}^{N_T}\left(z_{i,d}^{(t_k)}-\bar z_{i,d}\right)^2. \]

The cached equations have extraction damage. The two equations above restore notation from the adjacent prose specifying a head average of text-attention sums and an unbiased temporal variance averaged over channels; they do not introduce a new mechanism. Unbiased variance requires \(N_T>1\). Large variance is an empirical proxy for the usefulness of earlier refinement, not a theoretical guarantee that greater variance implies greater final error.

4. Joint ranking and staged upsampling: change when fine-grained computation arrives

The raw scores have different scales, so each is independently min-max normalized over the current token set, for example \(\hat A_i=(A_i-\min A)/(\max A-\min A+\epsilon)\), with the same operation for \(\hat V_i\). A weight \(\alpha\in[0,1]\) combines them, and an allocation ratio \(\rho\in(0,1)\) determines the early-upsampled subset:

\[ S_i=\alpha\hat A_i+(1-\alpha)\hat V_i,\qquad \mathcal S=\operatorname{TopK}\!\left(\{S_i\}_{i=1}^{M'},\lfloor\rho M'\rfloor\right). \]

A Gaussian perturbation on the order of \(10^{-6}\) breaks ties before sorting. Selected tokens undergo orthogonal upsampling, while the remainder retain low-resolution representations. Coordinate-bound noise is reinjected into the mixed sequence before \(N_2\) denoising steps. The third stage expands the remaining tokens, reinjects noise again, restores spatial ordering, and uses \(N_3\) steps to recover details. Coordinate binding supports consistency across resolution changes, but the cache supplies neither the orthogonal transform matrix nor a noise-amplitude equation, so those implementation details are not reconstructed here. The main text also does not specify the experimental values of \(\alpha\) or \(\rho\).

A Worked Example

Figure 4 uses a boy flying a kite in a park, showing a \(1024\times1024\) target, an initial \(512\times512\) stage, and transitions through mixed resolution to full resolution. The following is only a token-count illustration, not a reported sampling configuration. Suppose the target sequence contains 4096 tokens: downsampling leaves 1024. If \(\rho=0.25\), 256 low-resolution tokens expand first, each corresponding to 4 high-resolution positions. The mixed stage then processes \(256\times4+768=1792\) tokens before the final expansion to 4096.

For this illustrative input, text scoring might emphasize the boy and kite, while temporal variance might emphasize evolving contours and thin structures. This explains the mechanism without claiming that the paper reports exact scores for those locations. Unselected park-background tokens still participate in mixed-resolution denoising, but remain coarse for longer. The budget constrains how many tokens expand early; the prompt and sampling trajectory determine where they are.

Loss & Training

AViTS is a training-free inference strategy, with no additional loss, trainable selection network, or fine-tuning dataset. It retains the backbone's flow-matching velocity network and Euler integration from \(t=1\) to \(t=0\), changing latent-token resolution across stages. The Euler equation in the cache has missing symbols; this note preserves the solver description supported by the prose rather than copying the damaged equation.

Compatibility with distillation, quantization, and caching means token allocation can be added to those existing models or acceleration paths. It does not mean AViTS performs a new distillation training run. Reproduction still requires module choices, collection length, fusion weight, allocation ratio, and noise scheduling, none of which can be uniquely inferred from the result tables.

Key Experimental Results

Main Results

Text-to-image evaluation uses DrawBench with ImageReward and CLIP Score. Editing uses the full Chinese or English GEdit benchmark, where SC, PQ, and OS denote Semantic Consistency, Perceptual Quality, and Overall Score. Higher quality scores are better; lower latency and FLOPs are better. FLUX.1-dev is evaluated on NVIDIA A800 GPUs, while both editing models use NVIDIA H20 GPUs, so absolute seconds should not be compared directly across backbones.

The following selection from source Table 1 focuses on dynamic-resolution comparisons. Latency speedup and FLOPs reduction are separate ratios relative to the 50-step original model; FLOPs are measured in T.

Method	NFE	Latency s	Latency speedup	FLOPs T	FLOPs reduction	ImageReward	CLIP Score
FLUX.1-dev	50	25.78	1.00×	3719.50	1.00×	0.9719	32.325
Fresco	30	9.17	2.81×	1295.99	2.87×	0.9801	32.125
AViTS	30	8.21	3.14×	1401.03	2.65×	1.0104	32.476
RALU	18	6.33	4.07×	904.98	4.11×	0.9481	32.074
Fresco	18	5.72	4.51×	788.03	4.72×	0.9861	31.970
AViTS	18	4.73	5.45×	774.74	4.80×	0.9959	32.361
AViTS	14	3.65	7.06×	586.56	6.34×	0.9723	32.352
AViTS	9	2.64	9.78×	380.30	9.78×	0.9201	32.169

At 30 steps, AViTS uses more FLOPs than Fresco but has lower latency, illustrating why operation counts cannot replace end-to-end timing. At 14 steps, both quality scores are near or slightly above the original model. At 9 steps, ImageReward falls to 0.9201, so the maximum speedup is not lossless. The abstract's 6.34× corresponds to FLOPs reduction in the 14-step configuration, whose measured latency speedup is 7.06×.

The editing results below select rows from source Tables 2 and 3 and show only GEdit-EN(FULL), keeping Chinese and English scores out of the same column. Each backbone uses its own 50-step original as the reference.

Backbone / configuration	NFE	Latency s	Latency speedup	FLOPs reduction	SC	PQ	OS
FLUX.1-Kontext-dev original	50	50.20	1.00×	1.00×	6.80	7.26	6.51
FLUX.1-Kontext-dev + AViTS	18	11.51	4.36×	5.37×	7.07	7.12	6.70
FLUX.1-Kontext-dev + AViTS	11	7.25	6.92×	8.85×	7.10	6.98	6.57
Qwen-Image-Edit original	50	284.51	1.00×	1.00×	7.82	7.54	7.54
Qwen-Image-Edit + AViTS	30	87.54	3.25×	3.72×	7.95	7.54	7.62
Qwen-Image-Edit + AViTS	11	40.93	6.95×	8.65×	7.80	7.48	7.48

The 11-step Kontext configuration improves SC but reduces PQ from 7.26 to 6.98. The 11-step Qwen configuration likewise does not preserve every English quality metric. Consequently, "nearly 9×" should be read as a FLOPs reduction ratio, accompanied by the separate semantic and visual-quality changes.

Ablation Study

Source Table 5 compares random, even, edge-based, Fresco, and single- or dual-signal selection. The authors state that the upsampling ratio is fixed and label all accelerated rows NFE=30. However, the mixed-score row's 4.73 s, 0.9959, and 32.361 exactly match the NFE=18 row in source Table 1. This note preserves the reported data and flags the conflict rather than changing the ablation NFE to 18 or treating the compute-budget match as fully verified.

Selection policy	NFE as reported	Latency s	Latency speedup	ImageReward	CLIP Score
Random	30	5.25	4.91×	0.9257	31.488
Evenly	30	5.25	4.91×	0.9689	32.012
Edge detection	30	5.76	4.48×	0.9512	32.074
Fresco	30	5.72	4.51×	0.9861	31.970
AViTS: attention only	30	4.72	5.46×	0.9875	32.251
AViTS: step variance only	30	4.67	5.52×	0.9846	32.202
AViTS: mixed score	30	4.73	5.45×	0.9959	32.361

Calculated within this table, fusion adds 0.0084 ImageReward and 0.110 CLIP Score over attention alone, or 0.0113 and 0.159 over temporal variance alone. Either signal already outperforms random selection, while their combination further improves semantic-related quality. Repeated trials or error bars are not provided to establish statistical significance.

Key Findings

• The strongest evidence is the joint change in quality and latency, not the largest speedup in isolation. Under aggressive budgets, retaining text-relevant regions still cannot fully compensate for omitted early detail computation.
• Source Table 4 reports 1.76 s, 14.65× latency speedup, and 14.76× FLOPs reduction when combined with step distillation; the last number is not measured latency speedup. That table uses CLIP-IQA and an original-model CLIP baseline of 33.402 rather than the main table's 32.325, so comparisons should remain within that table. The combined configuration reaches CLIP Score 32.20, which is still lower.
• The main text does not provide full sensitivity curves for \(\alpha\), \(\rho\), or \(N_T\). The ablation supports complementary signals, but does not establish insensitivity to hyperparameter choices.

Highlights & Insights

• Importance combines task relevance with continued evolution. Text attention can miss dynamic detail, while variance can emphasize irrelevant change; combining them is more closely tied to allocating high-resolution computation than a single structural cue.
• Low-scoring tokens are refined later rather than permanently removed. They preserve global context during mixed-resolution denoising, and the final complete grid supports output refinement: the method changes computation order rather than cropping content.
• Collecting ranking signals cheaply before spending on high-resolution processing is a transferable strategy. Other adaptive spatial-computation settings are plausible targets, but the image experiments do not validate temporal consistency in video.

Limitations & Future Work

• Incomplete reproduction details: the cached main text does not specify all stage lengths, layer choices, fusion weights, upsampling ratios, or noise amplitudes, and it inherits orthogonal upsampling from prior work. The main text alone cannot reconstruct a complete implementation. The code link is supplied by the paper and was not checked online.
• Ambiguous reporting: the ablation/main-table NFE conflict, loose wording around acceleration ratios, and the changed quality baseline in Table 4 all require confirmation during reproduction. A note should not silently "correct" them.
• Proxy-based importance: high attention need not imply a need for early refinement, and high temporal variance may include irrelevant change. These are mechanism-level risks inferred here, not failure rates measured by the authors.
• Hardware and collection costs: attention reconstruction, snapshot storage, sorting, and mixed-length computation all incur overhead. A800/H20 results do not guarantee the same ratios on consumer devices or different attention kernels.
• Quality and coverage boundaries: ImageReward or PQ declines at extreme acceleration. The main text has no dedicated limitations section and reports neither video validation, multi-seed uncertainty, nor a complete hyperparameter sweep. Editing-category breakdowns, error statistics, and collection-cost profiling would strengthen the evidence.

Related Work & Insights

• Versus RALU: edge-driven selection emphasizes existing structures, whereas AViTS explicitly uses text semantics and cross-step states. This distinction matters when the edit target does not coincide with strong input edges; upsampling-consistency mechanisms are not entirely new contributions of AViTS.
• Versus Fresco: inter-channel variance describes differences within one state, whereas AViTS's temporal score describes changes in the same channel across steps and is supplemented by text attention. The statistical axis is the essential distinction.
• Versus TeaCache, ToCa, and TaylorSeer: caching or prediction reduces repeated computation across steps, while AViTS allocates spatial-resolution budgets. Combination experiments demonstrate compatibility, not universally cost-free quality preservation.
• Versus Bottleneck Sampling: both adjust sampling resolution, but AViTS additionally ranks regions for early upsampling. A reusable research question is how to measure the actual marginal benefit of refining a region early, beyond improving proxy heatmaps.

Rating

• Novelty: 4/5. Joint text-relevance and temporal-variance selection addresses a clear problem, but builds on existing dynamic-resolution and partial-upsampling frameworks.
• Experimental Thoroughness: 3/5. Three backbones, generation and editing, and composable acceleration are covered, but budget-label conflicts and missing sensitivity/error statistics limit verification.
• Writing Quality: 3/5. The three-stage framework and complementary signals are clear, while acceleration terminology and table consistency need improvement; cache extraction damage is flagged separately and does not establish errors in the original PDF.
• Value: 4/5. A training-free, composable approach to spatial-budget allocation, with deployment still requiring checks of quality, parameters, and end-to-end overhead.