Accelerating Text-to-Video Generation with Calibrated Sparse Attention

Conference: ECCV2026
Paper: ECCV official page: 3436
PDF: Full paper
Area: Video Generation
Keywords: sparse attention, offline calibration, spatial repetition, FlashAttention3, inference acceleration

TL;DR

CalibAtt compiles input-stable block sparsity and spatial row repetition into two attention execution paths without updating model weights, reducing Wan 2.1 14B's 720p generation latency from 1244 to 785 seconds while retaining comparable VBench performance.

Background & Motivation

Video diffusion Transformers compute attention over large collections of spatiotemporal tokens, making quadratic attention increasingly expensive as resolution and frame count grow. FlashAttention reduces memory traffic and intermediate storage through tiling and online softmax, but still evaluates every query-key interaction. Training-free acceleration methods remove low-contribution connections, yet static windows rely on prescribed positional patterns, while online selection introduces the cost of estimating which blocks matter during generation.

The paper's observation is not that all attention is local. Rather, for a fixed layer, head, and diffusion timestep, many weak connections remain weak across different prompts and initial noise samples. These patterns do not remain identical across all layers and heads, so one shared geometric mask would erase useful structural differences. Other heads resist block sparsification but repeatedly produce similar attention distributions across spatial rows within a frame, suggesting a different way to save computation.

Core idea: move input-stable computation selection offline, identify both skippable attention blocks and reusable spatial-row outputs, and let inference kernels execute those calibrated choices without rediscovering sparsity online.

Method

Overall Architecture

The inputs are a pretrained video generator, a fixed inference configuration, and calibration prompts. The outputs are block-mask dictionaries and spatial-repetition flags indexed by timestep, layer, and head. Calibration combines block energy selection and cross-prompt aggregation with a separate spatial repetition detection branch; precompiled dual-path execution then uses block-sparse attention for non-repetitive heads or anchor-row attention with output broadcasting for repetitive heads.

What becomes fixed is the choice of connections to compute, not their attention values. Retained connections still use the current input's queries, keys, and values, and the repetition path recomputes anchor outputs for the current input. This is neither a cache of entire video features nor the reuse of one prompt's attention outputs for another prompt.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Model and calibration prompts"] --> Energy["Block energy selection"]
    Energy --> Consensus["Cross-prompt aggregation"]
    Input --> Repetition["Spatial repetition detection"]
    Consensus --> Execution["Precompiled dual-path execution"]
    Repetition --> Execution
    Execution -->|Non-repetitive heads| Sparse["Skip unselected blocks<br/>Compute retained connections"]
    Execution -->|Repetitive heads| Anchors["Compute anchor-row attention<br/>Broadcast to neighboring rows"]
    Sparse --> Output["Continue denoising and generate video"]
    Anchors --> Output

Key Designs

1. Block energy selection: retain attention mass at an executable hardware granularity

For a fixed timestep, layer, and head, CalibAtt inspects post-softmax attention. Small individual elements do not automatically translate into work that a GPU can efficiently skip, so selection operates on FlashAttention blocks. Experiments use query and key blocks of size 128 by 128. A block's energy sums attention over its keys and averages that sum over its queries.

Let \(P\) denote post-softmax attention, \(\mathcal I_r\) and \(\mathcal J_c\) the query and key token sets for a block, and \(B\) its side length. Equation (5) defines:

\[ E_{r,c}=\frac{1}{B}\sum_{i\in\mathcal I_r}\sum_{j\in\mathcal J_c}P_{ij}. \]

The energies across all key blocks for one query block consequently sum to one. CalibAtt sorts these energies in descending order, keeps the shortest prefix whose cumulative mass reaches a threshold, and maps the selection back to its original positions to obtain a per-prompt binary mask. This minimizes the number of retained blocks for a specified average attention mass; it does not give every individual query token the same mass-retention guarantee.

The energy threshold changes over diffusion timesteps. Early, high-noise steps use more conservative selection, while later steps can discard more weak connections. The paper reports an exponential schedule with thresholds ranging from 0.99 to 0.84 for Wan. Equation (7) is corrupted in the local text extraction, so this note does not reconstruct the exact schedule or invent hyperparameter values omitted from the main text.

2. Cross-prompt aggregation: keep frequently needed blocks instead of reusing one sample's mask

A mask optimized for one prompt may reflect that particular scene rather than stable model behavior. CalibAtt repeats block energy selection across calibration prompts and averages the resulting binary masks elementwise. The result is a block's keep frequency, not its average attention energy: the two thresholding stages operate on different quantities.

Using \(\mathcal D\) for the calibration set and \(M_p\) for prompt \(p\)'s mask, and suppressing the shared timestep, layer, and head indices, Equation (8) and the accompanying explanation of Equation (9) give:

\[ \bar M_{r,c}=\frac{1}{|\mathcal D|}\sum_{p\in\mathcal D}(M_p)_{r,c},\qquad M_{r,c}=\mathbf 1[\bar M_{r,c}\geq\rho]. \]

The default uses 64 MovieGenBench prompts and an agreement threshold of \(\rho=0.5\). A higher threshold removes more disputed blocks and increases sparsity, potentially reducing fidelity; a lower threshold is more conservative. Figure 3 shows keep frequencies concentrated near zero and one, providing empirical support for offline aggregation, rather than a guarantee for arbitrary unseen inputs.

3. Spatial repetition detection: reduce query rows for heads that are difficult to sparsify

A separate calibration branch compares attention distributions across spatial rows within the same video frame. It flattens the attention associated with each spatial row, measures cosine similarity to the nearest anchor row, and averages over frames, rows, and calibration prompts. A timestep-layer-head combination is marked repetitive only when this mean spatial similarity exceeds the selected threshold.

The default uses five equally spaced anchor rows per frame and a similarity threshold of 0.87. During inference, only anchor rows compute attention against all keys and values; each anchor's output is broadcast to its nearest spatial rows within the frame. This neither discards whole frames nor removes all cross-frame interactions: anchor queries still have access to all keys and values.

For a frame with \(H\) spatial rows and \(W\) tokens per row, using \(k\) anchors reduces its query count from \(HW\) to \(kW\), giving interaction sparsity \(1-k/H\). Here, \(k\) denotes the anchor count and should not be confused with the identically named energy-schedule parameter in the paper. Figure 6 shows that high spatial similarity tends to coincide with low block sparsity, so the two strategies address different attention maps.

4. Precompiled dual-path execution: turn calibrated patterns into GPU work that is actually skipped

During calibration, a custom CUDA kernel accumulates block energies without materializing the full attention matrix. A separate batched PyTorch implementation computes spatial similarity. Each binary mask is then encoded into a read-only skip list of contiguous key-block intervals to compute for each query block, and these lists are loaded onto the GPU before generation.

At inference time, the current timestep, layer, and head select the appropriate record. Non-repetitive heads use a block-sparse kernel based on FlashAttention3, whereas repetitive heads use standard FlashAttention3 with reduced queries followed by broadcasting. These are alternative paths selected by the repetition flag, not two approximations that are necessarily stacked on every head.

Precompilation removes online importance estimation, but does not make all auxiliary operations free. Mask lookup, storage, and broadcasting remain part of execution. The relevant evidence is measured generation latency on an H100, not an assumption that theoretical sparsity translates directly into an identical speedup.

A Worked Example

Consider the calibration mask for a non-repetitive head. Each prompt first selects key blocks using the energy threshold for that timestep. The selections are then counted across 64 binary masks. With the default agreement threshold of 0.5, a block selected at least 32 times is retained; 32 is a derived illustration of the default rule, not an additional experimental measurement.

For a new prompt, the model retrieves that mask and computes current-input attention only in retained blocks. A different head marked spatially repetitive instead computes five anchor rows per frame and broadcasts their outputs, without using the block-mask path. One branch reduces key-block computation, while the other reduces query-row computation, and both feed back into the original denoising process.

Loss & Training

CalibAtt introduces no training loss and makes no updates to pretrained weights. Offline calibration consists of running the model, collecting attention statistics, and compiling execution records, not gradient-based fine-tuning. Training-free therefore does not mean free of preprocessing cost.

Wan 2.1 14B generates 81 frames in 50 diffusion steps. Mochi 1 generates 85 frames in 64 steps. LightX2V is a four-step distilled Wan 2.1 14B variant evaluated with the same 81-frame output and two resolutions as Wan. Calibration dictionaries correspond to a particular model and inference configuration; these experiments do not establish that one dictionary can be reused unchanged across configurations.

Key Experimental Results

Main Results

All experiments use a single NVIDIA H100. Calibration prompts come from MovieGenBench, while quality evaluation uses the official VBench prompts and metrics. Methods within each setting share pretrained checkpoints and inference settings. VBench Total summarizes generation quality and semantic performance; it does not measure pixelwise agreement with a same-seed dense output.

The following entries come from Tables 1 and 2. Sparsity is the fraction of skipped spatiotemporal query-key interactions, averaged over timesteps, layers, and heads. Latency is the average wall-clock duration of the diffusion process for one video and excludes one-time offline calibration.

Model and setting	Method	VBench Total (higher)	Sparsity (higher)	Latency (lower)	Speedup over FA3
Wan 2.1 14B, 480p, 50 steps	Dense FA3	80.29	0%	363 s	1.00x
Wan 2.1 14B, 480p, 50 steps	CalibAtt	80.40	68.1%	250 s	1.45x
Wan 2.1 14B, 720p, 50 steps	Dense FA3	79.35	0%	1244 s	1.00x
Wan 2.1 14B, 720p, 50 steps	SpargeAttention	78.85	49.9%	930 s	1.33x
Wan 2.1 14B, 720p, 50 steps	RadialAttention	77.95	54.7%	936 s	1.32x
Wan 2.1 14B, 720p, 50 steps	SVG2	79.30	46.3%	846 s	1.47x
Wan 2.1 14B, 720p, 50 steps	CalibAtt	79.69	62.5%	785 s	1.58x
Mochi 1, 480p, 64 steps	Dense FA3	74.54	0%	188 s	1.00x
Mochi 1, 480p, 64 steps	CalibAtt	74.57	69.1%	161 s	1.16x
LightX2V, 480p, 4 steps	Dense FA3	81.04	0%	14.5 s	1.00x
LightX2V, 480p, 4 steps	CalibAtt	81.26	70.9%	11.2 s	1.29x
LightX2V, 720p, 4 steps	Dense FA3	77.63	0%	48.3 s	1.00x
LightX2V, 720p, 4 steps	CalibAtt	77.84	73.9%	30.6 s	1.57x

Ablation Study

Section 4.2 varies calibration set size and the agreement threshold. The curves indicate that the sparsity-quality tradeoff stabilizes quickly as more prompts are added, whereas overly aggressive thresholds slightly reduce quality. The table records only the explicit Wan 2.1 14B 720p calibration-budget figures from the main text, without estimating exact quality scores from Figure 7.

Calibration setting	Prompts	Calibration cost	Quality and sparsity evidence
Default budget	64	89.6 H100 GPU-hours	Default calibration set size used in main experiments
Restricted budget	16	13.7 H100 GPU-hours	Main text reports minimal impact; further configuration details are in supplementary B.2

These figures do not establish that reducing prompt count alone always scales calibration cost by the same factor. Details of the restricted-budget setup are deferred to B.2, which is absent from the local cache. The main text also refers to a two-component ablation in B.4 without including its results, so the available material cannot quantitatively isolate each execution path's contribution.

Key Findings

• Across the five evaluated settings, CalibAtt has lower generation latency than dense FA3 and comparable or slightly higher VBench Total. The small score differences should not be presented as statistically significant quality improvements.
• Mochi 1 combines 69.1% sparsity with only a 1.16x speedup. Other model computation and actual kernel behavior constrain the overall benefit, making sparsity alone insufficient for predicting latency.
• Few-step generation still benefits: LightX2V at 720p falls from 48.3 to 30.6 seconds. This supports the practical value of avoiding online selection, but does not show that offline calibration pays for itself under a small request volume.

Highlights & Insights

• The important distinction is between stability across inputs and uniformity across the model. The former permits offline selection, while the latter would ignore heterogeneity across layers, heads, and timesteps.
• Spatial repetition addresses cases where block sparsification is less effective. Even a relatively dense attention map can admit fewer query computations when spatial rows have similar distributions.
• Calibration targets FlashAttention blocks and contiguous-interval encoding rather than merely reporting elementwise sparsity. This connects measured redundancy to GPU operations that the implementation can actually omit.

Limitations & Future Work

• The authors explicitly acknowledge that fixed offline masks can miss prompt-specific sparsity. Stability across calibration inputs is an empirical observation, not a quality guarantee under arbitrary prompts or distribution shifts.
• Upfront computation is substantial: default Wan 720p calibration requires 89.6 H100 GPU-hours. Deployment value depends on enough subsequent requests to amortize this cost, not just on lower latency for one generation.
• The authors report 21.5 GB of additional mask memory for Wan 720p, reducible to 3.6 GB with minimal impact. The implementation details are in supplementary A.2, absent from the cache, so this note does not speculate about the compression mechanism.
• Evaluation is limited to one H100 and specified models, resolutions, and frame counts. Gains on other hardware, longer videos, or new inference configurations require independent validation; similar VBench scores also do not imply same-seed reconstruction equivalence.
• The paper proposes exploiting correlations across timesteps, compacting skip lists, and using more granular threshold parameterizations. These are future directions rather than extra contributions demonstrated in the main experiments.

Related Work & Insights

• FlashAttention3: FA3 improves the execution efficiency of dense attention; CalibAtt builds on it to reduce the blocks or query rows actually computed. The methods are complementary rather than mutually exclusive replacements.
• RadialAttention: It uses spatial windows that shrink with temporal distance. CalibAtt instead calibrates patterns for individual layers, heads, and timesteps, relying less on one predefined geometric prior.
• SpargeAttention and SVG2: The former estimates block importance online, while the latter uses token clustering and inter-cluster interactions to select computation. CalibAtt moves selection offline at the cost of reduced input-specific adaptation.
• LiteAttention: It reuses online skip lists across timesteps, whereas CalibAtt builds a timestep-indexed dictionary offline. Reuse across time and reuse across inputs exploit different sources of redundancy.

Rating

These are the note author's subjective assessments, not experimental metrics reported by the paper.

• Novelty: 4/5. The combination of offline per-layer and per-head calibration with spatial repetition is clear, while building on existing sparse-attention observations.
• Experimental Thoroughness: 4/5. Evaluation covers two architectures, multiple resolutions, and few-step distilled generation, but the local material lacks supplementary component ablations and cross-hardware validation.
• Writing Quality: 4/5. Observations, design choices, and execution paths connect coherently; corrupted equations in the cached extraction should be distinguished from issues with the paper itself.
• Value: 4/5. Useful for video-generation deployments with sufficient memory and enough requests to amortize calibration, rather than a universal acceleration switch without preprocessing costs.