Accelerating Diffusion Models via Equal-Risk Caching¶

Conference: ECCV2026
Official Paper: 5300
Paper: Paper PDF
Authors: Can Zhang and Liangshun Zou
Area: Diffusion Inference Acceleration
Keywords: feature caching, equal-risk scheduling, offline risk probing, cumulative error surrogate, image and video generation

TL;DR¶

ERC probes how a single cache-window length affects final outputs at different timesteps, accumulates the resulting one-dimensional risk curve, and partitions it to allocate refreshes at lower offline cost than exhaustive interval evaluation, achieving, for example, 3.23x acceleration and LPIPS 0.1954 on FLUX.1 [dev] with a refresh budget of 15.

Background & Motivation¶

Diffusion generation repeatedly evaluates an expensive denoising network, even though intermediate features often change slowly between adjacent steps. Feature caching exploits this redundancy without necessarily removing sampling steps: some steps fully recompute the network, while others reuse earlier features. The consequential decision is where fresh computation is indispensable. Reusing features for too long in a sensitive phase lets deviations propagate through the sampling trajectory; refreshing too often in a stable phase wastes the available computation budget.

A fixed-interval schedule treats all timesteps as equally sensitive. Online methods such as TeaCache and DiCache use local signals to decide when to refresh, but small instantaneous changes need not imply a small final deviation after an entire reuse segment. LeMiCa instead measures final-output errors for different segment starts and lengths and searches for a global schedule. That stronger view comes with the cost of constructing a two-dimensional interval-risk map across candidate starting positions and cache lengths.

ERC observes that measured interval errors correlate with a quantity obtained by accumulating a one-dimensional temporal risk signal. This suggests probing a single cache length to identify sensitive regions, then assigning more refreshes where accumulated risk grows quickly. Core idea: replace exhaustive interval-risk evaluation with a cumulative surrogate built from fixed-length probes, and divide the refresh budget evenly along the risk axis rather than the time axis.

Method¶

Overall Architecture¶

ERC takes a pretrained diffusion model, a sampling configuration, a small offline prompt set, a probe length, and a refresh budget, and returns timesteps for full forward computation. Offline, it first generates uncached reference outputs, moves a fixed-length caching window across the trajectory, and measures the resulting final-output deviations. It then accumulates the measured risks and locates equal-risk boundaries. Online generation follows the resulting schedule: refresh steps update the cache, and intervening steps reuse it, without repeating the entire profiling procedure for every new prompt.

The scheduling index follows execution order, from 0 to the last sampling step. It should not be confused with a diffusion-time variable that runs backward from noise to data. ERC changes when the heavy network is evaluated; it neither trains a new denoiser nor simply replaces 50 sampling steps with 15 sampling steps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Model and offline prompts<br/>Probe length and refresh budget"] --> B["Fixed-Length Probing"]
    B --> C["Cumulative Risk Surrogate"]
    C --> D["Equal-Risk Budget Partitioning"]
    D --> E["Decoupled Scheduling and Reuse"]
    E -->|Refresh step| F["Full computation and cache update"]
    E -->|Cache step| G["Residual reuse or DCTA"]
    F --> H["Advance sampling and generate output"]
    G --> H

Key Designs¶

1. Fixed-Length Probing: measure temporal sensitivity through final-output deviation

Start with a reference generation that uses no caching. For a selected starting timestep, enable caching only within one consecutive segment, keep the remaining steps fully computed, and compare the resulting final output with the reference using LPIPS. The measured quantity is not an intermediate-feature change or a current-step prediction error. It measures the perceptual effect of the caching intervention after subsequent denoising has propagated that intervention to the final output. Even a short probe window therefore observes an end-to-end consequence.

An exhaustive risk map repeats this experiment for every candidate start and length. ERC fixes the length and varies only the starting position, treating each measurement as a temporal risk signal. Applying the same window length everywhere makes sensitivity across positions comparable. Here, ground-truth risk means a directly measured deviation from the original model's output, not agreement with a real photograph or a human preference label. A probe length of 5 works best in the reported FLUX ablation, but it is an empirical setting rather than a universal constant for all models and samplers.

2. Cumulative Risk Surrogate: compress expensive interval evaluations into an additive temporal signal

Let \(x_0\) denote the uncached output and \(x_{t,l}\) the final output when caching is used for \(l\) consecutive steps starting at \(t\). The essential relationships in the paper's Equations (4)-(6) can be written together as:

\[ \mathcal{E}(t,l)=\operatorname{LPIPS}(x_{t,l},x_0),\qquad r(t)=\mathcal{E}(t,l'),\qquad H(t,l)=\sum_{u=t}^{t+l-1}r(u). \]

Once the probe length \(l'\) is fixed, scoring another candidate interval does not require another output-generation experiment. Instead, ERC sums the temporal risks within that interval. This replaces the question of how many steps have elapsed since refreshing with how much sensitive territory has been traversed. Two equally long intervals can have very different accumulated risks if one crosses a sensitive phase. Prefix sums also let the interval score be obtained by subtracting cumulative values at its endpoints rather than rescanning every position.

The underlying assumption is empirical correlation between this accumulated surrogate and measured interval error, not equality of their numerical values. FLUX scatter plots and probe-length ablations support the relationship, but the paper does not establish additivity of true perceptual error or a rigorous error bound for a complete multi-segment caching schedule. The reduction from \(O(TL)\) to \(O(T)\) should be read as a reduction in probe evaluations: with the offline prompt count held fixed, profiling every start-length combination becomes probing each start at one length. Each probe still involves generating an output, so this is not a claim that the whole calibration costs a single ordinary sampling run. Nor is the original scheme necessarily quadratic in \(T\) unless the number of candidate lengths \(L\) grows with \(T\).

3. Equal-Risk Budget Partitioning: place refreshes according to risk rather than fixed spacing

The initial step must be fully computed, so cumulative risk starts at zero before step 1 and excludes a caching risk for step 0. In Algorithm 1, budget \(B\) is the number of returned refresh boundaries: the first is fixed at 0 and the last at \(T-1\). After reserving the initial refresh, the remaining budget determines equally spaced risk thresholds. Writing \(R(t)\) for the risk prefix sum before timestep \(t\), the central rule is:

\[ \Delta R=\frac{R(T)}{B-1},\qquad t_i=\min\{t\mid R(t)\geq i\Delta R\}. \]

Conceptually, place equally spaced marks on the cumulative-risk axis, then map those marks back to timesteps. A steep portion of the curve reaches the next threshold quickly and receives dense refreshes. A flat portion takes more steps to reach that threshold and permits a longer reuse interval. Changing the refresh budget only changes the thresholds and boundary lookup; it does not require measuring every possible interval again.

Because timesteps are discrete, the lookup selects the first position that crosses a threshold, so segment risks are only approximately equal. The algorithm also explicitly fixes the final boundary to \(T-1\); raw threshold inversions should not be mistaken for a complete implementation without endpoint handling. The text does not elaborate on degenerate cases such as several thresholds selecting the same timestep or zero total risk. Those cases need an explicit implementation policy. Most importantly, the partition equalizes surrogate mass, not a guaranteed upper limit on each segment's true LPIPS, and it does not prove global optimality of the selected refresh set.

4. Decoupled Scheduling and Reuse: a refresh set can be paired with different cache rules

Selecting refresh timesteps leaves another decision: how should a cache step use old features? By default, ERC follows LeMiCa and reuses the residual cache from the latest full computation. More elaborate reuse rules can extrapolate feature changes using the two most recent fully computed steps. On Qwen-Image and Wan2.1, the paper also evaluates DCTA, dynamic cache trajectory alignment from DiCache. ERC does not introduce DCTA, and DCTA is not a required component of its risk-curve construction.

This distinction matters when attributing experimental gains. Comparing default ERC and default LeMiCa more directly compares their scheduling choices. Comparing ERC with ERC+DCTA instead exposes the additional effect of the reuse strategy under the same schedule. Improvements from the combined system cannot all be credited to equal-risk partitioning. Likewise, the paper does not establish that a risk curve profiled under one reuse rule can be transferred unchanged to every other caching mechanism.

A Worked Example¶

Consider the reported FLUX setup with 50 original sampling steps and probe length 5. The 10-prompt configuration in the offline-prompt ablation reproduces the main-table results. Move the five-step caching window along the trajectory, measure its effect on the final image, and accumulate the resulting temporal risk curve. For \(B=15\), reserve the initial refresh, use \(R(T)/14\) as the risk spacing, and fix the final step as a refresh.

During generation, a scheduled refresh executes the full network and replaces the cache. Other timesteps still advance sampling but reuse the stored residual. Sensitive phases thus receive new features more frequently, while stable phases reuse them longer. The reported latency drops from 15.60 seconds to 4.83 seconds. The paper's main text does not list all 15 refresh indices, so a concrete index sequence cannot be supplied from this evidence.

Loss & Training¶

ERC requires no training or fine-tuning and adds no training loss. LPIPS is an offline measurement used to construct the risk profile, not a differentiable objective used to update model weights. The principal choices are probe length, offline prompt count, refresh budget, and cache reuse rule. A risk curve is associated with the model and sampling configuration used to obtain it; unrestricted transfer across configurations is not a validated guarantee.

Key Experimental Results¶

Main Results¶

Image experiments use 200 DrawBench prompts. Video experiments randomly select 200 VBench prompts and generate 81 frames at 480x832 resolution. Original inference uses 50 sampling steps by default, with FlashAttention-2 enabled. FLUX and Qwen-Image run on RTX PRO 6000 96GB GPUs, while Wan runs on RTX 4090D 24GB GPUs. FLUX disables the CFG unconditional branch by default; the other two models retain it. Absolute latencies across models therefore should not be interpreted as a controlled comparison of architectural efficiency.

Lower LPIPS and higher SSIM/PSNR indicate greater consistency with the uncached output. They do not independently establish better generation quality or prompt adherence. The following selection comes from Table 1; comparisons within each group use the same refresh budget and resolution.

Model / Budget	Method	LPIPS	SSIM	PSNR	Latency (s)	Speedup
FLUX 1024x1024 / 15	LeMiCa	0.1857	0.8637	24.9852	4.83	3.23x
FLUX 1024x1024 / 15	ERC	0.1954	0.8628	25.4227	4.83	3.23x
FLUX 1024x1024 / 26	LeMiCa	0.0886	0.9354	30.1025	8.21	1.90x
FLUX 1024x1024 / 26	ERC	0.0816	0.9442	31.5304	8.21	1.90x
Qwen 1664x928 / 17	LeMiCa	0.1506	0.9234	27.9708	15.94	2.84x
Qwen 1664x928 / 17	ERC	0.1268	0.9316	27.6859	15.94	2.84x
Qwen 1664x928 / 17	LeMiCa+DCTA	0.1194	0.9344	28.5831	15.94	2.84x
Qwen 1664x928 / 17	ERC+DCTA	0.0842	0.9477	28.9992	15.94	2.84x

The video results below come from Table 2. VBench provides a composite video evaluation; matching the original score does not mean human reviewers judged the videos identical.

Wan2.1-T2V-1.3B / Budget 17	LPIPS	SSIM	PSNR	VBench (%)	Latency (s)	Speedup
Original	N/A	N/A	N/A	74.80	276.58	1.00x
LeMiCa	0.1622	0.8912	29.5242	73.98	101.73	2.72x
ERC	0.1642	0.8890	29.3016	74.08	101.73	2.72x
ERC+DCTA	0.1242	0.9179	30.6336	74.80	101.73	2.72x

Ablation Study¶

The following rows reproduce the probe-length ablation from Table 3 on FLUX. The two budgets have fixed speedups of 3.23x and 1.90x, respectively; quality differences across budgets should not be attributed to probe length alone.

Probe Length	B=15 LPIPS	B=15 SSIM	B=15 PSNR	B=26 LPIPS	B=26 SSIM	B=26 PSNR
1	0.2333	0.8475	25.2465	0.1199	0.9279	30.8135
3	0.2117	0.8547	25.3649	0.0960	0.9385	31.2367
5	0.1954	0.8628	25.4227	0.0816	0.9442	31.5304
7	0.1990	0.8603	25.0275	0.0891	0.9400	31.1941
9	0.2082	0.8486	24.3353	0.0920	0.9361	30.6320
11	0.2186	0.8415	24.0448	0.0997	0.9284	29.5831

Key Findings¶

Length 5 is best on all three metrics at both budgets. The authors suggest that short windows can underestimate nonlinear accumulated error, while long windows mix regions with different sensitivities. This is an empirical explanation, not a theoretical error-expansion result.
In Table 4 at B=15, increasing the offline prompt count from 1 to 10 reduces LPIPS from 0.2350 to 0.1954; 50 prompts yield 0.1926. A small representative set is already useful, but improvements are not strictly monotonic across every metric.
On the 200-prompt GenAI OOD set in Table 5, ERC/LeMiCa LPIPS is 0.0822/0.0921 at B=26 but 0.1988/0.1960 at B=15. This supports some distribution robustness, not dominance at every budget.
Table 6 uses Qwen with DPM-Solver++ 2M and 30 steps. ERC obtains LPIPS 0.0694 at 1.95x speedup, compared with MagCache's 0.0708 at 1.95x. All methods in this setting directly reuse residuals without DCTA. The authors state that they could not reliably reproduce LeMiCa here because its DAG construction code was not released.

Highlights & Insights¶

A single reusable risk curve compresses the information needed for scheduling. Changing the computation budget no longer entails rebuilding a two-dimensional error map, which is attractive for serving multiple quality tiers.
Probes observe final-output consequences while retaining only a low-dimensional temporal structure. This occupies a clear middle ground between immediate local thresholds and exhaustive interval evaluation.
Separating the refresh budget from the reuse strategy makes performance attribution clearer. The transferable idea is to test whether a task's risk can be approximated by a low-dimensional cumulative quantity before adopting budget partitioning, rather than copying a particular refresh interval.

Limitations & Future Work¶

The surrogate depends on empirical correlation. The text supplies no proof of true-error additivity, final-error bounds, or optimality. Strong nonlinear phases and interactions between caching segments could break the approximation; online deviation monitoring is a plausible extension.
Offline profiling still requires generating probe outputs. The paper reports an asymptotic reduction in evaluated combinations but not calibration wall-clock time, peak resource use, or the number of serving requests needed to amortize calibration.
Evaluation covers three models, prompt-count sensitivity, OOD prompts, and a few-step setting. However, full probe-length robustness is not established for every model, and repeated-run variation or confidence intervals are not reported. Very small metric differences should not be treated as statistically established advantages.
Most metrics measure consistency with the original output. VBench adds video-quality evaluation, but systematic human preference, prompt-following, and longer-video tests remain absent. Reproduction also needs explicit details for offline prompt aggregation, probe windows near the trajectory endpoint, and duplicate discrete threshold hits.

Versus LeMiCa: LeMiCa explicitly evaluates intervals and performs graph search; ERC substitutes a cumulative surrogate and equal-risk partitioning for expensive risk-map construction. ERC needs fewer offline probes, but does not beat every quality metric at every budget.
Versus TeaCache and DiCache: These approaches use online signals to assess whether caching is appropriate, whereas ERC precomputes a global schedule. DCTA comes from DiCache and can serve as ERC's reuse component; competing scheduling approaches can still have compatible components.
Versus TaylorSeer: TaylorSeer emphasizes extrapolating feature evolution, while ERC emphasizes where full computation occurs. Joint calibration is worth investigating rather than assuming different extrapolation rules share an identical risk distribution.
Versus few-step sampling: DPM-Solver++ and related samplers reduce the steps needed for numerical integration; ERC reduces computation within retained steps. The 30-step experiment shows that the approaches can be combined, but does not establish the same caching opportunity for extremely few-step distilled models.

Rating¶

Novelty: 4/5. Cumulative one-dimensional risk and equal-mass partitioning provide a simple alternative for global scheduling, but rely on an empirical surrogate without a rigorous optimization guarantee.
Experimental Thoroughness: 4/5. Image, video, probe-length, prompt-count, OOD, and few-step evaluations are reasonably broad; measured calibration costs and statistical uncertainty are missing.
Writing Quality: 4/5. The motivation, algorithm, and comparison structure are clear, while budget notation, boundary handling, and some reproduction details need clarification.
Value: 4/5. Training-free multi-budget scheduling is a practical way to simplify expensive offline cache planning, provided calibration cost is included in the deployment assessment.