Accelerating Diffusion Transformers with Gaussian Process Rectified Feature Cache¶
Conference: ECCV2026
Paper: ECCV official paper page · Paper PDF
Code: https://github.com/Aredstone/GP-Refiner
Area: Image Generation
Keywords: Diffusion Transformer, feature caching, Gaussian process regression, online rectification, uncertainty-aware scheduling
TL;DR¶
GP-Refiner treats full-compute features on an accelerated trajectory as noisy observations, corrects cache predictions with an online Gaussian process, and triggers recomputation through posterior uncertainty; on Qwen-Image, it raises PSNR from 28.58 to 29.48 while reducing computation relative to a fixed-interval TaylorSeer configuration.
Background & Motivation¶
Diffusion Transformers are expensive because a substantial network must run repeatedly throughout sampling. Feature caching reduces this cost by computing intermediate features at selected steps and reusing or forecasting them elsewhere. TaylorSeer already goes beyond copying the latest feature: it extrapolates local feature dynamics using historical finite differences, supporting longer skip intervals than direct reuse. However, a better local forecast does not guarantee a correct trajectory: approximate features change subsequent latent states, so even a later full network evaluation produces features on a drifted trajectory rather than those of an unaccelerated run.
This creates a supervision problem for online correction. Running an unaccelerated reference alongside inference would undermine the intended savings, while treating accelerated full-compute features as exact reference targets ignores accumulated drift. The paper instead studies the discrepancy statistically and observes approximately zero-mean Gaussian residuals locally, motivating the use of available full-compute features as noisy observations rather than perfect labels. Core idea: retain the existing cache predictor, use a Gaussian process to rectify its outputs from a small online observation history, and use the same model's posterior variance to decide when full computation is necessary.
Method¶
Overall Architecture¶
The system still receives text conditioning and diffusion latents and ultimately generates an image; GP-Refiner changes how intermediate features are obtained, not the text encoder or generation objective. At full-compute steps, it updates the base cache predictor and performs Noisy Observation Construction to collect local prediction–observation pairs. At candidate skipped steps, the base predictor proposes a feature, and Gaussian Process Rectification produces both a corrected mean and posterior variance. Uncertainty-Gated Recomputation then accepts the corrected feature or executes a full forward pass and feeds the resulting observation back into the online history. Correction and scheduling therefore share one observation set and probabilistic model instead of relying on unrelated rules.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Text conditioning and latents<br/>Base cache predictor"] --> B["Noisy Observation<br/>Construction"]
B --> C["Gaussian Process<br/>Rectification"]
C --> D["Uncertainty-Gated<br/>Recomputation"]
D -->|"Low variance: accept corrected feature"| E["Continue denoising<br/>and generate image"]
D -->|"High variance: full forward and update observations"| B
Key Designs¶
1. Noisy Observation Construction: do not confuse full computation with a reference-trajectory target
Three feature types must be distinguished: features from an entirely unaccelerated reference trajectory, features obtained by a full forward pass on the accelerated trajectory, and features forecast by the cache predictor. The first two can differ at the same timestep because their input latents differ; the third also introduces forecasting approximation. The authors examine residuals between the first two using histograms over vector elements and Q–Q plots of random projections across residual vectors. Section 3.2 reports an average correlation coefficient above 0.98 over 100 random projections, with a minimum of 0.9424, supporting a local approximate Gaussian model rather than proving it for every input and timestep.
During actual accelerated inference, reference features are unavailable, so only full-compute features on the current trajectory are collected. After each full forward pass, the base predictor's state is updated; the updated predictor then generates an adjacent-step prediction, paired with the current full-compute feature. Section 3.3 explicitly uses adjacent prediction and observation indices, rather than ordinary same-step prediction–ground-truth supervision. The intended mapping combines one-step evolution and correction, keeping the regression local instead of fitting all nonlinear drift accumulated across a long skipped interval. Some equations in the cached text are malformed, so this note retains the supported adjacent-pair mechanism without reconstructing an exact indexing implementation.
2. Gaussian Process Rectification: regress in predicted-feature space using a short history
The Gaussian process takes the base predictor's features as inputs, not merely timestep coordinates, and uses the collected full-compute features as noisy target observations. It is therefore not simply another high-order polynomial fitted along the time axis: its estimate depends on how the current predicted feature relates to historical predictions in feature space. The base method remains responsible for producing an inexpensive candidate, while GP-Refiner uses nearby observed evidence to rectify that candidate. This separation permits TaylorSeer, HiCache, and ClusCa to retain different internal prediction mechanisms while sharing the same external correction framework.
The implementation uses an angular RBF kernel and a first-in, first-out window containing the 10 most recent feature observations. The kernel matrix represents similarities among historical predictions, while an observation-noise term prevents regression from having to interpolate every already-biased observation exactly. For a current candidate, the model returns a predictive mean as the corrected feature and a posterior variance as its uncertainty estimate. A bounded window limits history size and keeps attention on the current denoising regime, but it does not make high-dimensional feature storage, kernel operations, or data movement literally free. The fixed-interval ablation helps isolate this contribution: correction improves agreement with the unaccelerated output even without changing how frequently the network is evaluated.
3. Uncertainty-Gated Recomputation: fall back to the network when historical support is inadequate
A corrected mean alone can still extrapolate poorly outside the region supported by recent observations, so the system also needs a criterion for rejecting approximation. Section 3.4 compares posterior variance against a preset threshold \(l\): if \(\sigma_t^2 > l\), it executes full computation; otherwise, it uses the Gaussian process mean. The resulting full-compute feature supplies the current step without cache approximation and updates both the base predictor and the observation history. This recalibration does not imply a return to the exact unaccelerated trajectory: the newly computed feature is still evaluated on the current latent state and remains a noisy observation of the reference.
Unlike a fixed recomputation period, the policy tries to spend computation where the regression model is currently uncertain. Posterior variance expresses uncertainty conditional on the chosen kernel, noise model, and observations; it is neither directly measured image error nor a formal guarantee of generation quality. The authors state that the threshold is robust within a range but refer its selection details to an appendix not present in the available local main paper, so no threshold value, search range, or threshold-ablation result is inferred here.
A Worked Example¶
Consider an intermediate stage of generating a text-conditioned image with TaylorSeer. A full-compute step obtains a new feature, updates TaylorSeer's feature and finite-difference history, and contributes a local paired observation to GP-Refiner. At the next candidate skipped step, TaylorSeer extrapolates a feature, and the Gaussian process queries the recent window to produce a corrected estimate and uncertainty. Low uncertainty permits continued denoising with that estimate; high uncertainty rejects it, runs the full network, and updates the window. This example illustrates state updates rather than asserting that a particular prompt must trigger a particular number of recomputations.
Loss & Training¶
GP-Refiner is an inference-time plugin: the paper does not retrain the diffusion Transformer or require an offline collection of reference trajectories to train a correction network. “Training-free” means no offline fine-tuning of the generator, not an absence of online regression or matrix operations. The experiments specify the angular RBF kernel and a FIFO window of 10 observations, but the main text does not fully specify kernel hyperparameters, noise parameters, or initialization. These implementation details cannot responsibly be filled in from generic knowledge of Gaussian processes.
Key Experimental Results¶
Main Results¶
The selected results below come from Tables 1 and 2, evaluated on 200 DrawBench prompts with quality metrics averaged against unaccelerated reference images. FLUX.1-dev and Qwen-Image both use 50-step reference runs; this is not reconstruction against real photographs or an evaluation of human preference or text alignment. Higher PSNR and lower LPIPS indicate better reference agreement; lower latency and FLOPs indicate greater efficiency, and absolute latency should not be compared across different generators.
| Model / configuration | Latency (s) ↓ | FLOPs (T) ↓ | PSNR ↑ | LPIPS ↓ |
|---|---|---|---|---|
| FLUX: TaylorSeer, \(O=2,N=7\) | 3.60 | 670.44 | 28.671 | 0.4542 |
| FLUX: TaylorSeer + GP-Refiner, dynamic intervals | 3.42 | 643.13 | 29.550 | 0.3497 |
| Qwen-Image: TaylorSeer, \(N=6\) | 9.71 | 2583.97 | 28.58 | 0.46 |
| Qwen-Image: TaylorSeer, \(N=7\) | 8.99 | 2323.30 | 28.20 | 0.65 |
| Qwen-Image: TaylorSeer + GP-Refiner, dynamic intervals | 8.37 | 2085.26 | 29.48 | 0.29 |
Here \(N\) is the fixed caching interval and \(O\) the Taylor extrapolation order; the complete plugin uses dynamic scheduling, so these are not strictly matched-compute comparisons. Against Qwen-Image's \(N=6\) baseline, computation falls by approximately 19.3%, PSNR increases by 0.90, and LPIPS falls from 0.46 to 0.29; against \(N=7\), the PSNR gain is 1.28, and those baselines must not be mixed. Table 2 also reports the 8-step Qwen-Image-Lightning setting: at fixed \(N=3\), PSNR improves from 30.064 to 31.204 and LPIPS from 0.2962 to 0.1922, but latency rises from 4.56 to 4.69 seconds. The low-step result supports improved reference agreement, not the claim that every configuration also runs faster.
Ablation Study¶
This selection from Table 3 uses FLUX.1-dev with the same 200-prompt DrawBench evaluation and 50-step reference. The fixed-\(N=7\) pair isolates correction, whereas the final pair compares dynamic scheduling alone with the complete plugin at different compute budgets.
| Configuration | FLOPs (T) ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| TaylorSeer, \(O=2,N=7\) | 670.44 | 28.671 | 0.6237 | 0.4542 |
| Fixed \(N=7\) + GP rectification | 670.45 | 29.266 | 0.6622 | 0.4151 |
| TaylorSeer + Only Dynamic | 650.21 | 28.930 | 0.6604 | 0.3819 |
| Complete GP-Refiner | 643.13 | 29.550 | 0.6972 | 0.3497 |
Key Findings¶
- At fixed intervals, PSNR increases by 0.595 while FLOPs change only from 670.44T to 670.45T, supporting a genuine correction effect rather than improvement from more network evaluations.
- The complete plugin adds 0.620 PSNR over Only Dynamic, suggesting that timing alone cannot replace feature correction; however, the scheduling signals also differ, so this is not a perfectly isolated single-variable comparison.
- FLOPs reduction and wall-clock speedup are different: Table 1 reports a 5.78× FLOPs speedup but a 3.38× latency speedup for the complete FLUX configuration.
Highlights & Insights¶
- Replacing unavailable ideal supervision with available noisy observations is the central conceptual move. It acknowledges trajectory drift rather than pretending that one full evaluation restores the reference state.
- The regression mean supplies a feature, while posterior variance decides whether approximation should continue. Both decisions use the same local history and statistical model.
- Gains under fixed caching frequency are particularly informative evidence. They distinguish improved approximation from benefits purchased with extra network computation.
Limitations & Future Work¶
- Problem explicitly discussed by the authors: a full forward pass still cannot provide reference-trajectory ground truth online, so the method relies on noisy-observation modeling and mitigates rather than provably eliminates long-interval drift.
- Reader assessment: Gaussian-looking random projections do not establish independent, identically distributed or isotropic noise, nor do they validate the same model for every prompt and denoising stage.
- Evaluation scope: agreement with unaccelerated images is not equivalent to aesthetic quality, text adherence, or temporal consistency in video generation.
- Reproducibility gap: the available main paper omits the referenced appendix's threshold-selection details and does not fully state hardware and resolution conditions for latency reproduction; reported seconds are not universal deployment estimates.
- Internal reporting issue: Section 4.3 labels both 77 images and 119 images as requiring 8 evaluations, and its later prose disagrees with Figure 5 about step counts; no precise NFE distribution or mean is inferred from these inconsistent statements.
- Concrete next checks: threshold sensitivity, calibration between variance and actual error, and additional latency and memory overhead on fixed hardware should be established before claiming broader deployment generality.
Related Work & Insights¶
- vs TaylorSeer: TaylorSeer extrapolates temporal evolution through local finite differences; GP-Refiner preserves that predictor and rectifies its outputs using online feature observations.
- vs HiCache and ClusCa: these offer alternative caching backbones that can receive the plugin. The HiCache results in Table 1 do not improve every metric against every fixed-interval baseline, so universal dominance would be an overstatement.
- vs SpeCa and DiCache: these control drift through verification or probing signals; GP-Refiner emphasizes nonlinear regression from noisy observations and posterior-uncertainty-triggered recomputation, without proving that its signal is always more reliable.
- vs reducing sampling steps: caching retains approximate denoising updates while avoiding some expensive network computations; step reduction changes the sampling discretization itself, producing a different compute path and error source.
Rating¶
- Novelty: 4/5. A well-motivated combination of noisy observations, GP rectification, and uncertainty scheduling, built from established regression tools.
- Experimental Thoroughness: 3/5. Multiple backbones, a low-step generator, and fixed-interval ablations are useful, but reproduction details and broader quality measures remain limited.
- Writing Quality: 3/5. The supervision dilemma is clear, while indexing, statistical reporting, and equation rendering complicate exact reproduction.
- Value: 4/5. A relevant inference-time design for existing diffusion-model caches, with deployment benefits requiring environment-specific validation.