LinCa: Accelerating Diffusion Models via Learnable Decomposed Feature Caching

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/QHR69/LinCa
Area: Diffusion Model / Model Compression / LLM Efficiency
Keywords: Diffusion Acceleration, Feature Caching, Invertible Neural Network, Differentiated Polynomial Extrapolation, Diffusion Transformer

TL;DR

Addressing the cumulative degradation caused by cross-timestep and cross-dimension dynamics mismatch in existing feature caching, LinCa introduces a lightweight learnable invertible network to decompose cached features into sub-components with distinct temporal continuity and applies differentiated-order polynomial extrapolation, achieving 5-7× near-lossless acceleration on FLUX, Qwen-Image, and HunyuanVideo with under 0.2% additional parameters.

Background & Motivation

Diffusion models based on Diffusion Transformers (DiTs) have established state-of-the-art visual synthesis across text-to-image, image editing, and video generation. However, their iterative denoising sampling process necessitates multi-step sequential forward evaluations, resulting in prohibitive computational latency and high memory costs that severely constrain real-world deployment. To reduce the execution cost of each forward pass, feature caching has emerged as an appealing paradigm that avoids modifying the underlying model architecture. Typically, feature caching executes a full forward evaluation once every \(N\) steps to cache intermediate representations, and directly reuses or forecasts intermediate features across the intermediate \(N-1\) steps to bypass redundant deep layer computation.

Nevertheless, existing feature caching schemes encounter two fundamental bottlenecks: cross-timestep/cross-model dynamics mismatch and cross-dimension dynamics mismatch. On one hand, prominent approaches (such as FORA, ToCa, and TaylorSeer) implicitly presuppose that hidden representations follow uniform evolution characteristics across the entire denoising process, applying a static caching schedule across all timesteps and varied backbones. In practice, denoising trajectories exhibit heterogeneous continuity: early, middle, and late stages showcase vastly different temporal derivatives, transitioning from smooth continuous progressions to abrupt mutations where uniform caching inevitably accumulates severe estimation drift. On the other hand, at a fine-grained dimensional level, different channels within identical hidden tensors display divergent continuity dynamics—some dimensions evolve smoothly and suit high-order polynomial extrapolation, whereas others fluctuate with sharp fluctuations and only tolerate zeroth-order reuse. Crucially, these dimensions are intricately interleaved in the native feature space, rendering heuristic slicing ineffective.

Applying a uniform, single-order extrapolation strategy across all interleaved dimensions triggers catastrophic quality degradation under aggressive acceleration ratios. Core idea: employ a lightweight learnable invertible network to decompose interleaved features into sub-components with distinct continuity properties, apply differentiated-order polynomial extrapolation matched to each sub-space, and leverage mathematically exact invertible reconstruction and per-segment scheduling to realize high-speed, near-lossless feature prediction.

Method

Overall Architecture

LinCa introduces an end-to-end Decompose-Predict-Reconstruct feature caching pipeline. During computation timesteps, the model executes a standard forward pass, and the cumulative residual features at the final layer are cached. During prediction timesteps, LinCa projects the historical cached features through a lightweight learnable invertible network, splitting them along the channel dimension into \(M\) sub-components exhibiting distinct continuity characteristics. Differentiated-order Hermite polynomial extrapolation or direct reuse is then executed according to each sub-component's dynamics. Finally, all predicted sub-components are concatenated and losslessly mapped back to the native feature space via the analytical inverse of the invertible network.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Historical Cached Residual Features x_t"] --> B["Invertible Decomposition Mapping"]
    B --> C["Differentiated-Order Polynomial Prediction"]
    C --> D["Lossless Reconstruction & Per-Segment Adaptation"]
    D --> E["Output Target Predicted Features x_hat"]

Key Designs

1. Invertible Decomposition Mapping: Channel Disentanglement via Additive Coupling To separate interleaved dimensions with disparate continuity properties from the original feature manifold, LinCa formulates a lightweight Learnable Invertible Network \(\mathcal{E}_\theta\). The architecture comprises \(L\) cascaded reversible blocks. Each block integrates two strictly invertible transformations: an invertible \(1 \times 1\) convolution parameterized by an orthogonal matrix \(W\) for global channel mixing, followed by an additive coupling layer that partitions the features evenly into \(u_1, u_2\) along the channel dimension and executes non-linear transformations using lightweight MLPs \(F\) and \(G\): $\(v_1 = u_1 + F(u_2), \quad v_2 = u_2 + G(v_1)\)$ During backward reconstruction, exact inversion is computed by subtraction: \(u_2 = v_2 - G(v_1)\) and \(u_1 = v_1 - F(u_2)\), followed by multiplication with the inverse orthogonal matrix \(W^{-1}\). Because every layer maintains mathematical invertibility (\(\mathcal{E}_\theta^{-1} \circ \mathcal{E}_\theta = I\)), the mapping avoids the irreversible information loss inherent in conventional feed-forward autoencoders or MLPs.

2. Differentiated-Order Polynomial Prediction: Sub-Space Dynamics Matching Projected latent features \(z_t = \mathcal{E}_\theta(x_t)\) are partitioned along the feature dimension into \(M\) sub-components (with \(M=3\) adopted in practice): \(z_t = [z_t^{(0)} \mid z_t^{(1)} \mid \dots \mid z_t^{(M-1)}]\). Tailored prediction policies are assigned to match the continuity of each sub-component: - For the zeroth-order sub-component \(z_t^{(0)}\) (comprising unstable dimensions with weak temporal continuity and sharp mutations), direct nearest-cache reuse is applied: \(\hat{z}_t^{(0)} = z_{t_{\text{prev}}}^{(0)}\), avoiding the severe numerical divergence caused by high-order fitting; - For higher-order sub-components \(z_t^{(m)}\) (\(m \ge 1\)), which exhibit stable temporal continuity, \(m\)-th order Hermite polynomial extrapolation based on historical discrete differences is performed: \(\hat{z}_{t-k}^{(m)} = z_t^{(m)} + \sum_{i=1}^m \alpha_i(k) \Delta^i z_t^{(m)}\), reconstructing high-frequency details with negligible closed-form computational overhead.

3. Lossless Reconstruction & Per-Segment Adaptation: Multi-Stage Denoising Optimization Once all sub-components are extrapolated, they are concatenated and restored back to the original feature dimension through the analytical inverse mapping: \(\hat{x}_t = \mathcal{E}_\theta^{-1}([\hat{z}_t^{(0)} \mid \hat{z}_t^{(1)} \mid \dots \mid \hat{z}_t^{(M-1)}])\). Because diffusion denoising trajectories display macro-level dynamics shifts across different denoising phases, LinCa partitions the total sampling timesteps into \(S\) distinct segments (\(S=3\) empirically achieves near-optimal gains). An isomorphic yet independently parameterized predictor \(\mathcal{E}_\theta^{(s)}\) is trained for each segment, decoupling stage-dependent trajectory shifts and ensuring optimal alignment between feature clusters and regional dynamics.

Loss & Training

LinCa decouples entirely from the backbone diffusion model during training, requiring only 100-200 pre-generated intermediate feature snapshots. The training completes within 1 hour on a single GPU (requiring only 12GB VRAM), adding fewer than 0.2% parameters relative to the base diffusion model. For each timestep segment \(s\), the joint optimization objective balances end-to-end reconstruction fidelity and sub-component prediction accuracy: $\(\mathcal{L}^{(s)} = \mathcal{L}_{\text{feat}}^{(s)} + \mathcal{L}_{\text{comp}}^{(s)}\)$ The end-to-end feature prediction loss ensures global consistency in the original space: $\(\mathcal{L}_{\text{feat}}^{(s)} = \|\hat{x}_{t-k} - x_{t-k}\|_2^2\)$ The sub-component loss directly penalizes prediction errors within each sub-space: $\(\mathcal{L}_{\text{comp}}^{(s)} = \sum_{m=0}^{M-1} \|\hat{z}_{t-k}^{(m)} - z_{t-k}^{(m)}\|_2^2\)$ This component loss drives \(\mathcal{E}_\theta^{(s)}\) to cluster feature dimensions amenable to \(m\)-th order extrapolation into the corresponding \(m\)-th latent slice in an automated, data-driven manner.

Key Experimental Results

Main Results

On the flagship text-to-image model FLUX.1-dev (50 steps), LinCa was evaluated against leading feature caching and token pruning baselines across multiple caching intervals \(N\) using DrawBench (200 prompts) on NVIDIA A100 GPUs.

Method	Caching Interval \(N\)	Latency (s)	Latency Speedup	FLOPs (T)	FLOPs Speedup	ImageReward ↑	CLIP Score ↑
Original Baseline (50 steps)	-	23.10	1.00×	3719.50	1.00×	0.9930	32.61
60% steps direct sampling	-	14.87	1.55×	2231.70	1.67×	0.9693	32.50
\(\Delta\)-DiT	2	15.96	1.45×	2480.01	1.50×	0.9471	32.46
FORA	3	9.08	2.54×	1320.07	2.82×	0.9802	32.45
DBCache	F=8, B=8	15.05	1.53×	2384.29	1.56×	1.0097	32.72
TaylorSeer	N=3, O=2	8.83	2.61×	1320.07	2.82×	1.0018	32.58
FoCa	3	8.35	2.78×	1327.21	2.80×	0.9917	32.75
LinCa (Ours)	4	7.51	3.08×	1120.68	3.32×	1.0175	32.88
ToCa	N=6, R=75%	11.76	1.96×	924.30	4.02×	0.9830	32.25
DuCa	5	7.32	3.15×	978.76	3.80×	0.9982	32.41
TeaCache	l=0.8	6.42	3.58×	892.35	4.17×	0.8710	31.89
HyCa	5	6.83	3.38×	893.54	4.16×	1.0096	32.87
LinCa (Ours)	6	5.27	4.38×	823.21	4.52×	1.0228	32.97
TeaCache	l=1.0	7.24	3.19×	743.63	5.01×	0.8421	32.01
TaylorSeer	N=5, O=2	6.71	3.46×	893.54	4.16×	0.9793	32.63
FoCa	6	6.76	3.42×	745.39	4.99×	0.9741	33.10
Speca	Nmax=8, Nmin=2	6.61	3.48×	791.38	4.70×	1.0012	32.45
HyCa	6	6.09	3.79×	744.81	5.00×	1.0043	32.64
LinCa (Ours)	8	4.40	5.25×	674.64	5.51×	1.0162	32.72

Accelerations on Qwen-Image (50 steps, baseline latency 126.60s): - At \(N=3\), LinCa achieves 58.88s latency (2.75× speedup), with ImageReward of 1.2329, PSNR of 31.86, SSIM of 0.83, and LPIPS of 0.18; - At aggressive \(N=10\), LinCa achieves 23.19s latency (6.95× speedup), retaining 1.0524 ImageReward and 29.10 PSNR, while TaylorSeer, FORA, and DuCa collapse to 0.7318, 0.4812, and 0.4104 respectively.

On HunyuanVideo (50 steps, baseline latency 185.00s, VBench 80.66%): - At \(N=6\), LinCa reduces latency to 38.21s (5.50× FLOPs speedup) while preserving an 80.16% VBench score (-0.6% relative drop), outperforming TaylorSeer (79.93%), TeaCache (79.36%), and DuCa (78.72%).

On Qwen-Image-Edit, LinCa achieves 7.08× speedup at \(N=10\), achieving 7.27 (CN) and 7.40 (EN) overall scores on GEdit-Bench, significantly outperforming competitors.

Ablation Study

Ablations on FLUX.1-dev and Qwen-Image validate the distinct contributions of the invertible structure and differentiated order extrapolation:

Ablation Dimension	Configuration	Operational Mechanism	FLUX.1-dev (N=6) Performance	Analysis & Failure Modes
Decomposition Architecture	Feed-Forward MLP	Non-invertible forward compression loss	Degrades sharply as \(N\) increases	Fails identity inversion \(\mathcal{E}_\theta^{-1} \circ \mathcal{E}_\theta \ne I\), compounding errors
Decomposition Architecture	Random Projection	Untrained weights lack continuity clustering	Severe divergence at high speedups	Unable to group dimensions by temporal smoothness
Decomposition Architecture	Invertible Net (LinCa)	Strict bidirectional lossless mapping \(\mathcal{E}_\theta^{-1} \circ \mathcal{E}_\theta = I\)	Best (ImageReward 1.0228)	Lossless reconstruction preserves full fidelity
Prediction Strategy	Pure 0th-order (Reuse)	Ignores temporal derivatives completely	Blurry textures and motion distortion	Incapable of anticipating smooth trajectory shifts
Prediction Strategy	Pure 1st-order Extrapolation	Cannot capture curvature; sensitive to noise	Rapid degradation at high intervals	Linear extrapolation diverges on abrupt features
Prediction Strategy	Pure 2nd-order Extrapolation	Severe overfitting on unstable channels	Explodes numerically for \(N \ge 6\)	Extrapolating non-smooth channels triggers divergence
Prediction Strategy	0th + 1st + 2nd Order	High-order on smooth; 0th on abrupt	Superior quality & extreme robustness	Optimal order matching per sub-space dynamic

Compatibility with compression techniques: - FLUX.1-lite-8B (28-step model distillation): LinCa (\(N=3\)) delivers 2.32× speedup (3.34s vs 8.21s), boosting ImageReward to 0.9070; - FLUX.1-schnell (4-step few-step distillation): LinCa (\(N=3\)) yields 1.99× speedup (1.38s vs 2.48s), reaching 0.9843 ImageReward; - FLUX.1-dev-int8 (INT8 post-training quantization): LinCa (\(N=3\)) achieves 2.63× speedup (4.61s vs 12.55s), with ImageReward rising to 1.0036.

Key Findings

1. Invertibility is foundational for deep caching: Invertible networks eliminate representation drift across multi-step cascading predictions that otherwise cripple conventional autoencoders.
2. Sub-space decomposition averts polynomial divergence: Higher-order polynomial extrapolation diverges rapidly when applied across fluctuating channels; routing noisy dimensions to zeroth-order reuse and smooth dimensions to higher-order extrapolation stabilizes predictions.
3. Extreme efficiency and transferability: Requiring under an hour of single-GPU training and ~100 cached tensors, LinCa integrates seamlessly into image generation, video generation, instruction editing, and quantized/distilled backbones.

Highlights & Insights

• Paradigm shift toward dimensional dynamics: Prior feature caching works concentrated primarily on temporal skipping schedules (when to cache) or spatial token sparsity (where to cache). LinCa is the first to identify cross-channel dynamics heterogeneity, unlocking a fresh dimension in acceleration research.
• Theoretical invertibility guarantees practical zero-loss: Adopting additive coupling and orthogonal channel mixing guarantees \(\mathcal{E}_\theta^{-1}\mathcal{E}_\theta = I\), enabling unsupervised feature clustering purely driven by task-specific prediction losses.
• Universal compatibility: Demonstrates robust gains across modal architectures (text-to-image, text-to-video, image editing) and compounds cleanly with orthogonal efficiency methods like INT8 quantization and distillation.

Limitations & Future Work

• Weak data dependency for training: While training costs under an hour on a 12GB GPU, LinCa is not strictly training-free; shifting to entirely novel models requires extracting an initial offline cache dataset.
• Static segment and decomposition hyperparameters: Pre-defining \(S=3\) segments and \(M=3\) sub-components could be generalized to dynamic, adaptive timestep and channel clustering mechanisms.
• Scaling to ultra-long temporal video models: In long-form video diffusion, spatial-temporal cross-attention channels exhibit intricate 3D continuity, suggesting opportunities for spatio-temporal invertible operators.

Related Work & Insights

• From heuristic reuse to analytical dynamics forecasting: While early pioneers like FORA and ToCa demonstrated token reuse and TaylorSeer/FoCa introduced ODE solvers, LinCa mathematically resolves extrapolation mismatch across interleaved feature dimensions.
• Implications for foundation model compression: The principle of invertible manifold decomposition to separate stable from transient dimensions holds significant promise for compressing KV caches and hidden activations in multimodal autoregressive LLMs.

Rating

• Novelty: 4.5 / 5.0 (Pioneering invertible network decomposition for dynamics-matched feature caching)
• Experimental Thoroughness: 4.8 / 5.0 (Thorough evaluation across FLUX, Qwen-Image, HunyuanVideo, Qwen-Image-Edit, plus quantized/distilled variants and ablations)
• Writing Quality: 4.6 / 5.0 (Rigorous mathematical formulation, clear structural hierarchy, and detailed empirical reporting)
• Value: 4.7 / 5.0 (Sub-0.2% parameter footprint delivering 5-7× near-lossless acceleration with minimal training overhead)