Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption

Conference: ECCV 2026
Paper: ECCV 2026
Area: Model Compression / Video Generation
Keywords: KV cache compression, instance-specific parametric absorption, autoregressive video generation, test-time adaptation, local attention

TL;DR

ISPA fits the difference between local and full attention during a short video-generation warmup and absorbs historical effects into output projections, allowing LongLive ISPA-0.5 to retain only 54.2% of its KV cache with subject consistency changing from 96.10 to 95.61 in the 30-second evaluation, without implying strictly lossless behavior across all models and metrics.

Background & Motivation

Autoregressive video generation divides a long video into sequential chunks, each still denoised by a diffusion or flow model. Queries from the current chunk access historical key-value caches, allowing the model to reuse information about scenes, subjects, and motion without processing all earlier frames again. This shifts rather than eliminates the bottleneck: avoiding expensive joint generation leaves an accumulating external memory. The cache consumes GPU memory and increases subsequent attention traffic and computation, making historical-cache reduction a distinct deployment problem.

Deleting historical tokens does not necessarily mean deleting only redundant information. An earlier frame that is no longer visible may still constrain subject shape, background layout, or temporal continuity; removing it changes attention outputs, and the error propagates into later layers and generated chunks. Keeping the initial frame as an attention anchor can stabilize local inference, but does not automatically recover the contribution of intermediate history. The paper therefore asks a narrower question: for a particular video being generated, can historical contributions in some layers become stable enough to be approximated by a fixed linear projection correction?

This hypothesis neither requires all history to be linearly compressible nor assumes the same layers are compressible for every video. It only requires some layers to reconstruct full outputs accurately from local outputs on the current instance's warmup data, while other layers retain dynamic dependencies that resist absorption. Here, remembering means preserving history's effect on subsequent computation, not encoding past frames into an exactly retrievable image database. Core idea: use the current video to identify layers whose historical effects can be absorbed into output projections, then remove those layers' historical KV entries and replace part of external token memory with instance-specific parametric memory.

Method

Overall Architecture

The inputs are an existing autoregressive video generator, its generation conditions, and the current generation instance; the output remains a continuously generated video, without changing the task. Text-to-video experiments use text conditions, while the LiveAvatar extension uses a reference image and continuous speech. ISPA modifies self-attention cache usage and the following linear output projection rather than retraining the entire video generator. Figure 1 on page 5 divides the process into warmup, layer absorption, and subsequent memory-efficient inference.

During warmup, every layer generates with full context while producing both local and full attention outputs. The local context is not an arbitrary sliding window: it specifically contains the initial sink frame and the current frame, with intermediate history treated separately. At the transition, the method computes a candidate weight correction for each layer and uses reconstruction error to decide which layers switch. Selected L-Layers retain initial and current context, while unselected F-Layers continue using the full-context path.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Conditions and current video"] --> B["Decomposable Dual-Stream Warmup"]
    B --> C["Closed-Form Parametric Absorption"]
    C --> D["Instance-Specific Layer Selection"]
    D -->|Low reconstruction error layers| E["Local attention<br/>Update projection and evict history"]
    D -->|Remaining layers| F["Full attention<br/>Retain history"]
    E --> G["Subsequent video chunks"]
    F --> G

The diagram's branches represent different layers within one network, not independently generated videos that are later blended. The correction belongs to the current video instance and should not be treated as a compressed model reusable across every prompt. The budget determines how many layers are selected, while calibration signals from the current instance determine their identities. ISPA therefore combines numerical adaptation with instance-dependent changes to attention structure.

Key Designs

1. Decomposable Dual-Stream Warmup: obtain full and local signals from the same KV workload

Running full attention and local attention separately would duplicate computation and weaken the compression benefit. ISPA instead partitions keys and values into disjoint local and historical groups, letting the same current query attend to each group. The local group contains the initial sink frame and current frame; the historical group contains the intervening frames. Each group returns both its weighted value output and the Log-Sum-Exp (LSE) state associated with softmax normalization. The local branch directly supplies the local activation to be calibrated, while the full activation is recovered by combining both branches using LSE-derived weights. An ordinary average would be incorrect because the groups generally carry different amounts of total attention mass.

The two normalization constants determine each group's contribution to the full softmax. This uses online-softmax information already available in kernels such as FlashAttention, without materializing the full attention matrix. Pages 7–8, Equations (9)–(12), describe the decomposition; the dot products cover the same total KV population as full attention. However, the nearly free claim concerns signal collection, not the absence of costs for projection statistics or the transition-time linear solve. The authors report that fusion accounts for less than 1% of execution time, but this does not establish a universal transition-latency guarantee across hardware. The sink frame also remains in the compressed local path; it is not a temporary warmup input that is subsequently removed.

2. Closed-Form Parametric Absorption: fit the output difference caused by missing history

Let \(A_{loc}\) and \(A_{full}\) denote local and full attention activations collected during warmup, both immediately before the final output projection \(W\). The objective is not to reconstruct historical keys and values token by token, but to make local activations under the corrected projection approximate full activations under the original projection. Targeting the projection immediately after attention compensates for context loss near its origin, limiting propagation into subsequent layers. The fitted quantity is therefore a projected output residual, not merely a difference between attention probability maps. The following restates the residual and reconstruction error defined in the text on page 6; the cached rendering of Equations (4)–(6) loses characters, so the damaged inverse expression is not transcribed here.

\[ R=(A_{full}-A_{loc})W,\qquad \epsilon=\left\|A_{loc}(W+\Delta W)-A_{full}W\right\|_F^2. \]

Here, \(\Delta W\) is the instance-specific output-projection correction, and \(\epsilon\) is the remaining squared Frobenius reconstruction error after absorption. The solver regresses the target residual on local activations using regularized closed-form regression rather than gradient descent. Activation outer products and activation–residual cross-statistics can be accumulated during forward passes without retaining a back-propagation graph. The paper denotes activation width by \(D\) and describes the central transition-time solve as a single \(D\times D\) matrix inversion. These sufficient statistics explain why adaptation need not repeatedly iterate over all warmup samples, but the matrices still consume memory and the solve still requires computation.

After absorption, the active weights become \(W+\Delta W\), while local attention accesses only the retained KV entries. History is no longer stored as individually retrievable frame states; it influences the mapping from current activations to outputs. This is suited to historical contributions that remain relatively stable within the instance and admit linear compensation. If later dynamics differ from those observed during warmup, a fixed correction need not remain an adequate substitute for actual history. Good warmup reconstruction is therefore a selection criterion, not an error bound for arbitrary future frames.

3. Instance-Specific Layer Selection: absorb only layers that are easy to compensate for in the current video

Every layer receives a candidate correction and reconstruction error, but not every candidate is applied. For a network with \(N\) layers and a budget permitting \(K\) conversions, ISPA sorts layers by increasing \(\epsilon\) and selects the \(K\) lowest-error layers. These become L-Layers, receive their projection corrections, and discard intermediate historical KV entries. The other \(N-K\) layers remain F-Layers and preserve long-range dependencies that the current linear fit cannot adequately replace. This is more faithful to the paper's hypothesis than deleting caches at fixed layer indices: temporal relationships occupy different layers for different videos.

ISPA-x denotes \(K=xN\), where x is the fraction of layers converted to local attention, not the fraction of total GPU memory removed. Even when half the layers are converted, those layers retain sink and current-frame entries, so the remaining KV ratio is not exactly one half. Unconverted layers still require their original historical caches, so fixed-size historical representations in L-Layers do not imply constant memory for the entire model. The paper also proposes re-entering warmup after a major scene change to refresh both layer selection and weight corrections. However, the default experiments use a single transition and do not provide a complete scene-change detector or a separate repeated-recalibration benchmark. Recalibration is a proposed extension, not a demonstrated solution to arbitrary scene transitions.

A Worked Example

Consider the paper's LongLive ISPA-0.5 configuration: initially, all layers still use full context. The first 12 latent frames provide warmup signals; these must not be confused with 12 displayed frames or 12 seconds. At the transition, every layer computes its candidate projection correction and reconstruction error from accumulated local and full activations. The system selects the lowest-error half of the layers, applies their corrections, and removes their intermediate historical KV entries. When generating the next chunk, those layers obtain local inputs only from the initial and current frames, while the modified projections indirectly convey historical effects. The other half still accesses history directly, and both layer types participate in the subsequent denoising network. Table 2 reports 54.2% remaining KV and 95.61 subject consistency for this configuration; these are experimental outcomes, not quality guarantees for every new prompt.

Loss & Training

ISPA does not re-optimize the generator's original diffusion or flow-matching training objective and introduces no new offline training dataset. Its adaptation targets come from full attention outputs of the same instance during warmup, functioning as online teacher signals. The default warmup is \(T_{warm}=12\) latent frames, with regression regularization \(\lambda=0.001\) on pages 9–10. Closed-form solving replaces back-propagation, so the absence of gradient-based training does not mean the weights remain unchanged. The paper describes statistic accumulation and switching but does not fully specify batching, numerical precision, and solver details for every backbone in this full text. These choices affect peak memory and transition jitter and should be measured separately from steady-state attention acceleration when reproducing the method.

Key Experimental Results

Main Results

The following excerpts Table 2 on page 9: VBench evaluation of 30-second videos with MovieGen prompts. Higher scores are better; KV is remaining cache relative to each model's Vanilla configuration, not total GPU memory. LongLive and Reward are 1.3B models, while Krea is 14B; cross-backbone score differences are not used to attribute effects to ISPA.

Backbone and configuration	Remaining KV	Aesthetic quality	Background consistency	Imaging quality	Subject consistency	Temporal flickering score
LongLive Vanilla	100%	63.59	96.65	70.15	96.10	97.92
LongLive ISPA-0.5	54.2%	62.98	96.28	69.05	95.61	97.89
Reward Vanilla	100%	60.82	94.97	66.79	93.43	96.49
Reward ISPA-0.5	55.6%	60.45	94.95	66.55	93.36	96.64
Krea Vanilla	100%	62.15	94.64	71.20	94.85	96.85
Krea ISPA-0.5	58.3%	63.93	94.63	70.12	93.81	97.12

LongLive loses 0.49 score points in subject consistency and 1.10 in imaging quality, so not every change is below 1. Krea gains 1.78 points in aesthetics while losing 1.04 in subject consistency, illustrating a tradeoff across metrics. The temporal flickering column follows the original table's higher-is-better convention; it does not reward a greater amount of flickering. The authors interpret some improvements as filtering long-range attention noise, but the observed score changes alone do not establish that causal explanation.

Ablation Study

This table also comes from Table 2 on page 9 and isolates converted-layer ratios for LongLive under the same 30-second setting. It is a compression-budget analysis with exact numerical values; longer-term behavior at higher ratios is shown separately in Figure 4 on page 12, without estimating precise values from its curves.

Configuration	Converted layers	Remaining KV	Aesthetic quality	Imaging quality	Subject consistency
Vanilla	0%	100%	63.59	70.15	96.10
ISPA-0.3	30%	72.5%	63.32	69.38	95.77
ISPA-0.4	40%	63.3%	63.05	69.05	95.52
ISPA-0.5	50%	54.2%	62.98	69.05	95.61

Subject consistency does not decrease strictly monotonically with compression, so small fluctuations should not be interpreted as a reliable advantage. Figure 3 on page 12 provides qualitative component ablations: removing parametric absorption produces subject-shape drift, while removing sink frames causes oversmoothing. These are observations from the displayed examples; the paper supplies no per-component scores for that figure that could populate a numerical table. Figure 4 shows substantial quality deterioration at \(K=0.7N\), indicating the need to retain enough full-attention layers. Figure 5 on page 13 compares warmups of 6, 12, 18, and 24 frames: 6 performs poorly, behavior stabilizes from 12, and 24 postpones cache reclamation. This trend supports the default warmup choice, but does not establish a universal threshold across video content and temporal scales.

Key Findings

• Table 1 on page 9 reports imaging quality of 70.70 for 5-second LongLive ISPA-0.5 versus 70.74 for Vanilla, showing a small short-video difference that cannot replace the 30-second conclusion.
• Pages 13–14 and Figure 7 report 1.35× overall acceleration for ISPA, 1.89× acceleration within attention, and 1.86× overall acceleration with W8A8; these are different measurement scopes.
• Figure 6 on page 13 illustrates identity and mouth-shape preservation in LiveAvatar, but provides no separate quantitative lip-synchronization result here.

Highlights & Insights

• The compression target shifts from identifying important historical tokens to identifying historical effects reconstructible from local computation. Layer-output error thereby informs compression decisions instead of relying only on token attention scores.
• Sink retention and projection correction serve different purposes. The former anchors local attention, while the latter compensates for missing history; neither is a substitute for the other.
• Calibrating with global signals and deploying a local path is a reusable idea. Applying it to other streaming models still requires testing whether warmup covers future dynamics; this paper does not demonstrate universal cross-task applicability.

Limitations & Future Work

• The paper acknowledges degradation with insufficient warmup and excessive layer conversion. Static weight corrections have limited capacity to represent continuously changing long-range dependencies.
• The main numerical tables cover 5-second and 30-second videos, not validated hour-long generation. Recalibration after abrupt scene changes also remains primarily a methodological description.
• Historical representations in selected layers can be fixed-size, but other layers still use historical caches. The work reduces cache pressure without independently establishing constant system memory for unbounded duration.
• Ablations provide useful visual comparisons but lack corresponding No Absorb and No Sink scores and error bars. Whether selection error predicts future quality also deserves independent validation.
• The full text does not completely report a unified hardware setup, transition-time peak overhead, or repeated-run variance. Quantization compatibility and near-lossless behavior should be limited to the demonstrated settings.

Related Work & Insights

• Compared with StreamingLLM: Page 15 describes retaining initial sink and recent tokens; ISPA also preserves an anchor but additionally fits a projection to compensate for evicted history. This is a mechanistic comparison, not a direct numerical comparison supplied by the paper.
• Compared with H2O and SnapKV: These methods select retained tokens using attention statistics, whereas ISPA selects layers using post-absorption output reconstruction error. It introduces instance calibration and weight updates in exchange for avoiding explicit storage of some history.
• Compared with LongLive, Reward-Forcing, and Krea-Realtime: These are the generation backbones being compressed, not models newly trained by ISPA. Their differing responses motivate evaluation by model and duration rather than reliance on a single aggregate claim.

Rating

• Novelty: 4/5. Absorbing video KV history into instance-specific projections with dynamic layer selection offers an alternative to token eviction.
• Experimental Thoroughness: 3/5. Coverage spans backbones and durations, but numerical component ablations, recalibration experiments, and system-overhead details remain limited.
• Writing Quality: 3/5. The main pipeline is clear, but warmup discussion is repetitive and some near-lossless and constant-memory language exceeds the tabulated evidence.
• Value: 4/5. Useful for studying quality–memory tradeoffs in streaming video, with peak overhead and long-horizon distribution shifts still requiring deployment-time evaluation.