ProGVC: Progressive-based Generative Video Compression via Auto-Regressive Context Modeling

Conference: ECCV 2026
Paper: ECCV Official Page
Area: Model Compression
Keywords: perceptual video compression, progressive coding, visual autoregressive model, entropy coding, multi-scale residual quantization

TL;DR

ProGVC ports the next-scale prediction of Visual Auto-Regressive (VAR) models to video compression: a multi-scale residual quantizer turns video latents into a coarse-to-fine token pyramid, the encoder transmits all intra scales plus the first k inter scales, and the truncated high-frequency scales are generated on the fly by the same autoregressive context model — yielding a scalable bitstream, adaptive rate control, and competitive perceptual quality in the 300–2000 kbps regime.

Background & Motivation

Video compression is designed either for human viewing or for machine-oriented analysis, and this paper targets the former. Traditional standards (H.264/AVC, H.265/HEVC, H.266/VVC) and the recent generation of end-to-end neural video codecs (DCVC-FM, DCVC-B, SEVC) are all optimized against pixel-wise distortion metrics such as PSNR and SSIM. They preserve signal fidelity well, but at low bitrates the reconstructions become visibly unpleasant — blurring, blocking, and ringing artifacts that reveal how far human perception has drifted from those distortion measures. This pushed the field toward generative video compression. GAN-based codecs (the recurrent conditional GAN of PLVC, the generative latent coding of Qi et al.) synthesize sharp textures but suffer from unstable adversarial training and low fidelity; with diffusion models came codecs such as DiffVC that plug Stable Diffusion or a video diffusion transformer in as a decoding-time denoiser, using powerful priors to synthesize missing textures and motion.

Three structural weaknesses remain. First, diffusion-based codecs use the model essentially as a decoder-side refiner for transmitted pixels or latents, largely decoupled from entropy coding — however strong its spatio-temporal priors are, they do not translate into transmitting fewer bits, and those bits are exactly where the rate cost lives. Second, diffusion decoding needs multiple iterative denoising steps, which inflates latency and rules out low-delay or real-time streaming. Third, most perceptual codecs lack native support for scalable bitstreams and variable bitrate: when bandwidth fluctuates you must retrain or swap models, and PLVC supports neither scalability nor rate adaptation, while fidelity-oriented neural codecs support variable rate but cannot truncate a single bitstream into quality layers. This paper's goal is to fold three things that are currently separate — progressive scalability, efficient entropy coding, and detail synthesis — into a single autoregressive pipeline.

The angle of attack comes from VAR's next-scale prediction: a multi-scale residual quantizer encodes visual data into hierarchical token maps (scales), which a Transformer then predicts autoregressively. That coarse-to-fine structure is natively scalable — the encoder sends only the first few coarse scales and refines quality by transmitting additional high-frequency scales. The autoregressive model also produces an accurate conditional probability for every token, which is directly usable as an entropy context model. And since VAR samples in far fewer steps than diffusion denoises, decoding latency drops too. Core idea: encode video into a coarse-to-fine pyramid of discrete tokens via multi-scale residual quantization, then let one multi-scale autoregressive context model serve both as the probability model for entropy coding of the transmitted tokens and as the generative model for the truncated high-frequency scales — transmit every intra scale but only the first k inter scales, and generate the rest.

Method

Overall Architecture

ProGVC takes a video clip (one intra frame plus T inter frames) and returns a reconstructed video, in four steps. A causal video VAE first encodes the clip into continuous spatio-temporal latents. A multi-scale residual quantizer then discretizes those latents into K token maps whose spatial resolution and high-frequency content both grow with the scale index. A multi-scale autoregressive context model estimates the conditional distribution of every token, scale by scale. During encoding, all intra scales plus the first k low-frequency inter scales are losslessly written to the bitstream, while the remaining K−k high-frequency inter scales are discarded; at the decoder those discarded scales are recovered by taking the arg-max of the same model's distribution, after which multi-scale summation restores the latents and the VAE decoder reconstructs the video. The single network reused throughout is that autoregressive context model: an entropy model on the encoder side, and still a generative model on the decoder side.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["input video clip<br/>causal VAE extracts continuous latents"] --> B["hierarchical multi-scale residual quantization<br/>BSQ refines the latent level by level"]
    B --> C["multi-scale autoregressive context model<br/>estimates token distributions per scale"]
    C --> D["sparse attention mask<br/>self + previous scale + largest intra scale"]
    D --> E{"is the scale transmitted?"}
    E -->|"all intra + first k inter scales"| F["arithmetic coding/decoding<br/>lossless transmission"]
    E -->|"truncated K−k inter scales"| G["token generation on discarded scales<br/>take the most probable entry"]
    F --> H["multi-scale summation + VAE decoding<br/>reconstruct video"]
    G --> H

On scalability and rate control: transmitting coarse-to-fine works because the base scales establish global structure and layout, and later scales only add localized high-frequency detail — so any prefix subset is a valid bitstream. More scales in the bitstream means higher reconstruction quality, while truncating the tail produces a valid lower-rate reconstruction. In practice the bitrate is adjusted by truncating inter scales only, because inter tokens account for the overwhelming majority of transmitted bits and because intra scales are vital for the temporal alignment needed when modeling inter-scale distributions. The discarded high-frequency inter scales are not guessed blindly: they are generated conditioned on the already reconstructed scales.

Key Designs

1. Hierarchical multi-scale residual quantization: compressing continuous latents into a truncatable coarse-to-fine bitstream

The design problem is that "transmitting a subset of scales must equal a valid low-rate bitstream" — if scales were quantized independently and were not additive, dropping the tail would yield a broken representation rather than a lower-quality one. The solution is a residual chain. At scale k the module takes the current residual feature \(e_k\) (equal to the continuous latent itself when k=1), downsamples it into token vectors, and discretizes each element with Binary Spherical Quantization (BSQ), which binarizes by sign and normalizes by \(\sqrt{L_k}\) to give an \(L_k\)-bit binary code. The discrete tokens are then upsampled back to latent resolution and subtracted from the residual, and what remains feeds the next scale: \(e_{k+1}=e_k-U_k(r_k)\). Larger k means higher spatial resolution and progressively higher-frequency detail, so the upsampled sum of the first k scales approximates the original continuous feature more and more closely. This additivity is precisely what makes progressive delivery possible: any prefix subset sums into a valid latent that simply carries less detail. A less conspicuous but equally deliberate choice is BSQ itself — each token is literally a string of bits, so the bit-level probabilities the model outputs can drive the arithmetic coder without the index-remapping machinery a VQ codebook would require.

2. Multi-scale autoregressive context model: one set of probabilities serving both entropy coding and detail generation

This is the paper's core component and the sharpest departure from diffusion-based perceptual compression. Its autoregression is not raster-scan token-by-token prediction but next-scale prediction: intra scales are factorized by the chain rule, each conditioned on its coarser predecessors, and inter scales are additionally conditioned on all intra scales.

\[p(r^{I}_{1},\dots,r^{I}_{K})=\prod_{k=1}^{K}p\!\left(r^{I}_{k}\mid\{r^{I}_{k'}\}_{k'<k}\right)\]

\[p(r^{P}_{1},\dots,r^{P}_{\kappa_P})=\prod_{k=1}^{\kappa_P}p\!\left(r^{P}_{k}\mid\{r^{P}_{k'}\}_{k'<k},\,R^{I}\right)\]

To estimate the distribution of scale k, the model first sums the upsampled tokens of the first k scales, downsamples the result into an aggregated multi-scale representation, and feeds it as the current scale's input; several masked attention blocks then inject information from the prefix scales, and an MLP prediction head outputs the discrete token distribution. That distribution has two consumers: the arithmetic coder, which uses it for lossless coding of transmitted tokens, and the decoder, which takes the most probable discrete entry to generate discarded scales. Why does this work better than the diffusion route? A diffusion codec places its generative prior at the decoder as a refiner, where no amount of prior strength can save a single bit; here the conditional probabilities live on the encoder side and decide how many bits each token costs, so generative capability converts directly into rate savings. The ablation makes the point bluntly — replacing the context model with a uniform distribution more than doubles the bitrate.

3. Sparse attention mask: trading "self + previous scale + largest intra scale" for the rate-complexity sweet spot

In theory richer conditioning lowers conditional entropy and hence the expected bitrate, but naively widening the context is ruinous for video: the context grows along both spatial and temporal dimensions and the cost of autoregressive modeling explodes. Moreover the aggregated input at scale k already summarizes information from coarser scales, so attending to the full history is partly redundant. Neither existing mask fits. Infinity's block-wise causal mask lets every scale attend to all historical tokens and becomes expensive on long sequences; InfinityStar's more restrictive mask shrinks the receptive field so much that context utilization weakens and the bitrate actually rises. ProGVC's compromise lets each scale attend only to itself and its immediately preceding scale, so long-range dependencies are captured implicitly by stacking multiple attention layers rather than by explicitly attending to the full history. To retain temporal priors and temporal consistency, inter tokens are additionally allowed to attend to the largest intra scale, which operates at the finest spatial resolution, keeps the richest fine-grained detail, and shows the strongest temporal coherence with inter scales. The mask is used identically on the encoder and decoder sides — otherwise the two probability estimates would disagree and arithmetic coding would desynchronize. The ablation confirms the trade-off: against self-only attention all three BD-rate metrics improve, and against full causal attention compression is essentially on par (about 6.5% worse on DISTS BD-rate) while per-frame encoding/decoding time falls from 0.64/1.96 s to 0.56/1.53 s.

4. Token generation on discarded scales: generating high-frequency scales instead of transmitting them

Without recovering the K−k dropped high-frequency inter scales, the reconstruction would simply lack fine detail and fall back to blur. ProGVC's remedy is remarkably cheap: since the context model already produces their conditional distributions, the decoder greedily selects the most probable discrete entry, \(\hat r^{P}_{k}=\arg\max p\!\left(\hat r^{P}_{k}\mid\{\hat r^{P}_{k'}\}_{k'<k},\hat R^{I}\right)\) — one forward pass, no iterative denoising, no extra bits. The generated scales are summed with the already decoded ones and passed to the VAE decoder. This is where the rate-quality trade-off is actually executed: smaller k means more is generated and the bitrate drops, larger k means more is transmitted and quality rises. The price is that what gets restored resembles natural video rather than the specific source video — the model captures the natural video distribution and favors structural coherence and natural textures over strict pixel- or feature-level correspondence — so local textures may deviate slightly, which also explains why LPIPS is less flattering than DISTS and NIQE later on.

Loss & Training

Training runs in two stages. The causal VAE and multi-scale residual quantization inherit their base configuration from InfinityStar and are then finetuned jointly for 10K iterations on 256×256×81 clips with batch size 2 and learning rate \(5\times10^{-5}\); the objective combines entropy loss, commitment loss, L2 reconstruction loss, GAN loss, and LPIPS perceptual loss with weights 0.1, 0.25, 1, 0.01, and 4 respectively (the perceptual and adversarial terms dominate the reconstruction term, consistent with the perceptual-coding goal). The multi-scale autoregressive context model is trained separately for 60K iterations with AdamW, batch size 8, learning rate \(5\times10^{-5}\), and momentum coefficients \((\beta_1,\beta_2)=(0.9,0.97)\). Following the InfinityStar protocol, later inter scales are randomly dropped during training to improve efficiency, which also exposes the model to varying truncation lengths and aligns it with the variable-rate behavior at inference. The training corpus is a large high-quality set built from Pexels: aesthetic and clarity scores are computed per video, only samples above 4.5 and 0.65 respectively are kept, videos shorter than 10 seconds are discarded, and only clips with aspect ratios close to standard 720p are retained — roughly 480K videos in total.

Key Experimental Results

Main Results

Evaluation uses Xiph, HEVC Class B, and MCL-JCV, taking the first 81 frames of each sequence over a 300–2000 kbps range in the RGB domain, with HEVC-B and MCL-JCV downsampled to 720p to match the training configuration. Baselines span three families: the traditional VTM-17.0 (VVC reference, random access, GOP size 32 with one intra frame); fidelity-oriented neural codecs DCVC-FM, DCVC-B, and SEVC; and the perceptual codec PLVC, the strongest reproducible generative baseline. Perceptual quality is measured by DISTS and LPIPS (full-reference) plus NIQE (no-reference), fidelity by PSNR, temporal consistency by t-LPIPS, and rate in kbps. A negative BD-rate means fewer bits at equal quality.

BD-rate↓(%) / BD-metric↑ against the VTM-17.0 anchor on Xiph:

Method	DISTS	LPIPS	NIQE	PSNR
VTM-17.0	0.0 / 0.0000	0.0 / 0.0000	0.0 / 0.0000	0.0 / 0.0000
DCVC-FM	92.0 / −0.0257	40.4 / −0.0228	71.3 / −0.2525	111.1 / −1.8623
SEVC	87.7 / −0.0179	19.4 / −0.0062	41.8 / −0.1331	51.5 / −0.9035
DCVC-B	63.2 / −0.0148	17.3 / −0.0065	94.7 / −0.2306	56.8 / −1.0048
PLVC	−40.3 / 0.0109	−83.7 / 0.0565	169.9 / −0.0902	825.9 / −5.5261
ProGVC	−61.5 / 0.0300	3.5 / −0.0025	−62.2 / 0.2670	891.2 / −5.8437

On the other two datasets ProGVC posts the best DISTS BD-rate as well: −47.7% (BD-metric +0.0217) on HEVC-B and −46.2% (+0.0238) on MCL-JCV, with NIQE at −52.5% and −65.5% respectively. LPIPS is a different story: on HEVC-B it records 30.3%, behind PLVC's −30.5%, i.e. comparable rather than leading. PSNR trails across the board — on Xiph it needs roughly 8.9× the bitrate to match VTM-17.0 — which is the direct price of trading fidelity for perceptual quality.

Temporal consistency and complexity:

Method	t-LPIPS BD-rate↓ (Xiph)	Enc. time (s) ↓	Dec. time (s) ↓
DCVC-B	58.61	1.28	1.09
SEVC	−15.89	1.04	0.88
DiffVC	—	0.82	4.54
PLVC	709.87	—	—
ProGVC	−20.76	0.56	1.53

ProGVC leads all baselines on t-LPIPS at −20.76% (runner-up SEVC at −15.89%), which the authors attribute to hierarchical next-scale autoregressive generation and the spatio-temporal attention design, where each token attends to tokens from other frames at the current and previous scales to capture cross-frame temporal dependencies. On complexity, ProGVC has the lowest encoding time (0.56 s/frame) and a decoding time of 1.53 s/frame, on par with DCVC-B (1.09 s) and SEVC (0.88 s) and clearly below the diffusion-based DiffVC (4.54 s). Two caveats apply: PLVC is excluded from the timing comparison because its implementation is incompatible with the unified evaluation platform, and DiffVC is not open-sourced, so its architecture was reimplemented from the paper without training.

Ablation Study

Ablations are run on Xiph, reporting DISTS / LPIPS / PSNR BD-rate and BD-metric relative to the full model (negative is better):

Variant	DISTS BD-rate↓	LPIPS BD-rate↓	PSNR BD-rate↓	Note
Base (full model)	0.0	0.0	0.0	reference
w/o context model (uniform prob.)	122.4	119.6	119.4	entropy context contributes, more than doubling the rate
w/o token generation	N/A	280.9	189.9	no quality overlap for DISTS BD-rate
Attention: self-only	6.8	12.8	10.9	receptive field too narrow
Attention: full causal	−6.5	−3.8	0.5	marginally better than ours but 0.64 / 1.96 s
Reference: no intra scale	37.3	41.7	46.6	temporal prior discarded
Reference: smallest intra scale	42.7	40.4	46.8	low-frequency reference, weakest predictor
Reference: same-resolution intra scale	11.7	12.2	14.4	second best, still clearly behind largest scale

Key Findings

• The entropy context model is the single largest source of gain. Replacing it with a uniform distribution raises the bitrate by 122.4% (DISTS BD-rate), which the authors summarize as the context model halving the bitrate. The gain therefore comes less from generation than from the fact that accurate autoregressive probabilities can drive the arithmetic coder directly — exactly the link diffusion-based codecs lack.
• Token generation is what buys perceptual quality. Removing the generation of discarded scales pushes the LPIPS and PSNR rate penalties to 280.9% and 189.9%, and on DISTS the quality ranges no longer overlap at all, so BD-rate is reported as N/A. The effect of this step on perceptual metrics is order-of-magnitude.
• The sparse mask trade-off pays off. Against self-only attention, all three BD-rate metrics improve (6.8% → 0.0% on DISTS); against full causal attention the gap is only 6.5% on DISTS while per-frame encoding/decoding time drops by 12.5% / 22%. A small compression penalty buys a clear computational saving.
• Inter scales should reference the largest intra scale. No intra reference, the smallest intra scale, and the same-resolution intra scale cost 37.3%, 42.7%, and 11.7% extra DISTS BD-rate respectively, while the computational cost is nearly identical across variants — so picking the finest, detail-richest reference is essentially free.
• Perception and fidelity split cleanly. ProGVC dominates DISTS and NIQE (NIQE BD-rate −62.2% on Xiph) but is only comparable on LPIPS and needs several times the bitrate on PSNR. The authors' explanation is that the autoregressive context model learns the natural video distribution and favors structural coherence over strict correspondence, so slight local deviations are penalized heavily by LPIPS.
• Qualitative results win on both axes at once. On the Kimono sequence in Xiph, ProGVC reaches 355.6 kbps / DISTS 0.0621 while VTM-17.0 needs 444.15 kbps / 0.1174, SEVC 751.66 kbps / 0.1330, and PLVC 745.69 kbps / 0.0717; on videoSRC07 in MCL-JCV it is 300.2 kbps / 0.0590 against the best baseline's 302.2 kbps / 0.1283 — lower bitrate and better DISTS simultaneously.

Highlights & Insights

• Merging the entropy context model and the generative model into one network is the sharpest contrast with diffusion-based perceptual compression. A diffusion codec puts its generative prior at the decoder as a refiner, where prior strength cannot save bits; here the conditional probabilities sit on the encoder side and determine the cost of every token, so generative capability cashes out as rate savings. The "one distribution, two uses" pattern transfers to any setting with discrete tokens that must be transmitted — scalable image, audio, or point-cloud coding.
• Scalability grows out of the quantization structure rather than being bolted on. The residual chain guarantees that any prefix subset sums into a valid latent, so "truncate equals lower rate" needs no extra training and a single model covers the whole 300–2000 kbps range — exactly the gap left by perceptual codecs such as PLVC that must be retrained per rate.
• The mask design turns "exploit temporal priors" into a concrete, testable rule: inter tokens attend to the largest intra scale only. The 11.7%–42.7% BD-rate spread across reference choices shows this apparently arbitrary decision matters, and that fine-grained detail is a better predictive reference for inter scales than low-frequency layout.
• The real "aha" is latency. Diffusion decoders need iterative denoising; here the discarded scales come back from a single-step arg-max, giving 1.53 s/frame decoding at one third of DiffVC's cost and making perceptual compression plausible for low-delay streaming for the first time.

Limitations & Future Work

• Resolution and duration are capped by the autoregressive backbone. The authors acknowledge that supported video resolution and length are constrained by the AR model's training configuration, and that extending beyond 720p is hard given the computational cost of high-resolution training and the current maturity of AR video generative backbones; they plan to explore more general resolution- and length-agnostic architectures.
• Rate control is coarse. Bitrate is adjusted by transmitting or discarding entire scales, producing discrete rather than continuous rate points. The stated next step is finer-grained control such as token-level dropping for smoother adaptation in practical streaming.
• Perceptual metrics disagree with each other. ProGVC leads by a wide margin on DISTS and NIQE, is only comparable on LPIPS (losing to PLVC on HEVC-B), and needs about 8.9× the bitrate on PSNR. Scenarios with hard fidelity requirements — machine-vision downstream tasks, medical or forensic playback — are out of scope.
• Gaps I noticed. All ablations are on Xiph alone, so cross-dataset robustness of the conclusions is unverified; the DiffVC baseline is untrained, weakening the complexity comparison; every test uses the first 81 frames, while the cost of transmitting all intra scales is amortized over the GOP length, so long-sequence rate behavior is not discussed; and results are reported only in 300–2000 kbps, leaving the very-low-rate regime unknown.
• Concrete directions. Refine the rate knob from whole scales to token- or bit-level granularity and replace the discrete k grid with an explicit rate-distortion Lagrangian; train the VAE, quantizer, and context model end-to-end to reduce the mismatch between reconstruction and probability estimation.

Related Work & Insights

• vs VTM-17.0 (VVC reference): Traditional standards are designed around PSNR/SSIM and blur or block up at low rates; ProGVC saves 61.5% of the bitrate at equal DISTS but pays about 8.9× the bitrate on PSNR. They are a fidelity tool and a perception tool, not substitutes.
• vs DCVC-FM / DCVC-B / SEVC: Fidelity-oriented end-to-end neural codecs support variable rate but not scalable bitstreams, and look poor at low rates; ProGVC leads them on DISTS and NIQE, encodes faster (0.56 s vs 1.04–1.28 s), and decodes in comparable time (1.53 s vs 0.88–1.09 s).
• vs PLVC (GAN-based perceptual codec): PLVC is a strong reproducible generative baseline but supports neither scalability nor adaptive rate control, so each rate needs retraining; on Xiph ProGVC's NIQE BD-rate is −62.2% against PLVC's +169.9%, and a single ProGVC model spans the whole rate range.
• vs diffusion-based perceptual codecs (DiffVC et al.): The diffusion model acts as a decoding-time denoiser decoupled from entropy coding, so its prior does not reduce transmitted bits, and iterative denoising makes decoding slow (4.54 s vs ProGVC's 1.53 s). ProGVC feeds autoregressive conditional probabilities straight into arithmetic coding and recovers discarded scales with a single-step arg-max.
• vs VAR / Infinity / InfinityStar: VAR's next-scale prediction targets image generation, and Infinity and InfinityStar extend it to text-to-image and text-to-video; this paper is the first to apply it to video compression, repurposing "autoregressive over scales" from a generative mechanism into an entropy context model, with the attention mask redesigned for compression (wider than InfinityStar to use context, cheaper than Infinity).
• vs autoregressive progressive image coding (Zhang et al., ICLR 2026): The same "autoregressive plus progressive coding" idea has been validated for ultra-low-bitrate image compression; this paper's increment is the video-specific intra/inter dual pyramid and the cross-frame conditioning of inter scales on intra scales.

Rating

• Novelty: ⭐⭐⭐⭐ First to apply VAR-style next-scale prediction to video compression and to unify entropy context and detail generation in one model; but progressive autoregressive coding already exists for images, so it is a paradigm transfer plus task specialization rather than a wholly new mechanism.
• Experimental Thoroughness: ⭐⭐⭐⭐ Three datasets, five baselines, four metrics plus temporal consistency and complexity, and three well-targeted ablations; however ablations are Xiph-only, DiffVC is untrained, and there is no continuous rate-point analysis.
• Writing Quality: ⭐⭐⭐⭐ Clear motivation chain, restrained formulas, and an intuitive three-way mask comparison figure; some equations and the latent dimensionality are corrupted in the cached text (⚠️ refer to the original paper).
• Value: ⭐⭐⭐⭐ A viable and practically attractive new paradigm — one model, scalable, low latency, strong low-bitrate perceptual quality — though the large PSNR gap and discrete rate points limit near-term deployment.

Related Papers

• [ICCV 2025] Learned Image Compression with Hierarchical Progressive Context Modeling
• [ICCV 2025] Context Guided Transformer Entropy Modeling for Video Compression
• [ECCV 2026] Enhanced Neural Video Representation Compression with High Scalability
• [CVPR 2026] Generative Video Compression with One-Dimensional Latent Representation
• [NeurIPS 2025] Correlation Dimension of Auto-Regressive Large Language Models