Enhanced Neural Video Representation Compression with High Scalability¶

Conference: ECCV 2026
Paper: ECCV Official
Project: https://hmkx.github.io/nvrc-pp/
Area: Model Compression
Keywords: implicit neural representation, video compression, high-resolution feature grids, in-parameter coding structure, multi-dimensional grid entropy model

TL;DR¶

This paper introduces NVRC++, an implicit neural representation framework for video compression that employs high-resolution multi-scale feature grids, an in-parameter hierarchical B-frame coding structure with unconditional quantization, feature grid annealing masks, and a multi-prior entropy model, achieving constant decoding complexity across wide bitrate ranges and up to 7.6x faster decoding speed than NVRC.

Background & Motivation¶

Implicit neural representations (INRs) map individual video instances into compact parameters of neural networks or feature grids without relying on general training across large video corpuses, presenting promising advantages in decoding latency and rate-distortion efficiency. However, existing high-performance INR video codecs (such as HiNeRV and NVRC) suffer from a critical scalability bottleneck: to span multiple target quality and rate points, distinct model capacities must be trained for different bitrates, causing network parameters and decoding computational complexity to scale steeply with video quality. Under high-bitrate or high-resolution scenarios, the substantial decoding overhead negates the native low-complexity advantage of overfitted representations, presenting a major barrier to real-world deployment on resource-constrained platforms.

Conversely, while lightweight overfitted codecs attempt to bypass high decoder capacity by relying on feature grids, they encounter severe computational and memory barriers when scaling to long sequences. Optimizing high-dimensional grids with contextual autoregressive entropy models across entire long sequences causes GPU memory explosion during backpropagation; prior attempts had to partition sequences into independent spatial patches or short video clips, severely sacrificing global spatio-temporal redundancy. Furthermore, directly deploying dense high-resolution grids leads to spatial overfitting at low bitrates, failing to share smooth context across neighboring pixels and resulting in degraded rate-distortion efficiency.

This paper tackles the fundamental tension between model capacity and reconstruction quality by offloading high-frequency details onto dense feature grids while fundamentally restructuring parameter dependency modeling and optimization. Core idea: build an efficient representation centered around the lightweight HiNeRV++ architecture and multi-scale high-resolution feature grids, incorporate an in-parameter hierarchical B-frame coding structure with decoupled unconditional quantization and grid annealing masks, and pair it with a multi-prior grid entropy model, thereby delivering scalable, high-performance video compression across wide bitrates with fixed, ultra-low decoding complexity.

Method¶

Overall Architecture¶

The NVRC++ framework consists of an enhanced lightweight implicit neural representation network, HiNeRV++, an in-parameter coding structure for feature grid parameters, and a multi-dimensional grid entropy model combining spatial, temporal, and scale priors. The spatio-temporal dynamics of a long video sequence are first captured into temporal slices of multi-scale high-resolution feature grids, which are subsequently rendered into full-resolution frames through compact convolutional blocks and a final Pixel Shuffle layer. To eliminate gradient accumulation and memory explosion during backpropagation across long sequences, NVRC++ re-engineers slice dependency and utilizes decoupled rate-distortion steps with random sampling during overfitting.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long Video Sequence Input"] --> B["Multi-scale High-Res Grids & HiNeRV++"]
    B --> C["In-Parameter Hierarchical B-frame Slices"]
    C --> D["Feature Grid Annealing Masking"]
    D --> E["Multi-dimensional Grid Entropy Model"]
    E --> F["Constant-Complexity Real-time Video Decoding"]

Key Designs¶

1. Multi-scale High-Resolution Feature Grids and HiNeRV++: Decoupling Model Complexity from Quality Conventional NeRV-like codecs widen network channels or stack deeper stages to represent finer video details, which incurs excessive decoding MACs at higher bitrates. HiNeRV++ fundamentally shifts the representation burden from dense network weights to high-resolution multi-scale feature grids. Learned feature embeddings from diverse grid resolutions are directly injected into the first three stages of the network. Crucially, HiNeRV++ introduces a multi-reference input mechanism: instead of relying solely on trilinearly interpolated features, the network explicitly gathers neighboring raw, un-interpolated grid slices as direct context, providing high-fidelity spatio-temporal cues. To further minimize computational and memory footprints at the finest resolution, HiNeRV++ replaces the final deep ConvNeXt stages and head layers with an efficient Pixel Shuffle layer, constraining network parameters to as few as 1.23M and guaranteeing invariant, ultra-low decoding complexity across all bitrates.

2. In-Parameter Hierarchical B-frame Coding and Unconditional Quantization: Overcoming Memory Walls in Overfitting Directly optimizing high-dimensional feature grids on long sequences leads to severe GPU memory exhaustion when traditional sequential conditional entropy models are applied, because backward activation chains accumulate over hundreds of frames. NVRC++ introduces an in-parameter coding structure that partitions each single-resolution grid tensor along the temporal axis into \(T_{\text{grid}}\) slices \(\{z_t\}\), establishing a \(K\)-level hierarchical B-frame dependency where higher-level slices serve as temporal conditions for lower-level slices. Crucially, NVRC++ removes conditional quantization offsets and applies an unconditional, symmetric quantization step size \(\delta_l\) across the entire grid:

\[\hat{z}_t^l = \left\lfloor \frac{z_t^l}{\delta_l} \right\rceil \cdot \delta_l\]

This unconditional formulation completely decouples slice-level backpropagation. Consequently, rate estimation and distortion calculation can be performed independently: during distortion computation, entropy forward passes are skipped entirely; during rate estimation, random temporal sampling is applied to evaluate rate expectations without caching full-sequence activations. This design allows models with over 100M parameters to overfit a 600-frame 1080P sequence smoothly on a single consumer GPU with only 24GB of memory.

3. Feature Grid Annealing Masking: Regularizing Low-Bitrate Spatial Overfitting When high-resolution feature grids are introduced directly, codecs often suffer noticeable performance degradation at low bitrates. Lower-resolution grids naturally force neighboring pixels to share interpolated representations, exploiting spatio-temporal redundancy effectively; in contrast, high-resolution grids possess excessive degrees of freedom, easily overfitting local textures or isolated noise and impairing global redundancy extraction under constrained rate budgets. Inspired by Dropout, NVRC++ incorporates a random masking strategy applied to higher-resolution grid parameters during early training stages. By stochastically dropping fine-scale features, the model is compelled to establish foundational low-frequency representations via coarser grids. The masking probability is annealed gradually across training epochs and smoothly reaches zero in later iterations, ensuring optimal rate-distortion balance across both low and high bitrates.

4. Multi-dimensional Grid Entropy Model with Fused Priors: Balancing Rate and High-Throughput Parallelism Sequential 3D autoregressive entropy models suffer from prohibitive latency during bitstream decoding. NVRC++ designs an advanced multi-prior grid entropy model that integrates three complementary contexts. In the scale domain (Scale Prior), coarser-scale feature slices are encoded first and provide hierarchical condition \(\hat{z}_t^{l-1}\) via spatial interpolation; in the temporal domain (Temporal Prior), reference slices from higher hierarchical B-frame levels \(\hat{z}_{\tau^0_t}^l\) provide temporal guidance; in the spatial domain (Spatial Prior), each slice is partitioned into 1-4 step checkerboard masking patterns \(m_s(\hat{z}_t^l)\), predicting Gaussian parameters (mean \(\mu\) and scale \(\sigma\)) conditioned on previously decoded checkerboard segments:

\[p\left(m_s(\hat{z}_t^l) \mid \hat{z}_t^{l-1}, \hat{z}_{\tau^0_t}^l, \dots, m_0(\hat{z}_t^l), \dots, m_{s-1}(\hat{z}_t^l)\right)\]

By substituting full 3D spatial serialization with multi-step parallel slice estimation, NVRC++ achieves high coding efficiency alongside high entropy decoding throughput.

Loss & Training¶

The overall training objective is formulated as an end-to-end rate-distortion optimization:

\[\min_{\theta} \left( R(\hat{\theta}) + \lambda D(V, V^{\text{gt}}) \right)\]

where \(D\) measures the reconstruction fidelity against the ground-truth video \(V^{\text{gt}}\), and \(R(\hat{\theta})\) computes the expected bitrate from the multi-prior entropy distribution. Thanks to unconditional quantization and in-parameter decoupling, training alternates between computing pixel distortion without the entropy model overhead and estimating the bitstream cost on randomly sampled grid slices. This avoids long activation retention and enables stable convergence on long sequence benchmarks.

Key Experimental Results¶

Main Results¶

Evaluations are conducted on UVG (7 sequences, 300-600 frames at 1080P) and MCL-JCV (30 sequences, 120-150 frames at 1080P). BD-rate is calculated against x265 (veryslow), and decoding speeds are measured on a single NVIDIA RTX 4090 GPU using FP16 mixed precision. NVRC++ is benchmarked across four constant-complexity scales: S1 (ultra-lightweight), S2 (lightweight), S3 (balanced), and S4 (high-capacity).

Method	Type	UVG PSNR BD-rate (%)	UVG MS-SSIM BD-rate (%)	MCL-JCV PSNR BD-rate (%)	Dec. Complexity (kMACs/px)	Dec. Speed (FPS)
x265 (veryslow)	Conventional	0.00	0.00	0.00	-	-
HM-18.0 (RA)	Conventional	-44.54	-43.85	-39.91	-	-
VTM-20.0 (RA)	Conventional	-62.81	-61.74	-58.86	-	-
DCVC-FM	Deep Autoencoder	-66.30	-67.16	-58.33	369.6(I)/865.5(P)	3.7
DCVC-RT	Deep Autoencoder	-65.56	-68.05	-57.40	364.9(I)/166.8(P)	57.8 (141.9*)
HiNeRV	INR	-45.84	-63.04	-28.75	87.3 - 346.3	13.3 - 49.9
NVRC	INR (SOTA)	-73.74	-80.65	-51.61	173.4 - 930.6	14.0 - 33.2
C3	Lightweight Grid INR	-21.91	-14.06	-19.42	4.4	434.0
NVRC++ (S1)	Ours (Ultra-light)	-40.06	-67.59	-22.07	7.3 (Constant)	365.0
NVRC++ (S2)	Ours (Light)	-61.41	-76.78	-47.61	25.1 (Constant)	162.9
NVRC++ (S3)	Ours (Balanced)	-71.20	-81.25	-55.90	92.5 (Constant)	73.4
NVRC++ (S4)	Ours (High-capacity)	-74.82	-81.88	-51.84	357.7 (Constant)	34.0

Note: The starred decoding speed for DCVC-RT reflects custom CUDA kernel acceleration; NVRC++ reaches its reported decoding frame rates using native PyTorch without custom CUDA kernels.

Ablation Study & Architecture Comparison¶

Ablations are anchored on NVRC++ (S2) on the UVG dataset to inspect individual components; architecture comparison highlights parameter distribution and convergence between HiNeRV (S) and HiNeRV++ (S2).

Config / Variant	PSNR BD-rate Change (%)	MS-SSIM BD-rate Change (%)	Note
NVRC++ (S2) Full Model	0.00 (Anchor)	0.00 (Anchor)	Constant complexity 25.1 kMACs/px, 162.9 FPS
(v1) w/ Low-resolution grid only	+44.01	+59.00	Loss of high-dimensional grid capacity causes massive drop
(v2) w/o Grid Annealing Masking	+30.33	+39.77	Severe spatial overfitting occurs at low bitrates
(v3) w/o Spatial Prior	+1.42	+0.52	Omits checkerboard intra-slice spatial context
(v4) w/o Temporal Prior	+5.13	+4.00	Omits hierarchical B inter-slice dependency
(v5) w/o Scale Prior	+6.12	+6.60	Omits cross-scale coarse-to-fine guidance
(v6) w/ 2D Stem	+35.55	+31.64	Substituting multi-reference with 2D conv loses temporal depth
(v7) w/ 3D Stem	+7.76	+8.44	Pure 3D conv stem is suboptimal compared to multi-reference slices

Architecture Comparison: HiNeRV vs. HiNeRV++ (UVG Dataset):

Architecture	Grids / Layers Parameters	Complexity (kMAC/px)	Dec. Speed (FPS)	PSNR @37 Epochs	PSNR @300 Epochs
HiNeRV (S)	0.35M / 2.84M	87.3	35.5	33.70 dB	35.27 dB
HiNeRV++ (S2)	100.06M / 1.23M	24.8	162.9	37.36 dB	39.46 dB

Key Findings¶

Decoupled Grid Representation Unlocks Superior Speed and Convergence: HiNeRV++ compresses network layer parameters from 2.84M to 1.23M while allocating capacity into 100.06M feature grid entries, reducing computational complexity by 71.6% (87.3 \(\to\) 24.8 kMACs/px) and boosting decoding speed by 4.6x (35.5 \(\to\) 162.9 FPS). Impressively, training HiNeRV++ for merely 37 epochs achieves 37.36 dB PSNR, significantly surpassing the 300-epoch convergence limit of HiNeRV (35.27 dB).
High-Resolution Grids Require Annealing Regularization: Removing high-resolution grids (v1) impairs BD-rate by +44.01%, while omitting annealing masks (v2) degrades BD-rate by +30.33%, verifying that stochastic masking is essential to prevent high-frequency grid overfitting at low bitrates.
Constant Complexity across Unprecedented Rate Ranges: Under the S3 configuration, NVRC++ maintains fixed 92.5 kMACs/px across wide rate ranges, outperforming VTM and DCVC-RT while running up to 7.6x faster than NVRC.

Highlights & Insights¶

Complete Decoupling of Decoding Complexity from Bitrate: Overturns the established NeRV assumption that higher quality necessitates deeper networks, demonstrating that lightweight decoders paired with dense grids can span extreme quality scales at zero computational penalty.
In-Parameter Hierarchical B-frame Coding with Unconditional Quantization: Elegantly transfers classic video compression temporal structures into the latent grid parameter domain and removes conditional offsets, eliminating gradient retention and solving the memory wall for long sequences on consumer GPUs.
Multi-Prior Fast Entropy Modeling: Replaces slow 3D autoregressive models with multi-step checkerboard masking and cross-scale conditioning, maintaining superior compression density while unlocking high decoding throughput.

Limitations & Future Work¶

High Encoding Iteration Overhead: While decoding achieves real-time speeds (>70 FPS), the instance overfitting phase still requires multiple training epochs (encoding FPS around 0.005 - 0.020), which is unsuitable for low-latency interactive communications.
Memory Bandwidth Demand: Offloading capacity onto feature grids increases memory bandwidth utilization during grid interpolation; memory access efficiency on low-power mobile or edge hardware warrants further optimization.
Future Directions: Exploring parameter initialization via pre-trained video foundation models or few-step diffusion adaptation could drastically reduce overfitting duration, facilitating practical video-on-demand delivery.

vs. HiNeRV / NVRC: HiNeRV relies on heavy deep convolution stages, and NVRC suffers from bitrate-dependent complexity that explodes up to 930 kMACs/px at high quality. NVRC++ establishes constant decoding complexity (S1-S4 fixed), achieves up to 7.6x faster decoding, and enables long-sequence fitting without memory blowup.
vs. C3 / COOL-CHIC: C3 uses lightweight decoders with grids but partitions videos into independent spatial patches due to memory constraints, sacrificing sequence-level redundancy. NVRC++ uses hierarchical in-parameter temporal slicing to overfit entire video sequences globally, dramatically outperforming C3 in rate-distortion efficiency.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Seamlessly embeds hierarchical B-frame structures into grid parameters, introducing unconditional quantization and annealing masks with high elegance.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated on standard Full HD benchmarks (UVG and MCL-JCV) across four complexity scales, with extensive ablations and clear speed-rate trade-offs.
Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, detailed architectural descriptions, and compelling empirical analysis.
Value: ⭐⭐⭐⭐⭐ Resolves the long-standing scalability and decoding latency dilemmas in implicit neural video compression, marking a major milestone toward practical deployment.