MixCompress: Mixture of Experts for Variable Rate Learned Image Compression¶
Conference: ECCV 2026
Paper: ECCV 2026
Area: Signal & Communication
Keywords: Learned Image Compression / Variable Bit-rate / Mixture of Experts / Mixture of Depths / Gradient Conflict Mitigation
TL;DR¶
Addressing severe gradient conflict between high and low bit-rates in dense variable-rate codecs, MixCompress integrates rate-conditioned Mixture-of-Experts (MoE) and Mixture-of-Depths (MoD) with Conditional Auxiliary Transforms (CAT), surpassing individually optimized single-rate models under a single unified checkpoint.
Background & Motivation¶
Learned image compression (LIC) built upon nonlinear transform coding and variational hyper-priors has demonstrated rate-distortion performance superior to legacy handcrafted standards. Nevertheless, conventional state-of-the-art neural codecs are trained independently for each individual rate-distortion operating point governed by the Lagrange multiplier \(\lambda\). Serving a practical dynamic rate spectrum requires storing, deploying, and maintaining multiple large checkpoints, creating steep multiplication in training GPU-hours, disk footprints, and switching latency on edge devices.
Variable bit-rate (VBR) alternatives seek to alleviate this deployment burden by modulating a single shared backbone using conditional convolutions, latent affine transforms, or gain scaling. Despite offering rate adaptability, these shared models consistently suffer substantial coding efficiency drops compared to their dedicated single-rate counterparts. This performance gap stems from fundamental gradient interference during multi-objective optimization: low bit-rate objectives strongly penalize high-frequency residuals to enforce smoothing and save bits, yielding destructive gradients that directly corrupt the delicate parameters needed to preserve fine textures at high bit-rates. Forcing a single shared dense parameter set to approximate these mutually opposing transformations induces severe negative cosine gradient conflicts and structural capacity bottlenecks.
Because dense parameter modulation merely rescales feature magnitudes rather than resolving parameter interference, conditional capacity specialization is required. Core idea: incorporate sparsely gated Mixture-of-Experts (MoE) and progressive Mixture-of-Depths (MoD) into key transform bottlenecks alongside rate-conditioned Conditional Auxiliary Transforms (CAT), isolating conflicting rate gradients into orthogonal expert subspaces and establishing a superior Pareto frontier within a single model.
Method¶
Overall Architecture¶
MixCompress equips hierarchical transform-based LIC backbones (e.g., linear attention-driven LALIC or hybrid attention-driven LIC-TCM) with variable bit-rate capability. The input consists of an uncompressed image \(x\) and a target rate multiplier \(\lambda\), which is embedded into a 32-dimensional rate vector \(\lambda_e\). The overall architecture comprises an analysis transform (encoder), a synthesis transform (decoder), a hyperprior network, and a space-channel contextual entropy model. To decouple conflicting optimization trajectories without disturbing primary feature propagation, MixCompress replaces dense bottleneck layers at the ends of the analysis, synthesis, and hyperprior networks with rate-conditioned MixCompress blocks, while inserting symmetrical Conditional Auxiliary Transforms (CAT and iCAT) shortcuts along the encoder and decoder.
During encoding, input \(x\) is mapped to continuous latent \(y\) via the analysis transform while CAT decomposes the signal via 2D Haar wavelets and applies rate-adaptive subband energy reweighting; the hyper-encoder then computes hyper-latent \(z\). Quantized hyper-latents \(\hat{z}\) and spatial-channel contexts guide the hyper-decoder to estimate scale and mean parameters \((\mu, \sigma)\) for arithmetic coding of \(\hat{y}\). At synthesis, the decoder transforms \(\hat{y}\) back into reconstructed image \(\hat{x}\) complemented by inverse wavelet residual features from iCAT. Within critical bottlenecks, each MixCompress block combines a fixed shared expert with Top-\(K\) rate-specialized experts routed strictly via \(\lambda_e\), preventing smoothing gradients from overwriting texture-preserving weights.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
In["Input image x & target rate λ"] --> Emb["Rate embedding λ_e"]
In --> CAT["Conditional Auxiliary Transforms (CAT)<br/>Haar wavelet subband + FiLM energy modulation"]
subgraph Analysis["Analysis Transform ga (Encoder)"]
direction TB
EncFeat["Backbone feature extraction & downsampling"] --> MoE_Enc["Sparse MoE / MoD bottleneck<br/>Shared expert + Top-K routed experts"]
end
CAT -.->|Residual injection| Analysis
Emb --> MoE_Enc
Emb --> CAT
Analysis --> Quant["Latent quantization & hyper-prior coding<br/>Context model estimating μ, σ"]
Quant --> Synthesis
subgraph Synthesis["Synthesis Transform gs (Decoder)"]
direction TB
DecFeat["Backbone feature upsampling"] --> MoE_Dec["Sparse MoE / MoD bottleneck<br/>Shared expert + Top-K routed experts"]
end
Emb --> MoE_Dec
Emb --> iCAT["Inverse CAT (iCAT)<br/>Reciprocal scaling + IDWT residual reconstruction"]
iCAT -.->|Residual addition| Synthesis
Synthesis --> Out["Reconstructed image x̂ (target R-D point)"]
Key Designs¶
1. Conditional Auxiliary Transforms (CAT): rate-aware subband energy compaction While static wavelet shortcuts (such as AuxT) utilize 2D Haar discrete wavelet transforms (DWT) to split features into low-frequency (\(LL\)) and high-frequency (\(LH, HL, HH\)) subbands to offload de-correlation, their fixed subband scaling fails across disparate bit-rates: low rates require intense energy compaction while high rates demand fine detail preservation. CAT conditions subband scaling on the rate embedding \(\lambda_e\) through a FiLM modulation head that predicts adaptive scaling \(\boldsymbol{\gamma}_\lambda\) and shift \(\boldsymbol{\beta}_\lambda\): $$ [\boldsymbol{\gamma}\lambda, \boldsymbol{\beta}\lambda] = f_{\mathrm{film}}(\lambda_e), \quad s^{\mathrm{enc}}\lambda = \boldsymbol{\gamma}\lambda \odot s_{\mathrm{base}} + \boldsymbol{\beta}_\lambda $$ Wavelet tokens undergo element-wise modulation \(U \odot \exp(s^{\mathrm{enc}}_\lambda)\) and linear projection before residual injection into the encoder. Decoder-side iCAT mirrors this process with reciprocal scaling \(\exp(-s^{\mathrm{dec}}_\lambda)\) and inverse DWT (IDWT). This allows subband energy to adapt across operating points without modifying the core transform or entropy model.
2. Sparse structural MoE routing: decoupling adversarial gradient interference To counteract negative cosine gradient conflicts observed in dense modulation, MixCompress embeds \(N\) parallel routed experts and one always-active shared expert \(E_{\mathrm{global}}\) into critical bottlenecks. The shared expert preserves universally useful generic representations across all bit-rates, while a softmax gating network Top-\(K\) selects specific experts based on \(\lambda_e\) (defaulting to \(N=4, K=2\)). For two distant operating points \(\lambda_{\mathrm{low}}\) and \(\lambda_{\mathrm{high}}\), the gradient inner product within the routed subspace is strictly bounded by active expert overlap: $$ \langle g_{\mathrm{routed}}(\lambda_{\mathrm{low}}), g_{\mathrm{routed}}(\lambda_{\mathrm{high}}) \rangle = \sum_{i \in \mathcal{T}{\mathrm{low}} \cap \mathcal{T}}}} \langle \nabla_\theta \mathcal{L{\mathrm{low}}, \nabla\theta \mathcal{L}_{\mathrm{high}} \rangle $$ When non-overlapping experts are chosen, the gradient inner product is identically zero. Empirical analysis shows this reduces the proportion of encoder layers exhibiting negative gradient cosine similarity from 78% in dense modulation down to 31%, shielding high-rate textural representations from destructive low-rate updates.
3. Progressive Mixture-of-Depths (MoD): dynamic representational capacity scaling Standard MoE assigns identical topologies and FLOP budgets to all experts. However, high-rate reconstruction intrinsically demands greater non-linear capacity than coarse low-rate smoothing. MoD replaces homogeneous experts with a shallow-to-deep hierarchy: the \(i\)-th expert comprises \(i\) nested transformation blocks (cascaded Bi-RWKV blocks for LALIC or convolutional layers for TCM), formulated as \(E_i(f) = (h_i \circ \dots \circ h_1)(f)\). Retaining identical tensor interfaces, the router dynamically allocates deeper computational paths to higher bit-rates, shifting optimization dynamics further toward orthogonality and decreasing conflicting negative gradient layers down to 22%.
4. Noise annealing stabilization and zero-overhead precomputed routing Because routing decisions and CAT modulations depend solely on rate embedding \(\lambda_e\) rather than spatial data \(x\), naive training risks early representation collapse onto a subset of experts. MixCompress stabilizes training by adding Gaussian noise \(\epsilon \sim \mathcal{N}(0, \sigma(e)^2)\) to gating logits, annealed linearly from \(\sigma_0=1.0\) to 0 over the first 20 epochs to encourage early exploration. Crucially, because \(\lambda_e\) is static for any target bit-rate, expert assignments and FiLM vectors can be precomputed and cached in lookup tables (LUT), introducing zero dynamic routing overhead during inference.
Loss & Training¶
MixCompress optimizes a unified variable bit-rate objective where each mini-batch uniformly samples a rate multiplier \(\lambda \sim p(\lambda)\): $$ \min_\theta \mathcal{L}{\mathrm{VBR}}(\theta) = \mathbb{E}; \theta) \right] $$ with mean squared error (MSE) distortion }}} \left[ \lambda D(x, \hat{x}; \theta) + R(\hat{y}; \theta) + R(\hat{z\(D\) and \(\lambda \in \{0.0018, 0.0035, 0.0067, 0.0130, 0.0250, 0.0483\}\). Training follows a three-stage schedule on the OpenImages dataset with batch size 8 on a single NVIDIA H100 GPU: 40 epochs at learning rate \(1 \times 10^{-4}\) on \(256 \times 256\) crops, 4 epochs at \(1 \times 10^{-5}\), and 4 fine-tuning epochs on \(512 \times 512\) crops. The entire multi-rate model is trained for the same total duration as one single-rate baseline.
Key Experimental Results¶
Main Results¶
Evaluated against VTM-23.1 as the anchor (0.00% BD-Rate), Table 1 summarizes BD-Rate (PSNR) across Kodak, CLIC Validation (CLIC-V), CLIC Test (CLIC-T), and Tecnick datasets for dedicated single-rate models and variable-rate frameworks.
| Model Type | Method | VBR Support | Kodak (%) | CLIC-V (%) | CLIC-T (%) | Tecnick (%) |
|---|---|---|---|---|---|---|
| Handcrafted Baseline | VTM-23.1 [18] | ✓ | 0.00 | 0.00 | 0.00 | 0.00 |
| Single-Rate Checkpoints | Hyper-prior (Ballé 18) | ✗ | +27.09 | +34.27 | +35.38 | +35.88 |
| Single-Rate Checkpoints | Minnen18 [25] | ✗ | +6.89 | +2.95 | +0.98 | +3.30 |
| Single-Rate Checkpoints | Cheng20-Parallel [7] | ✗ | -1.77 | -4.12 | -5.45 | -3.52 |
| Single-Rate Checkpoints | ELIC [15] | ✗ | -8.95 | -12.22 | -14.35 | -15.07 |
| Single-Rate Checkpoints | TCM [24] | ✗ | -16.06 | -18.63 | -19.70 | -20.41 |
| Single-Rate Checkpoints | FAT [22] | ✗ | -17.12 | -19.07 | -20.04 | -23.07 |
| Single-Rate Checkpoints | LALIC (Anchor) [13] | ✗ | -18.81 | -23.96 | -26.43 | -26.03 |
| Dense VBR Baselines | LALIC-QRAF [29] | ✓ | -9.32 | -13.22 | -15.24 | -14.36 |
| Dense VBR Baselines | LALIC-CondConv [9] | ✓ | -15.12 | -19.22 | -22.05 | -21.60 |
| Ours | MixCompress-LALIC-MoE | ✓ | -18.81 | -24.16 | -26.80 | -27.05 |
| Ours | MixCompress-LALIC-MoD | ✓ | -20.12 | -25.73 | -28.71 | -28.88 |
Ablation Study¶
Component-wise ablation on the LALIC backbone reporting BD-rate savings relative to VTM-23.1:
| Configuration | Kodak (%) | CLIC-V (%) | CLIC-T (%) | Tecnick (%) |
|---|---|---|---|---|
| Variable-rate (Cond-Conv baseline) | -15.15 | -19.27 | -22.09 | -21.53 |
| + AuxT static wavelet shortcut | -17.06 | -20.76 | -23.37 | -23.50 |
| + CAT rate-adaptive wavelet modulation | -17.89 | -21.92 | -24.23 | -24.41 |
| + MoE homogeneous experts (\(N=4, K=2\)) | -18.81 | -24.16 | -26.80 | -27.05 |
| + MoD progressive-depth experts (\(N=4, K=2\)) | -20.12 | -25.73 | -28.71 | -28.88 |
Computational and Deployment Efficiency¶
Inference latency, memory footprint, and storage efficiency:
| Method | Enc Time (s) | Dec Time (s) | Mem (GB) | FLOPs (G) | Active / Total Params (M) |
|---|---|---|---|---|---|
| LALIC (Single rate) [13] | 0.234 | 0.184 | 0.875 | 283.39 | 66.13 / 66.13 (6 rates need ~397M) |
| MixCompress-MoE | 0.252 | 0.187 | 1.037 | 311.99 | 77.86 / 88.56 |
| MixCompress-MoD | 0.257 | 0.190 | 1.262 | 392.02 | 120.22 / 156.55 |
Key Findings¶
- Surpassing single-rate baselines: Despite receiving only \(1/6\)-th of effective rate-specific iterations during joint training, MixCompress-LALIC-MoD outclasses the individually trained LALIC baselines on all benchmarks (e.g., reaching -28.71% BD-Rate on CLIC-T vs. -26.43% for single-rate LALIC).
- Mitigating LayerNorm collapse in QRAF: QRAF adapts rates via latent scaling before quantization, which is inadvertently squashed by Bi-RWKV LayerNorm modules, blinding downstream entropy models. MixCompress routes features along structurally separated paths, altering representation direction rather than scale.
- High parameter efficiency: Covering 6 rate points with single-rate LALIC requires storing 397M parameters across 6 models, whereas MixCompress-MoD requires only 156.55M total parameters with only 120.22M active per pass and negligible latency overhead (\(<0.02\) s).
Highlights & Insights¶
- Multi-task optimization lens: The work identifies conflicting negative gradient inner products as the primary impediment in variable-rate learned compression, motivating structural orthogonality over dense modulation.
- Hierarchical conditional computation: Combining horizontal subband gating (CAT), transverse expert selection (MoE), and longitudinal capacity stretching (MoD) provides a multi-axis conditional computing framework.
- Inference zero-overhead deployment: Because routing is conditioned exclusively on the scalar rate parameter, routing choices and modulation scales can be cached offline, ensuring zero dynamic gating overhead during practical execution.
Limitations & Future Work¶
- Fine-grained continuous rate evaluation: Although interpolation at arbitrary unseen \(\lambda\) is demonstrated in the supplement, the transition behavior of discrete expert selection over continuous fine-grained rate grids requires further investigation.
- Extension to learned video coding: Expanding conditional structural routing to neural video codecs—where experts dynamically adapt to temporal motion magnitude and reference frame intervals—presents a promising future avenue.
Related Work & Insights¶
- From scalar gains to structural routing: Prior variable-rate approaches (Choi et al., ICCV 2019; Cui et al., CVPR 2021) relied on global feature scaling, limiting representational adaptation. MixCompress demonstrates that structural parameter specialization is vital to avoid capacity saturation.
- Wavelet synergy with deep networks: Building upon AuxT (Li et al., ICLR 2025), CAT establishes that coupling classical multiresolution wavelet decomposition with rate-aware FiLM conditioning stabilizes multi-rate neural training.
Rating¶
- Novelty: 4.5 / 5.0 (First application of MoE/MoD to learned compression transforms with thorough gradient conflict analysis)
- Experimental Thoroughness: 4.5 / 5.0 (Comprehensive benchmarks on 4 datasets against single-rate and VBR baselines with layer-wise gradient statistics)
- Writing Quality: 4.5 / 5.0 (Cohesive theoretical motivation and clean visual representations)
- Value: 5.0 / 5.0 (Significantly reduces multi-rate neural codec storage and training budgets while setting a new Pareto standard)