Skip to content

title: >- Towards Reconfigurable Visual Feature Compression description: >- [ECCV2026][Model Compression][Feature Compression] Proposes RFC, a reconfigurable multi-task feature compression framework for end-cloud VFM collaboration, achieving on-demand low-bitrate transmission and once-forward cloud inference via adapter factorization, cross-task hyperprior consolidation, and attentive recomposition. tags: - ECCV2026 - Model Compression - Feature Compression - End-Cloud Collaboration - Vision Foundation Models date: 2026-09-19 content_hash: 5f09c4c9b1bd95de

Towards Reconfigurable Visual Feature Compression

Conference: ECCV 2026
Paper: ECCV Official
Code: https://jhang2020.github.io/Projects/RFC/RFC.html
Area: Model Compression
Keywords: Multi-task Feature Compression, Feature Factorization, Cross-task Hyperprior, End-Cloud Collaboration, Feature Recomposition

TL;DR

Addressing transmission redundancy and inference inefficiency when deploying vision foundation models (VFMs) for arbitrary user-requested multi-task combinations in end-cloud collaboration, this paper proposes the Reconfigurable Feature Compression (RFC) framework based on a "factorization-consolidation-recomposition" paradigm, achieving demand-aware minimal bitrates and once-forward cloud multi-task inference.

Background & Motivation

Vision foundation models (VFMs) parameterized by large Transformer backbones have demonstrated remarkable multi-task representation capabilities across diverse dense prediction benchmarks. However, their massive computational footprints severely hinder on-device execution. The prevailing end-cloud collaborative paradigm partitions the VFM into a shallow frontend encoder deployed on edge devices and a deep backend deployed in the cloud. To mitigate high bandwidth consumption and safeguard raw visual privacy, intermediate feature compression has emerged as an indispensable bridge. Yet, conventional feature compression follows a rigid compress-and-reconstruct methodology that struggles with dynamic multi-task service demands.

Existing approaches to multi-task feature transmission are caught in a clear dilemma. On one hand, conventional multi-task feature compression encodes a monolithic intermediate representation embedding information for all conceivable tasks. Because real-world users rarely request all tasks simultaneously, this rigid pipeline transmits massive amounts of task-irrelevant information, leading to excessive bandwidth consumption. On the other hand, maintaining multiple standalone single-task models to support on-demand transmission introduces substantial computational redundancy: the frontend must execute multiple heavy Transformer passes, while cross-task shared visual priors are repeatedly encoded into separate bitstreams, and the cloud loses its ability to infer multiple tasks within a unified forward pass.

This paper breaks away from the dichotomy between static monolithic compression and disjoint single-task encoding by recasting multi-task feature transmission as a reconfigurable factorize-and-recompose system. Core idea: propose the Reconfigurable Feature Compression (RFC) framework, which utilizes shared attention with task-isolated adapters at the frontend to factorize features and strip task-irrelevant redundancy, exploits previously transmitted task features as auto-regressive hyperprior conditions to eliminate shared coding redundancy, and performs task-attentive recomposition at the cloud to retain once-forward multi-task inference.

Method

Overall Architecture

RFC operates through a cohesive three-stage pipeline: frontend feature factorization, consolidation coding, and cloud-side feature recomposition. Given an arbitrary task request \(T_Q \subseteq \mathcal{T}\), the shallow frontend backbone \(F_{front}(\cdot)\) extracts base intermediate representations, which are projected into atomic task-specific features via lightweight adapters. These separated features are then entropy-encoded in an auto-regressive sequence, where later tasks reuse decoded features from earlier tasks as hyperprior conditions, yielding an ultra-compact bitstream. At the cloud, decoded features are dynamically fused through a lightweight task-attentive recomposition module into a single aggregated representation, enabling the cloud backbone \(F_{cloud}(\cdot)\) to produce predictions for all requested tasks in a single forward pass.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image x_i & Target Request T_Q"] --> B["Frontend Shared Backbone F_front"]
    B --> C["Adapter Feature Factorization<br/>Parameter-isolated IB decomposition"]
    C --> D["Cross-Task Auto-Regressive Coding<br/>Prior decoded features as hyperpriors"]
    D --> E["Demand-Aware Compact Bitstream"]
    E --> F["Cloud Entropy Decoding"]
    F --> G["Task-Attentive Feature Recomposition<br/>Learnable task embeddings & self-attention"]
    G --> H["Cloud Backbone F_cloud Single Forward<br/>Multi-task heads predict final results"]

Key Designs

1. Adapter Feature Factorization: Parameter-Isolated Information Bottleneck Decomposition Directly training a unified encoder for all \(2^N - 1\) combinatorial task subsets is computationally intractable, while duplicating frontend backbones incurs prohibitive self-attention overhead. RFC employs a divide-and-conquer strategy: freezing the shared frontend ViT backbone while fine-tuning lightweight, task-specific adapters at the feature extraction exit. Grounded in Information Bottleneck (IB) theory, this design minimizes the mutual information \(I(x_i; f_i)\) between the representation and input image while maximizing task-relevant mutual information \(I(f_i; Y_i^{t_k})\), effectively purging task-irrelevant context. The optimization objective balances rate-distortion and feature alignment: $\(\mathcal{L}_{factor} = \mathcal{R}(f_i^{t_k}) + \lambda \mathcal{L}_{t_k}(\tilde{f}_i^{t_k}, Y_i^{t_k}) + \lambda' \|f_i^{t_k} - \tilde{f}_i^{t_k}\|_2\)$ where \(\tilde{f}_i^{t_k} = \mathcal{D}_{t_k}(\mathcal{C}_{t_k}(f_i^{t_k}))\) denotes the reconstructed feature from task-dedicated codec networks, and \(\lambda'\) regularizes feature consistency. This allows the frontend to execute a single expensive global attention pass while selectively outputting clean, task-pure representations.

2. Cross-Task Auto-Regressive Conditioning: Eliminating Shared Semantic Redundancy While factorization eliminates task-irrelevant noise, atomic features extracted from early layers still exhibit strong correlations and shared low-level visual priors. Compressing them independently re-introduces cross-task coding redundancy. RFC shifts redundancy reduction from the feature representation level to the entropy coding level. In standard hyperprior compression, an auxiliary downsampled latent \(z\) must be encoded to estimate the conditional Gaussian parameters \(\mu\) and \(\sigma\). In RFC, once an initial task \(t_j\) is transmitted and decoded at the cloud, its full-resolution reconstruction \(\tilde{f}_i^{t_j}\) is readily accessible to both sides. Subsequent task \(t_k\) directly exploits \(\tilde{f}_i^{t_j}\) as an undownsampled, high-fidelity hyperprior condition \(z_i^{t_j \to t_k}\) fed into prediction heads \(\mu'_{t_k}(\cdot)\) and \(\sigma'_{t_k}(\cdot)\): $\(z_i^{t_j \to t_k} \longrightarrow \mu'(z_i^{t_j \to t_k}), \sigma'(z_i^{t_j \to t_k}) \stackrel{f_i^{t_k}}{\longrightarrow} \mathcal{B}(f_i^{t_k})\)$ Because the condition is derived from an already transmitted feature, no separate hyperprior bits are sent for task \(t_k\), instantly cutting hyperprior bitstream overhead by approximately 30% while yielding sharper distribution parameters. To avoid sample-wise combinatorial search during inference, a global transmission order lookup table is precomputed during training, allowing instant, deterministic routing.

3. Task-Attentive Feature Recomposition: Preserving Single-Forward Cloud Throughput If the cloud were to feed \(|T_Q|\) decoded features individually through the deep backbone, inference latency would scale linearly with \(|T_Q|\), negating the core efficiency advantage of multi-task VFMs. RFC incorporates a compact task-attentive recomposition module accounting for only 0.4% of the backbone parameters. Decoded features are projected into a common space and reshaped to \(\tilde{f}_i^{all} \in \mathbb{R}^{(B \cdot N_p) \times |T_Q| \times C}\). After injecting learnable task embeddings \(TE[T_Q]\), multi-head self-attention (MHSA) is applied across the compact \(|T_Q|\) dimension to model task synergies and compute normalized fusion weights: $\(w = \text{sigmoid}\left(\text{FFN}\left(\text{MHSA}\left(\tilde{f}_i^{all} + TE[T_Q]\right)\right)\right)\)$ A weighted summation along the task dimension yields a unified representation \(\tilde{f}_i^{fuse} \in \mathbb{R}^{B \times N_p \times C}\), which undergoes a single forward pass through \(F_{cloud}(\cdot)\) to service all heads simultaneously. This preserves \(O(1)\) cloud forward complexity with negligible performance degradation.

Loss & Training

RFC adopts a decoupled three-stage training strategy: 1. Factorization and Codec Pretraining: Freeze the pre-trained backbone, and train task-specific adapters along with base compression networks \(\mathcal{C}_{t_k}, \mathcal{D}_{t_k}\) for 60k iterations using the rate-distortion objective in Eq. (4); 2. Cross-Task Hyperprior Learning: Freeze adapter and base codec parameters, and optimize cross-task distribution predictors \(\mu'_{t_k}(\cdot), \sigma'_{t_k}(\cdot)\) for 20k iterations strictly under the bitrate objective \(\mathcal{R}(\cdot)\); 3. Recomposition Module Tuning: Freeze all encoder, frontend, and codec weights, and train the lightweight recomposition module across enumerated task combinations \(T_Q\) for 20k iterations using the composite downstream task loss \(\sum_{t_k \in T_Q} \mathcal{L}_{t_k}\).

Key Experimental Results

Main Results

Under a simulated request stream protocol reflecting realistic traffic—Few-task dominated (F-request, where 1 to 3 task queries account for 85% of traffic) and Multiple-task dominated (M-request)—RFC is compared against traditional image/video codecs (HEVC, VVC), learned feature compression models (Hyperprior FC, VCM, OmniICM, LaMoFC), and standalone single-task models (Multiple ST FC) on PASCAL-Context (extracted from Table 1 of the original paper):

Method F-request Bitrate (bpp) ↓ F-request Seg. mIoU ↑ F-request Parse mIoU ↑ F-request Edge odsF ↑ M-request Bitrate (bpp) ↓ Cloud Forward Passes
Uncompressed Upper Bound - 78.95 66.83 72.4 - 1
HEVC 0.2648 17.12 19.16 24.1 0.2648 1
VVC 0.2907 45.98 37.38 44.3 0.2907 1
Hyperprior FC 0.1554 77.43 65.70 70.4 0.1554 1
VCM 0.2531 76.25 64.06 69.3 0.2531 1
OmniICM 0.1713 76.63 65.14 69.8 0.1713 1
LaMoFC 0.1894 77.21 65.02 70.4 0.1894 1
Multiple ST FC 0.0976 76.75 64.63 70.6 0.1557 $
RFC (Ours) 0.0725 77.61 66.31 72.0 0.0977 1

Ablation Study

To isolate the contribution of each core component under the maximum workload where all 5 tasks are requested (\(|T_Q| = 5\)), the ablation results on PASCAL-Context are reported below (extracted from Table 2 of the original paper):

Config Seg. mIoU ↑ Parsing mIoU ↑ Saliency maxF ↑ Normals mErr ↓ Edge odsF ↑ Bitrate (bpp) ↓ Forward Passes
Only Factorize (Full \(F_{front}\)) 76.98 65.79 84.49 13.63 71.9 0.2700 5
Only Factorize (Adapters) 77.85 66.56 85.00 13.54 72.5 0.2675 5
Factorize + Fusion 77.43 65.90 84.60 13.75 71.1 0.2675 1
Factorize + Consolidate + Fusion (Full Model) 77.43 65.90 84.60 13.75 71.1 0.1253 1

Key Findings

  • Consolidation coding drastically cuts shared bitrate: Comparing rows 3 and 4 of Table 2, introducing cross-task auto-regressive hyperprior consolidation slashes bitrate from 0.2675 bpp to 0.1253 bpp (a 53.16% reduction) without degrading any task accuracy metric, as the hyperprior purely refines lossless entropy estimation.
  • Adapters surpass full frontend tuning: Tuning lightweight adapters while sharing self-attention yields superior segmentation (77.85 vs 76.98 mIoU) and human parsing (66.56 vs 65.79 mIoU) compared to full frontend fine-tuning, successfully preventing overfitting while preserving backbone generalization.
  • Superiority in sparse request streams: In real-world F-request streams, RFC achieves 0.0725 bpp, outperforming the non-configurable Hyperprior FC baseline (0.1554 bpp) by more than 53% in transmission savings and reducing VVC bitrate by nearly 75% while delivering higher downstream accuracy.

Highlights & Insights

  • Entropy-level redundancy reduction over representation decoupling: Perfect semantic feature disentanglement at shallow layers is computationally prohibitive on edge devices. RFC circumvents this by reusing decoded task features as conditional hyperpriors during entropy coding, eliminating cross-task redundancy at zero representation cost.
  • Task-attentive recomposition unlocks unified cloud throughput: By learning dynamic task affinity weights through a self-attention module operating over task tokens, RFC condenses multi-stream inputs into a single composite representation, preserving single-forward batch processing for foundation models.
  • Offline sequence pre-computation eliminates runtime overhead: Pre-calculating optimal task encoding sequences during training and referencing them via lightweight bitstream headers eliminates online greedy search latencies during real-time serving.

Limitations & Future Work

  • Author-acknowledged limitations: The framework is tailored for intermediate visual feature compression in machine perception and does not reconstruct human-interpretable visual images. Furthermore, under certain mutually conflicting task subsets, full \(|T_Q| \to 1\) recomposition causes minor metric trade-offs (e.g., a ~0.42% drop in segmentation mIoU).
  • Future directions: Investigating adaptive clustering and multi-branch recomposition strategies (e.g., grouping synergistic tasks together while routing conflicting tasks to separate forward passes) represents a promising path to optimize the accuracy-computation trade-off.
  • vs OmniICM / LaMoFC: Conventional learned feature codecs rely on static end-to-end compression of monolithic representations, sending all latent information regardless of user demand. RFC introduces demand-aware reconfigurable compression, slashing bitrates drastically for targeted task subsets.
  • vs Multiple ST FC: Naive single-task model banks support on-demand transmission but demand \(|T_Q|\) separate forward inferences on both edge and cloud while ignoring shared visual information. RFC resolves this through shared frontend attention, auto-regressive hyperprior reuse, and single-forward cloud recomposition.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First to formulate reconfigurable multi-task feature compression with an elegant factorization-consolidation-recomposition paradigm.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across PASCAL-Context and NYUD v2 on 5+ dense prediction tasks, simulated traffic flows, and ViT-Large scaling.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly organized structure with clear mathematical framing, intuitive diagrams, and comprehensive ablation analysis.
  • Value: ⭐⭐⭐⭐⭐ Offers an impactful and practical blueprint for high-throughput, bandwidth-efficient VFM deployment in real-world end-cloud services.