Skip to content

ReTarget: Representation Transformation via Adversarial Regularization for Geometric Misalignment

Conference: ECCV 2026
Paper: ECCV 2026
Area: AI Safety
Keywords: split inference, data reconstruction attacks, privacy-preserving learning, adversarial regularization, geometric misalignment

TL;DR

Addressing the vulnerability of edge-cloud split inference (SI) to data reconstruction attacks (DRA) that recover sensitive raw inputs from transmitted intermediate activations, ReTarget shifts away from traditional information suppression assumptions and proposes a lightweight residual transformation with adversarial regularization and class-prototype guidance, perturbing the predictable local geometric organization of representations while preserving downstream utility.

Background & Motivation

The rapid growth in the parameter scale and computational demands of deep neural networks poses severe challenges for deploying state-of-the-art models directly on resource-constrained edge devices like mobile phones and IoT sensors. Split inference (SI) has emerged as an attractive collaborative computing paradigm: the client executes the shallow layers of a partitioned network locally and transmits only intermediate feature activations to a cloud server to finish downstream inference. For years, split inference was broadly regarded as privacy-friendly under the intuitive assumption that raw inputs never leave the client device and the server only observes high-level latent representations.

Recent advances in data reconstruction attacks (DRA) have shattered this assumption. Powered by guided diffusion models (e.g., DRAG), generative adversarial networks (e.g., GLASS), and optimization-based feature inversion algorithms, adversaries can invert transmitted intermediate representations and synthesize high-fidelity reconstructions of private user images. Existing defensesβ€”such as dynamic channel pruning (DISCO) and statistical distance correlation penalties (NoPeek)β€”rely on a shared foundational assumption: privacy leakage is strictly proportional to the amount of mutual information retained between the input and representation. Consequently, prior methods endeavor to suppress or decorrelate feature channels. Strikingly, even under heavy information suppression, reconstruction attacks frequently retain high perceptual fidelity.

The tension lies in the fact that empirical reconstruction success depends not solely on how much information is retained, but fundamentally on how representations are locally organized in feature space. Natural image encoders map nearby inputs to continuous, smooth, and predictable neighborhoods, allowing inverse models to readily fit stable inverse mappings. Core idea: mitigating reconstruction attacks requires disrupting the local geometric neighborhood structure of representations via lightweight residual transformations guided toward class-level prototypes, destabilizing instance-level inversion while strictly maintaining task-relevant discriminative semantics.

Method

Overall Architecture

The ReTarget framework consists of three sequential components: on-device intermediate feature extraction, lightweight local representation transformation, and server-side task prediction. Given a frozen client encoder \(f_c\), an input image \(x\) is encoded into an intermediate activation \(h = f_c(x)\). ReTarget places a lightweight residual multi-layer perceptron transformation module \(T_\theta\) on the client, mapping \(h\) to a transformed feature \(\tilde{h} = T_\theta(h)\). The server model \(f_s\) receives \(\tilde{h}\) and outputs downstream predictions \(\hat{y} = f_s(\tilde{h})\). During training, a patch-level reconstruction decoder with a gradient reversal layer (GRL) and a class-prototype semantic regularizer jointly optimize \(T_\theta\) and \(f_s\). At test time, the reconstruction decoder is discarded entirely, allowing the client to execute the lightweight residual transformation with negligible computational overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image x"] --> B["Client Encoder fc<br/>frozen feature extractor"]
    B --> C["Residual Geometric Transform<br/>TΞΈ module perturbing features"]
    C --> D["Transmitted Feature h_tilde"]
    D --> E["Adversarial Reconstruction Branch<br/>gradient reversal maximizing error"]
    D --> F["Prototype Semantic Regularization<br/>aligning with centroid prototype xtarget"]
    D --> G["Downstream Task Prediction<br/>server fs predicting class y"]

Key Designs

1. Residual Geometric Transformation: shifting feature manifolds while preserving capacity Conventional defenses rely on hard channel dropping or coarse global decorrelation, often degrading downstream utility or being bypassed by adaptive decoders. ReTarget models feature perturbation as a continuous geometric shift implemented via a lightweight residual MLP: \(T_\theta(h) = h + \text{MLP}(\text{LN}(h))\), where the MLP comprises two linear layers with hidden dimension \(d\) and an intermediate GELU activation. This formulation shifts the representation away from the high-density, easily invertible region of the feature space while preserving representation capacity. Crucially, the client encoder remains completely frozen, eliminating expensive backbone retraining on edge devices.

2. Adversarial Reconstruction Regularization: pushing representations toward inversion failure zones To actively prevent external decoders from establishing stable inverse mappings, ReTarget introduces a lightweight Transformer-based patch reconstruction decoder \(g_\phi\) during defense training (modeled after the MAE reconstruction head). The decoder is optimized to minimize the mean squared reconstruction error \(\mathcal{L}_{\text{rec}} = \|\hat{x} - x\|_2^2\), where \(\hat{x} = g_\phi(T_\theta(h))\). Concurrently, the transformation module \(T_\theta\) is trained adversarially via gradient reversal to maximize this reconstruction error (\(-\mathcal{L}_{\text{rec}}\)). This explicit adversarial pressure drives transformed representations into regions where standard inversion mappings destabilize, substantially increasing the geometric complexity required for any inverse model to reconstruct inputs.

3. Class-Prototype Semantic Regularization: neutralizing instance cues and preventing representation collapse Unconstrained adversarial reconstruction training easily leads to representation collapse, turning features into uninformative noise and destroying downstream task performance. ReTarget counteracts this vulnerability with a prototype guidance mechanism: for each class \(y\), it computes the feature centroid \(\mu_y = \frac{1}{N_y} \sum_{i: y_i = y} f_c(x_i)\) across training samples and designates the training image with the highest cosine similarity to \(\mu_y\) as the class-level prototype \(x_{\text{target}}\). The semantic consistency regularizer measures the cosine distance between the decoded image and the prototype in the client encoder's feature space:

\[\mathcal{L}_{\text{sem}} = 1 - \frac{\langle f_c(\hat{x}), f_c(\hat{x}_{\text{target}}) \rangle}{\|f_c(\hat{x})\|_2 \|f_c(\hat{x}_{\text{target}})\|_2}\]

During backpropagation, the decoder \(g_\phi\) is kept frozen, directing gradients solely into \(T_\theta\). The decoded image thus acts as a semantic proxy: instead of preserving instance-specific fine details, the reconstructed output is pulled toward common class semantics, penalizing instance identifiability while preserving task-discriminative structures.

Loss & Training

The overall training objective of ReTarget integrates the downstream classification loss, adversarial reconstruction objective, and prototype semantic regularization:

\[\mathcal{L} = \mathcal{L}_{\text{rec}} + \rho \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{sem}}\]

where \(\mathcal{L}_{\text{task}} = \text{CE}(f_s(T_\theta(h)), y)\) is the standard cross-entropy loss aligning the server classifier \(f_s\). Hyperparameters are set to \(\rho = 0.95\) and \(\lambda = 1.0\) by default. The training procedure proceeds in two alternating steps per iteration: - Step 1: Fix client encoder \(f_c\), transformation \(T_\theta\), and server \(f_s\), updating the reconstruction decoder \(g_\phi\) by minimizing \(\mathcal{L}_{\text{rec}}\); - Step 2: Fix encoder \(f_c\) and decoder \(g_\phi\), updating transformation module \(T_\theta\) and server \(f_s\) by minimizing \(\rho \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{sem}} - \mathcal{L}_{\text{rec}}\) using gradient reversal.

Key Experimental Results

Main Results

The defense is evaluated on a CLIP-ViTB/16 backbone trained on ImageNet-1K and tested against the state-of-the-art white-box diffusion inversion attack DRAG across MS COCO, FFHQ, and ImageNet-1K. Evaluation metrics include multi-scale structural similarity MS-SSIM (lower is better), perceptual distance LPIPS (higher is better), and semantic feature similarity measured via DINO ViT-S/16 (lower is better). Baselines include dynamic channel obfuscation (DISCO) and statistical distance decorrelation (NoPeek).

Dataset Split Point Defense Method MS-SSIM ↓ LPIPS ↑ DINO Similarity ↓
MS COCO Layer 3 (shallow) DISCO 0.7818 0.1025 0.9314
MS COCO Layer 3 (shallow) NoPeek 0.3760 0.4690 0.4263
MS COCO Layer 3 (shallow) ReTarget (Ours) 0.0209 0.6337 0.1695
MS COCO Layer 6 (mid-shallow) DISCO 0.5683 0.2195 0.9003
MS COCO Layer 6 (mid-shallow) NoPeek 0.3704 0.3923 0.6361
MS COCO Layer 6 (mid-shallow) ReTarget (Ours) 0.0588 0.7665 0.1869
MS COCO Layer 12 (deep) DISCO 0.5115 0.2960 0.8777
MS COCO Layer 12 (deep) NoPeek 0.0694 0.7645 0.2558
MS COCO Layer 12 (deep) ReTarget (Ours) 0.0134 0.8389 0.1846
FFHQ Face Layer 3 (shallow) DISCO 0.8942 0.0522 0.9645
FFHQ Face Layer 3 (shallow) NoPeek 0.6187 0.3095 0.5999
FFHQ Face Layer 3 (shallow) ReTarget (Ours) 0.0238 0.6170 0.1599
FFHQ Face Layer 12 (deep) DISCO 0.6372 0.2288 0.8126
FFHQ Face Layer 12 (deep) NoPeek 0.2940 0.5060 0.5054
FFHQ Face Layer 12 (deep) ReTarget (Ours) 0.0244 0.7285 0.2263

Ablation Study

To verify that privacy gains originate from geometric shifts rather than unconstrained corruption, feature displacement was analyzed under a frozen classifier (Flip@1, Overlap@5, Logit L2, Cosine Shift). Local neighborhood disruption was measured across 5,000 ImageNet validation images via Disrupt@10 (\(1 - \text{Overlap@10}\)). Robustness was additionally tested against an adaptive attack, DRAG++, where the inverse decoder is retrained directly on defended features \(\tilde{h}\).

Configuration / Mechanism Split Layer Local Disruption Disrupt@10 ↑ Prediction Flip@1 ↑ Task Acc ↑ Adaptive DRAG++ MS-SSIM ↓
w/o Prototype Guidance (\(\lambda = 0\)) Layer 3 0.6970 - 0.7410 -
Full ReTarget (\(\lambda = 1.0\)) Layer 3 0.8560 0.2465 0.7461 0.0481
w/o Prototype Guidance (\(\lambda = 0\)) Layer 6 0.7570 - 0.7520 -
Full ReTarget (\(\lambda = 1.0\)) Layer 6 0.7930 0.2460 0.7574 0.0375
w/o Prototype Guidance (\(\lambda = 0\)) Layer 12 0.5010 - 0.7850 -
Full ReTarget (\(\lambda = 1.0\)) Layer 12 0.6710 0.0579 0.7862 0.0188

Key Findings

  • Breakthrough at shallow split layers: At shallow cut points (Layer 3), intermediate representations retain dense spatial details. Baselines fail severely here (DISCO yields an MS-SSIM of 0.7818 on MS COCO). In contrast, ReTarget reduces MS-SSIM to 0.0209 and boosts LPIPS by 35% (from 0.4690 to 0.6337), overcoming the longstanding barrier in shallow split inference.
  • Divergence between mutual information and empirical privacy: Mutual information estimation via InfoNCE shows that DISCO suppresses instance-level mutual information \(\Delta I(X; H)\) even more aggressively at deeper layers, yet fails to prevent recognizable reconstruction. This confirms that information reduction alone is insufficient to predict vulnerability; preserving or perturbing local representation manifolds is the critical factor.
  • Robustness against adaptive adversaries: In adaptive DRAG++ attacks where decoders are explicitly retrained on defended representations \(\tilde{h}\), Layer 3 MS-SSIM only slightly increases to 0.0481, demonstrating that the geometric neighborhood disruption resists specialized inverse adaptation.

Highlights & Insights

  • Paradigm shift beyond information suppression: ReTarget moves beyond naive channel dropping, noise injection, or global decorrelation, establishing that disrupting local feature neighborhood geometry provides robust protection against inverse mappings without utility loss.
  • Elegant class-prototype semantic anchoring: Using class centroids prevents adversarial representation collapse while restricting reconstructions to generic category attributes rather than instance identities, realizing effective utility-privacy decoupling.
  • Minimal edge deployment friction: The client requires only a lightweight 2-layer residual MLP without retraining backbones, enabling seamless plug-and-play deployment on edge hardware.

Limitations & Future Work

  • Admitted limitation: While Disrupt@10 successfully explains global reconstruction failure, sample-level correlation with perceptual metrics like LPIPS shows mixed trends, indicating that local neighborhood disruption serves as an insightful interpretation rather than a fully closed theoretical proof.
  • Identified limitation: Constructing prototypes relies on offline class centroids from labeled datasets, which may require adaptation when applied to open-vocabulary classification, long-tailed distributions, or continuous spatial regression tasks.
  • Future directions: Extending geometric transformations to vision-language foundation models (e.g., LLaVA/CLIP multimodal split serving) and deriving formal differential privacy or geometric perturbation bounds.
  • vs DISCO (CVPR 2021): DISCO relies on input-conditioned channel pruning masks. Because unpruned channels preserve their native manifold geometry, generative decoders easily fill in missing structures; ReTarget modifies the continuous representation space, disrupting inverse learning across all channels.
  • vs NoPeek (ICDM 2020): NoPeek optimizes a distance-correlation penalty between inputs and activations, causing utility degradation and failing at shallow layers; ReTarget combines adversarial regularization with class-prototype guidance, achieving 74.61% accuracy while degrading reconstruction fidelity.
  • vs DRAG / DRAG++ (ICML 2025): DRAG demonstrates the power of guided diffusion inversion; ReTarget demonstrates strong resilience against both vanilla DRAG and adaptive DRAG++ across all split depths.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers a representation transformation and geometric misalignment paradigm, departing from traditional mutual information suppression]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluates across diffusion, GAN, and optimization attacks on three datasets with comprehensive ablation and adaptive threat modeling]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, structured narrative, rigorous theoretical grounding, and clean presentation]
  • Value: ⭐⭐⭐⭐⭐ [Provides a practical and computationally lightweight defense for edge-cloud collaborative computing]