---

title: >- [Paper Note] TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction description: >- [ECCV 2026][LLM Safety][Adversarial Reconstruction] Proposes TrustCLIP, a privacy-preserving visual feature framework against generative inversion attacks by inserting an identity-initialized residual projection layer and jointly optimizing task utility with adversarial reconstruction loss, substantially degrading image reconstructibility while maintaining downstream classification and VLM reasoning capabilities. tags: - ECCV 2026 - LLM Safety - Adversarial Reconstruction - Diffusion Inversion Attack - CLIP Representation Learning date: 2026-09-19 content_hash: d54d4cd28003b455

TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://atnikos.github.io/trustclip
Area: LLM Safety
Keywords: Visual Feature Privacy, Adversarial Reconstruction, Diffusion Inversion Attack, CLIP Representation Learning, Multimodal Safety

TL;DR

Addressing the severe privacy threat where generative diffusion models faithfully reconstruct sensitive raw images from frozen CLIP embeddings, TrustCLIP introduces an identity-initialized lightweight residual projection layer between the vision encoder and downstream heads, jointly optimizing downstream task utility against negative generative reconstruction loss to selectively eliminate reconstructible instance-level details while preserving high-level task semantics.

Background & Motivation

Vision-language pre-trained backbones such as CLIP have become foundational components across contemporary computer vision, powering open-vocabulary recognition, visual question answering, cloud-based multimodal large language models (MLLMs), and large-scale image retrieval systems. However, this remarkable multimodal alignment comes with severe, underexplored privacy risks. Because CLIP representations retain rich semantics closely aligned with natural image distributions, recent advancements in conditional diffusion models (such as IP-Adapter) demonstrate that an adversary can directly invert frozen CLIP feature tokens into photorealistic, high-fidelity reconstructions of the original input. In collaborative client-server inference or cloud feature caching, this inversion capability exposes sensitive user information including facial identities, perceived age, gender, medical imaging details, and private domestic environments.

Defending against this generative leakage presents two fundamental challenges. First, existing privacy-preserving representation techniques largely focus on discriminative proxy metrics—such as reducing the accuracy of an adversarial classifier predicting sensitive attributes. Such defenses fail against generative diffusion attackers, which leverage rich generative priors to synthesize visually coherent and privacy-revealing scenes even when attribute classifier accuracy is low. Second, the visual information exploited by a reconstruction attacker is deeply entangled with the semantic cues required by downstream tasks. Naive suppression strategies, such as adding isotropic Gaussian noise or aggressive quantization, indiscriminately corrupt both task-critical semantics and reconstruction cues, leading to catastrophic utility degradation without reliably preventing generative recovery.

The central insight of this work is that downstream tasks and generative reconstruction attackers do not depend on the same visual components to the same extent: tasks like classification and visual reasoning primarily rely on macro-level semantics—object categories, spatial layout, and coarse scene geometry—whereas faithful pixel-level reconstruction strictly requires instance-specific high-frequency details, fine textures, precise facial landmarks, and skin tones. The core idea of TrustCLIP is to treat the feature-conditioned generative reconstructor directly as an explicit adversarial training target, inserting a lightweight, zero-initialized residual projection layer after the frozen vision encoder and jointly optimizing task utility alongside negative reconstruction loss, thereby forcing the projection to selectively excise reconstructible cues while retaining the semantic subspace.

Method

Overall Architecture

The TrustCLIP architecture is designed around modular decoupling, plug-and-play deployment, and non-destructive initialization. The entire operational pipeline comprises four stages: feature extraction, lightweight projection defense, downstream task prediction, and adversarial reconstruction back-propagation. During training, an input image is mapped to a sequence of feature tokens by a frozen vision encoder (e.g., CLIP ViT-L/14); the lightweight residual projection layer transforms these tokens into protected representations; these protected tokens are then routed along two competing pathways: one branch feeds downstream task heads (such as a linear classification probe or a VLM multimodal projector and language model) under standard task loss, while the other branch feeds a frozen generative attacker (a pre-trained IP-Adapter combined with a Stable Diffusion U-Net) to evaluate reconstruction errors using both pixel and perceptual metrics. The projection is optimized to minimize task loss while maximizing reconstruction error. At inference deployment, the generative attacker is discarded, and the projection functions as a drop-in privacy barrier incurring negligible runtime overhead without requiring architectural modifications to the backbone encoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image x"] --> B["Frozen Vision Encoder f_v<br/>(CLIP ViT-L/14)"]
    B --> C["Original Vision Tokens z"]
    C --> D["Identity-Initialized Residual Projection P_theta"]
    D --> E["Protected Vision Tokens z_tilde"]
    E --> F["Downstream Task Heads h_psi<br/>(Linear Probe / LLaVA-SP)"]
    F --> G["Task Utility Loss L_task"]
    E --> H["Frozen Generative Attacker G_phi<br/>(IP-Adapter + SD U-Net)"]
    H --> I["Reconstructed Image x_tilde"]
    I --> J["Adversarial Reconstruction Loss L_rec<br/>(Pixel L2 + LPIPS)"]
    G -->|Forward Minimization| K["Jointly Update P_theta & Downstream Heads"]
    J -->|Negative Gradient Maximization| K

Key Designs

1. Identity-Initialized Lightweight Residual Projection: Cold-Start Utility Preservation

To prevent premature corruption of pre-trained feature manifolds during early training iterations, TrustCLIP adopts a non-invasive, token-wise projection architecture \(P_\theta(z) = z + f_\theta(z)\). The transformation \(f_\theta\) is parameterized as a compact multi-layer perceptron (MLP) with LayerNorm applied independently across tokens. Crucially, the final linear layer of \(f_\theta\) is initialized with all zeros. At the very onset of training, \(f_\theta(z) = 0\), rendering \(P_\theta(z)\) an exact identity mapping. This ensures seamless initialization directly from the native CLIP representation, allowing task and privacy gradients to co-adapt gradually and preventing the performance collapse typical of randomly initialized projection bottlenecks.

2. Generative Inversion Adversarial Joint Optimization: Directing Defense Against Generative Threats

Moving beyond discriminative proxy metrics that miss diffusion-based leakage, TrustCLIP directly incorporates a state-of-the-art conditional generator (IP-Adapter coupled with a frozen Stable Diffusion backbone) as the explicit white-box adversary \(G_\phi\). The reconstruction loss \(\mathcal{L}_{\text{rec}}\) combines low-level pixel distance with high-level perceptual discrepancy: $\(\mathcal{L}_{\text{rec}}(x, \tilde{x}) = \alpha \|x - \tilde{x}\|_p + (1 - \alpha) \text{LPIPS}(x, \tilde{x})\)$ where \(p \in \{1, 2\}\) and \(\alpha\) balances pixel fidelity and perceptual similarity. The defense parameters \(\theta\) and task head parameters \(\psi\) are jointly trained under the unified objective: $\(\mathcal{L}(\theta, \psi) = \mathbb{E}_{x, y} \left[ \mathcal{L}_{\text{task}}(h_\psi(\tilde{z}), y) - \lambda_{\text{rec}} \mathcal{L}_{\text{rec}}(x, G_\phi(\tilde{z})) \right]\)$ During back-propagation, the attacker \(G_\phi\) remains frozen, and the negative reconstruction gradient flows backward through the diffusion denoising pathway directly into \(P_\theta\). This dual-objective competition drives an orthogonal decomposition: features strongly aligned with classification and reasoning tasks are anchored by \(\mathcal{L}_{\text{task}}\), while instance-specific textures and identity cues critical for diffusion synthesis are actively suppressed by \(-\lambda_{\text{rec}} \mathcal{L}_{\text{rec}}\).

3. Adaptive Attacker Defense and Versatile Downstream Adaptation: Withstanding Worst-Case Adversaries

To guarantee that privacy is not an artifact of overfitting a fixed attacker, TrustCLIP introduces an adaptive evaluation protocol: once \(P_\theta\) has converged and is frozen, a brand-new generative attacker \(G_{\phi'}\) is trained from scratch directly on the protected feature distribution \(\tilde{z}\): $\(\phi' = \operatorname*{arg\,min}_\phi \mathbb{E}_x \left[ \mathcal{L}_{\text{rec}}(x, G_\phi(P_{\theta^\star}(f_v(x)))) \right]\)$ By the Data Processing Inequality, once the projection strips recoverable raw image information from \(\tilde{z}\), no downstream decoding algorithm can reconstruct what is no longer present. Furthermore, the learned projection integrates seamlessly across paradigms: for image classification, a linear probe is trained atop pooled tokens; for vision-language models (e.g., LLaVA-SP, configured as TrustLLaVA), \(P_\theta\) sits directly before the multimodal projector, co-training with standard language-modeling loss and LoRA adapters.

Loss & Training

For multimodal large language models, training stabilizes using a linear warm-up schedule: \(\lambda_{\text{rec}}\) is held at 0 for the initial \(N\) training steps to allow the task adapter and LLM to align with the visual tokens, before scaling linearly to its target value (e.g., \(\lambda_{\text{rec}} = 0.001\)). For image classification on SUN397, a constant trade-off parameter \(\lambda_{\text{rec}} \in [0.25, 1.0]\) is maintained. The framework optimizes using AdamW, freezing both the base vision encoder and all components of the generative attacker to ensure computational efficiency and avoid adversarial min-max instability.

Key Experimental Results

Main Results

Evaluated on the SUN397 scene classification benchmark across different loss variants and regularizer coefficients \(\lambda_{\text{rec}}\), TrustCLIP demonstrates minimal utility drop alongside drastic reductions in reconstruction fidelity. Lower PSNR and SSIM, together with higher LPIPS and DreamSim (DSIM), reflect superior privacy protection.

Data source: Table 1 in original paper

Setting	Pixel Loss	Perc. Loss	\(\lambda_{\text{rec}}\)	Top-1 Acc (%) ↑	Top-5 Acc (%) ↑	PSNR ↓	SSIM ↓	LPIPS ↑	DSIM ↑
CLIP (no defense)	—	—	—	83.36	97.76	13.581 ± 2.203	0.288 ± 0.147	0.522 ± 0.066	0.215 ± 0.055
TrustCLIP	\(\mathcal{L}_1\)	—	0.25	82.66	97.36	10.555 ± 1.608	0.191 ± 0.102	0.704 ± 0.051	0.522 ± 0.098
TrustCLIP	\(\mathcal{L}_2\)	—	0.25	82.49	97.41	10.527 ± 1.576	0.187 ± 0.097	0.702 ± 0.051	0.514 ± 0.097
TrustCLIP (recommended)	\(\mathcal{L}_2\)	LPIPS	0.25	82.92	97.58	10.687 ± 1.655	0.203 ± 0.109	0.699 ± 0.053	0.514 ± 0.097
TrustCLIP (high-privacy)	\(\mathcal{L}_2\)	LPIPS	1.00	82.76	97.67	10.617 ± 1.590	0.203 ± 0.093	0.776 ± 0.061	0.799 ± 0.068

Ablation Study

To confirm that learned adversarial suppression is fundamentally superior to isotropic corruption, TrustCLIP is benchmarked against additive Gaussian noise across various standard deviations \(\sigma\) under the identical fixed attacker on SUN397 test samples.

Data source: Table 2 in original paper

Defense Method	Top-1 Acc (%) ↑	LPIPS ↑	DSIM (DreamSim) ↑	Observations & Notes
CLIP (no defense)	83.9	0.58	0.34	High-fidelity reconstruction; zero privacy preservation
Gaussian Noise (\(\sigma = 0.01\))	84.1	0.59	0.34	Negligible privacy change
Gaussian Noise (\(\sigma = 0.05\))	83.8	0.58	0.33	Inversion quality identical to unprotected baseline
Gaussian Noise (\(\sigma = 0.10\))	82.7	0.58	0.33	Marginal utility drop; no privacy gain
Gaussian Noise (\(\sigma = 0.25\))	75.2	0.60	0.36	Top-1 drops 8.7%; privacy barely improves
Gaussian Noise (\(\sigma = 0.50\))	54.4	0.64	0.44	Severe utility collapse
Gaussian Noise (\(\sigma = 1.00\))	17.4	0.71	0.68	Utter utility failure; DSIM still inferior to TrustCLIP
TrustCLIP (Ours)	85.1	0.90	0.88	Preserves high utility while completely suppressing reconstruction

Furthermore, under multimodal large language model benchmarks (Table 3 in original paper): - Multimodal Utility Preservation: On POPE hallucination evaluation, TrustLLaVA attains 86.1 (comparable to unprotected LLaVA-SP at 86.6). Notably, it achieves gains on LLaVA-Bench (+3.6, rising from 61.7 to 65.3) and MM-Vet (+0.7, rising from 33.8 to 34.5), demonstrating that pruning noisy instance details can reduce spurious visual hallucinations. - Privacy Under Adaptive Attacks: When faced with a dedicated attacker retrained on protected tokens, TrustLLaVA elevates DSIM from 0.32 (LLaVA-SP) to 0.39 (identity-initialized) and 0.62 (standard MLP), with PSNR dropping to 7.13.

Key Findings

1. Asymmetric Impact on Semantics vs. Detail: Sub-category evaluation across MME-Perception and SEED-Bench (Table 4) reveals that high-level semantic categories (e.g., existence detection with 0.0 drop, artwork identification with 0.0 drop, and action prediction gaining +2.9%) are entirely preserved. Conversely, fine-grained tasks dependent on local pixel structure (e.g., OCR dropping -12.2% and counting dropping -14.5%) suffer modest declines. This empirical divergence firmly validates the premise that task semantics and reconstruction details inhabit separable feature dimensions.
2. Failure of Isotropic Feature Perturbations: When matched to the exact \(\ell_2\) feature displacement (\(\|\tilde{z} - z\|_2\)) of TrustCLIP, isotropic Gaussian noise collapses classification accuracy to 12.5% while achieving only 0.64 DSIM (compared to TrustCLIP's 82.9% Top-1 and 0.80 DSIM). This confirms that privacy stems from targeted information removal rather than perturbation magnitude.
3. Resilience Across Attacker Architectures: The defense holds not only against adaptive diffusion models retrained directly on protected features (retaining PSNR 11.04 vs. 13.58 on vanilla CLIP), but also transfers against un-cotrained architectures such as high-capacity IP-Adapter Plus (DSIM 0.88 vs. 0.80) and non-diffusion feed-forward CNN decoders, proving that information is genuinely eliminated at the feature level.

Highlights & Insights

• Targeting Real Generative Leakage: The framework shifts the paradigm away from shallow discriminative proxy metrics toward explicit defense against generative diffusion inversion, directly tackling the most practical and potent privacy threat to vision-language models.
• Identity-Initialized Residual Design: Utilizing zero-initialized terminal linear layers within a lightweight residual MLP prevents cold-start representation shock, allowing smooth co-adaptation of task and privacy gradients without fine-tuning the underlying visual backbone.
• Noise Elimination as an Implicit Regularizer: Rather than hurting general multimodal performance, eliminating fine-grained instance details improves benchmark scores on LLaVA-Bench and MM-Vet, demonstrating that stripping reconstructible pixel noise can act as an effective regularizer against visual hallucination in MLLMs.

Limitations & Future Work

• Performance Trade-Offs on Fine-Grained Tasks: Because pixel-level reconstruction and tasks like OCR, small-object counting, and dense bounding-box regression share overlapping high-frequency feature dependencies, aggressive suppression incurs a noticeable 6% to 12% accuracy drop on fine-grained benchmarks.
• Dependence on Differentiable Attacker Approximations: The current adversarial pipeline relies on back-propagating gradients through pre-trained diffusion adapters (IP-Adapter and Stable Diffusion U-Net). Extending defenses against non-differentiable or discrete token-based autoregressive generative decoders warrants further study.
• Dynamic Task-Conditioned Suppression: Future architectures could implement adaptive, instruction-aware routing that dynamically adjusts suppression strength depending on user prompts (e.g., enforcing maximal privacy for open-domain dialog while conditionally releasing high-frequency tokens when OCR instructions are detected).

Related Work & Insights

• vs SPAct / STPrivacy [7, 23]: Earlier video action privacy techniques optimized against discriminative identity classifiers; TrustCLIP is the first to optimize defenses directly against feature-conditioned generative diffusion attackers.
• vs DP-CLIP [18]: DP-CLIP injects differential privacy noise into contrastive pre-training to mitigate training-set membership inference, requiring exorbitant compute and degrading zero-shot utility; TrustCLIP provides a lightweight, post-hoc inference-time defense for deployed representations.
• vs NinjaDesc [32]: NinjaDesc modifies sparse geometric feature descriptors for SfM and visual localization; TrustCLIP secures dense, language-aligned multimodal tokens while preserving universal semantic similarity structures.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Pioneers an end-to-end adversarial defense framework against feature-conditioned diffusion inversion attacks]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Thoroughly evaluated across classification, modern MLLMs, adaptive retrained attackers, and transfer generator families]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clearly articulates the core tension and delivers a rigorous theoretical and empirical justification for asymmetric information separation]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, deployable privacy layer for edge-cloud collaborative MLLM inference and sensitive feature storage]

Related Papers

• [AAAI 2026] AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models
• [ICLR 2026] BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning
• [NeurIPS 2025] FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
• [ICML 2025] POPri: Private Federated Learning using Preference-Optimized Synthetic Data
• [ICLR 2026] Learning to Lie: Adversarial Attacks on Human-AI Teams and LLMs