Skip to content

SDSA: Shallow-Deep Squeezing Adapter for Vision-Language Models

Conference: ECCV 2026
Paper: ECCV 2026 Official
Code: https://github.com/haowang-ac/SDSA
Area: Multimodal VLM
Keywords: vision-language models, parameter-efficient fine-tuning, adapter, few-shot classification, cross-modal alignment

TL;DR

Addressing the issue where dense cross-modal interaction and unconstrained alignment capacity degrade novel-class generalization in vision-language model adapters, SDSA proposes a hierarchical two-stage "shallow-deep squeezing" adapter that imposes structured token sparsity via random masking and constrains alignment capacity via a shared low-rank subspace and cross-attention, significantly boosting few-shot base-to-novel generalization across 11 benchmarks.

Background & Motivation

Large-scale vision-language models (VLMs) such as CLIP acquire transferable multimodal representations through contrastive pretraining on paired image-text data. To adapt these frozen foundation models to downstream fine-grained recognition under data scarcity, parameter-efficient fine-tuning has developed rapidly through prompt learning (e.g., CoOp, CoCoOp, MaPLe) and lightweight adapters (e.g., CLIP-Adapter, MMA). In particular, multimodal adapters like MMA introduce bidirectional interactions between image and text branches, offering superior adaptation flexibility and reduced training overhead compared to heavy full-model tuning.

However, existing adapters encounter a critical conceptual bottleneck: cross-modal token-level interactions remain dense and fully coupled, while the alignment capacity across modalities lacks explicit regulation. Under few-shot supervision where training samples are scarce, such unrestricted dense coupling encourages the adapter to extend alignment into minor, unstable, or dataset-specific correlation directions. Consequently, while models fit base training classes effectively, their generalizability to unseen novel categories and out-of-distribution domains degrades severely due to overfitting and inter-class confusion.

The key insight to resolve this tension is to simultaneously govern the interaction density and the subspace capacity of cross-modal alignment: neither allowing unrestricted token-level coupling nor permitting alignment to drift freely in high-dimensional representations. Core idea: design a hierarchical Shallow-Deep Squeezing Adapter (SDSA) that utilizes token-level random masking in shallow layers to enforce structured sparsity, and employs a shared low-rank transformation combined with bidirectional cross-attention in deep layers to compress multimodal features into a compact dominant subspace.

Method

Overall Architecture

SDSA inserts lightweight squeezing adapters in parallel across Transformer layers of both the image and text encoders, while keeping the underlying pretrained CLIP backbone entirely frozen. Input images are converted into patch tokens alongside the class token (CLS), and candidate class labels are augmented with fine-grained visual descriptions generated by an LLM before text tokenization. Within each adapter stage, multimodal tokens first enter modality-specific "shallow squeezing" modules where token-level random Bernoulli masking filters out high-frequency noise and redundant activations. The sparsified features are linearly projected into a shared bottleneck dimension \(m\), entering the "deep squeezing" stage: a modality-shared low-rank transformation strictly confines cross-modal alignment to a dominant low-dimensional subspace, followed by bidirectional cross-attention for selective information refinement. Finally, the adapted features are projected back to backbone dimensions and aggregated via residual connections scaled by hyperparameters \(\alpha\) and \(\beta\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Images and Class Text"] --> B["Class Description Augmentation<br/>DeepSeek enriches fine-grained attributes"]
    B --> C["Backbone Encoding & Shallow Squeezing<br/>Token-level random Bernoulli masking"]
    C --> D["Shared Low-Rank Space Compression<br/>Low-rank bottleneck suppresses spurious directions"]
    D --> E["Bidirectional Cross-Modal Refinement<br/>Selective complementary cross-attention"]
    E --> F["Residual Re-injection & Prediction<br/>Dual scaling factors ฮฑ, ฮฒ aggregate representations"]

Key Designs

1. Class Description Augmentation: enriching fine-grained linguistic priors

A fundamental limitation in standard CLIP adaptation is the extreme information disparity between modalities: visual inputs comprise dense grids of patch tokens, whereas standard text inputs consist of concise class prompts like "a photo of a [class]". To balance token information content, SDSA employs a large language model (DeepSeek) to enrich class names into descriptive sentences capturing appearance, color, and contextual cues (e.g., transforming "a photo of a [tench]" into "The tench is a fish with olive-green skin and red eyes"). This textual augmentation introduces rich, discriminative linguistic priors into the text branch, establishing a balanced foundation for subsequent fine-grained cross-modal alignment.

2. Shallow Squeezing: token-level random sparsity to suppress noisy coupling

To prevent dense token interactions from learning fragile, dataset-specific cross-modal couplings, the shallow squeezing module acts as a lightweight pre-filter prior to deep alignment. During training, a Bernoulli-distributed random binary mask is sampled along the sequence dimension with a retention probability \(\rho = 1 - \text{mask\_ratio}\), independently dropping features across visual tokens \(E_i\) and textual tokens \(T_i\):

\[x_i^j \sim \text{Bernoulli}(\rho), \quad x_i \in E_i, \, j \in d_v\]
\[t_i^j \sim \text{Bernoulli}(\rho), \quad t_i \in T_i, \, j \in d_l\]

This random sparsification forces the model to reconstruct invariant representations from partial cues, breaking superficial local co-occurrences. The sparsified representations are subsequently mapped into a unified bottleneck dimension \(m\) (\(m = 128\)) via linear fully connected layers, converting dense token interaction into a compact, structured representation.

3. Shared Low-Rank Transformation: constraining capacity to dominant semantic directions

To prevent the adapter from fitting minor, unstable noise directions under limited few-shot supervision, the deep squeezing module applies a shared low-rank parameterization within the bottleneck space. Multi-modal bottleneck representations pass through shared projection matrices \(A \in \mathbb{R}^{m \times r}\) and \(B \in \mathbb{R}^{r \times m}\) (with rank \(r \ll m\), default \(r = 16\)), followed by ReLU activation and Dropout:

\[x_i = \text{Dropout}(\phi(x_i A) B), \quad t_i = \text{Dropout}(\phi(t_i A) B)\]

Sharing weights across visual and language branches enforces joint projection into a single shared low-dimensional manifold. This low-rank constraint geometrically restricts alignment capacity, filtering out dataset-specific spurious correlations and concentrating representation energy onto dominant, transferable semantic axes.

4. Cross-Modal Attention and Dual Residual Aggregation: selective refinement and calibrated injection

Following the low-rank subspace projection, bidirectional cross-attention selectively exchanges complementary cues across modalities. The text representations query visual patch features for concrete spatial grounding, while visual features query text representations for semantic context:

\[\hat{x}_i = \text{CrossAttn}(Q = x_i, K = t_i, V = t_i), \quad \hat{t}_i = \text{CrossAttn}(Q = t_i, K = x_i, V = x_i)\]

To preserve pretrained foundation capabilities while integrating adapted features, SDSA re-injects the transformed representations back into the original Transformer outputs using dual scaling factors \(\alpha\) and \(\beta\):

\[[c_j, E_j] = \mathcal{V}_j([c_{j-1}, E_{j-1}]) + \alpha \Phi([c_{j-1}, E_{j-1}]) + \beta \Phi([c_{j-1}, \hat{E}_{j-1}])\]
\[[T_j] = \mathcal{T}_j([T_j]) + \alpha \Phi([T_{j-1}]) + \beta \Phi([\hat{T}_{j-1}])\]

where \(\Phi\) denotes linear projection expanding features back to the backbone embedding dimension. Setting \(\alpha = 0.05\) and \(\beta = 0.01\) reflects the design principle that shared low-rank alignment carries the core generalizable representation, while cross-modal attention is injected conservatively to avoid disrupting stability.

Loss & Training

During the entire fine-tuning process, all original CLIP parameters are kept frozen, and only the SDSA adapter parameters are updated. Optimization is performed using standard cross-entropy contrastive loss over image-text cosine similarities:

\[\mathcal{L} = -\frac{1}{B} \sum_{k=1}^B \log \frac{\exp(\text{sim}(x_k, z_{y_k}) / \tau)}{\sum_{c=1}^C \exp(\text{sim}(x_k, z_c) / \tau)}\]

Using a ViT-B/16 backbone, the default hyperparameters are mask_ratio = 0.10, bottleneck dimension \(m = 128\), rank \(r = 16\), \(\alpha = 0.05\), and \(\beta = 0.01\). Under the 16-shot base-to-novel setup, the model is trained with the SGD optimizer at an initial learning rate of 0.02 for 20 epochs (10 epochs with batch size 32 for ImageNet; batch size 16 for all other datasets). Experiments are conducted on an RTX 5090 GPU.

Key Experimental Results

Main Results

On 11 benchmark recognition datasets (ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101) under the 16-shot Base-to-Novel evaluation setting, SDSA achieves state-of-the-art generalization on unseen novel classes and overall harmonic mean (HM).

Method Adaptation Type 11-Dataset Avg Base 11-Dataset Avg Novel 11-Dataset Avg HM EuroSAT (Novel/HM) FGVCAircraft (Novel/HM)
CLIP (Zero-Shot) None 69.34 74.22 71.70 64.05 / 60.03 36.29 / 31.09
CoOp Prompt Tuning 82.69 63.22 71.66 54.74 / 68.69 22.30 / 28.75
CoCoOp Conditional Prompt 80.47 71.69 75.83 60.04 / 71.21 23.71 / 27.74
MaPLe Multimodal Prompt 82.28 75.14 78.55 73.23 / 82.35 35.61 / 36.50
PromptSRC Self-Regulated Prompt 84.26 76.10 79.97 73.90 / 82.32 37.87 / 40.15
CoPrompt Consistency Prompt 84.00 77.23 80.48 78.57 / 85.84 39.33 / 39.76
MMA Multimodal Adapter 83.20 76.80 79.87 82.34 / 83.87 36.33 / 38.33
2SFS Two-Stage Tuning 85.55 75.48 80.20 67.09 / 79.29 35.51 / 40.63
SDSA (Ours) Shallow-Deep Squeezing 83.77 78.34 80.96 86.33 / 87.37 45.13 / 44.02

In cross-dataset evaluation (fine-tuned on ImageNet 16-shot and evaluated zero-shot across 10 remaining target datasets), SDSA attains an average accuracy of 67.32%, outperforming MMA (66.61%) and MaPLe (66.30%), achieving top ranks on 7 out of 10 target domains including OxfordPets (92.17%), FGVCAircraft (30.33%), and UCF101 (70.43%).

Ablation Study

The ablation investigations across all 11 datasets validate the critical contributions of the low-rank projection (\(\alpha\)), cross-attention (\(\beta\)), and key structural hyperparameters.

Config / Variant Module Setting Base Accuracy Novel Accuracy HM (Harmonic Mean) Note
Backbone baseline \(\alpha = 0, \beta = 0\) 68.85 75.15 71.86 Zero-shot backbone without adapters
Low-rank only \(\alpha = 0.05, \beta = 0\) 83.25 77.83 80.45 Shared low-rank provides foundational stability
Cross-attention only \(\alpha = 0, \beta = 0.01\) 72.82 76.31 74.52 Lacks low-rank regularization; unstable alignment
Full SDSA \(\alpha = 0.05, \beta = 0.01\) 83.77 78.34 80.96 Optimal balance of capacity and interaction
Mask ratio = 0.00 No token masking 83.83 78.18 80.91 Dense interactions slightly degrade novel generalization
Mask ratio = 0.10 Default masking 83.77 78.34 80.96 Best trade-off for structured sparsity
Mask ratio = 0.30 High masking 83.22 78.20 80.63 Drops informative visual tokens
Low-rank \(r = 8\) Minimal rank 82.94 78.22 80.51 Capacity too constrained
Low-rank \(r = 16\) Default rank 83.77 78.34 80.96 Optimal dominant semantic subspace
Low-rank \(r = 64\) High rank 84.69 77.68 81.03 Base overfits; novel performance drops

Key Findings

  • Low-rank bottleneck is essential for stable alignment: Retaining only cross-attention (\(\beta = 0.01\)) collapses Base accuracy to 72.82% and HM to 74.52%, whereas shared low-rank transformation alone (\(\alpha = 0.05\)) sustains an HM of 80.45%. Unconstrained token interaction diverges easily; the low-rank subspace constraint provides the stabilizing backbone.
  • Subspace capacity trade-off: Increasing rank \(r\) from 8 to 64 steadily increases Base accuracy (82.94% to 84.69%), but Novel accuracy peaks at \(r = 16\) (78.34%) and drops to 77.68% at \(r = 64\). Excess rank capacity allows adapters to encode spurious correlations specific to base training classes.
  • Structured sparsity regularizes dense coupling: Incorporating 10% random token masking outperforms the fully dense setting (ratio = 0.00) by boosting Novel accuracy from 78.18% to 78.34%, validating that controlled token dropping prevents reliance on isolated features.

Highlights & Insights

  • Constraining capacity beats expanding interactions: While prior VLM adapters continuously increase interaction complexity and degrees of freedom, SDSA proves that restricting cross-modal capacity through hierarchical compression (shallow random masking + deep shared low-rank transformation) is significantly more effective at preventing few-shot overfitting.
  • Symmetric parameter sharing enforces geometric alignment: Applying the exact same projection matrices \(A\) and \(B\) to both modalities enforces image and text features into a common low-dimensional manifold, guaranteeing geometric consistency that independent low-rank adapters (e.g., vanilla LoRA) fail to enforce.
  • Lightweight, generalizable plug-and-play design: Requiring only 4.79M trainable parameters on ViT-B/16, SDSA introduces negligible overhead and produces sharp diagonal alignment heatmaps with high intra-class compactness and inter-class separation.

Limitations & Future Work

  • Sensitivity under extreme few-shot supervision: In 1-shot and 2-shot regimes, the model exhibits sensitivity to scaling factors \(\alpha\) and \(\beta\), suggesting that dynamic, data-adaptive capacity scheduling could be beneficial.
  • Random uniform masking: The shallow compression stage currently relies on uniform Bernoulli random masking without considering semantic salience, which may occasionally drop critical foreground visual tokens.
  • Extension to autoregressive MLLMs: The current framework is evaluated on dual-encoder contrastive CLIP models; validating this hierarchical compression mechanism on generative multimodal LLMs represents an exciting future direction.
  • vs MaPLe & PromptSRC: MaPLe and PromptSRC couple modalities via learnable prompt tokens inserted into Transformer layers. SDSA demonstrates that parallel low-rank adapters with explicit sparsity filters yield sharper cross-modal alignment without prompt interference.
  • vs MMA (Multi-Modal Adapter): MMA enables bidirectional adapter tuning but suffers from dense interactions and diffuse alignment heatmaps. SDSA replaces dense full-rank adapters with shallow-deep squeezing, demonstrating noticeably cleaner diagonal logit distributions and superior novel-class accuracy on EuroSAT and FGVCAircraft.
  • vs LoRA / CLIP-Adapter: Standard LoRA tunes individual modalities separately, while CLIP-Adapter operates solely on final output embeddings. SDSA achieves end-to-end, layer-wise joint alignment in a unified shared bottleneck subspace.

Rating

  • Novelty: โญโญโญโญ [Introduces hierarchical shallow-deep squeezing to explicitly regulate VLM adapter interaction density and capacity]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 11 datasets, Base-to-Novel, cross-dataset, domain shift, and granular ablations]
  • Writing Quality: โญโญโญโญโญ [Clear motivation, well-formulated technical exposition, and thorough qualitative analysis]
  • Value: โญโญโญโญ [Provides practical regularization guidelines and lightweight adapter architecture for few-shot multimodal learning]