Subspace Alignment for Vision-Language Model Test-time Adaptation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zhichenz98/SubTTA_EMNLP26
Area: Multimodal VLM
Keywords: Vision-Language Models, Test-Time Adaptation, Subspace Alignment, Modality Gap, Semantic Projection
TL;DR¶
SubTTA addresses the failure modes of vision-language model test-time adaptation under severe distribution shifts by geometrically aligning visual-textual subspaces via Grassmannian chordal distance minimization and filtering task-irrelevant visual nuisances via hard textual subspace projection.
Background & Motivation¶
Pre-trained Vision-Language Models (VLMs) such as CLIP and ALIGN have established remarkable zero-shot transfer performance across a broad spectrum of visual recognition domains. This capability fundamentally relies on their shared high-dimensional joint embedding space, where visual features are globally aligned with descriptive textual concepts. In real-world deployments, however, models inevitably confront severe out-of-distribution shiftsโsuch as optical corruptions, adverse weather perturbations, and stylistic variations. To adapt pre-trained models on the fly without supervised target labels, test-time adaptation (TTA) has emerged as an essential operational paradigm. Existing TTA frameworks, spanning training-free memory banks (e.g., TDA, ECALP) and gradient-based parameter updates (e.g., TENT, TPT, MINT, BATCLIP), predominantly rely on the model's raw zero-shot predictions as pseudo-labels to guide online optimization.
Nevertheless, under distribution shifts, this foundational reliance on raw zero-shot predictions becomes precarious and often triggers catastrophic adaptation failures. An in-depth geometric diagnosis reveals two structural failure modes: First, the Modality Gap, where distribution shifts induce a global drift and rotation of the visual manifold away from the textual anchors. Principal angle measurements demonstrate that severe corruptions systematically widen the geometric angle between the visual and textual subspaces, causing visual representations (e.g., an image of a dog) to drift dangerously close to incorrect textual anchors (e.g., bird). Second, Visual Nuisance, where high-dimensional visual embeddings unavoidably capture extraneous background noise, lighting artifacts, and corruption textures. Under domain shifts, these task-irrelevant factors overwhelm core semantic signalsโas quantified by a steep drop in semantic concentrationโcausing instances like a dog set against a prominent blue sky to be misclassified as a bird through spurious correlations.
Consequently, pseudo-labels extracted from raw visual features are contaminated by these combined geometric distortions, causing standard TTA objectives to reinforce false decision boundaries rather than rectify them. Prior subspace methods were primarily designed for few-shot or unimodal settings and relied on rigid rank-to-rank distance metrics, which erroneously bind dominant statistical noise dimensions to prime textual bases. Core idea: propose SubTTA, a subspace-centric test-time adaptation framework that employs exponential moving average covariance estimation to extract principal subspaces, minimizes basis-invariant chordal distance on the Grassmannian manifold to resolve global modality drift, and projects aligned visual embeddings onto the task-specific textual subspace to filter out orthogonal visual nuisances before self-training.
Method¶
Overall Architecture¶
SubTTA re-engineers the standard test-time adaptation pipeline by establishing a purified, geometrically aligned subspace before evaluating or optimizing any adaptation objective. The workflow comprises an offline textual subspace initialization and a three-step online adaptation pipeline: given normalized textual prompts of all target classes, SubTTA computes the principal textual basis \(B_T\); for incoming test image batches, it dynamically updates a running visual covariance matrix using an exponential moving average (EMA) to derive the principal visual basis \(B_V\); it then adapts the image encoder's normalization layers to minimize the Grassmannian chordal distance between \(B_V\) and \(B_T\); finally, it projects the aligned visual representations onto \(B_T\) to eliminate orthogonal noise, producing purified features \(V_{\text{proj}}\) on which downstream TTA objectives (e.g., entropy minimization or cluster-based self-training) optimize decision boundaries.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Streaming Test Images & Class Prompts<br/>X~(k)~ and t~1~...t~C~"] --> B["Momentum Covariance & Principal Subspace Extraction"]
B --> C["Chordal Distance Geometric Alignment"]
C --> D["Textual Subspace Semantic Projection"]
D --> E["Purified Feature Self-Training & Class Prediction<br/>Optimize TTA objective on V_proj"]
Key Designs¶
1. Momentum Covariance & Principal Subspace Extraction: Mitigating Small-Batch Statistical Variance In realistic streaming TTA, test images arrive in small batches where raw sample covariance matrices exhibit high variance and fail to capture the true underlying data manifold. To resolve this statistical deficit, SubTTA leverages the compact, semantic-rich textual prompts as an anchor prior. Denoting normalized class text embeddings by \(T = [t_1, \dots, t_C]^\top \in \mathbb{R}^{C \times d}\), the static text covariance is formed as \(\Sigma_T = T^\top T \in \mathbb{R}^{d \times d}\). The visual covariance matrix is initialized directly with this semantic prior, \(\Sigma_V \leftarrow \Sigma_T\). For each arriving test batch \(X^{(k)}\) yielding normalized visual features \(V^{(k)} \in \mathbb{R}^{B \times d}\), SubTTA updates the visual covariance via an Exponential Moving Average (EMA): $\(\Sigma_V \leftarrow (1 - \alpha)\Sigma_V + \alpha (V^{(k)\top} V^{(k)})\)$ where \(\alpha \in (0, 1)\) denotes the momentum coefficient. Performing rank-\(r\) truncated eigendecomposition on \(\Sigma_T\) and \(\Sigma_V\) yields orthonormal principal bases \(B_T \in \mathbb{R}^{r \times d}\) and \(B_V \in \mathbb{R}^{r \times d}\), establishing smooth, noise-resilient subspace approximations without requiring large batch sizes.
2. Chordal Distance Geometric Alignment: Circumventing False Rank-to-Rank Correspondences Standard subspace metrics such as the Frobenius norm \(\|B_T - B_V\|_F\) or component-wise cosine similarities enforce a strict rank-to-rank correspondence, dictating that the \(i\)-th visual eigenvector must match the \(i\)-th textual eigenvector. However, eigenvectors are sorted strictly by statistical variance rather than semantic saliency; under severe corruptions, the leading visual eigenvector often captures dominant non-semantic artifacts (e.g., fog patterns or background brightness), causing rigid metrics to misalign noise with primary textual categories. To ensure that the alignment metric depends exclusively on the subspace geometry rather than the specific basis orientation, SubTTA models the subspaces as elements on the Grassmannian manifold \(\text{Gr}(r, d)\) and minimizes their chordal distance: $\(d_{\text{ch}}(B_T, B_V) = \sqrt{\sum_{i=1}^r \sin^2 \theta_i}\)$ where \(\theta_i\) denotes the \(i\)-th principal angle between \(\text{span}(B_T)\) and \(\text{span}(B_V)\). The squared chordal distance objective simplifies to an efficient Frobenius formulation: $\(\mathcal{L}_{\text{align}} = d_{\text{ch}}^2(B_T, B_V) = r - \|B_T B_V^\top\|_F^2\)$ Because \(\|B_T (Q B_V)^\top\|_F^2 = \|B_T B_V^\top\|_F^2\) holds for any orthogonal rotation matrix \(Q \in \mathbb{R}^{r \times r}\), the alignment objective is completely invariant to internal basis coordinate rotations. By keeping the textual anchor fixed and updating only the normalization layer parameters of the visual encoder via Adam with \(\nabla_\theta \mathcal{L}_{\text{align}}\), the visual manifold is rotated back into geometric congruence with the invariant text space.
3. Textual Subspace Semantic Projection: Eliminating Orthogonal Visual Nuisances While geometric alignment rectifies the global manifold drift, it leaves intra-sample nuisance features intact within the visual representations. Recognizing that the target classification semantics are fully captured within the span of the textual basis \(B_T\), SubTTA utilizes \(B_T\) as an explicit linear semantic filter. For any adapted image representation \(v \in \mathbb{R}^d\), the semantic projection is defined as: $\(\text{Proj}(v) = v B_T^\top B_T\)$ This orthogonal projection operator mathematically eliminates all visual feature components that lie orthogonal to the semantic span, stripping away corruptions, clutter, and irrelevant style dimensions while preserving core class semantics. The purified embeddings \(V_{\text{proj}}\) exhibit substantially higher semantic concentration and sharper inter-class margins, enabling downstream TTA objectives (such as cluster-based losses or entropy minimization) to compute dependable pseudo-labels without reinforcing spurious correlations.
A Worked Example¶
Consider an image of a Golden Retriever from ImageNet-C subjected to severe Fog corruption (Severity Level 5) against an expansive green grass field: 1. Raw Zero-Shot State: High-variance fog and grass textures dominate the high-dimensional representation; the principal angle between raw visual features and the text manifold reaches \(68^\circ\), while the semantic concentration ratio drops to 0.28. The baseline zero-shot model erroneously classifies the dog as Sheep with 0.72 confidence due to pasture background bias. 2. Momentum Subspace Estimation & Alignment: The test batch is processed through the EMA covariance estimator with \(\alpha = 0.5\). Truncated SVD yields the \(r = 64\) visual basis \(B_V\). Minimizing \(\mathcal{L}_{\text{align}} = 64 - \|B_T B_V^\top\|_F^2\) updates the visual encoder's LayerNorm parameters in a single gradient step, reducing the average principal angle to \(42^\circ\) and correcting the global manifold orientation. 3. Semantic Projection & Rectified Prediction: The updated feature \(\tilde{v}\) is projected via \(\tilde{v} B_T^\top B_T\). Orthogonal fog and grass components are discarded, raising the semantic concentration to 0.81. The model correctly identifies Golden Retriever with 0.89 confidence, providing a purified pseudo-label that successfully guides the subsequent self-training step without error accumulation.
Loss & Training¶
SubTTA follows a two-step alternating optimization per test batch: 1. Geometric Alignment Step: Compute \(V^{(k)} = f_v(X^{(k)})\), update \(\Sigma_V\), extract \(B_V\), evaluate \(\mathcal{L}_{\text{align}} = r - \|B_T B_V^\top\|_F^2\), and perform one Adam gradient update on the visual encoder normalization layers; 2. Adaptation Step: Re-extract features \(\tilde{V}^{(k)}\) using updated parameters, project onto the textual basis \(V_{\text{proj}} = \tilde{V}^{(k)} B_T^\top B_T\), compute the standard TTA loss \(\mathcal{L}_{\text{TTA}}\) (e.g., MINT cluster loss or entropy minimization), and perform a second Adam gradient step; 3. Hyperparameters: Subspace rank \(r = 64\) (for \(d = 512\)), momentum coefficient \(\alpha = 0.5\), batch size \(B = 64\), Adam optimizer with learning rate \(10^{-4}\).
Key Experimental Results¶
Main Results¶
The authors conduct comprehensive evaluations on three 15-corruption benchmarks at the highest corruption severity (Level 5) using the ViT-B-16 architecture, comparing against source zero-shot CLIP, training-free methods (TDA, DMN, VTE, ZERO, ECALP), and gradient-based adaptation methods (TENT, RoTTA, TPT, MEMO, WATT, STS, MINT, BATCLIP).
| Category | Method | ImageNet-C (Mean) | CIFAR-10-C (Mean) | CIFAR-100-C (Mean) |
|---|---|---|---|---|
| Unadapted | Source CLIP | 24.54% | 61.16% | 35.78% |
| Training-free | TDA | 26.22% | 62.37% | 36.72% |
| Training-free | DMN | 24.16% | 62.15% | 35.04% |
| Training-free | VTE | 25.61% | 64.16% | 35.02% |
| Training-free | ZERO | 24.90% | 62.11% | 36.62% |
| Training-free | ECALP | 25.16% | 63.96% | 36.98% |
| Gradient-based | TENT | 25.18% | 61.92% | 36.98% |
| Gradient-based | RoTTA | 24.71% | 61.85% | 36.16% |
| Gradient-based | TPT | 24.85% | 63.65% | 36.15% |
| Gradient-based | MEMO | 24.63% | 61.81% | 36.93% |
| Gradient-based | WATT | 26.18% | 66.57% | 41.91% |
| Gradient-based | STS | 27.06% | 66.66% | 39.29% |
| Gradient-based | MINT | 29.18% | 70.20% | 42.41% |
| Gradient-based | BATCLIP | 30.06% | 73.79% | 42.23% |
| Ours | SubTTA | 32.15% | 76.33% | 44.49% |
On ImageNet-C, SubTTA establishes a new state-of-the-art with 32.15% mean accuracy, outperforming the strongest baseline BATCLIP (30.06%) by 2.09% and the source baseline by 7.61%. Consistent gains are achieved on CIFAR-10-C (+2.54% over BATCLIP) and CIFAR-100-C (+2.08% over MINT). On natural distribution shift benchmarks (ImageNet-A, ImageNet-R, ImageNet-K), SubTTA consistently improves downstream TTA performance by an average of 0.56%.
Ablation Study¶
The ablation analysis examines the impact of the Exponential Moving Average (EMA) mechanism across benchmarks when updating the visual covariance:
| Dataset | Configuration | Source CLIP | TENT Gain | BATCLIP Gain | MINT Gain |
|---|---|---|---|---|---|
| ImageNet-C | w/o EMA (\(\alpha = 1.0\)) | 24.51% | 21.03% | 31.23% | 31.43% |
| ImageNet-C | w/ EMA (\(\alpha = 0.5\)) | 25.87% | 24.98% | 31.90% | 32.15% |
| ImageNet-C | \(\Delta\) (EMA Benefit) | +1.36% | +3.95% | +0.67% | +0.72% |
| CIFAR-10-C | w/o EMA (\(\alpha = 1.0\)) | 61.15% | 61.30% | 75.48% | 72.95% |
| CIFAR-10-C | w/ EMA (\(\alpha = 0.5\)) | 68.87% | 68.18% | 77.79% | 76.33% |
| CIFAR-10-C | \(\Delta\) (EMA Benefit) | +7.72% | +6.88% | +2.31% | +3.38% |
| CIFAR-100-C | w/o EMA (\(\alpha = 1.0\)) | 35.78% | 35.41% | 41.67% | 42.88% |
| CIFAR-100-C | w/ EMA (\(\alpha = 0.5\)) | 40.12% | 41.13% | 43.87% | 44.49% |
| CIFAR-100-C | \(\Delta\) (EMA Benefit) | +4.34% | +5.72% | +2.20% | +1.61% |
Component ablations (Figure 9) reveal: 1. w/o Geometric Alignment (-align): Projecting features onto an unaligned textual space maps samples to incorrect semantic axes; TENT suffers severe negative transfer under this setting, proving alignment is a strict prerequisite. 2. w/o Semantic Projection (-project): Aligning subspaces without subsequent orthogonal projection leaves task-irrelevant visual noise in the embedding, degrading accuracy by 1.5%โ2.8%. 3. Synergy: The full SubTTA architecture decisively outperforms both individual variants, demonstrating that global drift correction and local nuisance filtering are complementary.
Key Findings¶
- Elimination of Catastrophic Negative Adaptation: Conventional entropy minimization methods like TENT collapse under extreme noise (e.g., scoring only 5.20% on Gaussian noise vs. Source 11.14%); SubTTA consistently outperforms the unadapted Source baseline across all 15 corruption types.
- Pronounced Advantages on Weaker Backbones: On ViT-B-32, SubTTA achieves 30.50% on ImageNet-C, surpassing BATCLIP (27.37%) by 3.13%, confirming that subspace rectification provides vital compensation for lower-capacity models where pre-trained alignments are fragile.
- Class-Scalable Runtime Complexity: SubTTA's computational complexity \(\mathcal{O}(d^2 r)\) is independent of class count \(C\), scaling gracefully up to 1,000 classes where cluster-based TTA methods incur \(\mathcal{O}(B C d)\) scaling.
Highlights & Insights¶
- Grassmannian Chordal Distance Invariance: By measuring distance on the Grassmannian manifold rather than enforcing rigid Euclidean or cosine alignment, SubTTA avoids erroneously aligning high-variance visual noise with primary semantic text anchors.
- Asymmetric Textual Anchoring: Capitalizing on the intrinsic asymmetry between compact textual semantics and redundant visual signals, SubTTA preserves the textual space as a frozen geometric anchor to guide visual rectification.
- Plug-and-Play Generalizability: The framework adds minimal latency (~0.15s per batch for SVD) and seamlessly integrates with existing entropy-based or cluster-based TTA objectives as a front-end geometric purifier.
Limitations & Future Work¶
- Reliance on Batch Statistics: Subspace estimation relies on streaming mini-batch statistics; strictly episodic settings (single isolated test samples) require an auxiliary memory bank to accumulate historical features.
- Access to Model Gradients: Updating normalization layers requires white-box gradient access, preventing deployment on closed-source black-box model APIs.
- Future Directions: Exploring low-rank matrix tracking to replace exact SVD operations and extending subspace principles to prompt-based black-box adaptation.
Related Work & Insights¶
- vs STS (Dafnis & Metaxas, 2026): STS aligns text representations to visual embeddings, risking accommodation of severe visual noise; SubTTA keeps text fixed and rectifies visual geometry, ensuring semantic fidelity.
- vs MINT (Bao et al., 2025) & BATCLIP (Maharana et al., 2025): While cluster-based methods optimize local sample distributions, they ignore global cross-modal manifold rotation; SubTTA serves as a complementary geometric front-end, elevating MINT and BATCLIP to new empirical records.
- vs Classical Manifold Adaptation (SA / GFK): Classical subspace alignment requires access to offline source data and full target batches; SubTTA successfully adapts manifold alignment to an online, source-free VLM setting.
Rating¶
- Novelty: โญโญโญโญโญ [Elegant framing of VLM TTA failure through Grassmannian geometry and orthogonal subspace noise filtering]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 3 corrupted datasets, 3 out-of-distribution benchmarks, 3 ViT backbones, and detailed ablations]
- Writing Quality: โญโญโญโญโญ [Clear mathematical derivation, cohesive motivation narrative, and insightful geometric interpretations]
- Value: โญโญโญโญโญ [Establishes a robust, theoretically grounded paradigm for test-time adaptation in multimodal foundation models]