CS-TTA: Preserving Concept Sensitivity in Test-Time Adaptation

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Medical Imaging
Keywords: Test-Time Adaptation, Concept Sensitivity Drift, Concept Activation Vectors, Spurious Correlations, Cross-Hospital Generalization

TL;DR

Addressing the hidden danger that unsupervised test-time adaptation (TTA) can exacerbate reliance on spurious shortcuts while improving overall accuracy, this paper formalizes Concept Sensitivity Drift (CSD) via TCAV directional derivatives and proposes CS-TTA—a plug-in regularizer that penalizes excessive sensitivity drift to anchor feature representations to task-relevant concepts without requiring target labels or architectural changes.

Background & Motivation

Test-time adaptation (TTA) has become a primary paradigm for mitigating distribution shifts. Given a source-pretrained model, methods such as entropy minimization (Tent), mean-teacher consistency (CoTTA), and selective sample filtering (EATA) adapt network parameters—typically the affine coefficients of batch normalization layers—directly on unlabeled test streams. While substantial research has addressed practical issues including small batch sizes, catastrophic forgetting, and temporal sample correlation, evaluation remains almost universally anchored to a single standard: task classification accuracy. Any model achieving higher accuracy post-adaptation is deemed successful, leaving the underlying feature representations and decision rationale unexamined.

This evaluation blind spot presents severe risks in safety-critical domains susceptible to shortcut learning. In medical imaging (e.g., chest radiograph diagnosis), classifiers readily exploit hospital-specific text markers, device shadows, or acquisition artifacts rather than genuine pathological manifestations. Similarly, in natural computer vision, models frequently rely on background context or texture instead of core object shape. When standard TTA techniques adapt models by minimizing unsupervised objectives like prediction entropy, they often reduce uncertainty by amplifying reliance on these spurious correlations. Consequently, models can display superficial accuracy gains while suffering catastrophic degradation on minority groups or upon subsequent deployment across differing clinical centers.

This work directly investigates this critical tension: tracking the trajectory of human-interpretable semantic concepts across adaptation using directional derivatives from Concept Activation Vectors (TCAV). This reveals the phenomenon of Concept Sensitivity Drift (CSD)—where mainstream TTA methods systematically diminish model sensitivity to causal and pathological attributes while elevating dependence on spurious background or institutional nuisance features. Core idea: by anchoring semantic directions pre-extracted from source concept examples, CS-TTA introduces a target-label-free, plug-in regularization objective that selectively penalizes concept sensitivity deviations beyond a tolerance threshold, steering adaptation away from spurious shortcuts without altering network architecture or adaptation pipelines.

Method

Overall Architecture

CS-TTA operates in two distinct phases: offline concept extraction on the source domain and online sensitivity-preserving adaptation on the target test stream. During the offline phase, Concept Activation Vectors (CAVs) are trained via linear logistic regression on source activation representations (e.g., the penultimate residual block in ResNet-50 or the final dense block in DenseNet-121) using positive and random concept exemplars, and are subsequently frozen. During the online adaptation phase, the model processes incoming batches of unlabeled target samples. Alongside computing the base unsupervised TTA objective (such as Tent entropy or CoTTA teacher-student consistency), the model evaluates the batch-level concept sensitivity across monitored directions. For concepts whose drift relative to the source anchor exceeds a tolerance threshold \(\tau\), a sensitivity-preserving penalty \(L_{\text{preserve}}\) is engaged and combined with the primary TTA objective to guide parameter updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unlabeled target test mini-batch"] --> B["Forward pass & deep feature activations"]
    B --> C["Offline frozen source CAV bank"]
    B --> D["Compute base unsupervised loss LTTA<br/>(Entropy / Consistency / Filtering)"]
    C --> E["Concept Sensitivity Drift (CSD) detection"]
    B --> E
    E --> F["Gated preservation loss Lpreserve<br/>(|S_C(t) - S_C(0)| > tau)"]
    D --> G["Total weighted backpropagation<br/>Ltotal = LTTA + lambda * Lpreserve"]
    F --> G
    G --> H["Update BN affine parameters only (gamma, beta)"]

Key Designs

1. Concept Sensitivity Drift quantification: directional derivatives as dynamic probes To quantitatively measure how adaptation reshapes a network's feature dependencies, the method utilizes the continuous directional derivative of class logits with respect to intermediate activations along the concept normal vector: \(S_{C, k}(x) = \nabla_{a^{(l)}} f_k(a^{(l)}(x)) \cdot v_C\). Standard TCAV summarizes this as a binary ratio of positive directional derivatives, which is non-differentiable. Instead, CS-TTA computes the batch-averaged continuous sensitivity: $\(\bar{S}_C(\theta_t) = \frac{1}{|\mathcal{B}_t|} \sum_{x \in \mathcal{B}_t} S_{C, k(x)}(x; \theta_t)\)$ The resulting Concept Sensitivity Drift metric, \(\text{CSD}_C(t) = \bar{S}_C(\theta_t) - \bar{S}_C(\theta_0)\), demonstrates that conventional TTA causes sensitivity toward task-relevant concepts to decline sharply while driving sensitivity toward nuisance concepts steadily upward. Crucially, CSD accumulated early during adaptation (e.g., at step 100) strongly correlates with subsequent accuracy degradation (Spearman rank correlation \(\rho = 0.76\)), establishing CSD as an effective early diagnostic indicator of adaptation failure.

2. Gated sensitivity-preserving loss: targeted constraint without over-regularization Imposing unconditional parameter constraints across all activation channels restricts the model's capacity to adjust to genuine domain shifts. CS-TTA employs a threshold-gated quadratic penalty that selectively intervenes only when the batch-level sensitivity drift for a monitored concept surpasses threshold \(\tau\): $\(L_{\text{preserve}}(\theta_t) = \sum_{C : |\bar{S}_C(\theta_t) - \bar{S}_C(\theta_0)| > \tau} \left(\bar{S}_C(\theta_t) - \text{sg}(\bar{S}_C(\theta_0))\right)^2\)$ Here, \(\text{sg}(\cdot)\) denotes the stop-gradient operator treating the initial source sensitivity profile \(\bar{S}_C(\theta_0)\) as a fixed optimization target. With default hyperparameters \(\tau = 0.05\) and regularizer weight \(\lambda = 0.1\), this objective selectively penalizes deviant concept shifts while preserving full adaptation flexibility along unconstrained feature dimensions.

3. First-order projection surrogate: preserving performance with minimal overhead Differentiating the exact sensitivity \(\bar{S}_C\) with respect to model parameters introduces second-order Hessian terms because \(S_{C,k}\) involves the activation gradient \(\nabla_{a^{(l)}} f_k\). To circumvent significant computational latency and memory consumption during real-time test-time deployment, the authors propose a first-order activation projection surrogate \(P_C(\theta_t)\) that directly preserves the projection of layer activations onto the fixed CAV direction: $\(P_C(\theta_t) = \frac{1}{|\mathcal{B}_t|} \sum_{x \in \mathcal{B}_t} a^{(l)}(x; \theta_t)^\top v_C\)$ During online deployment, this reduces to simple dot-product operations between intermediate activations and concept vectors, scaling with complexity \(\mathcal{O}(|\mathcal{C}| \cdot B \cdot d)\). Ablation results confirm that this first-order surrogate yields accuracy and worst-group performance on par with exact second-order optimization (78.5% overall and 65.8% worst-group accuracy vs. 78.6% and 66.0% on Waterbirds) while cutting the computational overhead in half, adding merely 1.1%–2.8% to per-step execution wall-clock time.

Loss & Training

The overall test-time training objective combines the base adaptation loss with the sensitivity preservation loss: $\(L_{\text{total}} = L_{\text{tta}} + \lambda \cdot L_{\text{preserve}}\)$ Here, \(L_{\text{tta}}\) can be any off-the-shelf unsupervised objective, including Tent entropy minimization, CoTTA consistency loss, or DeYO prediction filtering. In alignment with established TTA protocols, optimization updates only the batch normalization scale (\(\gamma\)) and shift (\(\beta\)) parameters, keeping all convolutional kernels and linear layers strictly frozen. Optimization is conducted using SGD with learning rate \(10^{-3}\) and batch sizes of 64 for standard vision benchmarks and 16 for high-resolution medical imaging.

Key Experimental Results

Main Results

The framework was evaluated across five diverse benchmarks against five baseline TTA methods (Tent, EATA, CoTTA, SAR, DeYO). Benchmarks include standard corruption robustness (CIFAR-10-C, ImageNet-C), spurious correlation benchmarks (Waterbirds, CelebA), and cross-hospital clinical adaptation (CheXpert \(\to\) MIMIC-CXR evaluated over Pneumonia, Pneumothorax, and Enlarged Cardiomediastinum).

Benchmark / Task	Metric	Source (No Adapt)	Tent Baseline	Tent + CS-TTA (Ours)	DeYO Baseline	DeYO + CS-TTA (Ours)
CIFAR-10-C (WRN-28-10)	Mean Acc (%)	74.5 ± 0.2	79.8 ± 0.3	81.2 ± 0.3 (+1.4)	81.8 ± 0.3	82.4 ± 0.3 (+0.6)
ImageNet-C (ResNet-50)	Mean Acc (%)	39.2 ± 0.3	44.8 ± 0.4	45.9 ± 0.3 (+1.1)	46.5 ± 0.3	47.1 ± 0.3 (+0.6)
Waterbirds (ResNet-50)	Worst-Group WG (%)	58.1 ± 0.8	56.2 ± 1.0	65.8 ± 0.9 (+9.6)	63.8 ± 0.9	68.5 ± 0.8 (+4.7)
CelebA (ResNet-50)	Worst-Group WG (%)	47.2 ± 0.9	45.8 ± 1.1	54.3 ± 1.0 (+8.5)	51.2 ± 1.0	56.8 ± 0.9 (+5.6)
Cross-Hospital CXR (DenseNet-121)	Mean Clinical AUC	0.762 ± .004	0.778 ± .004	0.795 ± .003 (+0.017)	0.788 ± .003	0.802 ± .003 (+0.014)

Across all baselines and environments, CS-TTA consistently enhances performance. On spurious correlation benchmarks, conventional entropy-driven adaptation degrades worst-group performance below the unadapted source model (Tent drops WG from 58.1% to 56.2%; CoTTA drops to 55.1%). Integrating CS-TTA restores and substantially improves worst-group accuracy (Tent+CS-TTA achieves 65.8%, a 9.6-point gain over Tent). In cross-hospital chest X-ray adaptation, CS-TTA yields reliable 1.4 to 1.8 point gains in mean AUC.

Ablation Study

Ablations on Waterbirds using Tent+CS-TTA with a ResNet-50 backbone examine hyperparameter sensitivity, target network layers, concept supervision sources, and baseline regularizer controls:

Ablation Component	Specific Setting / Variant	Overall Acc (%)	Worst-Group WG (%)	Concept Drift CSD	Key Conclusion / Finding
Full Model (Default)	\(\lambda=0.1, \tau=0.05\), Layer 4, 1st-order	78.5	65.8	0.12	Optimal trade-off between stability and adaptation
Regularization Weight \(\lambda\)	\(\lambda = 0.01\) (Under-regularized)	77.2	62.3	0.25	Insufficient restraint allows marked concept drift
Regularization Weight \(\lambda\)	\(\lambda = 1.0\)	77.8	65.1	0.07	Stronger constraint slightly dampens target adaptation
Regularization Weight \(\lambda\)	\(\lambda = 10.0\) (Over-constrained)	74.5	63.5	0.03	Rigid source anchor impedes necessary distribution shift
Gating Threshold \(\tau\)	\(\tau = 0.02\) / \(\tau = 0.10\)	78.1 / 77.8	65.1 / 64.2	0.10 / 0.16	\(\tau=0.05\) optimally balances coverage and flexibility
Monitored Layer Depth	Early Layer (Layer 2)	77.2	62.8	0.18	Early features exhibit lower semantic separation
Monitored Layer Depth	Multi-Layer Monitoring	78.3	65.5	0.11	Yields comparable gains to Layer 4 with added compute
Concept Supervision	CLIP Zero-Shot Pseudo-Labels	77.9	64.5	0.14	Highly practical without fine human concept masks
Gradient Approximation	Exact Second-Order Gradient	78.6	66.0	0.11	Performance matches 1st-order at double the overhead
Regularizer Control	Blind Activation \(L_2\) Penalty	77.1	57.5	0.26	Non-selective feature preservation fails completely
Regularizer Control	Random Concept Directions	77.3	58.2	0.24	Confirms benefits arise from meaningful semantic vectors

Key Findings

• Necessity of Semantic Concept Selection: Imposing indiscriminate \(L_2\) regularization across all activations or regularizing along random concept directions yields worst-group accuracy of only 57.5% and 58.2% (matching unregularized Tent at 56.2% and falling short of the source baseline at 58.1%). Only directional anchoring along task-relevant semantic vectors successfully suppresses shortcut learning.
• Early Warning Property of CSD: Across all tested benchmarks, the aggregate CSD measured at step 100 demonstrates a strong positive rank correlation (\(\rho = 0.76, p < 0.001\)) with subsequent accuracy degradation over the following 400 steps, highlighting CSD's utility as an unsupervised early diagnostic for adaptation stability.
• Restoration of Pathological Focus: Qualitative GradCAM and per-concept CSD evaluations demonstrate that conventional TTA on chest radiographs amplifies attention to peripheral patient markers and hospital hardware while attenuating sensitivity to lung opacities. CS-TTA reverses this imbalance, preserving attention on anatomical pathology.

Highlights & Insights

• From Post-Hoc Interpretability to Active Optimization: Rather than confining interpretability techniques (such as TCAV and GradCAM) to static post-hoc model inspection, this paper repurposes concept directional derivatives as dynamic, real-time regularizers that steer model optimization during adaptation.
• Identification and Formalization of CSD: The paper establishes that unsupervised adaptation can improve superficial accuracy metrics while actively degrading feature integrity and worsening reliance on nuisance shortcuts—providing a crucial diagnostic for real-world deployment.
• Lightweight, Non-Invasive Plug-in Architecture: By leveraging a first-order activation projection surrogate, CS-TTA incurs less than 3% runtime overhead per adaptation step, requires no changes to the underlying model architecture, and demands zero inference from vision-language foundation models at test time.

Limitations & Future Work

• Reliance on Source Concept Coverage: The method assumes access to well-defined concept exemplars on the source domain. If an unknown shortcut in the target domain has no corresponding CAV representation, or if the source model was already heavily biased, preserving source sensitivity cannot rectify unmodeled distortions.
• Distinguishing Beneficial from Malign Drift: CS-TTA penalizes any sensitivity drift exceeding the threshold \(\tau\). However, in scenarios where the target domain presents valid contextual domain shifts, attenuating reliance on certain source attributes could theoretically be advantageous.
• Future Research Directions: Key opportunities include integrating automated concept discovery (e.g., Automated Concept Explanations, ACE) to dynamically identify emerging target artifacts and extending concept preservation to continual, long-horizon test-time adaptation settings.

Related Work & Insights

• vs. Conventional Unsupervised TTA (Tent, CoTTA): Standard TTA methods rely entirely on output entropy minimization or self-training consistency, making them blind to shortcut exploitation. CS-TTA introduces semantic geometric anchors in activation space to prevent feature-level degradation.
• vs. Disentanglement-Based Sample Filtering (DeYO): DeYO utilizes predefined patch perturbations and shape-texture metrics (PLPD) to discard noisy samples, but cannot easily address complex domain artifacts like clinical text or institutional markers. CS-TTA flexibly handles arbitrary human-interpretable concepts.
• vs. Concept Bottleneck Adaptation (CONDA): CONDA mandates specialized Concept Bottleneck Model architectures with intermediate linear concept layers. CS-TTA keeps concept analysis external to the network, functioning as a modular loss term compatible with arbitrary standard backbones.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates the Concept Sensitivity Drift failure mode in TTA and turns directional derivatives into an online regularizer.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across five baselines, general vision and clinical CXR domains, accompanied by gradient order analyses and GradCAM visualizations.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression connecting the diagnosis of shortcut learning to an elegant, minimally invasive solution.
• Value: ⭐⭐⭐⭐⭐ Solves a critical reliability vulnerability in adapting deep models to safety-critical environments like cross-hospital healthcare.