Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification

Conference: ECCV 2026
Paper: ECCV
Area: Others (Test-Time Adaptation / Robustness under Distribution Shift)
Keywords: test-time adaptation, underspecification, particle diversification, entropy minimization, distribution shift

TL;DR

This paper reinterprets entropy-based test-time adaptation (TTA) as a posterior inference problem under underspecification — many low-entropy solutions exist with drastically different decision boundaries — and instead of committing to a single point estimate it maintains K adaptation particles that only update normalization layers, explores different low-entropy basins, applies diversification at the output, parameter, optimizer, and input levels to prevent collapse, and aggregates the multi-hypothesis predictions at inference; it consistently beats Tent / SAR / DeYO on ImageNet-C under batch size one, label shifts, and mixed shifts, and works as a plug-and-play wrapper for existing TTA methods.

Background & Motivation

Deep models perform well when training and test data share a distribution, but degrade sharply in the wild, where distribution shift is unavoidable; test-time adaptation (TTA) has therefore become the standard remedy — leave the labels alone and update the model online using only unlabeled target data. The most common objective is prediction-entropy minimization (pioneered by Tent), with SAR adding sharpness-aware optimization and reliable-sample filtering, and DeYO combining entropy with augmentation-based consistency and sample selection. All of these update a single model instance, and the TTA setting provides no supervision at all to constrain which direction the update should take — and that is exactly where the trouble starts.

Entropy minimization is a heavily underconstrained objective: on the same batch of target data, many parameter configurations reach nearly identical low entropy while inducing completely different decision boundaries, with very different generalization under shift. The paper makes this concrete with a blunt experiment (Table 1). Running DeYO five times on ImageNet-C Zoom Blur (severity 5) with batch size one, changing only the random seed and keeping all hyper-parameters fixed, the final entropy is essentially constant (0.245 ± 0.003) while accuracy swings from 32.9% to 45.1% (mean 38.37 ± 4.80), and the adapted normalization parameters converge to different endpoints in parameter space. In other words, the entropy objective never pins down what a good adaptation looks like — the quality of the final model is decided by the randomness of the optimization trajectory. The authors conclude that existing TTA methods are implicitly computing a single maximum a posteriori (MAP) point estimate under an entropy-induced pseudo-likelihood; in underspecified regimes a small perturbation of the trajectory qualitatively changes the decision boundary, so entropy minimization should be treated not as a well-posed optimization problem but as a multi-hypothesis inference problem.

The obvious fix would be ensembling or data augmentation — the standard remedies for underspecification elsewhere — but their integration into TTA is largely unexplored (DeYO does use augmentation, yet its purpose is filtering samples and highlighting robust features, not resolving underspecification). More importantly, simply averaging several independently adapted models does not work: under entropy minimization, independent particles tend to follow highly similar trajectories and end up in the same low-entropy basin, so averaging buys little. Core idea: turn TTA from "find one low-entropy point estimate" into "maintain a population of mutually repulsive adaptation particles that approximates the posterior over low-entropy solutions" — coupling particle dynamics during adaptation through an explicit diversification regularizer (rather than averaging after the fact) so that particles occupy distinct low-entropy regions of parameter space, then aggregating their predictions.

Method

Overall Architecture

The method builds on the standard TTA setting: a source model f_θ is trained on the source domain, only its normalization-layer parameters are kept (denoted θ0, everything else frozen), and adaptation runs online on a stream of unlabeled target samples. The difference from conventional TTA is that instead of adapting one θ, the method initializes a collection of K particles Θ = {θ_i}_{i=1}^K from θ0, where each θ_i is an independent copy of the normalization parameters, while the non-normalization layers (convolutional / attention weights) stay frozen and are shared by all particles. Each particle follows its own entropy-minimization trajectory online, and the particles interact through a diversification regularizer Ω(Θ) that prevents them from piling onto the same solution; at inference the K predictions are aggregated (the illustrative figure uses a plain average, ŷ = (y1 + y2)/2), i.e. model averaging over multiple adaptation hypotheses replaces a single point estimate.

The overall objective sums each particle's entropy loss and the diversification term:

\[\min_{\Theta}\; \mathcal{L}(X) = \frac{1}{K}\sum_{i=1}^{K}\ell(X;\theta_i) + \Omega(\Theta)\]

Here ℓ(X; θ_i) is the standard entropy-minimization term and Ω(Θ) is the diversification regularizer, which is where the entire design space of the paper lives: it is decomposed into four complementary levels — output, parameter, optimizer, and input — each targeting a specific failure mode of single-model entropy minimization under shift (predictions converging, parameters collapsing into one basin, identical optimization geometry, overfitting a single input configuration). Crucially, Ω is not a post-hoc aggregation trick; it acts during optimization.

One further detail appears only in the paper's framework figure (Fig. 2): after a batch of test samples arrives, "non-harmful" samples are first identified following the appendix, and only those are back-propagated with the gradient-diversity measure, so that noisy or outlier samples do not derail the particles. The selection criterion itself is in the appendix; ⚠️ refer to the original paper.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source model θ0<br/>normalization layers only"] --> B["Particle population adaptation<br/>K normalization copies + frozen shared backbone"]
    S["Unlabeled target stream"] --> B
    B --> C["Output- and parameter-level repulsion<br/>KL divergence + SVGD kernel repulsion"]
    B --> D["Functional gradient diversification<br/>penalize inter-particle gradient alignment"]
    B --> E["Heterogeneous optimizers and input perturbation<br/>SGD/Adam/AdamW + flipped views"]
    S --> E
    C --> U["Per-particle entropy-minimization update"]
    D --> U
    E --> U
    U -->|target stream not finished| B
    U --> P["Aggregate K particle predictions"]
    P --> O["Target-domain output"]

Key Designs

1. Particle population adaptation: K copies of the normalization parameters sharing one frozen backbone

This is the substrate that carries every diversification mechanism. Copying only the normalization parameters rather than whole networks has two direct justifications. First, the TTA literature has already established that adapting normalization layers alone (the Tent / SAR / DeYO line) absorbs most of the shift at very low memory and compute cost. Second, restricting particles to this low-dimensional subspace makes inter-particle distances comparable and makes repulsion actually bite — the paper measures particle drift with ‖θ_final − θ0‖₂ and dispersion with the mean pairwise distance ‖Δθ‖₂ across seeds. Since all particles share the same frozen non-normalization layers, they all run through one feature extractor and differ only in their adapted normalization statistics and affine parameters; the resulting "multiple hypotheses" are therefore controlled and interpretable, rather than the uncontrolled differences you get from training separate networks. The particle count is set to K = 3 in all experiments, a choice justified by ablation (see the experiments).

2. Output- and parameter-level repulsion: KL divergence and SVGD kernel repulsion

The most direct form of diversification is to make particles disagree. At the output level, repulsion is applied to the predictive distributions: writing the predictive distribution of particle θ_i on input batch X as p_i(X), the method minimizes the negative pairwise KL divergence — note that it uses KL divergence (not a symmetric JS or a plain L2), and together with the negative sign this amounts to maximizing disagreement between particle predictions within the overall objective:

\[\Omega_{\mathrm{KL}}(\Theta) = -\frac{1}{K(K-1)}\sum_{i\neq j} D_{\mathrm{KL}}\!\left(p_i(X)\,\|\,p_j(X)\right)\]

Particles are thus pushed toward distinct low-entropy regions of the solution space, reducing the risk of collective collapse onto the same spurious decision boundary. At the parameter level the method borrows Stein Variational Gradient Descent (SVGD): the entropy loss is treated as a pseudo-likelihood, and a Gaussian prior centered at the source parameters θ0 is added on top (discouraging excessive drift away from the source domain), giving the pseudo-posterior p̃(θ) ∝ exp(−ℓ(X; θ)) · exp(−γ‖θ − θ₀‖²), where γ trades off adaptation flexibility against retention of the source-domain inductive bias. SVGD writes particle repulsion into the update direction through a kernel (e.g. an RBF kernel):

\[\theta_i \leftarrow \theta_i - \epsilon\,\phi(\theta_i),\qquad \phi(\theta_i) = \frac{1}{K}\sum_{j=1}^{K}\Big[\kappa(\theta_j,\theta_i)\,\nabla_{\theta_j}\log\tilde{p}(\theta_j) + \nabla_{\theta_j}\kappa(\theta_j,\theta_i)\Big]\]

The first term drives particles toward low-entropy regions and the second (the kernel gradient) pushes them apart, so the combination means "stay low-entropy while spreading out". ⚠️ The equations in the cached text are typeset corruptly; the SVGD update and the pseudo-posterior above are reconstructed from standard SVGD and the paper's prose — refer to the original paper.

3. Functional gradient diversification: penalizing inter-particle gradient alignment

Ablation shows this is the most effective of all the mechanisms, and it acts somewhere different from the other two. The previous mechanisms act on predictions and on weights respectively, whereas this one constrains the functional behavior of the particles directly: for each particle θ_i it takes the gradient of the entropy loss with respect to the input batch, g_i = ∇_X ℓ(X; θ_i), and penalizes positive inner products between the input gradients of different particles:

\[\Omega_{\mathrm{Grad}}(\Theta) = \frac{1}{K(K-1)}\sum_{i\neq j}\langle g_i, g_j\rangle\]

Minimizing this within the total objective is equivalent to reducing gradient alignment among particles. The target here is simplicity bias: if all particles descend along the same set of input directions, they are relying on the same (likely spurious) features and will inevitably converge onto a single trajectory. Suppressing the inner product forces each particle to attend to different input directions, producing different decision boundaries at the functional level rather than merely at the level of parameter values. The paper compares three formulations — dot product, ℓ2 distance between gradients, and cosine similarity — and finds the dot product best: cosine discards magnitude information and the ℓ2 difference penalizes scale rather than directional agreement, whereas the dot product directly penalizes "everyone sharing the same descent direction".

4. Heterogeneous optimizers and input perturbation: giving each particle a different optimization problem

The first three designs change the objective; these last two change the optimization problem each particle actually solves. At the optimizer level, the K particles are assigned different optimization algorithms (SGD, Adam, AdamW). Even though they all minimize the same entropy objective, momentum accumulation, second-order rescaling, and decoupled weight decay make the geometry of the parameter updates entirely different: SGD follows uniform gradient directions, Adam rescales steps according to estimated second-order statistics, and AdamW decouples weight decay from the gradient update. Consequently, even with identical entropy gradients, the particles trace distinct paths through parameter space — an implicit diversification mechanism that needs no extra regularizer. At the input level, structured data augmentation slightly deforms the loss surface each particle sees: for an input x the method builds a set of views A(x) = {x, x^(h), x^(v)} (original plus horizontal and vertical flips), and each particle's adaptation loss is summed over all views:

\[\ell_{\mathrm{aug}}(X;\theta_i) = \sum_{\mathbf{x}'\in\mathcal{A}(\mathbf{x})}\ell(\mathbf{x}';\theta_i)\]

Unlike DeYO, which uses augmentation for sample filtering, here augmentation is purely an optimization-time perturbation: on one hand it stops a particle from overfitting a single input configuration, and on the other hand, because different particles compute different gradients on different views, it amplifies the repulsive regularizers above and sends particles toward distinct local minima corresponding to different augmented perspectives.

A Worked Example

Walk through the extreme batch-size-one setting: the target stream delivers a single image corrupted by Fog (severity 5), with a ViTBase-LN backbone. Because the batch holds one image, the normalization statistics are extremely unstable (which is why DeYO reaches only 38.37% on that corruption with ±4.8 variance). What the framework does is this: copy three sets of normalization parameters from the source model (K = 3) and assign them SGD / Adam / AdamW respectively; build three views of the image (original, horizontal flip, vertical flip) and let each particle sum its entropy loss over the three views; during back-propagation each particle updates only its own normalization parameters while the shared Transformer weights stay frozen; at every step the inter-particle gradient alignment (Ω_Grad) and predictive divergence (Ω_KL) are computed and fed back together with the entropy term, with λ at the grid-searched value of 0.3; if SVGD is enabled, the update direction is further corrected by the kernel repulsion. After the target stream is consumed, the three particles' predictions on the current sample are averaged into the final output — note that this is not picking the best one but letting three distinct low-entropy hypotheses vote, so that a single particle going astray gets pulled back by the other two.

Loss & Training

The total loss is the mean entropy-minimization term over the K particles plus λ times the diversification regularizer (one of, or a combination of, Ω_KL, Ω_Grad, and the SVGD repulsion). Key hyper-parameters: particle count K = 3 (ablation shows the largest gain from 1 to 3, with saturation afterwards); the diversification weight λ requires grid search, with the reported optimum near λ = 0.3 (too large a λ clearly hurts, as repulsion overwhelms adaptation itself); the optimizer set is SGD / Adam / AdamW; the input views are the original image plus horizontal and vertical flips. Adaptation is purely online and label-free, and the backbone weights stay frozen throughout, so the extra cost comes mainly from updating K sets of normalization parameters and small-scale back-propagation.

Key Experimental Results

Main Results

Evaluation uses ImageNet-C (15 corruption types × 5 severities) with ResNet-50-GN and ViTBase-LN backbones, covering three wild scenarios: batch size one (single-sample adaptation), label distribution shift (infinite class-imbalance ratio), and mixed shifts over 15 corruption types; all numbers are accuracies averaged over five random seeds. Besides the Tent / SAR / DeYO baselines, the comparison includes a Naive variant that keeps the particle population without any diversification, plus the three diversification variants.

Scenario	Backbone	No Adapt	Tent	SAR	DeYO	Naive	SVGD	KL	Grad
Batch size 1	ViTBase-LN	29.91	51.58	55.31	60.57	62.17	62.94	63.36	63.14
Label shift	ViTBase-LN	—	53.10	55.98	62.37	62.84	63.29	62.24	63.36
Mixed shift (top@1)	ViTBase-LN	29.94	32.36	57.78	57.05	60.56	60.84	60.37	61.36
Batch size 1	ResNet-50-GN	30.60	30.65	35.82	44.40	45.01	45.11	45.58	46.90
Label shift	ResNet-50-GN	30.60	21.15	35.89	41.65	41.74	42.49	42.26	43.27
Mixed shift (top@1)	ResNet-50-GN	30.61	29.80	38.12	31.36	33.06	32.71	32.66	34.36

On CIFAR-100-C the paper also applies the framework to WaTT (a weight-averaging TTA method that adapts only CLIP's text encoder) and performs diversification directly on the image encoder, raising the mean from 45.57 to 47.86.

Method	CIFAR-100-C mean accuracy
CLIP (zero-shot)	29.43
TENT	35.19
TPT	30.46
TDA	22.08
DiffTPT	22.89
SAR	31.92
CLIPArTT	41.51
WATT-P	44.68
WATT-S	45.57
WATT-S + Aug	47.30
WATT-S + Aug + Reg (Ours)	47.86

Ablation Study

Config	Key metric	Note
Particle population only (Naive)	62.17	ViTBase-LN, batch size 1; already above DeYO's 60.57, so "multiple normalization copies" alone pays off
+ KL output-level repulsion	63.36	Predictive disagreement gives a steady gain
+ SVGD parameter-level repulsion	62.94	Close to KL, slightly lower
+ Grad functional repulsion	63.14	Strongest under mixed shifts (ViT 61.36 / ResNet 34.36)
Grad + flip augmentation	64.72	Strongest configuration overall, about 4% above DeYO
Grad + heterogeneous optimizers	63.08	Essentially on par with Grad (63.14); see discussion below
Gradient diversity form: dot product / ℓ2 / cosine	dot product best	Peak near λ ≈ 0.3; too large a λ degrades clearly
Particle count N = 1 → 3 → larger	large gain from 1 to 3	Beyond N = 3 gains are marginal, hence N = 3
Tent → Tent + Grad (plug-in check)	51.52 → 53.63	Average over a 7-corruption subset (DeYO: 52.87), showing the regularizer does not depend on this paper's population initialization

Key Findings

• The largest contributor is functional gradient diversification (Grad). Under mixed shifts it lifts ViTBase-LN from DeYO's 57.05 to 61.36 and ResNet-50-GN from 31.36 to 34.36, and at batch size one it lifts ResNet from 44.40 to 46.90; it can also be transplanted onto Tent (51.52 → 53.63), showing it is a generic stabilization mechanism rather than something tied to this paper's ensemble initialization.
• Merely maintaining multiple particles without any diversification (Naive) already yields a steady gain (ViT 60.57 → 62.17). This is an important enabling result: multiple trajectories alone alleviate the instability of a single point estimate. But independent particles converge along similar trajectories, so the explicit repulsion terms are what really open up the gap.
• The gains are largest in the most underspecified settings. Grad's advantage is biggest under mixed shifts and batch size one, especially on severe corruptions such as Fog; conversely, on a few corruptions (e.g. Zoom blur on ResNet, baseline 24.90) diversification adds little.
• On hyper-parameters, λ ≈ 0.3 balances entropy minimization against functional repulsion, while excessive repulsion hurts adaptation; the particle count saturates clearly, with 3 being enough.
• ⚠️ Several percentages in the prose do not match the tables: at batch size one the text claims the ensemble improves by 2.23% and Grad by 3.21% over DeYO for ViTBase-LN, whereas the table gives 62.17 vs 60.57 (+1.60) and 63.14 vs 60.57 (+2.57); the ResNet-50-GN figure of +2.50% does match the table. Under mixed shifts the text says "3% better than the best-performing baseline" (ResNet), but the actual comparison is against DeYO (31.36) — in that column the strongest baseline is SAR at 38.12, above every particle variant. The tables take precedence in all cases above.
• Computational cost scales approximately linearly with the number of particles, so K = 3 means roughly 3× the inference overhead; however, since only normalization layers are updated and the backbone is frozen, the additional optimization cost stays light. This is the key precondition for the method's practicality.

Highlights & Insights

• It pins the notion of underspecification — previously discussed in generalization theory and robustness — directly onto TTA with a five-seed controlled experiment (entropy nearly constant while accuracy swings ±4.8%). That is far more convincing than simply asserting that "entropy minimization is unstable". This diagnostic paradigm — showing that nearly identical objective values produce wildly different outcomes — transfers to any scenario that updates a model online with an unsupervised objective, such as test-time prompt tuning or online reward adaptation in RLHF.
• Diversification is decomposed into four orthogonal levels, each matched to a concrete failure mode, so the design is not a pile of modules: output level treats prediction convergence, parameter level treats basin collapse, gradient level treats identical feature reliance, and optimizer / input levels treat shared optimization geometry and input overfitting.
• Gradient diversification uses the plainest possible "inner product of input gradients" yet the ablation shows it beats both KL and SVGD. This is counter-intuitive but explainable: dispersion in parameter space does not equal dispersion in function, and directly constraining functional behavior (the response to inputs) is closer to the real goal — different decision boundaries — than constraining weight values.
• The plug-and-play design (touching only normalization layers, adding one additive regularizer term) lets it combine with any entropy-based TTA method, as the Tent + Grad check demonstrates; in a field as crowded as TTA, this kind of wrapper contribution often has more engineering value than yet another novel loss.

Limitations & Future Work

• It does not beat the strongest baseline everywhere: on mixed shifts with ResNet-50-GN, SAR (38.12) remains above every particle variant (best: Grad 34.36). The prose describes this row as "3% better than the best-performing baseline", but the actual comparison is against DeYO — the reporting basis matters.
• In the batch-size-one setting the prose percentages disagree with the tables (see the ⚠️ above), and in several rows the gaps between variants fall within one standard deviation (e.g. KL 62.24–63.36 versus Grad 63.36–63.14 cross over), so the ranking on individual corruptions is not stable.
• The evidence for heterogeneous optimizers is weak: after adding them (Table 4), ViTBase-LN variants reach 61.88 / 62.81 / 62.91 / 63.08, which does not improve on the homogeneous configuration's 62.17 / 62.94 / 63.36 / 63.14 — most differences lie within noise. The prose claim of "further improving robustness" is mainly relative to DeYO.
• The particle count is fixed at K = 3 and λ requires offline grid search; the framework does not adapt its exploration strength online, using the same configuration for very hard corruptions (where more particles could help) and easy ones (where a single model would save compute).
• Adapting only normalization layers inherits the usual TTA restriction, but it also means that when a shift mainly affects semantic representations outside the normalization layers, the space the particles can explore is quite limited; input-level diversification only uses horizontal and vertical flips, so the exploration surface is narrow.
• The main text relies on a "non-harmful sample" filtering step described only in the appendix (back-propagation is restricted to the selected samples); its criterion and sensitivity are never unfolded in the main paper, making that part a black box for reproduction.
• Improvement directions: use inter-particle disagreement as an uncertainty / difficulty signal to adaptively set K, λ, or even whether to adapt at all on the current sample (large disagreement → explore more; small disagreement → save compute); replace the plain mean with posterior-weighted or Bayesian model averaging; extend "multiple hypotheses" from normalization layers to parameter-efficient subspaces such as prompts or LoRA, and transfer it to test-time prompt tuning and dense prediction tasks such as segmentation and detection.

Related Work & Insights

• vs Tent: Tent performs entropy minimization on a single model to tune its normalization layers online; this paper keeps that loss but turns it into a population objective and adds diversification terms that keep particle trajectories apart. The Tent + Grad plug-in experiment (51.52 → 53.63) shows the two are complementary rather than competing.
• vs SAR: SAR's core devices for the wild setting are sharpness-aware optimization and reliable-sample filtering, still maintaining a single model; this paper argues that the problem is that even a single model's solution is not unique, so the issue lies in solution diversity. The two are orthogonal (SAR and this method trade wins across settings, and SAR is actually stronger on ResNet mixed shifts).
• vs DeYO: DeYO also uses augmentation, but to filter samples and highlight robust features — its "diversity" lives at the sample level (the PLPD score). This paper's diversity lives at the parameter, prediction, and gradient levels, and it explicitly argues that approaches like DeYO never touch the root of underspecification.
• vs naive ensembling: Naive ensembling trains members independently and averages afterwards; under entropy minimization independent particles converge along similar trajectories, so the diversity dividend is limited. The key difference here is that Ω(Θ) couples particle dynamics during adaptation, making disagreement part of the optimization rather than a by-product of the outcome.
• vs SVGD (Liu & Wang, 2016): SVGD is a general-purpose Bayesian inference algorithm; this paper imports it into TTA as the particle interaction mechanism and additionally adds a Gaussian prior centered at the source parameters (the γ term) to keep particles from drifting away from the source domain — that adaptation is the key step for the test-time setting.

Rating

• Novelty: ⭐⭐⭐⭐ Formally bringing underspecification into TTA and framing it as multi-particle posterior inference is compelling, though the individual components (KL repulsion, SVGD, gradient alignment, heterogeneous optimizers) are assemblages of existing tools.
• Experimental Thoroughness: ⭐⭐⭐⭐ Three wild scenarios, two backbones, CIFAR-100-C with WaTT, a plug-in check, and several ablations — but multiple prose percentages disagree with the tables, and some settings do not surpass the strongest baseline.
• Writing Quality: ⭐⭐⭐ The concept and motivation are stated clearly and directly, but the numerical basis is inconsistent (ensembling / Grad gains), the heterogeneous-optimizer conclusion does not match the tables, and the equations are typeset corruptly in the body.
• Value: ⭐⭐⭐⭐ Plug-and-play, normalization-only, with controllable cost; easy for follow-up TTA work to stack on top, and the "multi-hypothesis + diversification" idea transfers to test-time prompt tuning and other test-time optimization settings.

Related Papers

• [CVPR 2026] Neural Collapse in Test-Time Adaptation
• [NeurIPS 2025] Test-Time Adaptation by Causal Trimming
• [ICML 2025] Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
• [ICLR 2026] Prior-Free Tabular Test-Time Adaptation
• [ICML 2026] Private and Stable Test-Time Adaptation with Differential Privacy