Skip to content

From Local Geometry to Global Pseudo-Labeling for Robust Positive–Unlabeled Learning under Covariate Shift

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/fira7s/S-PUNA
Area: Others
Keywords: Covariate Shift Detection, Positive-Unlabeled Learning, Pseudo-Labeling, Manifold Structure, Spectral Entropy

TL;DR

Addressing the failure of classical PU learning under severe distribution overlap caused by covariate shift, S-PUNA introduces a geometry-aware framework that expands positive and negative pseudo-labels symmetrically on ViT feature manifolds, guided by a spectral entropy stopping criterion to prevent drift and rival fully supervised methods.

Background & Motivation

Modern deep neural networks are predominantly trained under the assumption that test data follow the same distribution as the training source. When encountering covariate shift—where the input marginal \(P(X)\) changes while the conditional labeling function \(P(Y \mid X)\) remains intact—conventional literature has largely emphasized achieving domain generalization, domain adaptation, or adversarial robustness. However, in safety-critical domains such as AI-generated image detection, financial fraud detection, and medical imaging, explicitly detecting distributional shifts and raising automated alerts is just as crucial as model generalization. Existing detection strategies rely almost exclusively on fully supervised training (e.g., DisCoPatch), requiring meticulously labeled negative samples from both standard and shifted distributions, which is often unrealistic in open-world deployments where novel shifts emerge dynamically.

Positive–Unlabeled (PU) learning naturally emerges as an appealing paradigm to relieve this annotation burden by training binary classifiers using only labeled positive data alongside an unlabeled mixture. Nevertheless, naively porting classical PU empirical risk estimators (such as nnPU or Dist-PU) to covariate shift detection incurs fundamental theoretical and empirical pitfalls. Standard PU estimators rely on unbiased risk formulations that presuppose sufficient separability between positive and negative classes. Under covariate shift, instances preserve identical semantic classes while varying only in perceptual style or sensor statistics; this causes positive and shifted distributions to heavily overlap in latent representation spaces, driving the total variation distance \(\|P^+ - P^-\|_{TV}\) towards zero and inflating the Bayes optimal classification error. Under this regime, global risk minimization estimators suffer from high variance, instability, and rampant pseudo-label contamination.

This work argues that under covariate shift, the negative distribution should not be estimated globally across the ambient space. Instead, grounded in the manifold hypothesis, the shifted negative component must be progressively uncovered from local geometric structures. Core idea: formalize covariate shift detection as a bidirectional, geometry-aware manifold pseudo-label expansion process, introducing S-PUNA to dynamically annotate positive and shifted negatives while leveraging a spectral entropy stopping criterion to prevent contamination before distilling into deep networks.

Method

Overall Architecture

S-PUNA aims to reliably isolate covariate-shifted instances using only a known Positive Domain (PD) training bank and an unlabeled pool containing a mixture of in-distribution and shifted data. The overall pipeline proceeds through four sequential phases: first, intermediate visual features are extracted from the 6th block of a pre-trained Vision Transformer (ViT) to retain low-level geometric nuances; next, positive and negative seed banks are initialized using nearest-neighbor distances to the labeled positives; third, a bidirectional symmetric \(k\)-NN manifold expansion progressively assigns pseudo-labels to unlabeled candidates; during this iterative expansion, the spectral entropy of the negative covariance matrix is continuously tracked, terminating the process as soon as entropy drops to arrest semantic drift; finally, the non-parametric annotations are distilled into a multi-layer perceptron (MLP) to yield a smooth, generalizable decision boundary.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Labeled Positives & Unlabeled Mixture Pool"] --> B["ViT Intermediate Feature Extraction<br/>Select Block 6 to preserve geometric details"]
    B --> C["Bidirectional Seed Initialization & Manifold Expansion<br/>Maintain dynamic positive and negative neighbor banks"]
    C --> D["Spectral Entropy Stopping Criterion<br/>Monitor covariance eigenvalue spectrum to halt drift"]
    D -->|Entropy drops, terminate expansion| E["Deep Network Knowledge Distillation<br/>Fit parametric classifier for smooth decision boundary"]

Key Designs

1. Bidirectional symmetric neighborhood pseudo-labeling: eliminating decision boundary drift Classical \(k\)-NN-based PU techniques propagate labels unidirectionally from the positive set, leaving negative instances as an unguided residual that easily degrades in overlapping boundary zones. S-PUNA enforces a symmetric expansion mechanism that explicitly constructs both classes simultaneously. During initialization, for each unlabeled candidate \(x \in \mathcal{X}_U\), the algorithm computes its minimum distance to the positive training features, denoted as \(s_{orig}(x)\). The top-\(\alpha\) samples with the smallest distances form the initial pseudo-positive set \(\hat{\mathcal{X}}_{U,P}^{(t_0)}\), while the top-\(\alpha\) samples with the highest distances initialize the pseudo-negative set \(\hat{\mathcal{X}}_{U,N}^{(t_0)}\). In subsequent iterations \(t\), candidate samples in the remaining pool are evaluated against the evolving positive bank \(\mathcal{X}_P^{(t)}\) and negative bank \(\mathcal{X}_N^{(t)}\): $\(s_t(x) = d_N(x) - d_P(x)\)$ where \(d_P(x)\) and \(d_N(x)\) denote the mean distances from \(\phi(x)\) to its \(k\)-nearest neighbors in the positive and negative banks, respectively. Candidates with the highest \(s_t(x)\) are added to the positive set, while those with the lowest scores expand the negative set by step size \(\beta\). By pushing the pseudo-negative set outward along directions of maximum geometric discrepancy from the positive support, this symmetric mechanism widens the effective margin between empirical supports.

2. Spectral entropy stopping criterion: halting contamination in overlapping manifolds A pervasive risk in iterative pseudo-labeling is semantic drift, wherein the growing negative set eventually infiltrates overlapping regions densely populated by true positives, catastrophically contaminating the pseudo-labels. S-PUNA establishes an adaptive stopping rule derived from the spectral decomposition of the discovered negative covariance matrix. At iteration \(t\), the empirical covariance matrix \(\Sigma_t\) of all discovered pseudo-negatives \(\hat{\mathcal{X}}_{U,N}^{(t)}\) is computed, and its normalized eigenvalues \(\{\lambda_1^{(t)}, \dots, \lambda_d^{(t)}\}\) (satisfying \(\sum_i \lambda_i^{(t)} = 1\)) are extracted. The spectral entropy is formulated as: $\(H(\Lambda_t) = -\sum_{i=1}^d \lambda_i^{(t)} \log \lambda_i^{(t)}\)$ Spectral entropy characterizes the dispersion of feature variance across orthogonal principal axes, reflecting intrinsic manifold dimensionality. As the negative manifold is legitimately explored along diverse genuine shift directions, variance spreads across multiple dimensions and \(H(\Lambda_t)\) increases. However, as proven in Lemma 1, once positive in-distribution samples contaminate the negative set, the resulting multi-modal mixture creates an anisotropic distortion along the mean difference axis, causing the spectral entropy to strictly decrease. S-PUNA triggers an immediate halt when \(H(\Lambda_t) - H(\Lambda_{t-1}) < 0\), precisely preserving the purity of the discovered support.

3. Intermediate feature selection and deep parametric distillation: reconciling geometry with generalization While non-parametric \(k\)-NN label propagation captures intricate local manifold geometry without structural bias, direct nearest-neighbor inference incurs substantial variance and excessive computational complexity at test time. Once the stopping criterion is met, S-PUNA distills the accumulated pseudo-labeled dataset \(\mathcal{D}_{anno} = \mathcal{X}_P^{(t)} \cup \hat{\mathcal{X}}_{U,N}^{(t)}\) into a parametric multi-layer perceptron (MLP) \(g_\omega\). The distillation objective minimizes standard cross-entropy loss: $\(\mathcal{L}(\omega) = -\frac{1}{|\mathcal{D}_{anno}|} \sum_{(x, \hat{y}) \in \mathcal{D}_{anno}} \mathrm{CE}\big(g_\omega(\phi(x)), \hat{y}\big)\)$ This distillation step smooths out local high-variance noise while preserving the geometric separation identified by \(k\)-NN. Moreover, extracting features exclusively from Block 6 of a pre-trained ViT strikes an optimal trade-off: it filters high-frequency pixel noise present in early layers while avoiding the semantic invariance of deeper layers (e.g., Block 11) that tends to compress out fine-grained covariate variations.

Loss & Training

During the pseudo-labeling phase, S-PUNA is an optimization-free non-parametric manifold search parameterized by neighbor size \(k\), initial seed size \(\alpha\), and expansion step \(\beta\), terminated autonomously when \(\Delta H_t < 0\). In the subsequent distillation phase, the lightweight MLP classifier \(g_\omega\) is trained on the frozen ViT Block 6 representations using standard Adam optimization and cross-entropy loss over \(\mathcal{D}_{anno}\), converging stably in few epochs.

Key Experimental Results

Main Results

The authors evaluate S-PUNA on ImageNet-1K, TinyImageNet, and EuroSAT under both Near Shift (e.g., ImageNet-V2, GenImage, EuroSAT-D) and Far Shift (e.g., ImageNet-C, ImageNet-R, EuroSAT-C) conditions, benchmarking against competitive PU learning estimators and nearest-neighbor baselines.

Benchmark Dataset Method Category & Name Near Shift AUROC (%) ↑ Near Shift FPR95 (%) ↓ Far Shift AUROC (%) ↑ Far Shift FPR95 (%) ↓ Average AUROC (%) ↑ Average FPR95 (%) ↓
ImageNet-1K k-NN [40] 54.40 ± 0.00 62.63 ± 0.00 74.79 ± 0.00 80.84 ± 0.00 64.59 ± 0.00 71.73 ± 0.00
Dist-PU [41] 84.48 ± 0.40 35.05 ± 0.93 96.70 ± 0.11 18.06 ± 0.58 90.59 ± 0.25 26.56 ± 0.75
saPU [9] 82.93 ± 0.27 69.58 ± 1.01 95.09 ± 0.46 26.78 ± 2.33 89.01 ± 0.37 48.18 ± 1.67
DC-PU [24] 60.61 ± 2.44 85.80 ± 1.03 70.51 ± 3.96 89.38 ± 10.83 65.56 ± 3.20 87.59 ± 5.93
LaGAM [26] 83.01 ± 0.54 65.30 ± 1.96 96.50 ± 0.13 17.46 ± 0.71 89.75 ± 0.33 41.38 ± 1.33
S-PUNA w/o C (Ours) 95.81 ± 0.00 19.22 ± 0.00 95.95 ± 0.00 19.92 ± 0.00 95.88 ± 0.00 19.57 ± 0.00
S-PUNA w/ C (Ours Full) 97.89 ± 0.13 9.69 ± 0.48 98.51 ± 0.08 7.36 ± 0.49 98.20 ± 0.10 8.52 ± 0.48
EuroSAT Dist-PU [41] 99.78 ± 0.13 0.72 ± 0.46 89.46 ± 0.50 49.97 ± 2.81 94.62 ± 0.31 25.35 ± 1.64
LaGAM [26] 99.33 ± 0.56 3.01 ± 3.39 86.24 ± 0.90 54.83 ± 3.24 92.79 ± 0.73 28.92 ± 3.32
S-PUNA w/ C (Ours Full) 99.79 ± 0.05 0.65 ± 0.36 100.00 ± 0.00 0.00 ± 0.00 99.90 ± 0.03 0.33 ± 0.18

Ablation Study

The ablation investigates the sensitivity of S-PUNA and competing baselines across varying proportions of shifted samples (\(p\)) in the unlabeled pool, as well as the impact of ViT feature depth.

Config / Method Feature Layer \(p = 5\%\) AUROC (%) \(p = 25\%\) AUROC (%) \(p = 50\%\) AUROC (%) \(p = 75\%\) AUROC (%)
DC-PU Block 6 73.08 68.00 65.56 72.19
Dist-PU Block 6 71.04 86.40 90.59 92.17
LaGAM Block 6 85.08 87.45 89.75 90.92
saPU Block 6 60.24 75.53 89.01 92.03
S-PUNA w/ C (Ours) Block 11 - - 97.03 -
S-PUNA w/ C (Ours) Block 6 91.61 95.75 98.20 95.53

Key Findings

  • Breakthrough under Near Covariate Shift: On the challenging ImageNet Near Shift benchmark, the best existing PU baseline (Dist-PU) achieves only 84.48% AUROC with a high FPR95 of 35.05%. S-PUNA w/ C achieves 97.89% AUROC and cuts FPR95 to 9.69%, significantly outperforming prior weak supervision methods and rivaling fully supervised detectors (DisCoPatch averages 89.8% AUROC).
  • Smoothing Benefits of Distillation: Comparing S-PUNA w/o C (non-parametric \(k\)-NN scores) with S-PUNA w/ C demonstrates that distilling labels into an MLP lowers ImageNet Near Shift FPR95 from 19.22% to 9.69%, verifying that the parametric head eliminates local neighborhood classification noise.
  • Robustness Under Severe Class Imbalance: When the shifted sample ratio in the unlabeled pool drops to an extreme 5%, baselines like saPU plummet to 60.24% AUROC, whereas S-PUNA maintains a strong 91.61% AUROC, demonstrating superior resilience against unknown class priors.

Highlights & Insights

  • Formulates covariate shift detection under the PU learning framework for the first time, rigorously elucidating why classical risk estimators degrade when total variation distance approaches zero.
  • Introduces an elegant spectral entropy stopping criterion based on covariance spectrum dynamics, theoretically grounding the detection of early manifold contamination during pseudo-labeling.
  • Identifies intermediate ViT representations (Block 6) as an optimal sweet spot for distribution shift detection, retaining subtle geometric and textural perturbations that are discarded by semantics-oriented deeper layers.

Limitations & Future Work

  • The approach relies on expressive pre-trained visual representations; if the backbone encoder fails to separate certain physical distortions, the local manifold assumptions may become compromised.
  • Iterative \(k\)-NN graph lookups scale super-linearly with the size of the unlabeled pool, posing potential memory and latency bottlenecks on massive web-scale corpora.
  • The framework currently focuses on offline batch evaluation; extending the spectral geometric principles to continuous streaming data under online concept drift represents a compelling future direction.
  • vs Dist-PU / nnPU: Classical PU frameworks assume global risk decomposability and separable supports, leading to severe gradient variance under overlapping covariate shifts; S-PUNA bypasses global risk estimation by progressively building support sets on local manifolds.
  • vs DisCoPatch / DRCT: Fully supervised methods require explicit negative examples during training, struggling to generalize to unseen perturbation types; S-PUNA achieves comparable or superior detection using only positive reference data.
  • vs Conventional OOD Detectors (ReAct / ASH / ViM): Standard OOD methods focus on logit-level semantic divergence, rendering them ineffective for covariate shift where semantic classes remain intact; S-PUNA highlights the necessity of exploiting geometric feature structures.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Pioneers PU formulation for covariate shift detection with solid mathematical grounding for spectral entropy stopping.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across 3 datasets, 21 comparative baselines, varying shift severities, and imbalanced mixing ratios.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from decision-theoretic bounds to geometric pseudo-labeling mechanics and empirical verification.
  • Value: ⭐⭐⭐⭐☆ Offers a practical, label-efficient solution for real-world distribution monitoring, deepfake detection, and model reliability auditing.