A Mechanism-Driven Theory of Phase Transitions in Active Learning¶

Conference: ECCV 2026
arXiv: 2607.00144
Code: https://github.com/juliamachnio/PALM
Area: Medical Images
Keywords: Active Learning, Phase Transition, Generalization Bound, Sampling Strategy, Medical Image Analysis

TL;DR¶

This paper proposes a mechanism-driven theory of phase transitions in active learning: the four components of the PAC generalization bound (empirical risk, distribution discrepancy, model complexity, confidence term) are reinterpreted as dynamically interacting terms, proving that a shift in their dominance along the labeling trajectory is inevitable. Measurable proxy metrics and piecewise regression are used to identify a "data-driven - transition - model-driven" three-phase taxonomy, explaining why representativeness, coverage, and uncertainty strategies excel at different budget stages.

Background & Motivation¶

Background: Active learning (AL) reduces annotation costs by iteratively selecting the most informative samples. Extensive empirical studies show that the effectiveness of different AL strategies strongly depends on the annotation budget, typically described as early, middle, and late budget stages: representativeness-based methods (e.g., TypiClust) perform well in the early stage, coverage-based methods (e.g., Coreset) are effective in the middle stage, and uncertainty-based methods (e.g., Entropy) dominate in the late stage.

Limitations of Prior Work: The definitions of these "stages" rely entirely on heuristic label counts (e.g., 5,000, 25,000 labels), which are highly ungeneralized across different datasets, network architectures, and representation qualities. This definition scheme lacks structural interpretability and provides almost no principled guidance for designing sampling strategies.

Key Challenge: While the effectiveness of AL varies with the budget, it remains unclear whether the underlying cause is merely heuristic label counts or some intrinsic transition of generalization mechanisms during the sampling process. Existing works fail to address this theoretically.

Key Insight: The authors start from the decomposition of the PAC generalization bound. Under sample bias and covariate shift, the true risk can be decomposed into four components: empirical risk, distribution discrepancy (IPM distance), model complexity (Rademacher complexity), and a confidence term. The authors' core hypothesis is that the stages of AL are not defined by fixed label counts, but by which of these four components "dominates" the generalization bottleneck—as sampling progresses, a natural shift in the dominant mechanism occurs.

Core Idea: Replace heuristic budget boundaries with the shift in the dominance of generalization bound components, interpreting AL dynamics as structural phase transitions, and introducing measurable proxy metrics and piecewise regression to detect the transition points.

Method¶

Overall Architecture¶

The input of the proposed method is a complete active learning trajectory (multi-round sampling + model retraining), and the output is a three-phase partition along the annotation budget axis alongside the "dominant generalization mechanism" label for each phase. The overall pipeline consists of three steps: (1) tracking the trajectories of measurable proxy metrics along the budget axis; (2) detecting structural breakpoints using piecewise linear regression; (3) partitioning the trajectory into data-driven, transition, and model-driven phases based on these breakpoints. The core theoretical guarantee comes from Theorem 3.2: under adaptive sampling, the dominant terms among the four components of the generalization bound must shift ("moving bottlenecks"), making phase transitions a structural inevitability rather than an empirical coincidence.

Key Designs¶

1. Dynamic Reinterpretation of Generalization Bound Components and "Moving Bottlenecks" Theory

To address the fundamental question of why AL strategies behave differently under varying budgets, the authors directly manipulate the decomposition form of the PAC generalization bound under sample bias (Theorem 3.1). At each sampling step \(t\), the induced query distribution \(P_Q^{(t)} \neq P_X\) introduces sample bias, and the upper bound of the true risk decomposes into four components: empirical risk \(\hat{R}_S(h)\), distribution discrepancy \(d_{\mathcal{F}}(P_X, P_Q)\) (measured by IPM), model complexity \(2\text{Rad}(\ell \circ \mathcal{H} \circ S)\), and a confidence term \(\alpha\sqrt{2\log(4/\delta)/m}\).

The key insight is that, unlike classical i.i.d. learning where the four components monotonically decrease with the sample size, under adaptive sampling, the empirical risk and complexity can increase non-monotonically (Lemma 1 and Lemma 3 provide constructive proofs: newly sampled difficult samples can push up the empirical risk, and geometric expansion in orthogonal directions can increase the Rademacher complexity), while the distribution discrepancy and confidence term eventually approach zero (Lemma 2). This inevitably creates "moving bottlenecks"—the dominant component shifts at different budget values. Theorem 3.2 proves that this shift is structurally unavoidable as \(m_t \to \infty\).

2. Design and Decoupling of Measurable Proxy Metrics

Since the four components of the generalization bound are not directly computable for deep networks, the authors design empirical proxy metrics for each component that can be tracked along the AL trajectory. Each proxy metric is designed to be "order-preserving," meaning that the order of the proxy values corresponds to that of the true components rather than their absolute numerical equality (Assumption 1 and Proposition 2 in the appendix prove this consistency).

The specific proxy metrics are as follows:

Empirical Risk (ER): Measured by the difference in cross-entropy loss on the newly acquired batch \(A_{t-1}\) before and after the model update, \(ER(t) = \hat{R}_{\text{pre}}^{(t)} - \hat{R}_{\text{post}}^{(t)}\), reflecting the "marginal informativeness" of newly labeled samples.
Label Discrepancy (LD): Measures the class distribution discrepancy between the labeled set and the entire dataset using the total variation distance \(\text{LD}(S) = \frac{1}{2}\sum_{c=1}^{C}|p_S(c) - p_X(c)|\), directly capturing representativeness.
Feature Discrepancy (FD): Measured by the average nearest-neighbor cosine distance from the labeled set to the reference set \(\text{FD}(S) = \frac{1}{|D|}\sum_{x \in D}\min_{z \in S}(1 - \cos(\phi(x), \phi(z)))\), which is an "entangled" proxy metric reflecting both distribution representativeness and geometric coverage.
Geometric Coverage (GC): Measured by the average nearest-neighbor cosine distance within the labeled set \(\text{GC}(S) = \frac{1}{|S|}\sum_{z \in S}\min_{z' \in S \setminus \{z\}}(1 - \cos(\phi(z), \phi(z')))\), specifically quantifying the geometric dispersion of labeled samples in the feature space, structurally independent of LD/FD.
Complexity (Comp): Approximates the Rademacher complexity using the sum of the \(\ell_2\) norms of the model parameters \(\sum_w \|w\|_2\), reflecting the constraint/expansion of the hypothesis space by the geometric structure of the labeled set.
Confidence Term (Conf): Directly calculated at the deterministic rate \(\alpha m_t^{-1/2}\), serving as a "vanishing baseline."

The distinction between GC and FD/LD is a major contribution of this work: distribution representativeness measures how well the labeled set reflects the data distribution, whereas geometric coverage measures the spatial dispersion of the labeled set on the feature manifold. The two handle different roles in generalization dynamics, but prior literature failed to explicitly decouple them.

3. Phase Transition Detection via Piecewise Regression

With the proxy trajectories, structural phase transition points need to be identified from noisy data. The authors fit a piecewise linear regression to each proxy \(Z_k(t)\):

\[Z_k(t) = \beta_{k,0} + \beta_{k,1} t + \sum_{j=1}^{J} \gamma_{k,j} (t - \tau_j)_+ + \epsilon_t\]

where \((t - \tau_j)_+ = \max(0, t - \tau_j)\) is the hinge function introducing a slope change \(\gamma_{k,j}\) at the breakpoint \(\tau_j\). The number of breakpoints \(J\) and their locations \(\{\tau_j\}\) are automatically selected via BIC (Bayesian Information Criterion).

The definition of a "global phase transition" (Definition 3) requires satisfying two conditions simultaneously: (1) at least two components exhibit statistically significant slope changes around \(\tau\) (\(\gamma_{k,j} \neq 0\)); (2) the dominant component \(B_{\star}(t)\) indeed shifts at \(\tau\). This two-condition definition effectively filters out single-proxy noise fluctuations, ensuring that detected transition points reflect genuine structural shifts.

4. Functional Alignment Hypothesis and the Three-Phase Taxonomy

Based on the above theory, the authors propose the "Functional Alignment" hypothesis (Proposition 1): the effectiveness of an AL strategy depends on how well its selection bias (inductive bias) matches the current dominant generalization mechanism. This naturally leads to three-phase alignment predictions:

Phase I (Data-driven): Distribution discrepancy dominates. Representativeness-based methods (e.g., TypiClust) yield the highest marginal gain by minimizing LD/FD, as the labeled set does not yet reflect the data distribution.
Phase II (Transition): Geometric coverage and complexity dominate. Coverage-based methods (e.g., Coreset) reduce bottlenecks by expanding the support of the labeled set on the feature manifold, while representativeness-based methods produce redundancy due to oversampling dense regions.
Phase III (Model-driven): Empirical risk optimization dominates. Uncertainty-based methods (e.g., Entropy, BADGE) reduce residual training loss most effectively by selecting ambiguous samples near decision boundaries.

This taxonomy explains the empirical observations in prior literature that "representativeness works well early, uncertainty works well late," but elevates the underlying cause from heuristic label counts to the level of generalization mechanisms. Importantly, it predicts that the transition point \(\tau\) varies with dataset complexity and representation quality—occurring earlier on low-complexity datasets (e.g., CIFAR-10), delayed on high-complexity datasets (e.g., CIFAR-100), and compressed when using self-supervised pre-trained representations. This prediction is systematically validated in the experiments.

A Complete Example¶

Taking the supervised training trajectory on CIFAR-100 as an example: starting from an empty labeled set, random sampling is used as a cold start, after which the strategies diverge. In the 0–6,700 label range (Phase I), LD and FD are the top-ranked proxy metrics. TypiClust maintains an accuracy advantage of approximately 2% by prioritizing "typical" samples within clusters to rapidly reduce distribution discrepancy. Crossing \(\tau_1 = 6700\) leads into Phase II (6,700–37,900), where GC and Comp become the bottlenecks. The differences in LD/FD between methods tend to vanish, and Coreset begins to outperform TypiClust by maintaining lower GC values. Crossing \(\tau_2 = 37900\) leads to Phase III, where all proxy metrics converge across methods, making ER the only remaining signal source. Entropy and BADGE achieve the highest final accuracy by selecting boundary-ambiguous samples. TCU (hard-switching TypiClust-Coreset-Uncertainty at \(\tau\)) validates the algorithmic utility of the transitions: it retains TypiClust's early gains and inherits coverage and uncertainty benefits in later stages.

Loss & Training¶

Training follows the standard pool-based AL protocol: in each round, a batch is selected from the unlabeled pool, and the model is retrained. In the supervised setting, ResNet-18/50 is trained from scratch (SGD, 100–200 epochs). In the self-supervised setting, the SSL backbone (SimCLR/BYOL/MoCo v2+/v3) is frozen, and only a linear classification head is trained (Adam, lr=\(10^{-3}\), 100 epochs) to decouple sampling geometry from representation drift. Training cold-starts from an empty labeled set, and the first batch for model-dependent methods is acquired via random sampling to ensure a fair starting point. Before running piecewise regression, proxy metrics are min-max normalized to \([0,1]\) to ensure that SSE and BIC are comparable across proxies of different scales (e.g., weight norm \(\sim 10^3\) vs. normalized entropy \(\sim [0,1]\)).

Key Experimental Results¶

Main Results¶

Validation results of the three-phase taxonomy under different datasets and representation conditions. \(K=3\) achieves R2 > 0.93 across all benchmarks, supporting the universality of the three-phase macrostructure.

Dataset-Representation	BIC for K=3	R2 for K=3	\(\tau_1\)	\(\tau_2\)	Observation
CIFAR-10 (Supervised)	-43235	0.998	5000	8300	Low complexity, earliest transition points
CIFAR-100 (Supervised)	-34349	0.990	6700	37900	A 10x class count delays \(\tau_2\) by ~4.6x
CIFAR-100 + SimCLR	-33014	0.943	700	11000	SSL compresses \(\tau_2\) from 37,900 to 11,000
ISIC 2019 (Supervised)	-1485	0.937	7000	10000	Medical data is class-imbalanced; R2 is slightly lower but remains 0.93+

Ablation Study¶

Model selection for different number of segments \(K\), validating that \(K=3\) is the optimal balance for describing AL dynamics.

Dataset	K=1 R2	K=2 R2	K=3 R2	K=4 R2	K=5 R2	Conclusion
CIFAR-10	0.960	0.997	0.998	0.998	0.999	Marginal improvement in SSE diminishes after K=3, BIC is optimal
CIFAR-100	0.985	0.988	0.990	0.991	0.992	K=3 already captures the primary structural changes
CIFAR-100 + SimCLR	0.815	0.921	0.943	0.962	0.967	Under SSL, K=4 shows improvement, but the three-phase structure remains clear
ISIC 2019	0.828	0.892	0.937	0.956	0.967	Medical data is noisy; increasing K continuously improves, but core three-phase structure is preserved

Key Findings¶

Three-Phase Universality: Although the absolute position of the transition point \(\tau\) varies with the dataset and representation quality, the sequence of Phase I (Data-driven) \(\to\) Phase II (Transition) \(\to\) Phase III (Model-driven) remains consistent across all benchmarks, indicating this is a structural property of the AL process rather than a coincidence.
Representation Quality Compresses Early Phases: SimCLR pre-training reduces the \(\tau_2\) of CIFAR-100 from 37,900 to 11,000 labels (a 3.4x compression). This occurs because the structured feature space provided by SSL pre-aligns part of the distribution and geometric coverage, letting the AL bottleneck shift faster to empirical risk optimization. This finding explains why AL behavior on strong pre-trained models differs drastically from scratch training.
Empirical Validation of Functional Alignment: The proxy-method alignment plot shows that the "best proxy = LD/FD" in Phase I aligns with "best method = TypiClust"; the dominance of GC in Phase II aligns with the superiority of Coreset; and the dominance of ER in Phase III aligns with the rise of Entropy/BADGE. Theory and empirical observations are almost perfectly synchronized.
Hidden Computational Efficiency Gains: Freezing the SSL backbone brings a 30x–160x training speedup (e.g., on ImageNet-50, Margin accelerates by 162x, dropping from 123 minutes to 0.68 minutes). Meanwhile, clustering methods like TypiClust suffer from sharp computational growth under large budgets (a ratio of 6.79 on CIFAR-10). This implies that transition-aware switching strategies carry both theoretical and practical significance—timely switching to lightweight uncertainty methods after representativeness-based methods face diminishing marginal utility is computationally wise.

Highlights & Insights¶

Explaining Empirical Intuition with Generalization Bound Components, Providing a Unified Theoretical Framework: Prior AL literature described "representativeness is good early, uncertainty is good late" as an empirical observation. This work unifies them under "Functional Alignment"—the success of each strategy depends on whether its inductive bias matches the current generalization bottleneck. This not only explains known phenomena but also predicts new transition behaviors.
Decoupling Representativeness and Coverage: Literature often uses representativeness and coverage interchangeably. This work explicitly distinguishes them via LD (label distribution total variation distance) and GC (labeled set internal nearest-neighbor distance) as two orthogonal dimensions of generalization, proving they dominate at different phases of the generalization trajectory. This decoupling is highly generalizable for examining any sampling strategy along these two dimensions.
Piecewise Regression as a Lightweight Phase Transition Detector: The BIC-driven piecewise regression design is simple and elegant. It does not require training extra detector models; instead, it directly fits a hinge function to the proxy trajectories to identify structural breakpoints. This approach can adapt to other scenarios needing detection of "dynamic behavior shifts," such as phase detection in reinforcement learning, curriculum learning stage partitioning, or loss landscape phase transition analysis during model training.
SSL Compressing AL Phases is a Practically Instructive Finding: This finding implies that when utilizing strong pre-trained models, AL strategy design should be reconsidered. If the early data-driven phase is drastically compressed by SSL, coverage and uncertainty strategies should be leveraged from the very start, bypassing the traditional route of starting with representation-based methods.

Limitations & Future Work¶

Gap Between Theoretical Assumptions and Actual Networks: Theorem 3.1 requires bounded Lipschitz loss and finite Rademacher complexity, which might not hold strictly in deep, over-parameterized networks. Although the "order-preserving" property (Assumption 1) of proxy metrics is reasonable and supported by the separation guarantees of Proposition 2, in practice, the SNR of proxies might be insufficient, biasing the detected phase transition points. The authors acknowledge that the proxy metrics "assess dominant terms rather than provide exact PAC bound magnitudes."
Simplification of Fixed Representations: Freezing the encoder in SSL experiments is done to decouple sampling geometry from representation drift, but this does not equal actual AL scenarios where fine-tuning occurs. When the encoder updates alongside AL iterations, the feature manifold changes. This complicates the dynamics of the generalization bound components and may require expanding the framework to handle phase transitions under "moving targets."
Real-time Phase Transition Detection: The phase transition detection in this study is retroactive: the entire AL trajectory must run before using piecewise regression to locate breakpoints. How to predict or perceive upcoming phase transitions online is a critical step toward true transition-aware AL algorithms, which remains unaddressed here.
Limitations in Medical Images: Although the ISIC experiments validate the framework's robustness against class imbalances and noisy data, ISIC remains a relatively small-scale medical dataset. For larger-scale, higher-resolution medical images (e.g., whole slide pathographical images, CT volumetric data), AL sampling dynamics might show new phase transition modes, requiring further validation.
Coarseness of TCU Hard-Switching: TCU is a proof-of-concept retroactive hard-switching scheme that applies a fixed sequence of strategies at transition points without supporting soft transitions or adaptive selection. Practical transition-aware AL should dynamically determine when and where to switch based on real-time proxy metric states.

vs. Traditional AL Literature (Sener & Savarese 2018, Hacohen et al. 2022, Yehuda et al. 2022): These works focus on the design of explicit sampling strategies and empirical budget evaluations, whereas this paper asks "why" these strategies work under different budgets, providing a meta-level explanatory framework. The contribution lies not in proposing new strategies, but in unifying the understanding of existing strategy behaviors.
vs. Generalization Bound Analysis (Menden et al. 2025, Cortes et al. 2008): This work builds directly on the sample bias generalization bound of Menden et al., but reinterprets it from a "single-step bound" to a "trajectory-level dynamic system." It introduces mechanism dominance and phase transitions, a perspective unexplored in prior generalization bound analyses.
vs. Hybrid/Switching Strategies (BADGE, UHerding): Hybrid strategies combine multiple signals (e.g., uncertainty + diversity) but use fixed blending weights that do not adapt to the budget. The theory in this paper predicts that the advantages of hybrid strategies should concentrate in the transition phase (Phase II) because no single mechanism dominates. This observation is empirically supported. This implies that hybrid strategies with "budget-adaptive weights" could be designed in the future.
vs. Self-Supervised + AL Studies: The analysis of SSL compressing AL phases is a pioneering contribution to this intersection. Existing works mostly study how SSL improves the absolute accuracy of AL, whereas this work reveals how SSL alters the phase transition structure of AL—a perspective that may reshape AL benchmarking and strategy evaluation in the pre-trained model era.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Establishes a causal link between generalization bound dynamics and AL phase transitions, using phase-transition language for a highly original perspective.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 4 datasets, multiple representation settings, and diverse baselines with comprehensive ablations (choice of K, proxy importance, and runtime analysis).
Writing Quality: ⭐⭐⭐⭐⭐ Progressive structure of theory-proxies-experiments, complete Theorem-Lemma chain, with rich illustrations highly self-consistent with the text.
Value: ⭐⭐⭐⭐⭐ Provides a missing "theoretical map" for the AL domain, explaining both past empirical successes of various methods and laying principled foundations for future transition-aware AL algorithm designs.