Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomy¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://vtoborek.github.io/unifying-taxonomy-cl-nlp/
Area: LLM Reasoning
Keywords: Curriculum Learning, Unifying Taxonomy, Difficulty Evaluation, Training Schedulers, Monotonicity Properties
TL;DR¶
Addressing more than a decade of empirical confusion and incomparability in NLP curriculum learning (CL), this paper establishes the first unified dual taxonomy that analytically disentangles difficulty evaluation (attribution source and task dependence across four quadrants) from training scheduling (formalized via expected training contribution \(\pi_t\), five retention regimes, and three monotonicity properties), backed by an extensive empirical survey of 107 strategies from 90 ACL papers.
Background & Motivation¶
Data-level curriculum learning (CL) is a time-honored training paradigm in machine learning that aims to guide model training by deliberately regulating the exposure of training instances over time. Drawing direct inspiration from human developmental learning, its guiding hypothesis posits that neural networks benefit significantly from mastering "easier" examples before tackling "harder" ones, thereby enhancing optimization stability, acceleration of convergence, and ultimate generalization performance. Yet, despite sustained interest across multiple generations of NLP architectures—from LSTMs to pre-trained Transformers and contemporary large language models—the research community has failed to converge on principled guidelines regarding which difficulty function or scheduler should be selected for any given linguistic task.
A critical investigation into this stagnation reveals that the primary obstacle is not an absence of empirical exploration, but the profound absence of a shared, rigorous conceptual framework capable of specifying what a curriculum strategy actually implements. In the existing NLP literature, difficulty is defined in an extraordinarily ad-hoc and fragmented manner: practitioners borrow metrics interchangeably from readability formulas, psycholinguistic complexity, target output length, dataset tiering, or model training dynamics. Consequently, studies published under the identical "CL" label frequently implement structurally incompatible mechanisms that capture entirely different phenomena—some reflecting human perceptual cognitive costs, others capturing model uncertainty, and still others conflating task difficulty with data corruption and noise filtering.
This conceptual ambiguity is exacerbated on the scheduling front, where descriptions typically rely on informal prose declaring an "easy-to-hard" progression without formalizing how probability mass or gradient updates are allocated over time. Implementations combining sampling and loss weighting remain opaque, obscuring whether an empirical gain arises from the sequential curriculum, sample filtering, or distinct data distribution shifts. Core idea: systematically disentangle curriculum learning into an orthogonal two-part taxonomy separating difficulty evaluation (across attribution source and task dependence) from training scheduling (formalized through expected training contribution \(\pi_t\), retention regimes, and three-tiered monotonicity properties), thereby exposing the structural root causes of empirical incomparability and establishing a mathematically sound foundation for curriculum design.
Method¶
Overall Architecture¶
The proposed taxonomy treats any data-level curriculum learning strategy as a clean decomposition of two orthogonal modules: a Difficulty Function \(d_t(x)\) that assigns scalar difficulty estimates to instances at stage \(t\), and a Training Scheduler \(\mathcal{S}\) that translates these evaluations into actual parameter update dynamics. The difficulty dimension is categorized by analyzing its auxiliary resource dependencies into four foundational quadrants according to attribution source (human-centric vs. model-centric) and task dependence (task-agnostic vs. task-dependent). Meanwhile, the scheduler is mathematically formalized via the unified abstraction of the expected training contribution \(\pi_t\), enabling mechanism-agnostic comparison across sampling and weighting schemes through two rigorous axes: five discrete data retention regimes and three independent monotonicity properties.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Training Pool & Auxiliary Resources"] --> B["Difficulty Evaluation in Four Quadrants<br/>TA-H / TA-M / TD-H / TD-M"]
B --> C["Expected Training Contribution Formalization<br/>Unified Mapping πt(x) ∝ pt(x) × wt(x)"]
C --> D["Data Retention Regime Analysis<br/>Evolution of Admissible Support At"]
D --> E["Three-Tiered Monotonicity Verification<br/>Selection / Frontier / Exposure Monotonicity"]
E --> F["Attributable Strategy Comparison & Evaluation"]
Key Designs¶
1. Difficulty Evaluation in Four Quadrants: Disentangling Cognitive Assumptions from Task Dependence
To resolve the chronic conflation of disparate difficulty proxies, the authors characterize the information requirements of difficulty functions \(d_t: \mathcal{D} \to \mathbb{R}\). Letting \(\mathcal{R}_T\) denote target-task-specific resources (such as ground-truth labels, reference outputs, or annotator agreement) and \(\mathcal{R}_M\) denote signals derived from one or more models (such as loss, perplexity, or representation geometry), difficulty functions are mapped into four distinct quadrants:
- TA-H (Task-Agnostic, Human-Centric): \(\mathcal{R} \cap \mathcal{R}_T = \emptyset \land \mathcal{R} \cap \mathcal{R}_M = \emptyset\). Relies entirely on human-designed, label-free surface properties (e.g., input sequence length, syntactic parsing depth, Flesch-Kincaid readability, or compositional depth).
- TA-M (Task-Agnostic, Model-Centric): \(\mathcal{R} \cap \mathcal{R}_T = \emptyset \land \mathcal{R} \cap \mathcal{R}_M \neq \emptyset\). Employs model-derived signals independent of task supervision (e.g., auxiliary language model perplexity, static word vector norms, SBERT Euclidean/cosine geometry, or out-of-the-box LLM rating prompts).
- TD-H (Task-Dependent, Human-Centric): \(\mathcal{R} \cap \mathcal{R}_T \neq \emptyset \land \mathcal{R} \cap \mathcal{R}_M = \emptyset\). Exploits human references or performance on the target task (e.g., reference discrepancy metrics such as ROUGE or Levenshtein edit distance, target parse depth, gold argument injection, and human annotation time).
- TD-M (Task-Dependent, Model-Centric): \(\mathcal{R} \cap \mathcal{R}_T \neq \emptyset \land \mathcal{R} \cap \mathcal{R}_M \neq \emptyset\). Directly measures task-specific interaction between model and data (e.g., training cross-entropy loss, confidence margins, teacher-student distillation discrepancy, or Fisher Information Matrix traces).
This classification exposes the fundamental epistemic assumptions behind each strategy: whether "difficulty" is assumed to reside in intrinsic human linguistic complexity, model representational geometry, or task-specific optimization barriers.
2. Expected Training Contribution Formalization: Unifying Sampling and Loss Weighting
Prior taxonomies created artificial boundaries between sampling-based pacing schedules and loss-weighting curricula. The authors introduce a unifying mathematical abstraction: any training scheduler is defined as a mapping \(\mathcal{S}: (d_t, t) \mapsto \pi_t\), where \(\pi_t \in \Delta(\mathcal{D})\) denotes a probability distribution over the dataset \(\mathcal{D}\) at stage \(t\). The quantity \(\pi_t(x)\) represents the expected training contribution of instance \(x\) to the model parameter update at step \(t\). Regardless of whether a curriculum is executed via instance sampling probability \(p_t(x)\) or sample loss weighting \(w_t(x)\), their joint impact on the gradient update collapses into:
$\(\pi_t(x) \propto p_t(x) \times w_t(x)\)$
From \(\pi_t\), key structural sets are derived: the admissible pool \(\mathcal{A}_t = \text{supp}(\pi_t) = \{x \in \mathcal{D}: \pi_t(x) > 0\}\), the retained set \(\mathcal{X}_t = \{x \in \mathcal{D}: \pi_t(x) > 0 \land \pi_{t-1}(x) > 0\}\), and the increase-set \(\mathcal{C}_t = \{x \in \mathcal{D}: \pi_t(x) > \pi_{t-1}(x)\}\). This abstraction strips away optimization-specific artifacts (e.g., Adam optimizer states or learning rate decays), allowing direct structural comparison across diverse algorithms.
3. Data Retention Regime Analysis: Mechanism-Agnostic Characterization of Admissible Support
The taxonomy formalizes how instance support \(\mathcal{A}_t\) evolves across consecutive stages, defining five partially ordered retention regimes:
- (1) Unconstrained Regime: Imposes zero restrictions on the temporal trajectory of \(\pi_t\). Samples are free to enter, exit, and re-enter support dynamically (predominant in adaptive, loss-based re-clustering).
- (2) No Reentrance Regime: Satisfies \(\forall x \in \mathcal{D}\) and \(t' \le t \le t''\), \(\pi_{t'}(x) > 0 \land \pi_{t''}(x) > 0 \implies \pi_t(x) > 0\). An instance may contribute over a single contiguous interval of stages, but once removed, it can never re-enter (e.g., progressive domain adaptation or linguistic pruning).
- (3) No Retention Regime: Enforces \(\forall t > 1, x \in \mathcal{D}\), \(\pi_t(x) > 0 \implies \forall t' \neq t, \pi_{t'}(x) = 0\). Each instance contributes to exactly one stage, meaning \(\mathcal{X}_t = \emptyset\) (e.g., disjoint bucket scheduling or one-pass partitioned training).
- (4) Full Retention Regime: Enforces \(\forall 1 \le t' < t, x \in \mathcal{D}\), \(\pi_{t'}(x) > 0 \implies \pi_t(x) > 0\). Once admitted, an instance remains in the admissible pool indefinitely (\(\mathcal{A}_{t-1} \subseteq \mathcal{A}_t\)), serving as the standard paradigm for competence-based curricula and Baby-Step pacing.
- (5) Universal Regime: For all \(t \ge 1\) and \(x \in \mathcal{D}\), \(\pi_t(x) > 0\), maintaining \(\mathcal{A}_t = \mathcal{D}\) throughout training. Progression is manifested purely through continuous redistribution of contribution mass rather than hard gating.
4. Three-Tiered Monotonicity Verification: Deconstructing Pseudo-Monotonicity and Local Attribution Traps
To replace vague claims of "easy-to-hard progression", the authors define three mathematically distinct monotonicity properties over sequences of difficulty functions \((d_t)_{t \ge 1}\):
- Selection Monotonicity: Measures the difficulty of instances actively driving the stage transition in the increase-set \(\mathcal{C}_t\):
$\(\mu_{\text{sel}}(t) = \frac{1}{|\mathcal{C}_t|} \sum_{x \in \mathcal{C}_t} d_t(x), \quad \text{monotone if } \mu_{\text{sel}}(t+1) \ge \mu_{\text{sel}}(t)\)$
- Frontier Monotonicity: Tracks the maximum admission difficulty admitted into \(\mathcal{A}_t\), anchored to the entry step \(\tau(x) = \min\{t: x \in \mathcal{A}_t\}\):
$\(\iota(t) = \max_{x \in \mathcal{A}_t} d_{\tau(x)}(x), \quad \text{monotone if } \iota(t+1) \ge \iota(t)\)$
- Exposure Monotonicity: Tracks the global expected difficulty experienced by the model under distribution \(\pi_t\):
$\(\mu_{\text{exp}}(t) = \mathbb{E}_{x \sim \pi_t}[d_t(x)], \quad \text{monotone if } \mu_{\text{exp}}(t+1) \ge \mu_{\text{exp}}(t)\)$
Crucially, these properties do not imply one another: a full retention scheduler is structurally frontier monotone, yet if the mass allocated to newly admitted difficult instances is negligible, exposure monotonicity fails. Furthermore, under local difficulty evaluation (where \(d_t\) updates dynamically with model parameters), observed monotonic trends often reflect the model becoming better at predicting samples rather than structural progression imposed by the scheduler. Thus, the authors mandate classifying a scheduler as monotone only when the property is structurally guaranteed by design.
Benchmark & Evaluation Protocol¶
Rather than testing a single model checkpoint, this study takes as its empirical subject the entire corpus of data-level curriculum learning research in the ACL Anthology. A rigorous two-phase pipeline was executed: First, an automated retrieval queried titles and abstracts for "curriculum learning" or ("curriculum"/"curricula" + "learning"), restricted to ACL-sponsored conferences and Findings while filtering out multimodal, speech, and medical imaging domains, yielding 199 candidate papers. Second, a comprehensive manual inspection was conducted by domain experts, excluding non-English works, educational curricula, and task-level curricula (where label spaces shift incompatibly). The resulting curated dataset encompasses 90 published papers spanning 107 distinct CL strategies. Every strategy was systematically annotated across difficulty quadrants, evaluation modes (global, local, in-sample), retention regimes (1 to 5), and three-tiered structural monotonicities, forming an open-source, interactive benchmark repository.
Key Experimental Results¶
Main Results: Macro-Distribution of CL Strategies in the ACL Corpus¶
A comprehensive survey of all 107 curriculum learning strategies reveals striking structural imbalances across the NLP landscape:
| Categorical Dimension | Class / Regime | Strategy Count (Total=107) | Proportion (%) | Key Characteristics & Landmark Citations |
|---|---|---|---|---|
| Difficulty Quadrant | TD-M (Task-Dep., Model-Centric) | 45 | 42.1% | Dominant paradigm; model loss, confidence, teacher signals (Feng et al., 2019) |
| TA-H (Task-Agn., Human-Centric) | 24 | 22.4% | Default baseline; token length, readability, syntax (Platanios et al., 2019) | |
| TD-H (Task-Dep., Human-Centric) | 18 | 16.8% | Target complexity, ROUGE overlap, annotation time (Magooda & Litman, 2021) | |
| TA-M (Task-Agn., Model-Centric) | 11 | 10.3% | Pre-trained embedding geometry, LLM rating (Pan et al., 2025) | |
| Mixed Quadrants | 9 | 8.4% | Composable multi-stage signals; TA-H coarse + TD-M fine (Liu et al., 2023) | |
| Data Retention Regime | (4) Full Retention | 35 | 32.7% | Most prevalent; competence-based cumulative expansion (Platanios et al., 2019) |
| (5) Universal Regime | 29 | 27.1% | Full support \(\mathcal{A}_t=\mathcal{D}\); mass redistribution or partition mixing (Zhang et al., 2025b) | |
| (3) No Retention | 16 | 15.0% | Mutually exclusive stage buckets; one-pass traversal (Bengio et al., 2009) | |
| (1) Unconstrained Regime | 13 | 12.1% | High volatility; arbitrary entry/exit driven by local loss (Maharana & Bansal, 2022) | |
| (2) No Reentrance | 9 | 8.4% | Contiguous active intervals; targeted domain removal (Elhady et al., 2025) | |
| Unspecified | 5 | 4.7% | Lacks concrete procedural specifications in published text |
Cross-Analysis: Retention Regimes vs. Structural Monotonicity Guarantees¶
Analyzing the intersection of retention mechanics and structural monotonicity demonstrates that "easy-to-hard" claims are routinely violated in practice:
| Retention Regime | Primary Quadrant | Structural Frontier Monotone (F.) | Structural Selection Monotone (S.) | Structural Exposure Monotone (E.) | Root Cause of Non-Monotonicity / Failure Modes |
|---|---|---|---|---|---|
| (1) Unconstrained | TD-M (predominantly local) | ✗ (0/13 guaranteed) | ✗ (0/13 guaranteed) | ✗ (0/13 guaranteed) | Dynamic thresholding causes erratic support shifts; zero guarantees |
| (2) No Reentrance | TA-H / TD-M | ✓ (Implementation dep.) | ✓ (Implementation dep.) | ✓ (Implementation dep.) | Not guaranteed by regime definition; hinges strictly on partition sorting |
| (3) No Retention | TA-H / TD-H | ✓ (Standard buckets) | ✓ (Standard buckets) | ✓ (Standard buckets) | Epoch-wise resets or intra-bucket shuffling break macro-monotonicity |
| (4) Full Retention | TA-H / TD-M / TD-H | ✓ Guaranteed by Design | ✗ (Requires aligned mass) | ✗ (Requires aligned mass) | Frontier is strictly monotone; but skewed weighting can lower exposure difficulty |
| (5) Universal | TD-M / TA-M / TA-H | — (Trivially Inactive) | ✗ (Requires aligned mass) | ✗ (Requires aligned mass) | Full support renders frontier vacuous; batch-level reweighting rarely monotone |
Key Findings¶
- The Conflation of Local Evaluation with Scheduling Monotonicity: Over 40% of surveyed curricula reside in the TD-M quadrant with local evaluation. In these setups, difficulty estimates \(d_t(x)\) drift alongside model parameters. What prior authors reported as a successful "curriculum progression" was frequently an illusion: the model was merely learning the data, altering difficulty outputs, while the scheduler itself provided no structural monotonic guarantee.
- Divergent Training Distributions under Identical Pacing: As demonstrated in the worked SST-2 sentiment analysis case (\(n=67,349\)), using the exact same difficulty function and pacing schedule yields radically different gradient fields under different retention regimes. At stage \(t=3\), a No-Retention scheduler concentrates 100% of probability mass on the newest tranche (\(\pi_3^{\text{NR}}(x) \approx 7.42 \times 10^{-5}\)), whereas Full-Retention dilutes mass uniformly across all admitted instances (\(\pi_3^{\text{FR}}(x) \approx 2.47 \times 10^{-5}\)). Consequently, empirical outcomes cannot be compared without specifying the retention regime.
- Curriculum Learning Confused with Data Cleansing: A substantial fraction of published curricula treat noisy, mislabeled, or out-of-domain samples as "hard" instances. Scheduling these instances toward the end of training (or downweighting them entirely) acts as noise filtering rather than human-like concept scaffolding, conflating sample utility with curriculum ordering.
Highlights & Insights¶
- Orthogonal Disentanglement of Difficulty and Scheduling: By cleanly separating the epistemic question ("What makes an example difficult?") from the distribution dynamics ("How does expected gradient contribution evolve?"), this taxonomy eliminates more than a decade of semantic ambiguity in machine learning.
- Formulation of Expected Training Contribution \(\pi_t(x)\): Bridging the historical divide between discrete instance sampling and continuous loss reweighting through \(\pi_t(x) \propto p_t(x) \times w_t(x)\) provides a mathematically unified foundation that directly applies to modern LLM pre-training and alignment mixtures.
- Deconstruction of the Monotonicity Illusion: Formalizing the distinction between Frontier Monotonicity (expanding the difficulty ceiling) and Exposure Monotonicity (increasing average batch difficulty) reveals why many intuitive curriculum algorithms fail to deliver expected optimization advantages.
Limitations & Future Work¶
- Empirical Boundary Restricted to the ACL Corpus: While the mathematical formulations of \(\pi_t\), retention regimes, and monotonicities are universally applicable to any data-level machine learning paradigm, empirical validation in this paper was conducted exclusively on NLP papers in the ACL Anthology, leaving computer vision and speech CL unmapped.
- Descriptive Rather than Prescriptive Theory: The framework excels at dissecting why prior experiments are incomparable and identifying architectural defects, but it does not yet provide closed-form predictive bounds to determine which specific \((\text{Quadrant}, \text{Regime})\) pair will maximize downstream performance for an unseen dataset.
- Automated Disentanglement of Objectives: Although the paper pinpoints the widespread conflation of curriculum ordering with noise filtering and domain selection, establishing automated statistical metrics to isolate the exact performance gain attributable to data ordering versus data pruning remains an open research direction.
Related Work & Insights¶
- vs. Wang et al. (2021b) & Soviany et al. (2022) [Foundational CL Surveys]: Prior surveys categorized curricula superficially by implementation mechanisms (e.g., "heuristic vs. automated", "discrete vs. continuous"). This paper provides the first formalization of schedulers via expected contribution \(\pi_t\) and introduces mechanism-independent retention regimes, proving that implementations previously deemed identical actually expose models to entirely different training distributions.
- vs. Platanios et al. (2019) [Competence-Based Curriculum Learning]: While Platanios et al. established competence functions as the gold standard for full-retention curricula, this taxonomy demonstrates that competence-based methods only guarantee frontier monotonicity, showing that under non-uniform sampling or local evaluations, exposure monotonicity can easily collapse.
- vs. Zhou et al. (2024) [CurBench Benchmark]: CurBench attempted to evaluate diverse CL algorithms via end-to-end downstream benchmark scores. As this paper demonstrates, black-box benchmarking across incomparable strategies obscures the actual sources of empirical gains; progress requires component-level ablation over quadrants, retention regimes, and monotonicity types.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first unified dual taxonomy for curriculum learning, decoupling difficulty sources from expected training contribution and formalizing three-tiered monotonicity.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive audit of 199 candidate ACL papers, providing rigorous manual annotations across 107 strategies in 90 publications with an interactive online repository.
- Writing Quality: ⭐⭐⭐⭐⭐ Masterful exposition, mathematically rigorous, and delivers a much-needed, unflinching diagnostic critique of the NLP literature.
- Value: ⭐⭐⭐⭐⭐ An indispensable methodological milestone that will fundamentally guide data mixture design for LLM pre-training, instruction fine-tuning, and curriculum optimization.