Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: Model Compression
Keywords: dataset distillation; learning with noisy labels; trajectory matching; sample reweighting; second-split forgetting

TL;DR¶

On top of a trajectory-matching dataset distillation backbone (DATM), this paper adds two modules — Selective Guidance Reweighting (SGR), which fuses dynamic KNN neighborhood consistency with static second-split-forgetting timestamps through a curriculum-style convex interpolation to purify the teacher's parameter trajectory, and Teacher-Inspired Auxiliary Targets (TIAT), which derive an auxiliary alignment target from a denoised teacher checkpoint fine-tuned on a high-confidence subset — yielding distilled datasets that stay robust under symmetric, asymmetric, and real-world label noise, with gains of up to 4.7–6.6 points over DATM on CIFAR-10 at 40% symmetric noise.

Background & Motivation¶

Dataset distillation (DD) aims to replace an entire training set with a synthetic set of a few dozen to a few hundred images: knowledge is compressed into a handful of synthetic samples so that a model retrained on them approaches one trained on the full real dataset, enabling efficient retraining, fast adaptation, and privacy-friendly data sharing. Recent work (e.g. the observation that dataset distillers act as natural label denoisers) further suggests that distillation itself can act as a filter separating informative clean signals from noise patterns. Yet mainstream DD methods — trajectory matching (DATM), distribution matching (DANCE), convexified implicit gradients (RCIG), and efficiency-oriented RDED — almost all assume clean supervision. Real web-curated data carries annotation errors on a large scale, caused by human bias, crowdsourcing inconsistency, and semantic ambiguity, and once supervision is corrupted DD's reliability collapses. Conventional learning with noisy labels (LNL) — the small-loss trick in sample selection, loss reweighting, label correction — couples noise estimation and model optimization inside a single loop: the model is both noise detector and predictor, and this self-referential feedback amplifies confirmation bias, making the model increasingly confident in wrong labels. Such methods also typically need clean validation anchors or repeated iterative refinement, which sits badly with the plug-and-play supervision that distillation wants.

The problem setting of this paper therefore becomes: under noisy supervision, how can both the quality and the capacity of distilled data be improved? The authors split this into two complementary challenges — ❶ Learning Better, making the teacher's guidance itself more trustworthy and retaining more clean knowledge; ❷ Learning More, letting the synthetic set absorb more high-quality supervision through richer teacher-student interaction. The obstacle to the first is that existing DD has no mechanism to separate clean from corrupted signals (the paper reports a 2–3% accuracy drop on CIFAR-10 at 20% noise and 50 IPC); the obstacle to the second is that the synthetic set is parameterized as fixed-size image tensors (e.g. 50 IPC), so its representational capacity is inherently bounded and information is compressed prematurely, before noise and clean signals are disentangled.

The angle of attack is plain: do not touch the labels and do not carve out a clean subset — repair the teacher instead. The framework behaves like a teacher who first sharpens their own expertise (purifying their own parameter trajectory) and then assigns homework (auxiliary tasks) to consolidate the student's learning. Core idea: move denoising up to the teacher-trajectory level — train a cleaner teacher trajectory with a curriculum-style hybrid reweighting of static forgetting priors and dynamic neighborhood consistency, then screen a reliable subset using cross-teacher confidence means and variances, fine-tune a noise-suppressed teacher checkpoint on it, and optimize the synthetic set against both the original trajectory alignment loss and this auxiliary target.

Method¶

Overall Architecture¶

Start with notation and the backbone. The real training set is \(\mathcal{D}_{\text{real}}=\{(x_i,\tilde y_i)\}_{i=1}^N\), where \(\tilde y_i\) is the observed (possibly corrupted) label; the test set is assumed fully clean and is used to measure generalization. The goal is to synthesize a compact set \(\mathcal{S}\) with \(|\mathcal{S}|\ll|\mathcal{D}_{\text{real}}|\) such that a model trained only on \(\mathcal{S}\) generalizes better than one trained on the full noisy set.

The paper follows trajectory-matching DD. The "trajectory" here is the entire path of teacher parameters as the teacher trains on the noisy real set \(\tau^*=\{\theta^*_t\}_{t=1}^{T}\) — not a diffusion sampling trajectory and not an RL decision trajectory; and the object being distilled is not the trajectory but the synthetic set \(\mathcal{S}\), which must induce a student parameter trajectory that hugs the teacher's. The procedure is bilevel. In the inner loop, starting from an anchor \(\hat\theta_t=\theta^*_t\), the student takes \(N\) gradient steps on mini-batches of \(\mathcal{S}\), simulating the student trajectory \(\hat\tau=\{\hat\theta_{t+n}\}_{n=0}^{N}\). In the outer loop, the student's endpoint must approach the teacher's parameters \(M\) steps later, \(\theta^*_{t+M}\), with a normalized alignment error:

\[\mathcal{L}_{\text{match}}=\frac{\big\|\hat\theta_{t+N}-\theta^{*}_{t+M}\big\|_2^2}{\big\|\theta^{*}_{t+M}-\theta^{*}_{t}\big\|_2^2}\]

The denominator is the teacher's own parameter displacement over \([t,t+M]\); it acts as a scaling factor so that the loss is comparable across anchors (scale invariance). Without it, later anchors have smaller displacements and hence smaller losses, and optimization is dominated by the early anchors.

On this backbone the proposed framework is a three-stage pipeline. Stage one trains the teacher on \(\mathcal{D}_{\text{real}}\): during training, Selective Guidance Reweighting (SGR) assigns each sample a weight reflecting its estimated reliability, and these weights modulate both the teacher's parameter updates and the outer-loop guidance for \(\mathcal{S}\), producing a purified teacher trajectory; \(P\) teacher trajectories are trained with blending coefficients spread out for diversity. Stage two initializes the synthetic set \(\mathcal{S}\) (sampled from Gaussian noise or selected from real samples, with soft or hard labels), simulates the student's inner-loop training on it, and introduces Teacher-Inspired Auxiliary Targets (TIAT): auxiliary consistency signals drawn from intermediate, high-confidence teacher states act as regularization beyond trajectory alignment. Stage three updates \(\mathcal{S}\) in the outer loop using the normalized trajectory matching loss over multiple anchor steps. Together, SGR improves the trustworthiness of the source of supervision and TIAT improves how efficiently trajectory alignment is absorbed by the student.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Noisy real set D_real"] --> B["Hybrid reweighting SGR<br/>KNN consistency × SSFT forgetting prior"]
    B --> C["Multiple diverse teacher trajectories<br/>blending coefficients spread over 0–1"]
    C --> D["Confidence aggregation & stability filtering<br/>cross-teacher mean + variance → D_sub"]
    D --> E["TIAT auxiliary targets<br/>denoised checkpoint from reliable subset"]
    E --> F["Normalized trajectory matching<br/>joint loss updates synthetic set S"]
    F --> G["Distilled set S (used to train students)"]

Key Designs¶

1. Hybrid reweighting SGR: folding "learned how late" and "what the neighbors think" into one weight

SGR targets the pain point that under noise the teacher trajectory itself is untrustworthy. It estimates a reliability weight per training sample from two complementary signals and then interpolates them as training progresses, rather than simply multiplying the loss by a static coefficient.

The first signal is dynamic KNN consistency. In feature space, take the \(K\) nearest neighbors of sample \(i\) and measure how much their predicted distributions \(\hat y_j\) agree with its own observed label \(\tilde y_i\) (treated as a one-hot distribution), using the Jensen–Shannon divergence:

\[W_{\text{knn}}^{(i)}=1-\frac{1}{K}\sum_{j=1}^{K}\mathrm{JS}(\tilde y_i,\hat y_j)\]

The score lies in \([0,1]\): the more the neighbors endorse that label, the higher the value. It captures local evidence — if a sample's label contradicts all of its feature-space neighbors, the label is very likely wrong.

The second signal is the static forgetting prior (SSFT). Following the second-split forgetting idea, two timestamps are recorded per sample while training the teacher — \(t^{(i)}_{\text{learn}}\), the first epoch at which the sample is correctly classified, and \(t^{(i)}_{\text{forget}}\), the first epoch after which it is misclassified again — and converted into a difficulty score \(s^{(i)}\) (learned late and forgotten early means a higher score), which is flipped into a reliability weight \(W_{\text{ssft}}^{(i)}=1-s^{(i)}\). ⚠️ The formula for this score is OCR-corrupted in the cached text; the form below is reconstructed from the paper's verbal description of \(s^{(i)}\) (learned late + forgotten early ⇒ more likely noisy/hard) and from the stated balancing role of \(\lambda\in[0,1]\). Refer to the original paper for the exact expression:

\[s^{(i)}=\lambda\cdot\frac{t^{(i)}_{\text{learn}}}{\max_j t^{(j)}_{\text{learn}}}+(1-\lambda)\left(1-\frac{t^{(i)}_{\text{forget}}}{\max_j t^{(j)}_{\text{forget}}}\right)\]

This signal captures global evidence: forgetting behavior is a trace the sample leaves over the whole training history and is unaffected by the quality of the representation at any single moment. These timestamps are computed once before distillation and reused across all experiments, so the cost is a one-time preprocessing overhead.

The reliability of the two signals varies with the training stage, so SGR combines them with a time-dependent coefficient \(\alpha_t=\min\big(\alpha_{\max},\ \tfrac{t}{T_{\text{warmup}}}\cdot\alpha_{\max}\big)\), i.e. a linear ramp from 0 to \(\alpha_{\max}\) over the first \(T_{\text{warmup}}\) epochs (default \(T_{\text{warmup}}=60\), roughly 20% of training):

\[W^{(i)}=(1-\alpha_t)\cdot W_{\text{ssft}}^{(i)}+\alpha_t\cdot W_{\text{knn}}^{(i)}\]

The curriculum-style handover has a concrete justification: early in training the network is still in its early-learning phase, the representation space is unreliable and KNN neighbors may themselves cluster around noise, so the globally accumulated forgetting prior is the safer bet; later, once the network begins to memorize noise and the feature space becomes meaningful, local consistency captures the sample's current state better. In addition, when \(P\) teacher trajectories are trained, the \(p\)-th one uses \(\alpha^{(p)}_{\max}=(p-1)/P\), spreading the teachers uniformly from "static-prior heavy" to "dynamic-consistency heavy" and supplying the diversity that the cross-teacher consensus later exploits.

2. Confidence aggregation and stability filtering: the mean is not enough — the variance is the evidence of stable reliability

The weight from a single trajectory is noisy, so the first step of TIAT aggregates the opinions of many teacher trajectories into a confidence score while looking at both mean and variance. The confidence of sample \(i\) is defined as its expected reliability under all teacher views, approximated in practice by the mean over \(P\) trajectories:

\[W_{\text{conf}}^{(i)}:=\mathbb{E}_{p\sim\mathcal{U}(\mathcal{P})}\big[W_p^{(i)}\big]\approx\frac{1}{P}\sum_{p=1}^{P}W_p^{(i)}\]

where \(W_p^{(i)}\) is the weight assigned to sample \(i\) by the \(p\)-th teacher trajectory in its final stage. The mean answers "is this sample trustworthy on the whole," while the cross-trajectory variance \(\mathrm{Var}_p(W_p^{(i)})\) answers "is its trustworthiness stable": low variance means teachers with different static-dynamic blends all reach a similar verdict, and only such samples count as consistently reliable. A reliable subset \(\mathcal{D}_{\text{sub}}\) is selected from the two complementary criteria: \(W^{(i)}_{\text{ssft}}\ge\delta_{\text{sup}}\) (statistically confident enough) and \(\mathrm{Var}_p(W_p^{(i)})\le\sigma_{\text{inf}}\) (dynamically stable enough).

Using the variance rather than only a threshold is what sets this design apart from the usual "select high-confidence samples" recipe: a sample may score high under one trajectory and low under another, and the mean alone can be fooled by that averaging illusion, whereas the variance exposes exactly this kind of lucky high score. ⚠️ The concrete values of the thresholds \(\delta_{\text{sup}}\) and \(\sigma_{\text{inf}}\) are not given in the main text; refer to the original paper.

3. TIAT auxiliary targets: align the student to a denoised teacher checkpoint instead of regressing the raw trajectory

SGR only fixes the teacher side — it makes the teacher trajectory cleaner but provides no mechanism governing how the student absorbs those signals during alignment. Residual noise and locally unstable regions of the trajectory can still be fitted by the student, showing up as decision-boundary shifts and limited generalization. TIAT therefore adds a regularizer on the student side: using the reliable subset \(\mathcal{D}_{\text{sub}}\) from the previous step, the teacher checkpoint \(\theta^*_t\) is briefly fine-tuned (same learning protocol as the original teacher, but restricted to high-confidence samples) to yield a "noise-suppressed continuation of the trajectory" \(\theta^{\text{ft}}_{t+M}\); the student's displacement after \(N\) steps on the synthetic set is then aligned to this denoised checkpoint's displacement, using a loss structurally identical to the main one:

\[\mathcal{L}_{\text{aux}}=\frac{\big\|\hat\theta^{(i)}_{t+N}-\theta^{\text{ft}}_{t+M}\big\|_2^2}{\big\|\theta^{\text{ft}}_{t+M}-\theta^{*}_{t}\big\|_2^2}\]

Why this is more stable than regressing the teacher trajectory directly: the original endpoint \(\theta^*_{t+M}\) still carries the parameter perturbation caused by noisy samples, so fitting it treats that perturbation as the correct answer, whereas \(\theta^{\text{ft}}_{t+M}\) is obtained by continuing training on the clean subset and represents "where the teacher should have arrived without that noise" — aligning to it simultaneously pushes the noise direction away. Nothing is relabeled and no sample is removed, so the pipeline stays label-preserving and lightweight. One point deserves emphasis: these high-confidence real samples are used only to refine teacher checkpoints and to construct auxiliary targets; they are not paired with, indexed against, or mapped onto individual synthetic samples, so the synthetic set \(\mathcal{S}\) does not become a copy of real samples.

Loss & Training¶

The final objective is a weighted sum of the main trajectory alignment loss and the auxiliary loss, where \(\beta\in[0,1]\) controls the influence of the denoised auxiliary trajectory relative to the original teacher trajectory (⚠️ this equation is OCR-corrupted in the cached text; the form below is reconstructed from the paper's description of \(\beta\) and its ablation behaviour — refer to the original paper):

\[\mathcal{L}_{\text{total}}=(1-\beta)\cdot\mathcal{L}_{\text{match}}+\beta\cdot\mathcal{L}_{\text{aux}}\]

Key hyper-parameters: \(P=100\) teacher trajectories (following DATM); for SGR, KNN neighborhood \(K=10\), SSFT balance coefficient \(\lambda=0.6\), warm-up length \(T_{\text{warmup}}=60\); for TIAT, a 60% high-confidence subset ratio and auxiliary weight \(\beta=0.1\). CIFAR-10/100 use a three-layer ConvNet and Tiny-ImageNet a four-layer ConvNet, with student and teacher sharing the architecture; evaluation reports mean and standard deviation of test accuracy over 5 independently initialized networks trained on the distilled set.

Key Experimental Results¶

Main Results¶

Four noise settings on CIFAR-10 (symmetric/asymmetric × 20%/40%). Each column also lists the strongest baseline in that column, so that the backbone's contribution is not mistaken for a SOTA claim:

Noise setting (CIFAR-10)	IPC	Ours	DATM	Best baseline in column	vs DATM
Symmetric 20%	10	64.5	62.7	68.7 (DANCE)	+1.8
Symmetric 20%	1000	83.3	81.9	81.9 (DATM)	+1.4
Symmetric 40%	10	64.8	58.2	65.8 (DANCE)	+6.6
Symmetric 40%	1000	81.4	76.7	76.7 (DATM)	+4.7
Asymmetric 20%	10	63.7	61.9	69.2 (DANCE)	+1.8
Asymmetric 20%	1000	82.1	80.4	80.4 (DATM)	+1.7
Asymmetric 40%	10	58.7	55.5	66.1 (DANCE)	+3.2
Asymmetric 40%	1000	75.0	71.4	71.4 (DATM)	+3.6

Other datasets and real-world noise (CIFAR-N uses the multi-annotator CIFAR-10N/100N; the "Full Dataset" column is a reference that trains directly on the full noisy set without distillation):

Dataset / noise	IPC	Ours	Best baseline	Full dataset
CIFAR-100 symmetric 40%	100	52.8	44.4 (DATM)	39.9
CIFAR-100 asymmetric 40%	100	40.7	36.0 (DATM)	33.0
CIFAR-100N real-world noise	10	41.0	42.0 (DANCE)	44.4
CIFAR-100N real-world noise	50	45.9	43.9 (DATM)	44.4
CIFAR-10N Aggre (9.03%)	1000	83.4	83.0 (DATM)	82.5 (full set as reported by DANCE)
CIFAR-10N Worst (40.21%)	1000	79.2	75.3 (DATM)	67.1
Tiny-ImageNet symmetric 20%	10	30.8	30.4 (DATM)	30.0
Tiny-ImageNet symmetric 40%	10	30.1	28.2 (DATM)	22.6

Ablation Study¶

Adding modules one at a time on top of DATM (CIFAR-10; gains over DATM in parentheses):

Config	Symmetric 20% @50 IPC	Symmetric 40% @10 IPC	Asymmetric 40% @1000 IPC
DATM	71.7	58.2	71.4
+ SGR	72.8 (↑1.1)	62.9 (↑4.7)	73.3 (↑1.9)
+ SGR + TIAT	73.5 (↑1.8)	64.8 (↑6.6)	75.1 (↑3.7)

Sensitivity to the auxiliary weight \(\beta\) and the high-confidence subset ratio (CIFAR-10, IPC=50):

β	Subset 60%, symmetric 20%	Subset 60%, symmetric 40%
0.1 (default)	73.5	71.5
0.5	73.4	72.2
1.0	69.6	71.7

Efficiency overhead (CIFAR-10, single RTX 3090, both DATM and this method use the same TESLA-style memory-saving implementation):

IPC	Peak memory	DATM time	Ours time	Overhead
10	2.5 GB	6.5 h	7.0 h	1.08×
50	6.2 GB	17 h	25 h	1.47×
500	10.4 GB	24 h	36 h	1.50×
1000	11.6 GB	60 h	70 h	1.17×

Key Findings¶

The two modules are complementary and clearly specialized. SGR gains most where noise is heavy and IPC is low (+4.7 at 10 IPC under 40% symmetric noise), i.e. trajectory-level denoising pays off most when corruption is severe and synthetic samples are scarce; TIAT delivers larger relative gains when the noise has a systematic bias (+3.7 at 1000 IPC under 40% asymmetric noise), i.e. uncertainty filtering plus a clean checkpoint helps especially with structured noise. Overall TIAT contributes more under high noise, while both are stable in low-noise regimes.
Under noise, distillation genuinely acts as a filter. On Tiny-ImageNet at 40% symmetric noise the method reaches 30.1% versus 22.6% for training directly on the full noisy set; on CIFAR-100N at 50 IPC, 45.9% also overtakes the 44.4% full-dataset reference. Conversely, the random Subset baseline gets only 3–20% under the same budget, showing that compression itself is not the source of the gain — the way it is compressed is.
DANCE remains stronger at low IPC under asymmetric noise. At 10 IPC with 20% asymmetric noise DANCE reaches 69.2 versus 63.7 here (5.5 points higher), and 7.4 points higher at 10 IPC with 40% asymmetric noise. Distribution-matching methods do not rely on nested optimization and are steadier with extremely small synthetic sets; the paper acknowledges this, and the advantage of this method concentrates at high IPC and high noise rates.
\(\beta\) must not be large. With \(\beta=1.0\) (essentially optimizing only the auxiliary objective) accuracy drops to 69.6 under 20% symmetric noise, 3.9 points below \(\beta=0.1\), confirming that the auxiliary target must ride on top of trajectory alignment and can only serve as a regularizer. \(\beta=0.1\) and \(0.5\) are close; \(0.1\) was chosen because it is stable under both noise rates. The subset ratio is insensitive over 50%–70%, and 60% with \(\beta=0.1\) is the default combination.
Teacher diversity helps. Spreading the \(\alpha_{\max}\) of the \(P\) trajectories uniformly over \([0,1]\) (Diverse Sampling) is clearly more stable than fixed \(\alpha_{\max}\) values (0/0.5/1.0) as noise rises; the fixed ones show non-negligible degradation with changing noise rate, suggesting that injecting diversity is itself a source of robustness.
The overhead is mostly training time, not memory. Reliability estimation and auxiliary target construction cost 1.08×–1.50× training time (most expensive at IPC=500), while peak memory stays essentially on par with DATM.

Highlights & Insights¶

Two orthogonal noise signals scheduled across training stages. The SSFT timestamps answer "how has this sample behaved across the whole training history" (global, static, computed once), while KNN consistency answers "what do its neighbors think of it right now" (local, dynamic, updated every round), and \(\alpha_t\) ramping linearly from 0 to \(\alpha_{\max}\) executes the handover from "trust the prior early" to "trust the local view late." The switch point is well motivated: during early learning the feature space is still unreliable and KNN can cluster noisy samples together around their wrong labels, so trusting it too early is actively harmful.
Cross-trajectory variance as a reliability criterion, not just the mean. The classic blind spot of ensembles is the sample whose mean looks high while the members disagree sharply; putting the variance explicitly into the selection criterion demands that a sample be judged clean under very different static-dynamic blends. It is a cheap and effective source of robustness that transfers directly to any multi-trajectory/multi-view distillation or pseudo-label screening pipeline.
The role of the "clean subset" is repurposed. Conventional LNL uses clean samples to select data or correct labels; here they only fine-tune a denoised teacher checkpoint, and everything else is handled by an auxiliary alignment term structurally identical to the main loss. This preserves label invariance and avoids the sample loss caused by hard filtering, and the idea transfers to other distillation/fine-tuning pipelines that can build a reference trajectory from a clean anchor (for instance, refining a reference model from a clean preference subset to constrain the policy in RLHF).
The normalized alignment loss is worth reusing. The numerator measures how far the student still is from the target, the denominator how far the teacher itself moved over the same interval; the latter cancels the scale differences across anchors and training stages, and this scale-invariant form keeps multi-anchor joint optimization from being dominated by the early anchors.

Limitations & Future Work¶

Validated only on image classification. CIFAR-10/100, Tiny-ImageNet, and CIFAR-N are the entire experimental scope; there is no larger-scale evaluation (e.g. an ImageNet-1K subset) and no downstream detection or segmentation task, so the generalization evidence is limited to low-resolution classification.
Tightly coupled to the trajectory-matching backbone, at a real cost. The method is built on DATM and must first train \(P=100\) teacher trajectories — 60→70 hours on a single GPU at IPC=1000. It cannot be applied directly to distribution-matching DD methods (DANCE, M3D, etc.) that avoid nested optimization, which limits its reach.
No sensitivity analysis for the thresholds \(\delta_{\text{sup}}\) and \(\sigma_{\text{inf}}\). The paper grids the subset ratio and \(\beta\) but does not report the values of, or ablate, these two screening thresholds (⚠️ not given in the main text; refer to the original paper), so the construction of the reliable subset is not fully reproducible.
Still behind at low IPC under asymmetric noise. Distribution-matching methods are clearly stronger with very small synthetic sets, and the paper only offers a qualitative explanation; porting the SGR weighting logic into a distribution-matching objective might close this gap.
Small numeric inconsistencies with the baselines. For CIFAR-10 at 40% asymmetric noise, DATM is reported as 55.5 in Table 1 and 55.9 in Table 6, and the results here at 50/1000 IPC are 67.0/75.0 (Table 1) versus 67.1/75.1 (Table 6). The differences are within standard-deviation range; each figure is quoted as it appears in its own table rather than being harmonized.
The naming is easy to misread. The title's "trajectory distillation" actually refers to dataset distillation within a trajectory-matching framework, unrelated to trajectory distillation in diffusion-model or reinforcement-learning contexts; readers of the abstract should keep the context in mind.

vs DATM (difficulty-aligned trajectory matching): DATM is this paper's backbone — also trajectory matching, but selecting matching intervals in a difficulty-aligned way; this paper adds SGR (changing the per-sample weights of the teacher trajectory) and TIAT (adding an auxiliary alignment target) on top of it, and the ablation also starts from DATM. The key difference is that DATM fully assumes clean labels and has no defense once noise enters.
vs DANCE (dual-view distribution alignment): DANCE performs distribution matching without nested optimization, so it stays very competitive at low IPC under asymmetric noise (5.5 points ahead at 10 IPC with 20% asymmetric noise), but its accuracy saturates as IPC grows (on CIFAR-100N, 50 IPC is actually below 10 IPC) — it cannot extract more clean supervision from more synthetic samples, whereas the trajectory-matching backbone here keeps benefiting from larger IPC.
vs RDED / RCIG: RDED targets efficiency and realism, RCIG uses convexified implicit gradients; both degrade quickly under noise — RDED stops reporting results at high IPC (the drop is too large) and RCIG is clearly below this method in every setting. Their efficiency advantage does not buy robustness in noisy regimes.
vs conventional LNL (small-loss selection, Co-teaching, Meta-Weight-Net-style reweighting, label correction): those methods couple noise estimation with model optimization in one loop, with the model acting as both detector and predictor, so confirmation bias is self-amplifying, and they usually need clean validation anchors or repeated iterative refinement. This paper moves denoising up to the teacher-trajectory level, touching neither labels nor sample removal and requiring no clean anchors — exactly what the "compact, plug-and-play supervision" goal of distillation asks for, and the most load-bearing argument in the paper's motivation.

Rating¶

Novelty: ⭐⭐⭐ Both modules are recombinations and transfers of existing ideas (SSFT forgetting priors, KNN consistency, cross-teacher variance screening, auxiliary distillation targets), but the combined perspective of "move denoising to the teacher-trajectory level, then use a denoised checkpoint as the auxiliary target" is valuable for this specific problem, and the motivation (LNL's coupling dilemma versus DD's plug-and-play requirement) is brighter than the method itself.
Experimental Thoroughness: ⭐⭐⭐⭐ Covers symmetric/asymmetric/real-world noise, four datasets, four IPC levels, with ablations on modules, \(\beta\), subset ratio, and efficiency; weaknesses are the missing threshold sensitivity analysis, the absence of larger-scale tasks, and two small numeric discrepancies between prose and tables.
Writing Quality: ⭐⭐⭐ The motivation and the three challenges are clearly stated and the pipeline figure is complete, but formula rendering is poor (several equations are already broken when extracted from the PDF), some symbol definitions are missing (threshold values, the exact form of \(s^{(i)}\)), and the title's "trajectory distillation" invites confusion with diffusion/RL contexts.
Value: ⭐⭐⭐⭐ Gives a lightweight, label-preserving plugin that can be attached to any trajectory-matching DD pipeline for the "data compression under noisy supervision" intersection, and it makes the robustness gain real without touching labels (+4.7–6.6 at 40% noise), which is directly useful for practical data pipelines.