ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/hyhy892/ACTD
Area: LLM Reasoning
Keywords: Cross-Tokenizer Distillation, Knowledge Distillation, Anchor Loss, Residual Regularization, Multi-Teacher Distillation
TL;DR¶
ACTD tackles vocabulary/sequence misalignment and long-tail projection noise in cross-tokenizer distillation via first-subtoken vocabulary mapping, nearest monotonic boundary alignment, a Top-k anchor loss, and an L1 residual regularization that suppresses non-anchor probability mass, yielding up to +4.5 AVG@8 gains on reasoning benchmarks and unlocking robust multi-teacher fusion.
Background & Motivation¶
Large language models (LLMs) have achieved remarkable breakthroughs on multi-step reasoning, mathematical proof, and symbolic synthesis tasks. However, deploying frontier reasoning models in low-latency environments or resource-constrained edge devices remains impractical due to excessive parameter counts and quadratic autoregressive decoding latency. Knowledge distillation (KD) serves as the primary technique to compress these capabilities by transferring soft predictive distributions from capable teacher models to lightweight student networks. Nevertheless, standard distillation fundamentally presumes a shared vocabulary between teacher and student. When attempting to distill across distinct model families—such as transferring mathematical reasoning from LLaMA or DeepSeek-R1 teachers to compact Qwen student backbones—the severe heterogeneity in tokenization schemes, subtoken segmentations, and vocabulary sizes invalidates direct distribution-matching objectives like Kullback-Leibler (KL) divergence.
Recent studies on cross-tokenizer distillation (e.g., optimal transport in ULD, cross-attention in DSKD, and approximate likelihood matching in ALM) endeavor to construct global vocabulary transformation matrices or project sequence distributions. However, empirical analysis shows that these global distribution transports incur severe representation distortion and noise accumulation. At the vocabulary level, a single token in the teacher vocabulary often fragments into multiple subwords under the student tokenizer, causing irreversible semantic drift and probability mass redistribution during reverse mapping. At the sequence level, disparate tokenizers generate different sequence lengths and boundaries for identical input texts. Most critically, analyzing token probability distributions reveals that while individual long-tail tokens have negligible probabilities, their vast quantity aggregates substantial cumulative probability mass across the student's non-target vocabulary regions after projection. Distilling over the full vocabulary forces the student model to over-smooth unsupported token categories (mode-averaging), diluting predictive certainty and harming rigorous reasoning precision.
Extending existing cross-tokenizer methods to the multi-teacher setting further exacerbates these pathologies: alignment-induced projection noise compounds exponentially across heterogeneous teachers, causing multi-teacher variants to suffer negative transfer and fall below single-teacher baselines. To resolve this dilemma, rather than striving for a lossy, noisy global distribution match over the full vocabulary, cross-tokenizer distillation should focus supervision strictly on the teacher's sharp, high-confidence probability support set while neutralizing long-tail projection artifacts. Core idea: Bridge structural tokenizer mismatch through first-subtoken mapping and nearest monotonic character boundary alignment, optimize an anchor loss solely over the teacher's Top-k high-probability tokens, and impose an L1 residual regularization to penalize accumulated probability mass on the remaining vocabulary, cutting off long-tail projection noise and enabling robust, complementary multi-teacher distillation.
Method¶
Overall Architecture¶
The ACTD framework consists of two sequential phases: structural heterogeneity alignment and residual-regularized anchor distillation. In the first phase, tokenizers are unified through first-subtoken vocabulary mapping and character-level nearest monotonic sequence boundary alignment. In the second phase, the teacher's Top-k probability distribution is filtered, renormalized, and mapped to a sparse student anchor subset to compute the anchor L1 loss, while an L1 residual loss directly penalizes the aggregate probability mass allocated to non-anchor tokens. For multi-teacher distillation (Multi-ACTD), logits are cached independently per teacher, individual ACTD losses are computed, and their arithmetic mean forms the final multi-teacher supervision.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Input problem and prefix-truncated rationale<br/>OpenThoughts-114k Math"] --> B["Tokenizer and sequence boundary alignment<br/>First-subtoken mapping + Monotonic character boundary match"]
B --> C["Teacher Top-k anchor subset construction<br/>Top-128 probability renormalization mapped to At"]
C --> D["Anchor distillation and residual regularization<br/>Anchor L1 loss + Residual subset probability mass penalty"]
D --> E["Multi-teacher supervision aggregation<br/>Arithmetic mean of individual teacher losses (Multi-ACTD)"]
E --> F["Update student model parameters<br/>Qwen3-1.7B-Base / Llama-3.2-3B"]
Key Designs¶
1. First-subtoken and nearest monotonic boundary alignment: Bridging vocabulary-level and sequence-level structural mismatch
Distillation across heterogeneous model families requires unambiguous correspondences across both vocabulary tokens and sequence decoding steps. For vocabulary alignment, ACTD defines a deterministic mapping function \(\Gamma: \mathcal{V}_T \to \mathcal{V}_S\). Tokens shared identically across teacher and student vocabularies are mapped directly to their corresponding token IDs. For unshared tokens, the teacher token is first decoded into raw text and re-tokenized by the student tokenizer, selecting the first student subtoken as the target match. This preserves unambiguous 1-to-1 anchors without artificial vocabulary expansion. For sequence alignment, because different tokenizers segment sentences into divergent lengths, ACTD aligns positions through character-level boundaries: the text is tokenized by the student model to obtain character offsets, truncated to valid student token coverage, and re-tokenized by the teacher. Both tokenizations are converted into character boundary coordinates, and each student decoding position is monotonically aligned to the nearest teacher position, producing an exact position-to-position supervision map.
2. Anchor Loss: Focusing supervision on teacher high-confidence support regions
Empirical noise analysis demonstrates that cross-tokenizer projection noise predominantly stems from the vast number of low-probability long-tail tokens. Consequently, ACTD filters the teacher predictive distribution at decoding position \(t\) to retain only its Top-\(k\) tokens (set to \(k=128\)) and renormalizes their probabilities to form \(p_t^T = (p_{t,1}^T, \dots, p_{t,k}^T)\). These top tokens are projected into the student vocabulary via \(\Gamma\), yielding the student anchor subset \(\mathcal{A}_t = \{a_{t,j}\}_{j=1}^k \subset \mathcal{V}_S\). The student model's predicted probability on the \(j\)-th anchor token is \(\hat{p}_{t,j}^S = p_t^S(a_{t,j})\). The anchor loss computes the L1 distance between the teacher's high-confidence probabilities and the student's anchor predictions:
This formulation bypasses dense, expensive vocabulary-wide transport matrices and completely isolates the student from low-confidence teacher noise.
3. Residual Regularization: Suppressing non-anchor probability leakage and mitigating mode-averaging
Supervising anchor tokens alone leaves the remaining student vocabulary unconstrained, which can allow the student to disperse probability mass across irrelevant tokens, leading to high predictive entropy. ACTD defines the residual subset as all student tokens outside the anchor set, \(\mathcal{R}_t = \mathcal{V}_S \setminus \mathcal{A}_t\). Because the teacher's long-tail probabilities become corrupted during cross-vocabulary projection, ACTD discards their individual values, sets their target vector to zero, and penalizes the student's aggregate probability mass allocated to the residual subset:
Crucially, an L1 norm penalty is adopted rather than standard KL divergence. KL divergence over low-probability regions exhibits strong mode-averaging tendencies, causing excessive distribution smoothing. In contrast, directly minimizing the accumulated L1 mass forces the student to contract non-anchor probabilities, concentrating density back into the anchor set, lowering output entropy during inference, and preserving sharp decision boundaries.
4. Robust multi-teacher distillation (Multi-ACTD): Decoupled caching and noise-free complementary aggregation
Extending cross-tokenizer distillation to multiple teachers traditionally fails because mapping errors from different tokenizers compound and trigger negative transfer. Multi-ACTD decouples multi-teacher supervision: given \(M\) teachers \(\{T_m\}_{m=1}^M\), Top-\(k\) logits are cached independently for each teacher. During training, teacher-specific ACTD losses are computed per sample position:
The multi-teacher objective is then formulated as the unweighted arithmetic mean across all teachers:
Because residual regularization cleanses the projection noise for each individual teacher, combining multiple teachers produces synergistic, noise-free learning that exploits complementary reasoning trajectories across distinct model families.
Loss & Training¶
The position-level training objective is \(\mathcal{L}_{\text{ACTD}}(t) = \mathcal{L}_{\text{anchor}}(t) + \lambda_{\text{res}} \mathcal{L}_{\text{res}}(t)\), averaged across all supervised target tokens. Training uses prefix-truncated reasoning data from OpenThoughts-114k: completions are partitioned into reasoning prefixes and answer suffixes (using </think> tags or the last \boxed{} formula); prefixes exceeding 2,000 student tokens are truncated back to the nearest sentence boundary within a 600-character window and closed with a forced </think> token to maintain coherence within a 4,096 sequence budget. Optimization uses FSDP2 full-parameter fine-tuning on two GPUs with bfloat16, effective batch size 36, 300 steps, paged AdamW 8-bit, cosine learning rate schedule peaking at \(1.0 \times 10^{-5}\) with 30 warmup steps, \(\lambda_{\text{res}} = 1.0\), and \(k = 128\).
Key Experimental Results¶
Main Results¶
Primary evaluations use Qwen3-1.7B-Base as the compact student, trained on 30,000 math examples from OpenThoughts-114k. Inference runs under an 8,192 token limit (\(T=1.0\), \(top\_p=0.8\)) across five challenging benchmarks: AIME 2024, AIME 2025, AMC 2023, Minerva, and MATH500.
| Model / Distillation Setting | Setting | AIME24 (AVG8) | AIME25 (AVG8) | AMC23 (AVG8) | Minerva (AVG8) | MATH500 (AVG8) | Mean AVG@8 | Mean PASS@8 |
|---|---|---|---|---|---|---|---|---|
| Teacher Models | ||||||||
| DS-R1-Distill-Llama-8B (T1) | Heterogeneous | 26.7 | 21.7 | 73.1 | 32.4 | 81.3 | 47.0 | 67.6 |
| DS-R1-Distill-Qwen-7B (T2) | Semi-heterogeneous | 40.8 | 32.5 | 80.6 | 40.7 | 87.9 | 56.5 | 74.2 |
| DS-R1-0528-Qwen3-8B (T3) | Semi-heterogeneous | 6.7 | 33.3 | 30.0 | 33.0 | 66.1 | 33.8 | 46.3 |
| Student Baselines | ||||||||
| Qwen3-1.7B-Base (S) | Base model | 2.1 | 1.3 | 27.5 | 17.2 | 51.8 | 20.0 | 43.1 |
| SFT (Hard Labels) | Train set SFT | 5.0 | 3.3 | 27.2 | 20.0 | 57.1 | 22.5 | 44.3 |
| KD (Homogeneous) | T3 (same base) \(\to\) S | 8.3 | 4.6 | 34.7 | 25.1 | 60.8 | 26.7 | 48.7 |
| Single-Teacher Cross-Tokenizer (T1 \(\to\) S) | ||||||||
| DSKD | Dual-space attention | 5.0 | 2.9 | 29.4 | 22.0 | 57.4 | 23.3 | 42.8 |
| ALM | Approx. likelihood | 5.0 | 1.7 | 29.7 | 18.4 | 53.5 | 21.6 | 44.7 |
| ULD | Optimal transport | 4.6 | 3.8 | 30.3 | 18.4 | 57.4 | 22.9 | 42.8 |
| ACTD (Ours) | Anchor + Residual | 9.2 | 5.0 | 40.0 | 22.8 | 62.0 | 27.8 | 50.7 |
| Single-Teacher Cross-Tokenizer (T2 \(\to\) S) | ||||||||
| DSKD | Dual-space attention | 3.8 | 2.5 | 31.9 | 20.2 | 57.1 | 23.1 | 45.3 |
| ALM | Approx. likelihood | 4.2 | 1.3 | 25.9 | 18.4 | 53.4 | 20.6 | 40.7 |
| ULD | Optimal transport | 7.1 | 3.3 | 33.4 | 19.7 | 57.4 | 24.2 | 44.8 |
| ACTD (Ours) | Anchor + Residual | 7.1 | 5.4 | 35.6 | 24.8 | 61.4 | 26.9 | 48.8 |
| Multi-Teacher Distillation (T1+T2+T3 \(\to\) S) | ||||||||
| SeqKD | Mixed output CoT | 4.6 | 3.3 | 27.8 | 17.3 | 51.6 | 20.9 | 44.3 |
| Multi-ULD | Multi-teacher ULD | 4.6 | 5.0 | 30.6 | 21.6 | 58.8 | 24.1 | 42.1 |
| Multi-ACTD (Ours) | Loss aggregation | 10.0 | 6.3 | 40.9 | 24.0 | 63.6 | 29.0 | 51.1 |
In cross-architecture generalization experiments (DS-R1-Distill-Qwen-7B \(\to\) Llama-3.2-3B-Instruct), ACTD scores 18.4 AVG@8, outperforming DSKD (15.8), ALM (15.2), and ULD (12.1). Distilling from GLM-4-9B-Chat \(\to\) Qwen3-1.7B-Base, ACTD achieves 26.0 AVG@8 and 47.2 PASS@8, exceeding all baselines. On code generation (HumanEval+ Pass@1 distilling DS-R1-Distill-Llama-8B \(\to\) Qwen3-1.7B-Base), ACTD obtains 23.2, beating ULD (21.3) and the base model (20.7).
Ablation Study¶
Ablation investigations over top-\(k\) size, residual loss weight \(\lambda_{\text{res}}\), and prefix truncation under the T1 \(\to\) S distillation setting provide clear empirical insights:
| Configuration / Variable | Detailed Setting | AIME24 | AIME25 | AMC23 | Minerva | MATH500 | Mean AVG@8 | Observations |
|---|---|---|---|---|---|---|---|---|
| Top-\(k\) Size \(k\) | \(k=64\) | 8.3 | 5.0 | 34.7 | 22.4 | 62.8 | 26.6 | Under-captures teacher support mass |
| (T1 \(\to\) S setting) | \(k=96\) | 9.2 | 5.4 | 35.9 | 23.0 | 60.1 | 26.7 | Incremental improvements |
| \(k=128\) (Default) | 9.2 | 5.0 | 40.0 | 22.8 | 62.0 | 27.8 | Optimal trade-off between coverage and noise | |
| \(k=256\) | 8.3 | 5.8 | 36.9 | 22.0 | 60.0 | 26.6 | Long-tail noise introduces negative transfer | |
| Full vocabulary | 9.2 | 5.8 | 35.9 | 20.8 | 58.0 | 25.9 | Global projection noise causes 1.9 pt drop | |
| Residual Weight \(\lambda_{\text{res}}\) | \(\lambda_{\text{res}} = 0\) (w/o residual) | - | - | - | - | - | 26.1 (PASS@8 49.5) | Unconstrained non-anchor mass inflates entropy |
| (T1 \(\to\) S setting) | \(\lambda_{\text{res}} = 0.5\) | - | - | - | - | - | 26.3 (PASS@8 50.0) | Insufficient suppression strength |
| \(\lambda_{\text{res}} = 1.0\) (Default) | - | - | - | - | - | 27.8 (PASS@8 50.7) | Best balance between anchor fit and entropy control | |
| \(\lambda_{\text{res}} = 2.0\) | - | - | - | - | - | 26.6 (PASS@8 47.6) | Excessive penalty dampens top-k signal transfer | |
| Prefix Truncation | DS-Qwen-7B w/o truncation | - | - | - | - | - | 26.1 (vs 26.9) | Full reasoning traces degrade performance by 0.8 |
| DS-Llama-8B w/o truncation | - | - | - | - | - | 27.1 (vs 27.8) | Full reasoning traces degrade performance by 0.7 | |
| DS-Qwen3-8B w/o truncation | - | - | - | - | - | 25.5 (vs 27.1) | Long CoT teacher suffers severe degradation (-1.6) |
Key Findings¶
- Larger top-\(k\) is counterproductive: Expanding the supervision support from \(k=128\) to \(k=256\) and full vocabulary degrades AVG@8 from 27.8 to 26.6 and 25.9. This directly confirms the core thesis: in cross-tokenizer alignment, long-tail tokens carry disproportionate mapping noise rather than useful pedagogical signal.
- Residual regularization prevents mode-averaging and lowers generation entropy: Visualizing probability distributions and output token entropies demonstrates that students trained with \(\lambda_{\text{res}}=1.0\) exhibit consistently lower entropy across inference decoding steps. The L1 penalty prevents probability mass dispersion over non-target tokens, maintaining sharp decision bounds.
- Multi-teacher distillation yields positive synergy only when projection noise is tamed: Multi-ACTD pushes AVG@8 to 29.0, whereas Multi-ULD (24.1) drops below its best single-teacher baseline (24.2), and SeqKD stagnates at 20.9. Suppressing residual projection noise is the prerequisite for multi-teacher knowledge fusion across heterogeneous tokenizers.
Highlights & Insights¶
- Lightweight deterministic alignment beats heavy cross-attention: Instead of training complex cross-model attention layers or running costly Sinkhorn iterations, first-subtoken mapping and monotonic character boundary matching provide a robust, compute-free geometric bridge.
- L1 residual regularization as an anti-smoothing mechanism: Departing from conventional KL divergence, applying an L1 penalty directly against the aggregate residual probability mass prevents mode-averaging and sharpens model predictions without needing reverse-KL instability.
- Unlocks cross-family multi-teacher knowledge distillation: By decoupling offline logit caching and suppressing independent projection noise, Multi-ACTD enables practitioners to harvest collective reasoning strengths from distinct open-weight models (e.g., DeepSeek, Qwen, GLM, and LLaMA) into a single compact model.
Limitations & Future Work¶
- Requirement for teacher logit access: ACTD operates under white-box/grey-box assumptions requiring the teacher's Top-\(k\) logits, precluding its application to API-only commercial models (such as GPT-4o or Claude 3.5 Sonnet).
- Static subtoken mapping edge cases: In heavily agglutinative languages or complex domain-specific token combinations, first-subtoken matching can occasionally introduce boundary fragmentation, pointing to contextual subtoken mapping as a potential future refinement.
- Multi-teacher logit storage overhead: Generating and caching Top-128 logits for 30,000 reasoning traces across multiple large teacher models requires non-trivial offline GPU compute and disk capacity, which scales with teacher parameter count.
Related Work & Insights¶
- vs Universal Logit Distillation (ULD): ULD relies on optimal transport to solve an entropic Gromov-Wasserstein or Earth Mover's problem across vocabularies, which is computationally heavy and leaks noise across low-probability bins; ACTD achieves superior reasoning transfer via sparse Top-\(k\) anchors and an L1 residual penalty at negligible overhead.
- vs Dual-Space Knowledge Distillation (DSKD): DSKD bridges token spaces using cross-model attention on hidden states, adding trainable parameters and memory bottlenecks; ACTD functions purely at the output distribution level, keeping student fine-tuning clean and naturally supporting multi-teacher expansion.
- vs Approximate Likelihood Matching (ALM): ALM approximates sequence likelihood across different tokenizations, yet exhibits significant performance drops on complex mathematical reasoning; ACTD utilizes monotonic character boundary matching and reasoning prefix truncation, outperforming ALM by up to 6.2 points on AVG@8.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Pinpoints the core bottleneck of long-tail projection noise in cross-tokenizer distillation and resolves it with a clean anchor plus L1 residual formulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across five reasoning benchmarks, multiple teacher-student pairs, multi-teacher settings, code generation, and entropy/distribution analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Thorough motivation, elegant mathematical definitions, clean empirical validation, and intuitive design comparisons.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, robust paradigm for cross-family LLM distillation and heterogeneous multi-teacher knowledge integration.