Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/josephattieh/green_kd
Area: Model Compression
Keywords: Knowledge Distillation, Carbon Accounting, Life Cycle Assessment, Machine Translation, Green AI
TL;DR¶
By incorporating the Machine Learning Life Cycle Assessment (MLCA) framework into neural machine translation distillation, this paper decomposes carbon footprints across teacher training, distillation overhead, and student inference—accounting for both operational and amortized embodied hardware emissions—revealing that knowledge distillation is not inherently green, as its break-even deployment volume is highly serving-dependent and shifts by multiple orders of magnitude under batched inference.
Background & Motivation¶
Knowledge distillation (KD) has long been established as a cornerstone methodology for compressing oversized neural models into compact, efficient students. In neural machine translation (NMT) and broader language modeling, smaller student architectures are widely celebrated for reducing per-request latency, lowering active power draw, and unlocking edge deployment. Consequently, the research community often takes for granted that distilling large models into smaller ones is fundamentally an environmentally sound, "green" endeavor. However, traditional efficiency evaluations examine inference gains in isolation or juxtapose test-set translation quality metrics (such as COMET or BLEU) against runtime speedups, entirely brushing aside the substantial computational energy and carbon expenditures sunk during the distillation pipeline itself. Teacher-side beam decoding across multi-million sentence parallel corpora, iterative forward passes over teacher logit distributions, and complex student training routines collectively accumulate a non-negligible carbon debt before a single user query is ever served.
The core tension of this paradigm lies in the distinct economic nature of the carbon expenditures: the inference benefits of a student model accumulate as a marginal per-token saving across downstream requests, whereas the production phase—comprising allocated teacher training, synthetic target decoding, logit extraction, and student training—acts as a fixed, irreversible carbon investment. Prior evaluations typically lack a rigorous life cycle accounting framework that unifies both operational electricity consumption and the amortized embodied manufacturing emissions of underlying hardware infrastructure. Without factoring in actual downstream request volume (served volume), applying distillation naively in low-throughput regimes can end up emitting drastically more carbon than simply deploying the uncompressed baseline or teacher model.
To establish whether and when distillation delivers genuine environmental advantages, this paper incorporates the Machine Learning Life Cycle Assessment (MLCA) framework into machine translation distillation pipelines. Core idea: frame the environmental viability of knowledge distillation as a quality-constrained amortization problem, decomposing life cycle emissions across teacher training, distillation overhead, and downstream inference to empirically determine the break-even deployment volumes across distillation regimes and serving batch sizes.
Method¶
Overall Architecture¶
The paper formulates an end-to-end carbon accounting and evaluation protocol tailored to the complete life cycle of machine translation systems. The system boundary is compute-centric, rigorously set on the data-center server side to encompass three active compute phases: model training (\(\text{TRAIN}\)), distillation preprocessing and student optimization (\(\text{DISTILL}\)), and online query inference (\(\text{INFER}\)). In terms of environmental scope, the methodology quantifies both operational emissions (electricity consumed by GPU hardware scaled by data center Power Usage Effectiveness and local electricity grid emission intensity) and embodied manufacturing emissions (the cradle-to-gate hardware carbon footprint linearly amortized across device expected lifetime and active utilization rates).
The overarching assessment pipeline connects four foundational components: first, an explicit Functional Unit (FU) is formalized to anchor environmental costs to serving \(X\) decoded target tokens under a specified translation quality threshold; second, phase-level carbon equations decouple emissions into fixed production footprints and marginal per-token inference slopes; third, structural cost trade-offs are analyzed between sequence-level distillation (Seq-KD, Seq-Inter, Seq-Rep) and word-level distribution matching (Word-KD, Sel-KD, TIE-KD); finally, mathematical break-even deployment volumes (\(X^\star\)) are derived relative to teacher and compact baseline systems, uncovering the dramatic impact of inference batching on student amortization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["MT Task & Deployment Volume Spec<br/>Define Functional Unit & Quality Target"] --> B["Phase-Level MLCA Decomposition<br/>Quantify Operational & Embodied Emissions"]
B --> C["KD Paradigm Life Cycle Costing<br/>Sequence Synthetic Decoding vs Word Logit Matching"]
C --> D["Amortization & Break-Even Modeling<br/>Fixed Production Intercept & Batched Inference Slopes"]
D --> E["Pareto-Optimal Model Selection<br/>Workload-Aware Green Deployment Decision"]
Key Designs¶
1. Phase-Level MLCA Carbon Accounting: Bridging the Blind Spots of Embodied and Infrastructure Footprints
Conventional green AI tracking predominantly records direct GPU energy during execution, missing data-center overheads and hardware manufacturing emissions. Adopting the MLCA framework, the total carbon footprint \(I^{(\phi)}_e\) (in \(\text{kgCO}_2\text{e}\)) for each compute phase \(\phi \in \{\text{TRAIN}, \text{DISTILL}, \text{INFER}\}\) on computing device \(e\) is formally defined as the sum of operational use emissions and amortized embodied manufacturing emissions: $\(I^{(\phi)}_e = \text{PUE} \cdot E^{(\phi)}_e \cdot \text{EGM}_g + F_e^{(\text{prod})} \cdot \frac{t^{(\phi)}_e}{t_{\text{life},e} \cdot \text{AUR}_e}\)$ where \(E^{(\phi)}_e\) denotes measured phase electrical consumption in \(\text{kWh}\), \(\text{PUE}\) accounts for data-center cooling and power infrastructure overhead (fixed at 1.24), \(\text{EGM}_g\) represents the regional grid carbon intensity (\(0.033\text{ kgCO}_2\text{e/kWh}\) for the Puhti supercomputer in Finland), \(F_e^{(\text{prod})}\) is the embodied manufacturing carbon footprint (\(150\text{ kgCO}_2\text{e}\) per NVIDIA V100 GPU), \(t^{(\phi)}_e\) represents phase runtime, \(t_{\text{life},e}\) is hardware lifetime (5 years), and \(\text{AUR}_e\) is the active utilization rate (0.8 during training/distillation, 0.2 during inference). This unified formulation standardizes energy consumption and runtime into comprehensive carbon equivalents across all experimental phases.
2. KD Paradigm Life Cycle Costing: Dissecting Production Footprint Divergence Across Families
The study systematically breaks down how computational expenditures materialize across the two dominant MT distillation families. Sequence-level KD (Seq-KD) operates by having the teacher autoregressively decode high-probability synthetic targets across the entire parallel training corpus via beam search (\(B=5\)), constructing synthetic parallel data \(\hat{\mathcal{D}} = \{(x^{(i)}, \hat{y}^{(i)})\}_{i=1}^N\). The student subsequently trains via standard cross-entropy: $\(\mathcal{L}_{\text{CE}}(\theta_S; \hat{\mathcal{D}}) = \sum_{(x, \hat{y}) \in \hat{\mathcal{D}}} \sum_{t=1}^{|\hat{y}|} \mathcal{H}(\delta_{\hat{y}_t}, p_{\theta_S})\)$ The primary distillation overhead in Seq-KD is heavily front-loaded into teacher synthetic generation, scaling linearly with dataset token count; multi-candidate filtering strategies like Seq-Inter and Seq-Rep further inflate production footprints due to candidate evaluation and scoring. Conversely, word-level KD (Word-KD) aligns token prediction distributions at every timestep via Kullback-Leibler divergence: $\(\mathcal{L}_{\text{KD}}(\theta_S; \theta_T) = \sum_{(x, y) \in \mathcal{D}} \sum_{t=1}^{|y|} D_{\text{KL}}(p_{\theta_T} \parallel p_{\theta_S})\)$ In online settings, Word-KD demands synchronized teacher forward passes at every optimization step, substantially inflating student training duration; in offline settings, it requires extracting and caching large arrays of Top-K logits before student training. This structural breakdown exposes the disparate life cycle cost dynamics between synthetic target generation and distribution extraction.
3. Amortization and Break-Even Modeling: Modeling Fixed Intercepts and Batched Inference Slopes
To quantify when distillation achieves net environmental savings, the total life cycle footprint of a deployed system \(m\) is expressed as an affine function of served token volume \(X\): \(I_m(X) = I_m^{\text{prod}} + X \cdot c_m^{\text{infer}}\), where \(I_m^{\text{prod}}\) represents the fixed production footprint (incorporating teacher training allocation fraction \(\lambda_T \in [0, 1]\) and all distillation preprocessing/training emissions) and \(c_m^{\text{infer}}\) represents the marginal per-token inference carbon slope. For a distilled student \(S\) compared against a baseline teacher \(T\), the break-even volume \(X^\star\) is solved analytically: $\(X^\star = \frac{I_S^{\text{prod}} - I_T^{\text{prod}}}{c_T^{\text{infer}} - c_S^{\text{infer}}}\)$ Because a non-distilled compact baseline (No-KD) shares an identical student architecture and inference slope (\(c_S^{\text{infer}} = c_{\text{No-KD}}^{\text{infer}}\)), no deployment volume break-even exists between them; choosing KD over No-KD is strictly quality-constrained by whether the added production footprint yields acceptable COMET gains. Crucially, the inference slope \(c^{\text{infer}}\) is highly sensitive to the serving environment. Under batched serving, hardware saturation dramatically improves for both teacher and student models, causing the difference in inference slopes \((c_T^{\text{infer}} - c_S^{\text{infer}})\) to compress sharply, which in turn pushes the break-even volume \(X^\star\) outward by several orders of magnitude.
Loss & Training¶
In the bespoke encoder-decoder experiments, the teacher model is Transformer-Big (209M parameters), and students are Transformer-Base (65M parameters, compression ratio 3.21) and Transformer-Tiny (16M parameters, compression ratio 13.06). Word-KD optimizes an interpolated objective: $\(\mathcal{L}_{\text{WORD-KD}}(\theta_S; \theta_T) = (1 - \alpha) \mathcal{L}_{\text{CE}}(\theta_S) + \alpha \mathcal{L}_{\text{KD}}(\theta_S; \theta_T)\)$ Training uses the Adam optimizer with an inverse square root learning rate schedule on a single NVIDIA V100 GPU to maintain strict hardware control. In the decoder-only LLM setup, Qwen3-8B and Qwen3-4B act as teachers distilling into Qwen3-0.6B via LoRA adaptation (\(r=32, \alpha=64\)), utilizing offline quantized Top-20 logit distribution caching and a sunk teacher-cost allocation (\(\lambda_T = 0\)).
Key Experimental Results¶
Main Results¶
Evaluations are conducted on WMT 2024 constrained-track parallel data across three language pairs with distinct scales (EN→IS: 23M sentence pairs; EN→HI: 13M sentence pairs; EN→ES: 6M sentence pairs). Translation quality is evaluated on FLORES+ devtest using COMET (Unbabel/wmt22-comet-da).
| Model / KD Config | Language Direction | Params | COMET | Production Footprint (kgCO2e) | Unbatched Break-Even (M tokens) |
|---|---|---|---|---|---|
| Teacher (TF-Big) | EN→IS (23M) | 209M | 83.21 | 1.83 | Reference Baseline (0.0) |
| No-KD Baseline | EN→IS (23M) | 65M | 81.25 | 0.49 | None (Equal Slope) |
| Word-KD | EN→IS (23M) | 65M | 83.05 | 2.65 | 1.002* |
| TIE-KD | EN→IS (23M) | 65M | 83.18 | 2.38 | 0.671* |
| Seq-KD | EN→IS (23M) | 65M | 82.50 | 3.03 | 1.468 |
| Seq-Inter | EN→IS (23M) | 65M | 82.80 | 2.57 | 0.907 |
| No-KD Baseline | EN→IS (23M) | 16M | 74.32 | 0.16 | None (Equal Slope) |
| Word-KD | EN→IS (23M) | 16M | 76.54 | 2.33 | 0.439* |
| Seq-Inter | EN→IS (23M) | 16M | 77.20 | 2.50 | 0.595* |
| Seq-Rep | EN→IS (23M) | 16M | 76.85 | 2.27 | 0.382 |
(Note: Asterisks denote configurations lying on the global COMET vs. production-footprint Pareto frontier; break-even is computed against the teacher under unbatched sentence-level inference.)
In the decoder-only LLM distillation experiment (Qwen3 family, sunk teacher cost \(\lambda_T = 0\), 100K sampled pairs), translation quality and carbon emissions are summarized as follows:
| Teacher Model | Direction | Method | Student COMET | \(\Delta\text{Base}\) | KD Train (gCO2e) | Distilled Data (gCO2e) | Break-Even (M tokens) |
|---|---|---|---|---|---|---|---|
| Qwen3-8B | EN→ES | Seq-KD | 77.88 | +1.82 | 18.76 | 12.34 | 10.45 |
| Qwen3-8B | EN→ES | Word-KD | 80.76 | +4.70 | 91.97 | 1047.58 | 13.19 |
| Qwen3-8B | EN→IS | Seq-KD | 48.18 | +19.88 | 48.96 | 10.45 | 8.65 |
| Qwen3-8B | EN→IS | Word-KD | 47.99 | +19.69 | 79.53 | 854.22 | 10.81 |
| Qwen3-4B | EN→ES | Seq-KD | 78.02 | +1.96 | 17.31 | 11.05 | 8.65 |
| Qwen3-4B | EN→ES | Word-KD | 80.06 | +4.00 | 97.48 | 983.10 | 11.94 |
| Qwen3-4B | EN→IS | Seq-KD | 38.99 | +10.69 | 25.73 | 8.65 | 7.30 |
| Qwen3-4B | EN→IS | Word-KD | 46.36 | +18.06 | 80.92 | 756.72 | 9.26 |
Batched Serving Ablation Study¶
To quantify how inference batching shifts the amortization threshold against the teacher model, break-even volumes are measured across batch sizes on EN→IS:
| Batch Size | 16M Seq-Inter | 16M Word-KD | 65M TIE-KD | 65M Word-KD |
|---|---|---|---|---|
| Batch = 256 | 386.97M | 285.07M | 733.37M | 1.09B |
| Batch = 1024 | 973.63M | 717.23M | 2.34B | 3.49B |
| Batch = 4096 | 1.28B | 942.02M | 1.38B | 2.06B |
| Batch = 16000 | 1.60B | 1.18B | 2.51B | 3.75B |
Key Findings¶
- Batching shifts break-even volume by up to three orders of magnitude: While single-sentence unbatched inference amortizes fixed distillation costs within hundreds of thousands or a few million tokens, high-throughput batched serving (Batch 256 to 16,000) drives the break-even threshold into hundreds of millions or billions of tokens (e.g., 65M Word-KD shifts from 1.0M tokens to 3.75B tokens).
- Compression depth dictates optimal KD paradigm: Under moderate compression (65M student), Word-KD variants consistently dominate the Pareto frontier, matching teacher COMET without incurring expensive full-corpus beam generation. Under aggressive compression (16M student), capacity bottlenecks emerge and Seq-KD variants become competitive because synthetic targets simplify training, particularly on smaller datasets where teacher decoding costs are lower.
- Offline LLM Word-KD incurs massive data extraction overhead: In the Qwen3 case study, offline extraction and storage of Top-20 distributions generated 756 to 1047 \(\text{gCO}_2\text{e}\) in preprocessing emissions—nearly an order of magnitude higher than the entire LoRA student training phase (17 to 97 \(\text{gCO}_2\text{e}\))—rendering offline Word-KD substantially more carbon-intensive despite modest COMET gains.
Highlights & Insights¶
- Reframes green AI model compression from narrow inference latency speedups into an end-to-end life cycle amortization problem governed by deployment volume.
- Uncovers the paradoxical effect of batched inference: as batch size increases, hardware utilization saturates across all models, narrowing the relative per-token efficiency gap between teacher and student and severely delaying environmental amortization.
- Establishes a rigorous multi-objective Pareto frontier decision procedure, underscoring that distillation should always be benchmarked against an un-distilled compact baseline (No-KD) to prevent unjustified carbon expenditures.
Limitations & Future Work¶
- Experimental profiling was conducted on a single NVIDIA V100 GPU under a low-carbon Finnish electricity grid; cross-hardware profiles (e.g., Hopper architecture, edge NPUs) and regions with high fossil fuel grid intensities warrant broader investigation.
- Quality assessment relies primarily on COMET scores; qualitative dimensions such as translation hallucination, lexical consistency, and human preference in low-resource settings remain unexamined.
- Upstream data filtering for Seq-KD must be investigated: filtering is environmentally beneficial only if it curtails the amount of teacher generation required; post-hoc cleaning after full corpus decoding fails to alleviate the dominant generation footprint.
Related Work & Insights¶
- vs Traditional KD in MT (e.g., Kim and Rush, 2016; Zhang et al., 2023): Conventional distillation papers assume smaller students are inherently more cost-effective and focus exclusively on translation quality and inference latency; this paper introduces the first MLCA-based accounting pipeline demonstrating that distillation increases net emissions in low-volume scenarios.
- vs Standard Carbon Tracking Tools (e.g., CodeCarbon, Carbontracker): Prior tooling primarily monitors operational training energy or counts theoretical FLOPs; this paper demonstrates that equal FLOPs do not translate to equal emissions and highlights the necessity of factoring in amortized hardware embodied impacts and data center PUE.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first end-to-end life cycle assessment and break-even amortization framework for knowledge distillation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Spans custom encoder-decoder MT and modern decoder-only LLMs across varying dataset sizes, compression levels, and inference batching configurations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical definitions, and compelling empirical evidence directly addressing real-world deployment trade-offs.
- Value: ⭐⭐⭐⭐⭐ Provides indispensable actionable guidance and environmental accountability standards for researchers and practitioners deploying compressed models.