Length-Adaptive Decoding for Masked Diffusion Machine Translation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Entropy-Valley/Entropy-Valley
Area: Multilingual & Machine Translation
Keywords: Masked Diffusion Models, Machine Translation, Canvas Length Adaptation, All-Mask Predictive Entropy, Minimum-Entropy Decoding
TL;DR¶
Addressing the structural defect in fixed-canvas masked diffusion machine translation where target canvas length must be pre-allocated before denoising (causing truncation or hallucination), this paper introduces Entropy-Valley (EV), a training-free test-time length selector that scores candidate canvas lengths by mean predictive entropy across all-mask forward passes, closing 33.0%–65.3% of the COMET-22 gap to reference lengths across En↔Zh and En→De while matching or exceeding matched-data autoregressive LLaMA-3-8B.
Background & Motivation¶
Masked diffusion language models (dLLMs) such as LLaDA and Dream have recently achieved performance competitive with autoregressive (AR) LLMs on open-ended text generation and mathematical reasoning. However, machine translation (MT) poses a fundamentally different and unforgiving challenge: a faithful translation system must completely preserve source semantic units and actions, accurately reorder syntactic constituents across divergent languages, select appropriate target morphology, and terminate precisely at the sentence boundary. While standard autoregressive decoders inherently determine sequence length dynamically by emitting the end-of-sequence token <EOS> when translation is complete, fixed-canvas masked diffusion decoders require the exact target length to be allocated before any denoising steps begin. Because supervised training places <EOS> directly after the target canvas, the model inevitably attempts to populate whatever canvas size it receives at test time; an undersized canvas truncates critical content, whereas an oversized canvas creates spurious empty slots that induce repetitions or hallucinations.
Existing research on masked iterative generation and diffusion decoding has almost exclusively focused on the unmasking order schedule—exploring whether confidence-based reveals, learned policies, or task-specific heuristics optimize generation quality. Consequently, the prerequisite decision of selecting target canvas length has remained largely overlooked despite its decisive impact on translation adequacy. Standard academic evaluations circumvent this issue either by taking the ground-truth reference length as an oracle upper bound (which constitutes an unachievable evaluation setup rather than a deployable inference method) or by applying a static corpus-level source-to-target length ratio derived from training statistics. The latter rigidly forces a single compression rule across diverse sentences, leading to catastrophic omissions when translating sentences containing entity placeholders, numerical expressions, short imperatives, or idiomatic phrases. Adapting classic non-autoregressive translation (NAT) techniques—such as dedicated length predictor modules, fertility models, or edit-based Levenshtein architectures—requires redesigning model backbones and retraining with specialized auxiliary objectives, making them unsuitable for plug-and-play inference on frozen diffusion foundational models.
To overcome this dilemma, this paper formulates length selection as an internal uncertainty query directly posed to the frozen backbone itself. On an all-mask target canvas, each candidate length establishes a distinct predictive probability distribution across all target positions. When a canvas is too short, the model is forced to over-compress source semantics into insufficient slots, triggering sharp token conflicts and elevated predictive uncertainty. Conversely, when a canvas is overly long, the model must guess unsupported tokens for ungrounded positions, yielding diffuse and uncertain probability distributions. Only when the canvas length naturally aligns with the semantic capacity of the source sentence does the model exhibit sharp, confident token predictions across positions, forming a pronounced local minimum in average predictive entropy. Core idea: Introduce Entropy-Valley (EV), an entirely training-free length selector that evaluates a compact grid of candidate canvas lengths via single all-mask forward passes, chooses the canvas exhibiting minimum mean predictive entropy (the "entropy valley") where the backbone is most prepared to denoise, and decodes the selected canvas via standard minimum-entropy decoding.
Method¶
Overall Architecture¶
Entropy-Valley introduces a lightweight pre-decoding length selection stage for frozen masked diffusion translation systems. Given a source sentence prompt and a pre-configured language-specific ratio grid, the inference procedure executes four decoupled steps: first, candidate canvas lengths are generated based on source tokenized length and deduplicated; second, an all-mask sequence is constructed for each candidate length and evaluated via a single forward pass through the frozen backbone to compute its mean predictive entropy; third, the canvas length achieving the global minimum mean entropy is selected as the optimal "entropy valley"; fourth, standard 32-step minimum-entropy decoding (MED) is executed exclusively on this selected canvas, truncating the decoded sequence at the first emitted <EOS>.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Source Prompt<br/>prompt(x)"] --> B["Candidate Canvas Grid Construction"]
B --> C["All-Mask Mean Predictive Entropy Probing"]
C --> D["Single-Canvas Minimum-Entropy Decoding"]
D --> E["Target Translation Output<br/>Truncated at First EOS"]
Key Designs¶
1. Candidate Canvas Grid Construction: Avoiding Global Ratio Distortion and Unbounded Search Overhead via Prior Grids
A uniform global ratio fails to capture sentence-specific compression demands, while searching unconstrained length spaces wastes massive forward-pass compute and introduces linguistically absurd hypotheses. This design defines a compact direction-specific candidate ratio set \(\mathcal{R}\) consisting of 5 ratios centered around the WMT19 training corpus median (En→Zh median 0.80, grid \(\{0.70, 0.75, 0.80, 0.85, 0.90\}\); Zh→En median 1.24, grid \(\{1.00, 1.10, 1.20, 1.30, 1.40\}\); En→De median 1.48, grid \(\{1.50, 1.60, 1.70, 1.80, 1.90\}\)). For a source sentence with tokenized length \(|x|\), candidate canvas lengths are calculated as:
$\(\mathcal{C}(x) = \{\max(1, \lfloor r \cdot |x| \rfloor) + 1 : r \in \mathcal{R}\}\)$
where the trailing \(+1\) explicitly reserves the designated target <EOS> slot following the supervised training format. For short source inputs, distinct ratio floats frequently map to identical floor integers; duplicates are eliminated, resulting in fewer than five distinct forward passes in practice. This guarantees a bounded, plausible candidate space tailored to the source sentence length.
2. All-Mask Mean Predictive Entropy Probing: Scoring Unmasking Preparedness Directly from Frozen Backbone Uncertainty
Once candidate canvases are established, the key challenge is scoring candidate suitability without executing the multi-step denoising loop or training external scoring models. For each candidate length \(L \in \mathcal{C}(x)\), the model constructs an input sequence concatenating the source prompt with \(L\) mask tokens: \([\text{prompt}(x)] \parallel [\text{MASK}]^L\). A single forward pass through the frozen model yields token probability distributions \(q_1^L, \dots, q_L^L\). Because the terminal slot is pre-allocated for <EOS>, its predictive entropy naturally drops close to zero and would artificially bias longer canvases; the algorithm therefore excludes the terminal position and computes the mean predictive entropy across the remaining \(L-1\) target positions:
$\(\bar{H}(L) = \frac{1}{L - 1} \sum_{i=1}^{L-1} H(p_\theta(y_i \mid x, [\text{MASK}]^L))\)$
where \(H(p) = -\sum_{v \in \mathcal{V}} p(v) \log p(v)\). The canvas length minimizing mean entropy is selected: \(L^\star = \arg\min_{L \in \mathcal{C}(x)} \bar{H}(L)\). This formulation directly captures the backbone's internal confidence; mismatched lengths inflate entropy, whereas the entropy valley reflects the canvas geometry the model can denoise with maximal certainty.
3. Single-Canvas Minimum-Entropy Decoding: Preventing Decoupled Multi-Candidate Reranking Overhead
A naive decode-and-rerank strategy would require generating complete translations across multiple candidate canvases before selecting one by sequence log-probability, multiplying inference compute (empirical evaluations show that decoding five candidate lengths costs \(\sim 3.5\times\) compute and actually degrades translation quality due to length normalization biases). In contrast, EV finalizes the global canvas selection at step zero and performs standard iterative denoising exclusively on the single selected canvas \(L^\star\). Using standard minimum-entropy decoding (MED) over \(T=32\) steps, the decoder iteratively unmasks the \(K_t\) lowest-entropy positions per step and truncates at the first emitted <EOS>. Probing at most 5 candidate lengths adds at most 5 forward passes to the baseline's 32 denoising steps (an overhead of \(\le 15.6\%\)), preserving high inference throughput without multi-candidate memory bloat.
A Worked Example¶
The paper highlights concrete translation cases on WMT22 En→Zh that substantiate how EV restores missing source content: 1. Preserving Action Verbs: For the imperative sentence "Tap Reset Now." (ground-truth reference length \(L^*=6\), "现在点击重置。"), the fixed ratio baseline selects \(L=5\), which forces the model to drop the action verb and collapse into "现在。" (COMET-22 of 0.47). Entropy-Valley identifies the entropy valley at \(L=6\) and produces "点击重置。" (COMET-22 of 0.91, \(\Delta=+0.44\)). 2. Preserving Named-Entity Placeholders: For "Under #PRS_ORG#, tap Sign out." (reference length \(L^*=12\), "在#PRS_ORG#, 点击退出。"), the ratio baseline allocates \(L=10\) and drops the placeholder ("在私人组织下按退出。", COMET-22 of 0.43). EV allocates \(L=12\) based on entropy probing, preserving the literal tag exactly: "在#PRS_ORG#, 按下退出。" (COMET-22 of 0.87, \(\Delta=+0.44\)).
Loss & Training¶
The underlying masked diffusion system is built on LLaDA-8B-Base (an 8.02B parameter discrete diffusion Transformer) fine-tuned on 200k WMT19 sentence pairs per translation direction using LoRA. LoRA adapts 7 linear projection modules per Transformer block (\(\{q, k, v, o\}\) in multi-head attention and \(\{ff\_proj, up\_proj, ff\_out\}\) in the FFN) with rank \(r=64\), scaling \(\alpha=128\), and dropout 0.05, totaling \(\sim 157\text{M}\) trainable parameters (\(1.95\%\) of the backbone). Optimization uses AdamW with peak learning rate \(2\times 10^{-4}\), cosine decay with 5% warm-up, and a uniform masking schedule over 3 epochs. Crucially, Entropy-Valley requires zero additional training parameters, zero fine-tuning on development sets, and zero modifications to the underlying denoising weights, operating purely as an inference-time decoding strategy.
Key Experimental Results¶
Main Results¶
On the WMT22 News Translation benchmark (\(N=2,037\) per direction) using 32-step MED decoding across three independent training runs, Entropy-Valley (EV) is evaluated against the reference length oracle upper bound (Length Oracle) and the corpus-level fixed ratio baseline (Fixed Ratio):
| Direction | Method | COMET-22 (Mean ± Std) | COMET Gap Closure | sacreBLEU (Mean ± Std) | BLEU Gap Closure |
|---|---|---|---|---|---|
| En→Zh | Oracle length (Upper Bound) | 0.8610 ± 0.0013 | 100% | 40.81 ± 0.23 | 100% |
| Fixed ratio 0.8 (Baseline) | 0.8345 ± 0.0017 | 0% (baseline) | 36.72 ± 0.22 | 0% | |
| Entropy-Valley (Ours) | 0.8517 ± 0.0006 | 64.9 ± 7.4% | 38.57 ± 0.13 | 45.3% | |
| Zh→En | Oracle length (Upper Bound) | 0.8519 ± 0.0007 | 100% | 27.93 ± 0.23 | 100% |
| Fixed ratio 1.2 (Baseline) | 0.8266 ± 0.0010 | 0% (baseline) | 23.65 ± 0.21 | 0% | |
| Entropy-Valley (Ours) | 0.8431 ± 0.0004 | 65.3 ± 0.8% | 25.28 ± 0.33 | 38.1% | |
| En→De | Oracle length (Upper Bound) | 0.7382 ± 0.0090 | 100% | 22.55 ± 0.77 | 100% |
| Fixed ratio 1.8 (Baseline) | 0.7170 ± 0.0090 | 0% (baseline) | 20.73 ± 0.68 | 0% | |
| Entropy-Valley (Ours) | 0.7240 ± 0.0078 | 33.0 ± 8.4% | 21.55 ± 0.85 | 45.1% |
Ablation & Diagnostic Studies¶
The following table reports literal source content retention rates (placeholders, numbers) and translation quality across compression ratio buckets on WMT22 En→Zh (\(N=2,037\)):
| Diagnostic Category | Sample Count N | Fixed Ratio Baseline | Entropy-Valley (EV) | Difference \(\Delta\) (EV − Ratio) | 95% Bootstrap CI |
|---|---|---|---|---|---|
| Placeholder Retention (#PRS_ORG#) | 58 | 66.9% | 89.1% | +22.1 pp | [+12.4, +32.5] |
| Number / Decimal / Currency Retention | 290 | 77.6% | 81.3% | +3.7 pp | [+1.4, +6.1] |
| High-Expansion Bucket (r ≥ 0.8 COMET) | 1005 | 0.8056 | 0.8402 | +0.0346 | [+0.0295, +0.0401] |
| Mid-Compression Bucket (0.6 ≤ r < 0.8 COMET) | 796 | 0.8630 | 0.8678 | +0.0048 | [+0.0016, +0.0081] |
| High-Compression Bucket (r < 0.6 COMET) | 236 | 0.8449 | 0.8503 | +0.0054 | [+0.0016, +0.0093] |
Furthermore, in matched-data calibration using LLaMA-3-8B + LoRA-SFT trained on identical data, the autoregressive baseline achieves COMET-22 scores of \(85.07 \pm 0.03\) (En→Zh), \(83.68 \pm 0.07\) (Zh→En), and \(81.19 \pm 0.11\) (En→De). LLaDA-8B with EV essentially ties the AR baseline on En→Zh (85.17 vs. 85.07) and outperforms it on Zh→En (84.31 vs. 83.68).
Key Findings¶
- Canvas length selection dominates unmasking reveal order: Controlled experiments with reference lengths fixed show that varying source-guided reveal schedules spans only 0.0081 COMET-22. In contrast, under a fixed MED schedule, varying canvas length selection spans 0.0265 COMET-22 (3.3× larger variation). This establishes that length selection, not unmasking order, is the primary performance bottleneck in fixed-canvas diffusion MT.
- EV selects denoising-friendly lengths rather than human reference lengths: EV's length mean absolute error (MAE) against reference translations is 2.51 tokens (vs. 0.00 for Oracle), yet it recovers \(\sim 65\%\) of the COMET-22 Oracle gap. A canvas optimal for diffusion denoising does not have to match human surface length exactly.
- Drastic reduction in literal omission: On En→Zh, EV improves placeholder preservation from 66.9% to 89.1% (+22.1 percentage points) and numerical retention from 77.6% to 81.3%, directly remedying the truncation failures caused by short static canvases.
- Cross-backbone validation: Across Dream-v0-Base-7B (Qwen2.5 initialization) and DiffuLLaMA-7B (Llama-2 initialization), EV consistently outperforms Fixed Ratio across all directions, closing 54%–69% of the Oracle gap on Dream-Base.
Highlights & Insights¶
- Inverting an external decoding hurdle into internal model uncertainty: Rather than treating sequence length as an external variable that must be modeled by auxiliary regressors, the authors leverage the natural convexity of all-mask predictive entropy to let the model self-select its easiest canvas at zero parameter cost.
- Minimal inference overhead with zero training footprint: EV avoids expensive multi-candidate full decoding pipelines (which inflate compute by \(3.5\times\)); by probing up to 5 candidates with a single forward pass, it introduces only \(\le 15.6\%\) forward-pass overhead at \(T=32\).
- Reframing non-autoregressive generation priorities: The paper offers an essential empirical corrective to recent trends that over-index on intricate token unmasking orders, proving that pre-allocation of canvas geometry accounts for the vast majority of translation quality variance.
Limitations & Future Work¶
- Constraint of a fixed candidate ratio grid: EV selects lengths from a predefined 5-ratio set. When an atypical sentence requires extreme compression outside this range (e.g., \(r < 0.60\) while the En→Zh grid lowest bound is \(0.70\)), EV overshoots the target length, causing degradation relative to lucky ratio baselines. Future work should explore entropy-guided adaptive expansion of candidate bounds.
- Backbone-dependent cross-lingual performance ceilings: On En→De, EV achieves only 33.0% gap closure, and the diffusion model substantially lags behind the matched AR LLaMA-3-8B (72.40 vs. 81.19), a shortfall that persists even under Oracle lengths (73.82). This confirms that German translation quality is bottlenecked by the base pre-training distribution rather than canvas selection errors alone.
- Human evaluation scope restricted to bilingual En↔Zh: Professional human evaluation was conducted exclusively on English-Chinese translation; broader human studies across German, French, and low-resource language pairs remain to be investigated.
Related Work & Insights¶
- vs. Classical Non-Autoregressive Translation (NAT / CMLM / Levenshtein Transformer): Classical NAT architectures require training auxiliary length predictors, maintaining small length beams, or adopting complex token insertion/deletion mechanisms. EV operates entirely post-hoc at inference time on frozen decoder-only diffusion models without auxiliary heads or architectural modifications.
- vs. Variable-Length Diffusion Models (DAEDAL / CAL / \(\rho\)-EOS): DAEDAL dynamically inserts mask tokens during iterative denoising, leading to high slot latency (En→Zh: 1175 slots, 1.80s); CAL calibrates confidence after the first full denoising step. EV probes at step 0 via all-mask entropy, achieving superior COMET scores over DAEDAL (+0.0166) and CAL (+0.0068) on En→Zh while operating at lower latency (0.72s).
- vs. Token Reveal Order Scheduling (MaskGIT / OeMDM / LogicDiff): Reveal order methods optimize which positions to unmask first given an allocated canvas. EV addresses the orthogonal upstream question of determining canvas size. Controlled experiments show that canvas choice exerts more than \(3\times\) the impact on translation quality compared to order scheduling.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Clever and elegant insight showing that all-mask predictive entropy exposes an intrinsic "valley" for target sequence length selection without extra training]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 3 language pairs, 3 diffusion backbones, matched-data AR baselines, reveal order diagnostics, and professional human evaluations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous problem formulation, transparent analysis of failure modes and boundaries, and exceptionally clear prose]
- Value: ⭐⭐⭐⭐⭐ [Provides an essential, highly practical, training-free decoding recipe for deploying discrete diffusion language models to constrained sequence generation tasks]