Predict, Don’t Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Hsu1023/PILL
Area: LLM Reasoning
Keywords: diffusion language models, infilling, adaptive length, bidirectional attention, multi-slot parallel decoding
TL;DR¶
Addressing the sensitivity to initial length presets and substantial inference overhead in diffusion language model infilling, PILL freezes the backbone to directly predict target lengths via a single probing pass, decodes candidate spans in parallel using slot-wise attention and RoPE interpolation, and identifies the best span via single-pass causal scoring, requiring only two extra forward passes to outperform top baselines while running 1.82x faster.
Background & Motivation¶
Diffusion Language Models (DLMs) have emerged as a compelling alternative to the autoregressive paradigm, leveraging bidirectional attention and any-order generation to iteratively denoise sequences. In text and code infilling tasks where a middle span must be synthesized conditioned on both a given prefix and suffix, autoregressive models are structurally constrained by causal attention and can only ingest right-side context via Fill-in-the-Middle (FIM) prompt rearrangements. In contrast, DLMs natively condition on bidirectional context simultaneously, making them theoretically ideal for boundary-constrained text generation.
However, standard DLMs require the number of masked positions to be predetermined and fixed prior to decoding, whereas the ground-truth infilling length is unknown a priori and exhibits massive variance across diverse instances. Crucially, generation quality is exquisitely sensitive to this predetermined length: in code infilling, adding or omitting even a single token such as an indentation or variable delimiter frequently causes syntax invalidity or execution failure. Existing adaptive-length strategies attempt to overcome this rigidity but face severe drawbacks. Iterative search methods like CAL rely on early denoising confidence to update length estimates, becoming easily trapped in local minima near a preset initial length while introducing dozens of extra forward passes; other approaches insert discrete length-modifying operations that require costly and capability-compromising backbone fine-tuning.
This core tension between length sensitivity and the prohibitive inference cost of iterative search severely limits practical deployment. The authors break the conventional mindset of exploring lengths through iterative trial-and-error, proposing instead to decouple length determination from generation via direct prediction and parallel evaluation. Core idea: keep the DLM backbone frozen, probe target lengths directly from the bidirectional hidden state of an inserted mask token, decode a compact radius of length candidates in parallel via slot-wise attention and positional interpolation, and perform post-hoc selection in a single causal scoring pass—achieving preset-free adaptive infilling with only two extra forward passes.
Method¶
Overall Architecture¶
PILL (Probing-based InfiLling with preset-Length-free decoding) decomposes the adaptive infilling task into three lightweight, decoupled stages: Length Probing (Stage I), Multi-slot Parallel Decoding (Stage II), and Post-hoc Selection (Stage III). The entire pipeline strictly freezes the pretrained DLM backbone and incurs exactly two additional forward passes across inference (one for length estimation, one for candidate scoring). In Stage I, a single mask token inserted between the prefix and suffix aggregates full bidirectional context, allowing a lightweight 3-layer MLP to directly predict a central target length \(\hat{L}\) without any preset initialization. In Stage II, the predicted length is expanded into a local candidate set, and all candidates are denoised simultaneously in a packed sequence using a slot-wise attention mask and continuous position ID interpolation to prevent inter-slot interference and boundary gaps. In Stage III, candidates and their trailing context are scored in a single forward pass via a dual visible-probe causal attention mechanism, selecting the span that optimizes both internal fluency and suffix alignment.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prefix P and Suffix S"] --> B["Bidirectional Context Length Probing<br/>Single mask token + MLP predicts central length"]
B --> C["Slot-wise Parallel Decoding with Position Interpolation<br/>Expand local candidate radius and reuse KV context"]
C --> D["Single-Pass Joint Causal Post-Hoc Selection<br/>Dual-block mask scoring balances fluency and suffix fit"]
D --> E["Output Optimal Adaptive Infilled Span"]
Key Designs¶
1. Bidirectional Context Length Probing: Eliminating Preset Length Sensitivity via Single-Pass Regression Prior adaptive infilling frameworks require an initial length \(L_0\) to begin iterative search, making them highly vulnerable to poor initializations that trap the search trajectory in sub-optimal local minima. PILL exploits the fact that DLMs utilize full bidirectional attention, meaning the hidden state of an intermediate token naturally pools information across both the preceding prefix and following suffix. Given prefix \(P\) and suffix \(S\), the model constructs a probing sequence \(x_{\text{probe}} = P \oplus [\text{M}] \oplus S\) and executes a single forward pass of the frozen DLM. To enrich the representation against localized noise, the probe input vector concatenates the mean-pooled states of the last four prefix tokens, the mask token state, and the mean-pooled states of the first four suffix tokens: \(h = [\text{MeanPool}(h_{p-4:p-1}); h_{[\text{M}]}; \text{MeanPool}(h_{s:s+3})] \in \mathbb{R}^{3d}\). A lightweight 3-layer MLP regressor \(f_{\phi}(\cdot)\) then maps \(h\) to the target length in log-space: $\(\hat{L} = \lfloor \exp(f_{\phi}(h)) \rceil\)$ where \(\lfloor \cdot \rceil\) denotes rounding to the nearest positive integer. Formulating the regression target as \(\log L\) rather than raw \(L\) dampens the disruptive gradients of long-tailed sequence length distributions and strictly enforces positivity. This design determines an accurate length anchor in a single forward evaluation without requiring iterative search passes or hyperparameterized initial length guesses.
2. Slot-wise Parallel Decoding with Position Interpolation: Resolving Candidate Cost Explosion and Boundary Misalignment Because DLM generation is acutely sensitive to span lengths, relying exclusively on a single point prediction \(\hat{L}\) leaves the system susceptible to discrete rounding errors. PILL expands the prediction into a local candidate radius \(r\), yielding \(N = 2r + 1\) candidate lengths \(\{\hat{L}-r, \dots, \hat{L}+r\}\). To avoid the prohibitive \(N\)-fold inference overhead of separate generation passes, PILL packs all candidate slots alongside a shared context into a unified sequence: \(P \oplus [\text{M}]^{\hat{L}} \oplus S \oplus [S_1] \dots [S_N]\). An isolated slot-wise attention mask is enforced such that each candidate slot \(S_j\) attends exclusively to its own tokens and the shared prefix/suffix context, but cannot attend to any other slot \(S_{j'} (j' \ne j)\). This completely eliminates cross-candidate leakage and enables all candidates to be denoised simultaneously within a single decoding loop while fully reusing the static KV cache of the shared context. Crucially, when assigning position IDs to candidate slots of varying lengths \(\ell_j\), standard left-padding or right-padding leaves an unassigned position gap between the candidate tokens and one boundary of the context. In RoPE-based DLMs, the model misinterprets this structural gap as remaining unfilled space and under-generates content. PILL resolves this by permanently anchoring the suffix start at position \(p_S = p_P + \hat{L} + 1\) and linearly interpolating the position IDs for each candidate slot \(S_j\): $\(\text{pos}(S_j, k) = (p_P + 1) + (k - 1) \cdot \frac{\hat{L} - 1}{\ell_j - 1}\)$ This continuous mapping guarantees that every candidate spans the exact same interval \([p_P + 1, p_S - 1]\) regardless of its token count \(\ell_j\). It removes all discrete boundary gaps without requiring fine-tuning, preserving the native generation fidelity of the backbone while decoding \(N\) candidates at the latency cost of a single slot.
3. Single-Pass Joint Causal Post-Hoc Selection: Enabling Non-Autoregressive Likelihood Scoring in One Forward Pass Autoregressive language models can straightforwardly score candidate sequences by summing left-to-right next-token log-probabilities. In contrast, DLMs are trained on masked reconstruction, so the output logits at already-committed, unmasked token positions do not represent predictive distributions and cannot quantify sequence likelihood. Calculating pseudo-log-likelihood by masking tokens individually would require an impractical number of forward passes. PILL overcomes this limitation by designing an attention-masked scoring block that evaluates all \(N\) candidates in a single extra forward pass. For candidate \(C_j\) and a selected sparse suffix subset \(\tilde{S}\), the input is structured as a visible block \(v = C_j \oplus \tilde{S}\) aligned with a parallel block of scoring mask probes \(\mu = (\mu_1, \dots, \mu_{\ell_j+m})\) sharing identical position IDs. The global attention mask restricts visible tokens \(v_k\) to causal visibility (attending to prefix \(P\) and \(v_{\le k}\)), while scoring probes \(\mu_k\) strictly attend to \(P\) and strictly preceding tokens \(v_{< k}\) without self-visibility or cross-probe visibility. Reading the distribution at \(\mu_k\) yields the exact conditional log-likelihood \(\log p_{\theta}(v_k \mid P, v_{< k})\) with zero future label leakage. To balance internal generation fluency against seamless transition into the downstream context, the selection metric computes a convex combination of an inner score \(s_{\text{in}}^j\) and a suffix continuation score \(s_{\text{suf}}^j\): $\(j^{\star} = \arg\max_{j} \left[ \alpha \cdot \frac{1}{\ell_j} \sum_{k=1}^{\ell_j} \log p_{\theta}(c_k \mid P, c_{<k}) + (1 - \alpha) \cdot \frac{1}{m} \sum_{t=1}^{m} \log p_{\theta}(\tilde{s}_t \mid P, C_j, s_{<t}) \right]\)$ The sparse suffix \(\tilde{S}\) incorporates both transition tokens (the first few suffix tokens) and high-attention tokens (suffix tokens receiving the strongest cross-attention during Stage II denoising), providing a compact \(m \in [4, 8]\) window that captures both local coherence and long-range syntactic dependencies in a single forward computation.
Loss & Training¶
The DLM backbone remains completely frozen throughout training and inference. The only trainable component is the lightweight 3-layer MLP length probe in Stage I (hidden dimensions 512 and 128, GeLU activation, dropout rate 0.1). Probes are trained via self-supervised objectives on auxiliary corpora consisting of 196k samples per backbone, derived from public code repositories (Py150, LeetCode, CodeContests) and cleaned web text (C4). In each sample, a contiguous segment is randomly masked to form the supervisory target length, with the surviving text serving as prefix and suffix. Training minimizes Mean Squared Error (MSE) loss against target \(\log L\) using AdamW (learning rate \(1 \times 10^{-3}\), weight decay \(1 \times 10^{-4}\), batch size 16) for up to 50 epochs. Training a probe on a single NVIDIA A40 GPU requires approximately 100 minutes per model family, representing a one-time amortized offline setup cost that generalizes across all downstream code and text infilling benchmarks without task-specific adaptation.
Key Experimental Results¶
Main Results¶
PILL is comprehensively evaluated across five prominent DLM backbones (LLaDA-8B-Base, LLaDA-8B-Instruct, LLaDA-MoE-Base, Dream-7B-Base, Dream-Coder-7B-Base) on eight code and natural language benchmarks. Baselines include vanilla fixed-length decoding (Backbone), dynamic canvas expansion (DAEDAL), and iterative confidence search (CAL), all evaluated under a prior-free protocol averaging over initial lengths \(L \in \{4, 8, 16, 32\}\). Code tasks report Pass@1 functional correctness; text benchmarks report BLEU-2 / ROUGE-L.
| Backbone Model | Method | Human-S (Pass@1) | MBPP-S (Pass@1) | Java (Pass@1) | C/C++ (Pass@1) | WikiText (BLEU-2/ROUGE) | ArXiv (BLEU-2/ROUGE) |
|---|---|---|---|---|---|---|---|
| LLaDA-8B-Base | Backbone (Avg L) | 44.68 | 37.45 | 34.09 | 34.84 | 19.90 / 34.48 | 13.23 / 30.30 |
| DAEDAL (Avg L) | 44.99 | 38.86 | 35.34 | 36.03 | 10.82 / 20.78 | 6.65 / 16.61 | |
| CAL (Avg L) | 64.74 | 54.49 | 50.31 | 49.22 | 23.00 / 37.25 | 15.69 / 32.69 | |
| PILL (Ours) | 71.35 | 57.49 | 57.10 | 54.57 | 29.29 / 44.96 | 20.28 / 39.82 | |
| PILL (Oracle) | 81.90 | 71.56 | 60.98 | 60.08 | 41.51 / 56.76 | 29.35 / 50.43 | |
| LLaDA-8B-Instruct | Backbone (Avg L) | 49.73 | 37.47 | 32.42 | 33.40 | 18.07 / 32.77 | 12.57 / 29.44 |
| CAL (Avg L) | 69.77 | 48.64 | 49.20 | 48.66 | 20.52 / 34.93 | 14.77 / 31.33 | |
| PILL (Ours) | 67.86 | 52.27 | 52.66 | 52.35 | 25.58 / 41.91 | 18.94 / 38.13 | |
| Dream-7B-Base | Backbone (Avg L) | 42.79 | 40.12 | 33.90 | 34.30 | 19.12 / 33.75 | 13.56 / 30.97 |
| CAL (Avg L) | 64.48 | 56.00 | 47.92 | 48.00 | 21.68 / 36.68 | 15.98 / 33.52 | |
| PILL (Ours) | 68.34 | 57.97 | 58.98 | 56.30 | 32.36 / 48.13 | 22.89 / 42.79 | |
| Dream-Coder-7B | Backbone (Avg L) | 49.64 | 43.94 | 36.89 | 37.66 | - | - |
| CAL (Avg L) | 68.56 | 61.35 | 51.14 | 49.28 | - | - | |
| PILL (Ours) | 71.35 | 62.59 | 59.87 | 59.01 | - | - |
Inference efficiency profiling on HumanEval-S with LLaDA-8B-Base demonstrates the extreme efficiency of the two-pass probing and scoring design:
| Method | Pass@1 (%) | Extra Forward Passes | Wall-Clock Time (s) | Overhead vs. Backbone |
|---|---|---|---|---|
| Backbone (Avg L) | 44.67 | - | 1.10 s | +0% |
| DAEDAL | 44.80 | 11.98 | 2.29 s | +117% |
| CAL | 64.74 | 13.79 | 2.13 s | +94% |
| PILL (Ours) | 71.35 | 2.00 | 1.18 s | +7% |
Ablation Study¶
The impact of key structural choices in Stage II (position ID assignments) and Stage III (post-hoc selection objectives) on HumanEval-S using LLaDA-8B-Base (\(r=2\)):
| Ablation Dimension | Configuration | Pass@1 (%) | Oracle Pass@1 (%) | Note / Mechanism |
|---|---|---|---|---|
| Position-ID Assignment | Left-padding | 28.2 | 73.2 | Prefix aligned; leaves gap before suffix causing suffix under-generation |
| Right-padding | 31.6 | 74.2 | Suffix aligned; leaves gap after prefix causing initial token truncation | |
| Independent single-slot | 52.9 | 83.6 | Candidates decoded in separate sequences (theoretical upper bound) | |
| Interpolation (Ours) | 51.6 | 81.9 | Eliminates gaps across all slots, closely matching independent decoding | |
| Post-hoc Selection | Random candidate | 51.62 | 81.90 | Valid candidate present in pool, but random choice fails |
| Predicted length only | 66.02 | 66.02 | Evaluates center candidate \(\hat{L}\) without multi-slot selection | |
| Inner-only score (\(\alpha = 1\)) | 69.02 | 81.90 | Rewards internal fluency while neglecting downstream suffix coherence | |
| Suffix-only score (\(\alpha = 0\)) | 63.99 | 81.90 | Overfits to immediate suffix transition, degrading span syntax | |
| Joint Scoring (\(\alpha = 0.5\)) | 71.35 | 81.90 | Balances internal consistency and continuation alignment |
Key Findings¶
- Breakdown of Per-Stage Overhead: Compared to a controlled single-slot decoding baseline at matched length \(\hat{L}\) (0.967s), length probing adds 0.086s (+8.9%), multi-slot parallel generation adds only 0.033s (+3.4%), and causal post-hoc scoring adds 0.091s (+9.4%), totaling a modest 21.7% overhead. Generating all \(2r+1\) candidates concurrently via slot-wise attention introduces negligible extra latency.
- Robustness Against Initialization Traps: When varying the initial preset length from \(L=4\) to \(L=32\) on HumanEval-S, CAL's prediction MAE deteriorates from 3.56 to 24.13, while Acc@1 plummets from 0.58 to 0.02. In contrast, PILL requires no initial length, achieving the lowest overall MAE (2.50) with an Acc@5 of 86.35%, proving that the candidate window \(\{\hat{L}-2, \dots, \hat{L}+2\}\) reliably encompasses the target length.
- Hyperparameter Saturation: Increasing candidate radius beyond \(r=2\) (5 candidates) plateaus final Pass@1 (71.35 at \(r=2\) vs. 70.1 at \(r=4\)) despite raising the oracle upper bound (from 81.9 to 84.0), as larger candidate pools introduce selection noise. Similarly, suffix scoring length peaks at \(m=4\), confirming that the continuation signal is highly concentrated at the boundary.
Highlights & Insights¶
- Predicting Rather than Searching: Decoupling the problem of determining span length from content generation eliminates the traditional necessity of iterative confidence sweeps, transforming a multi-pass search into a single feed-forward probing operation.
- Rotary Position Interpolation for Multi-Slot Packing: In modern RoPE architectures, position gaps induce severe token under-generation; continuous linear interpolation over a unified coordinate span elegantly aligns multiple distinct lengths to an anchored suffix without modifying positional embeddings.
- Causal Mask Probes for Non-Autoregressive Models: Constructing parallel visible-probe token blocks inside a global attention mask allows bidirectional diffusion architectures to compute true non-leaking causal log-probabilities in a single forward evaluation, resolving the long-standing absence of likelihood scoring in DLMs.
Limitations & Future Work¶
- Model Scale and Architecture Boundaries: Evaluated primarily on 7B/8B parameter models on single-GPU hardware; validating probe accuracy and selection efficacy on massive trillion-parameter DLMs or deeply recurrent MoE structures remains open.
- Complex Multi-Span and Long-Context Infilling: While preliminary results show promise on disjoint multi-span synthesis, complex nested infilling, cross-file repository-level code patching, and hierarchical code repair require more sophisticated multi-probe mechanisms.
- Uncertainty-Aware Adaptive Budgeting: Preliminary exploration reveals that probe prediction variance correlates with ground-truth span length (\(r=0.32\)); implementing dynamic candidate radius adjustment based on prediction uncertainty presents a viable path for optimizing compute allocation.
Related Work & Insights¶
- vs. CAL (Liu et al., 2026): CAL estimates span length by probing confidence across early denoising steps, incurring approximately 13.8 extra forward passes per sample and remaining trapped by initial length choices; PILL requires no preset length, uses only 2 extra passes, and executes 1.82x faster while achieving superior accuracy.
- vs. DAEDAL (Li et al., 2025a): DAEDAL expands generation canvases dynamically by appending tokens to the sequence tail, but cannot enforce right-hand boundary constraints required for true infilling, resulting in severe degradation on text benchmarks; PILL natively conditions on both prefix and suffix throughout.
- vs. DreamOn (Wu et al., 2026): DreamOn achieves variable-length infilling by injecting explicit length-changing operations during diffusion, which requires full backbone fine-tuning; PILL keeps the pretrained backbone entirely frozen, preserving general-purpose capabilities and transferring seamlessly across domains.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Decouples length determination from generation via bidirectional probing, coupled with continuous RoPE interpolation and single-pass causal probe scoring.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 5 DLMs, 8 benchmarks, comprehensive per-stage runtime profiling, error mode analysis, and structural complexity ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative structure, clear mathematical formulations, and highly intuitive conceptual illustrations.
- Value: ⭐⭐⭐⭐⭐ Resolves the foundational efficiency bottleneck of diffusion language models in infilling tasks, offering a practical, training-free deployment framework.