Skip to content

Training-Free Character-Length Control in Summarization with Diffusion Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/DFKI-NLP/delta-diffusion
Area: Text Generation
Keywords: diffusion language models, length control, text summarization, inference-time decoding, character budget

TL;DR

This paper proposes DELTA (Diffusion Expected Length Targeting Algorithm), an inference-time training-free method for masked diffusion language models (such as LLaDA and Dream) that dynamically inserts and deletes mask tokens based on continuous expected character-length estimation, reducing character-length error by an order of magnitude across XSum, CNN/DailyMail, and arXiv while maintaining competitive summary quality.

Background & Motivation

In production workflows such as news headline publishing, mobile push notifications, social media previews, video subtitling, and space-constrained user interfaces, generated text summaries must strictly comply with predetermined character-length limits. Even an overshoot of a handful of characters can cause disruptive layout truncation, UI overflow, or broken visual templates. Existing approaches for length-controlled summarization are largely rooted in the autoregressive framework and face severe trade-offs: training-based methods (such as CTRLsum or EOS token loss weighting) require expensive task-specific fine-tuning or architectural alterations, whereas zero-shot prompting techniques (such as countdown prompts, statistical word-to-character ratio conversions, and post-hoc rejection filtering) achieve only loose, approximate compliance that rapidly degrades on longer targets.

The root cause of this failure lies in the unidirectional causal factorization of autoregressive generation: once an autoregressive model emits an initial sequence of tokens, it cannot directly backtrack, revise, or edit earlier decisions within the same decoding pass, making real-time correction of cumulative character drift impossible. In stark contrast, discrete masked diffusion language models (such as LLaDA and Dream) operate by iteratively refining an entire canvas buffer of masked tokens over multiple bidirectional denoising steps. Because these models are naturally trained to produce coherent text across variable sequence lengths, injecting or pruning mask tokens intermittently during the denoising process does not catastrophically disrupt generative fluency or factual coherence.

DELTA capitalizes on this structural affordance of masked diffusion models, transforming the non-differentiable discrete character count constraint into a dynamic tracking objective over continuous token probability distributions. Core idea: at each denoising step, estimate the total expected character length by combining exact counts from committed tokens with the probability-weighted character expectations of remaining mask positions, and dynamically insert or remove mask tokens at low-confidence locations to guide generation toward the exact character budget without any fine-tuning.

Method

Overall Architecture

DELTA functions entirely within the iterative denoising loop of an off-the-shelf masked diffusion language model. Given an input conditioning prompt \(p\), a target character budget \(L^*\), and an initial sequence buffer size \(m\) derived from dataset-level character-per-token ratios, the model initializes a fully masked sequence canvas \(x = [p; \text{[MASK]}^m; \text{[EOT]}]\). During each denoising step, the model computes token probabilities and prediction confidences via a bidirectional forward pass. The algorithm then coordinates three core operations: computing the overall expected character length, dynamically adjusting the sequence buffer via insertion or removal when expected length deviates from \(L^*\), and committing tokens at high-confidence positions where confidence exceeds a designated threshold \(\tau\). By interleaving buffer modifications strictly with commitment steps, the algorithm eliminates cyclic oscillation and runs until all mask tokens are completely resolved.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input prompt p and target budget L*"] --> B["Initialize sequence canvas<br/>x = [p; MASK^m; EOT]"]
    B --> C["Model forward pass<br/>Compute distribution q_i and confidence c_i"]
    C --> D["Expected character length estimation<br/>Committed lengths + mask expectations L_hat"]
    D --> E{"Does L_hat match L*?<br/>(and adjustment enabled)"}
    E -->|L_hat > L*| F["Low-confidence mask pruning<br/>Remove lowest-confidence masks and lock"]
    E -->|L_hat < L*| G["Low-confidence mask insertion<br/>Insert masks at lowest-confidence slots and lock"]
    E -->|Matched or adjusted| H["Confidence-threshold unmasking<br/>Commit tokens where c_i > tau and unlock"]
    F --> H
    G --> H
    H --> I{"Do MASK tokens remain?"}
    I -->|Yes| C
    I -->|No| J["Return final summary x satisfying character budget"]

Key Designs

1. Expected character length estimation: bridging committed certainty and predictive distributions

Whereas autoregressive decoders cannot anticipate the character footprint of future tokens, masked diffusion models compute predictive distributions across all vocabulary tokens at every unmasked slot simultaneously. DELTA exploits a character-length mapping function \(\ell: V \to \mathbb{N}_0\), which maps each vocabulary token to its decoded string length (assigning length 0 to special tokens such as [MASK], [PAD], and [EOT]). The expected total character length \(\hat{L}\) across sequence canvas length \(m\) is explicitly decomposed into committed deterministic lengths and expected probabilistic lengths:

\[\hat{L} = \sum_{i=1}^{m} \hat{\ell}_i, \quad \text{where} \ \hat{\ell}_i = \begin{cases} \ell(x_i) & \text{if } x_i \neq \text{[MASK]} \\ \sum_{v \in V} q_{i,v} \cdot \ell(v) & \text{if } x_i = \text{[MASK]} \end{cases}\]

Special tokens with zero decoded character length are masked out from the logits prior to evaluating \(q_i\), ensuring that degenerate or padding tokens do not distort the estimation. This allows the system to accurately evaluate whether the planned sequence overshoots or undershoots the target budget right from step 1โ€”even when the canvas contains only mask tokens.

2. Dynamic buffer adjustment: targeted pruning and expansion at low-confidence sites

When the estimated length \(\hat{L}\) deviates from the target \(L^*\), DELTA intervenes precisely where the model exhibits the least semantic certainty, preserving firmly established lexical structures. If \(\hat{L} > L^*\) (the output is projected to be too long), candidate mask positions are sorted in ascending order of prediction confidence \(c_i = \max_{v \in V} q_{i,v}\) (with permutation \(\sigma\)). The algorithm greedily eliminates \(n_{\text{remove}}\) mask positions to minimize the remaining discrepancy:

\[n_{\text{remove}} = \arg\min_k \left| \hat{L} - \sum_{j=1}^{k} \hat{\ell}_{\sigma(j)} - L^* \right|\]

Crucially, mask removal is strictly restricted to positions where confidence falls below a safety threshold \(\tau_{\text{low}}\), preventing the premature deletion of positions for which the model has already formed confident predictions. Conversely, if \(\hat{L} < L^*\) (the output is projected to be too short), new [MASK] tokens are inserted adjacent to the lowest-confidence masked positions. The number of inserted tokens is calculated as \(\lfloor (L^* - \hat{L}) / \bar{\ell} \rfloor\), where \(\bar{\ell}\) denotes the current average expected character length per mask token.

3. Damped alternating commitment: provable anti-oscillation mechanism

Unconstrained buffer modifications risk entering infinite ping-pong oscillationsโ€”repeatedly alternating between adding and deleting mask tokens at adjacent thresholds without advancing generation. DELTA enforces an alternating state machine controlled by a boolean latch can_adjust. Buffer adjustments are permitted only immediately following a token commitment step. Once an insertion or deletion occurs, can_adjust is immediately set to false, mandating that the subsequent iteration execute a token unmasking step (committing tokens where \(c_i > \tau\), with at least the single highest-confidence token guaranteed to commit). Because each commitment step permanently unmasks at least one position, the underlying state undergoes a strictly monotonic transition, providing a mathematical guarantee of convergence in no more than \(\lfloor L^* / 2 \rfloor\) steps.

Loss & Training

DELTA is a purely inference-time, training-free decoding algorithm that requires zero parameter fine-tuning, no auxiliary reward models, and no specialized sequence loss functions. It is deployed off-the-shelf on pretrained and instruction-tuned discrete diffusion language models: LLaDA-8B-Instruct and Dream-7B-Instruct. Hyperparameters were tuned via grid search on validation sets: on CNN/DailyMail and XSum, \((\tau, \tau_{\text{low}})\) is set to \((0.65, 0.95)\) for Dream and \((0.65, 0.90)\) for LLaDA; on arXiv, Dream uses \((0.90, 0.82)\) and LLaDA uses \((0.71, 0.71)\). Initial buffer sizes \(m\) are dynamically scaled according to the reference character-per-token density of the validation split.

Key Experimental Results

Main Results

Evaluations were performed across three benchmark datasets spanning varied length regimes: XSum (extreme single-sentence compression, average reference length 125 characters), CNN/DailyMail (multi-sentence news summaries, average reference length 312 characters), and arXiv (long-document scientific abstracts, average reference length 969 characters). Below are the main comparative results against strong autoregressive baselines and fine-tuned models (G-Eval scores use GPT-5-mini on a 1โ€“5 scale):

Dataset Method Length Error (MAE) Compliance (<10 chars, %) ROUGE-1 ROUGE-L BERTScore-F1 G-Eval Con G-Eval Coh
XSum DELTA+LLaDA 2.3 99.6 0.286 0.217 0.882 4.21 4.61
DELTA+Dream 2.3 99.7 0.333 0.255 0.890 4.14 4.76
LA-SF-AR (Llama-3.1-8B) 5.2 84.2 0.273 0.204 0.881 4.38 4.79
GPT-4o-mini 13.1 48.4 0.264 0.194 0.880 4.73 4.96
Lead-char-snap 15.7 60.5 0.166 0.122 0.850 4.72 3.08
TextRank 14.5 50.1 0.178 0.130 0.845 4.79 4.07
CNN/DM DELTA+LLaDA 2.4 99.6 0.363 0.259 0.876 4.59 4.58
DELTA+Dream 3.0 99.3 0.409 0.281 0.880 4.72 4.75
LA-SF-AR (Llama-3.1-8B) 10.3 59.7 0.399 0.254 0.879 4.61 4.94
GPT-4o-mini 96.6 11.1 0.393 0.245 0.877 4.86 4.99
BART-CNN (Fine-tuned) 104.7 5.7 0.441 0.306 0.882 4.83 4.60
Lead-char-snap 59.4 21.4 0.363 0.230 0.868 4.92 4.22
TextRank 17.3 32.4 0.356 0.235 0.866 4.80 3.38
arXiv DELTA+LLaDA 2.8 98.4 0.443 0.234 0.837 4.35 4.43
DELTA+Dream 3.1 98.3 0.431 0.236 0.853 4.42 3.60
LA-SF-AR (Llama-3.1-8B) 44.9 22.9 0.466 0.250 0.845 4.28 4.78
GPT-4o-mini 112.2 3.7 0.432 0.217 0.839 4.65 4.97
Lead-char-snap 96.5 7.0 0.351 0.183 0.837 4.99 4.11
TextRank 46.9 12.0 0.406 0.212 0.837 4.55 3.00

Ablation Study

To demonstrate that length compliance stems specifically from the dynamic buffer adjustment rather than the initial buffer heuristic of diffusion models, an ablation was conducted comparing DELTA to a fixed-buffer variant (disabling mask insertion and removal while retaining identical models, prompts, confidence schedules, and initialization). Comparison was also made against DAEDAL, an existing training-free variable-length diffusion decoding method (G-Eval scores use GPT-4o-mini):

Dataset Config / Method MAE (chars) Compliance (<10 chars, %) ROUGE-1 ROUGE-L BERTScore-F1 Time (s) Note
XSum DELTA+LLaDA (full model) 2.30 99.6% 0.286 0.217 0.882 0.69 Full adaptive insertion & removal
Fixed-buffer LLaDA 15.55 38.4% 0.285 0.218 0.883 0.68 Removing adjustment inflates error 7x
DAEDAL (LLaDA) 74.50 7.7% 0.251 0.196 0.845 2.50 Unconstrained variable decoding misses budget
CNN/DM DELTA+LLaDA (full model) 2.40 99.6% 0.363 0.259 0.876 8.59 Full adaptive insertion & removal
Fixed-buffer LLaDA 27.35 23.9% 0.423 0.282 0.878 8.50 Disabling dynamic edits drops compliance to 23.9%
Fixed-buffer Dream 26.20 24.6% 0.433 0.288 0.881 8.52 Dream exhibits identical length degradation
DAEDAL (LLaDA) 192.30 3.4% 0.367 0.248 0.858 5.94 Fails character budget compliance
arXiv DELTA+LLaDA (full model) 2.80 98.4% 0.443 0.234 0.837 22.80 Exceptional precision on long targets
Fixed-buffer LLaDA 113.94 4.4% 0.448 0.235 0.838 22.10 Over 100 character MAE without adjustment
Fixed-buffer Dream 92.31 8.1% 0.432 0.238 0.854 20.60 Massive drift from cumulative token variances
DAEDAL (LLaDA, 100-sample diag.) 6337.50 1.0% 0.106 0.073 0.796 306.56 Severe divergence and intractable latency

Key Findings

  • Zero drift scaling across long contexts: For autoregressive models, length deviation scales catastrophically with target length (GPT-4o-mini's MAE explodes from 13.1 on XSum to 112.2 on arXiv, with compliance collapsing to 3.7%). In contrast, DELTA maintains an MAE of 2.8โ€“3.1 characters and >98% compliance across all domains, completely eliminating target-length drift.
  • Dynamic buffer edits are the critical driver: In the ablation experiments, fixed-buffer setups achieve only 4.4%โ€“38.4% compliance. Activating DELTA's dynamic insertion and deletion surges compliance to 98.3%โ€“99.7% while preserving equivalent ROUGE and BERTScore values, confirming that quality is uncompromised.
  • Blind human evaluation validates linguistic quality: A double-blind human evaluation on 50 CNN/DailyMail articles (two annotators across relevance, consistency, coherence, and fluency with 91.5% inter-annotator agreement within one point) ranked DELTA+Dream (4.27) on par with LA-SF-AR (4.32) and GPT-4o-mini (4.31), proving that stringent character budgets do not induce awkward phrasing or unnatural truncation.

Highlights & Insights

  • Exploiting bidirectional non-autoregressive affordances: Instead of battling the causal limitations of left-to-right generation, the method leverages the inherent flexibility of masked diffusion models, demonstrating that non-autoregressive architectures can unlock powerful control capabilities previously deemed unattainable.
  • Continuous probabilistic feedback for discrete constraints: By integrating full vocabulary probability distributions over remaining mask tokens into an expected character sum, the algorithm establishes smooth, informative signals early in decoding before commitments are cast in stone.
  • Robust anti-oscillation through alternating state control: The simple yet mathematically grounded can_adjust latching mechanism prevents cycling between mask insertions and deletions, ensuring bounded inference runtimes without costly backtracking buffers.

Limitations & Future Work

  • Inference latency overhead: Because masked diffusion models require bidirectional full-sequence forward passes at every denoising iteration and cannot utilize standard causal KV caching, inference is computationally heavier (requiring 20.75โ€“22.80s on arXiv compared to 13.22s for LA-SF-AR). Integrating emerging accelerated diffusion kernels such as FlashDLM is a promising next step.
  • Cross-lingual tokenizer sensitivity: The current evaluation is restricted to English. In morphologically rich languages or logographic scripts (e.g., Chinese, Japanese), tokenizer token-to-character distributions exhibit higher entropy and variance, requiring further empirical validation.
  • Dataset-dependent hyperparameter thresholds: The commitment threshold \(\tau\) and safe-removal bound \(\tau_{\text{low}}\) were tuned via grid search on validation sets; developing adaptive, dynamic threshold schedules would improve out-of-the-box domain generalization.
  • vs LA-SF-AR (Retkowski and Waibel, 2025): LA-SF-AR relies on heuristic word-to-character multipliers and iterative post-hoc sample rejection over autoregressive LLMs. DELTA adjusts sequence dimensions dynamically within a single diffusion generation trajectory, avoiding multi-pass rejection overhead and maintaining an order of magnitude lower character error.
  • vs DAEDAL (Li et al., 2026) & DreamOn (Wu et al., 2026): DAEDAL addresses open-ended variable-length generation and struggles to enforce explicit budgets, while DreamOn introduces custom expand/delete tokens that mandate expensive supervised fine-tuning. DELTA provides exact, training-free targeting compatible with off-the-shelf checkpoints.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering adaptation of discrete masked diffusion dynamics for fine-grained character-level budget control without retraining]
  • Experimental Thoroughness: โญโญโญโญโญ [Spans three diverse benchmark datasets, thorough ablations against fixed-buffer setups, automated LLM judging, and blind human evaluation]
  • Writing Quality: โญโญโญโญโญ [Clear mathematical formulations, cohesive algorithmic exposition, and thorough ablation reporting]
  • Value: โญโญโญโญโญ [Offers an immediately deployable, zero-training solution for exact length-constrained natural language generation in space-critical production interfaces]