Discourse Dependency: A Continuous Criterion for Translation Difficulty¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/trotacodigos/ddp.git
Area: Multilingual & Machine Translation
Keywords: Discourse Dependency (DDP), Translation Difficulty, Document-Level MT, Automatic Post-Editing, Coreference Resolution
TL;DR¶
The paper reframes machine translation difficulty from traditional output-based quality estimates into a continuous, metric-free source-side criterion called Discourse Dependency (DDP) based on referential reach, demonstrating that frontier LLMs fail to exploit long-range document context and that standard automatic metrics suffer from systemic blindness on context-demanding segments.
Background & Motivation¶
Recent breakthroughs in large language models have driven machine translation quality to historical highs, causing aggregate benchmark scores across standard general-domain test sets to saturate and fail to differentiate state-of-the-art systems. To re-establish discriminatory power, the machine translation community has increasingly pursued harder evaluation benchmarks, either by filtering difficult segments from existing corpora via automated score discrepancies or by synthesizing targeted adversarial perturbations. However, nearly all these attempts operate under a common flawed premise: translation difficulty is estimated post hoc from automatic translation quality scores such as COMET and BLEU. Relying on automatic metrics to quantify difficulty inevitably inherits their known unreliability and compressed dynamic range in ceiling-level quality regimes, running the grave risk of optimizing evaluations toward what metrics can easily calculate rather than what truly challenges translation models.
More fundamentally, conventional evaluation paradigms treat difficulty as an intrinsic property of an isolated segment, ignoring the fact that translation difficulty is inherently relational. A segment that is syntactically trivial and composed of basic vocabulary can be contextually demanding if resolving its pronouns, entities, and honorific agreements requires looking back across dozens of preceding sentences in the document. While document-level machine translation literature has long highlighted the necessity of discourse phenomena such as coreference, lexical cohesion, and consistency, prior diagnostic efforts have remained strictly phenomenon-specific, categorizing sentences via binary labels or bespoke contrastive minimal pairs. Such approaches lack a continuous, generalizable metric to quantify exactly how much referential reach a given sentence requires, and they leave the true contextual difficulty profile of standard benchmarks unmeasured.
To break this impasse, the authors shift the evaluative focus away from post hoc translation outputs and ground difficulty directly in the source document's referential structure. Core idea: formalize translation difficulty as discourse dependency (DDP), a metric-free, source-side lower bound computed from the maximum distance between named entity re-mentions and pronominal coreference chains, establishing a continuous spectrum that exposes frontier LLMs' failure to exploit long-range context and uncovers the severe blindness of standard automatic metrics.
Method¶
Overall Architecture¶
Discourse Dependency (DDP) operates directly on the source document to assign each segment a non-negative continuous score measuring the maximum sentence distance it must look back to resolve its referential expressions. Using DDP as a continuous analytical lens, the evaluation framework decouples the effects of context capacity versus context selection across five distinct context injection strategies within an English-to-Korean (En-Ko) automatic post-editing (APE) setup, comparing frontier open and proprietary LLMs against professional human post-editing.
The complete operational and diagnostic pipeline is illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Source Document"] --> B["Discourse Anchor Extraction & Chain Distance Accumulation<br/>NER tracking + pronominal coreference chaining"]
B --> C["DDP Stratification & Dual-Assumption Context Injection<br/>Decoupling Context Size vs Selection"]
C --> D["Post-Editing Preference & Metric Diagnostic Auditing<br/>Human ranking vs automated metric blind spots"]
D --> E["Continuous Difficulty Profile & Evaluation Findings"]
Key Designs¶
1. Discourse Anchor Extraction & Chain Distance Accumulation: Metric-Derived Post Hoc Estimates -> Dual-Rule Source Grounding -> Metric-Free Lower-Bound Difficulty Guarantee
Traditional difficulty estimation depends on model generation outputs and scoring functions, making it impossible to audit benchmark difficulty prior to model inference. DDP addresses this by extracting two foundational discourse anchors from the English source text: named entities (restricted to PERSON, ORG, GPE, and LOC via spaCy en_core_web_trf) and referential pronouns (categorized into MASC, FEM, NEUT, and PLUR, with expletives filtered out via dependency parsing). For each anchor in sentence \(s_i\), distance is assigned via two linguistically motivated rules:
- Rule 1 (Named Entities): The first mention of an entity introduces new information (\(d=0\)). Subsequent re-mentions measure the distance back to the earliest mention sentence \(s_j\) in the document (\(d = i - j\)).
- Rule 2 (Pronouns): Pronouns cannot serve as first mentions and must resolve to the nearest compatible named entity or head noun \(s_k\) (\(k < i\)). If anchored to a named entity, the chain accumulates that entity's own distance from its original first mention; if anchored to a head noun, the distance is simply \(i - k\); if no antecedent exists, a default minimal distance of 1 is assigned.
Segment-level DDP is the maximum distance across all anchors in the segment. Because unmodeled phenomena (e.g., bridging anaphora, lexical chains) only lower DDP rather than inflate it, DDP provides a rigorous one-sided lower bound. A high-DDP segment is mathematically certified to demand long-range document context (validated against OntoNotes 5.0 gold annotations, where 99.2% of segments satisfy the one-sided lower bound, with only 0.8% error due to NER false positives).
2. DDP Stratification & Dual-Assumption Context Injection: Coarse Domain Conflation -> Continuous DDP Stratification -> Systematic Decoupling of Context Size vs Selection
Topical domain labels (e.g., News vs Literary) are commonly used as crude proxies for text complexity, but DDP analysis reveals that within-domain variation vastly exceeds cross-domain variance. To isolate whether models actually utilize provided document context, the framework sets up an automatic post-editing (APE) paradigm on English-Korean translation and explores five strategies designed around two fundamental assumptions: - Size Assumption (more context is better): Compares APEseg (zero context, current segment only), APEseq (sliding window of preceding \(k=5\) segment pairs), and APEfull (entire document context minus the current segment). - Selection Assumption (proximity implies relevance): Compares APEseq (\(k=5\) immediately preceding segments), APErel (\(k=5\) most semantically relevant segments retrieved via LaBSE multilingual embeddings), and APEexp (structured declarative knowledge of genre, participant roles, and register formality extracted by prompting Claude Opus 4.6).
Stratifying by continuous DDP allows the evaluation to decouple raw translation competence from contextual retrieval efficiency, testing whether models can leverage expanding context as referential dependencies stretch further into the document.
3. Post-Editing Preference & Metric Diagnostic Auditing: Discrete Coarse Metrics -> Professional Human Ranking & LOWESS Smoothing -> Exposing Frontier LLM Collapse and Metric Insensitivity
To bypass the compressed resolution of automatic metrics, the evaluation conducts double-blind human relative ranking (1 to 4 scale, ties allowed, Kendall's \(\tau = 0.37\)) with 4 professional En-Ko translators across ~1,000 segments, benchmarking model post-edits against professional human post-editing (ยจ HPE). Simultaneously, 7 automatic metrics (xCOMET, xCOMET-Kiwi, MetricX, MetricX-QE, d-BLEU, SLIDE, doc-COMET) are computed across 229,230 segment-level observations and analyzed via LOWESS local non-parametric regression (\(\text{frac}=0.4\)).
This auditing design yields two crucial findings: First, while LLMs match or surpass human editors on self-contained segments (\(DDP \lesssim 10\)), human preference for professional human edits widens monotonically as DDP grows (\(DDP \ge 15\)), with no context injection strategy keeping pace. Second, all automatic segment- and document-level metrics remain virtually flat across the DDP spectrum, failing completely to detect the severe contextual degradation that human evaluators penalize.
Formal Definition and Evaluation Protocol¶
Formal Formulation of Discourse Dependency (DDP)¶
Given a document composed of sentences \((s_1, s_2, \dots, s_n)\), let \(A_i\) be the set of discourse anchors in sentence \(s_i\). For any named entity mention \(e \in A_i\), let \(s_j\) (\(j < i\)) be the earliest sentence containing a mention of the same entity: $\(d(s_i, e) = \begin{cases} 0, & \text{if no prior mention exists} \\ i - j, & \text{otherwise} \end{cases}\)$
For any referential pronoun \(p \in A_i\) of gender-number category \(g \in \{\text{MASC}, \text{FEM}, \text{NEUT}, \text{PLUR}\}\), let \(s_k\) (\(k < i\)) be the sentence containing the most recent compatible named entity \(e_k\) or head noun: $\(d(s_i, p) = \begin{cases} (i - k) + d(s_k, e_k), & \text{if antecedent is named entity } e_k \\ i - k, & \text{if antecedent is head noun} \\ 1, & \text{if no antecedent exists} \end{cases}\)$
The sentence-level and segment-level DDP scores are computed via maximum aggregation: $\(ddp(s_i) = \max_{a \in A_i} d(s_i, a), \quad ddp(\text{seg}) = \max_{s_i \in \text{seg}} ddp(s_i)\)$
Experimental Protocol & Evaluation Setup¶
- Evaluation Corpus: WMT24++ English-Korean (En-Ko) professionally post-edited subset across News, Social, and Literary domains (59 documents, 849 unique source segments).
- Evaluated Models: Six diverse state-of-the-art LLMs: open-weight systems Gemma-4-31B, Qwen3.5-27B, HyperCLOVA-X-Seed-Think-32B (Korean-specialized), DeepSeek-V4-Pro (1.6T MoE), and closed-source systems GPT-5.4 and Gemini-3-Flash. All models are evaluated with an identical 128K context limit, fixed decoding parameters (\(\text{temperature}=1.0\), \(\text{top\_p}=0.95\), \(\text{seed}=42\)), and thinking mode disabled for fair comparability.
- Human Evaluation & Significance: Professional human relative ranking on two non-overlapping subsets (~500 segments each) evaluated with Wilcoxon signed-rank tests and bootstrap 95% confidence intervals.
- Failure Filtering: Outputs with Translation Edit Rate \(\text{TER} > 100\) are categorized as generation hallucinations and filtered.
Key Experimental Results¶
Main Results¶
The table below presents the segment-level DDP distribution across domains in the WMT24++ En-Ko evaluation benchmark:
| Domain | Docs | Segments | Mean DDP (Std) | Median | 95th Percentile (p95) | Maximum | Zero-Dependency Share (% ddp=0) |
|---|---|---|---|---|---|---|---|
| News | 17 | 149 | 9.7 (12.4) | 4 | 35 | 52 | 28.9% |
| Social | 34 | 494 | 3.8 (14.3) | 0 | 17 | 166 | 62.3% |
| Literary | 8 | 206 | 52.8 (46.9) | 44 | 141 | 171 | 13.1% |
| Total | 59 | 849 | 16.7 (33.2) | 1 | 101 | 171 | 44.5% |
In WMT25, the Literary mean DDP reaches 222.6 (maximum 482), and Social rises to 21.3 (maximum 186), demonstrating that newer benchmarks contain higher dependency segments, yet remain predominantly skewed toward low-DDP sentences.
The table below summarizes human rater preference for professional human post-editing (ยจ HPE) over model post-editing (APEseq) alongside automatic metric differentials across DDP strata:
| DDP Stratum | Segments (n) | Human Rank 1st Share (ยจ HPE) | Human Preference Gap (ยจ HPE vs APEseq) | Statistical Significance (p-value) | xCOMET Mean Gap (ยจ HPE vs APEseq) |
|---|---|---|---|---|---|
| Overall Spectrum | 498 | 48.6% | +0.16 | \(p = 1.2 \times 10^{-9}\) (Spearman \(\rho=0.28\)) | -0.002 (Not significant) |
| Low DDP (\(DDP < 15\)) | 335 | 39.2% | +0.05 | \(p > 0.10\) (Insignificant human gap) | +0.001 (Not significant) |
| High DDP (\(DDP \ge 15\)) | 163 | 60.1% | +0.35 | \(p = 3.0 \times 10^{-5}\) (95% CI: [+0.18, +0.52]) | -0.005 (\(p = 0.40\), Metric blind spot) |
Ablation Study¶
The table below provides a comprehensive comparison of computational overhead across models and context curation strategies on a single NVIDIA RTX PRO 6000 Blackwell GPU:
| Model | Strategy | Avg Input Tokens | Input Ratio | Avg Output Tokens | Latency / Seg (sec) | Latency Ratio |
|---|---|---|---|---|---|---|
| GPT-5.4 | APEseg APEseq APErel APEexp APEfull |
216 644 730 347 10,688 |
1.0ร 3.0ร 3.4ร 1.6ร 49.5ร |
68 60 62 72 62 |
โ โ โ โ โ |
API model network excluded |
| GEMINI-3-FLASH | APEseg APEseq APErel APEexp APEfull |
208 636 718 343 17,556 |
1.0ร 3.1ร 3.5ร 1.6ร 84.4ร |
32 32 32 32 34 |
โ โ โ โ โ |
Highest context token inflation |
| GEMMA-4-31B | APEseg APEseq APErel APEexp APEfull |
243 665 747 372 10,371 |
1.0ร 2.7ร 3.1ร 1.5ร 42.7ร |
57 54 56 57 57 |
2.90 3.26 3.40 3.27 3.89 |
1.00ร 1.12ร 1.17ร 1.13ร 1.34ร |
| HCX-32B | APEseg APEseq APErel APEexp APEfull |
220 613 690 343 8,947 |
1.0ร 2.8ร 3.1ร 1.6ร 40.7ร |
47 44 46 46 49 |
4.09 7.45 7.90 2.81 13.18 |
1.00ร 1.82ร 1.93ร 0.69ร 3.22ร |
| QWEN3.5-27B | APEseg APEseq APErel APEexp APEfull |
222 631 708 358 9,471 |
1.0ร 2.8ร 3.2ร 1.6ร 42.7ร |
53 51 53 54 56 |
2.49 2.79 2.95 2.60 12.33 |
1.00ร 1.12ร 1.18ร 1.04ร 4.95ร |
| DEEPSEEK-V4-PRO | APEseg APEseq APErel APEexp APEfull |
218 686 781 364 12,080 |
1.0ร 3.1ร 3.6ร 1.7ร 55.4ร |
880 927 952 934 906 |
โ โ โ โ โ |
High output tokens due to tokenizer |
The table below breaks down the generation failure rate (\(\text{TER} > 100\)) across DDP strata:
| Model | Overall Failure (%) | \(DDP = 0\) (%) | \(DDP \in [1, 4]\) (%) | \(DDP \in [5, 14]\) (%) | \(DDP \ge 15\) (%) |
|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 3.2% | 5.1% | 1.9% | 1.9% | 0.7% |
| HyperCLOVA-X-32B | 7.2% | 10.1% | 6.4% | 4.5% | 2.8% |
| Gemma-4-31B | 10.6% | 13.2% | 12.3% | 6.4% | 4.2% |
| GPT-5.4 | 11.5% | 13.8% | 16.0% | 6.4% | 2.8% |
| Qwen3.5-27B | 13.8% | 17.7% | 13.2% | 10.0% | 7.0% |
| Gemini-3-Flash | 26.9% | 44.2% | 16.9% | 15.5% | 4.9% |
Key Findings¶
- Full Document Context (APEfull) Yields Negative Marginal Returns: Despite consuming 40โ84ร more tokens and incurring up to 5ร latency, APEfull consistently underperforms the localized sliding window (APEseq) in human preference across nearly the entire DDP range. This proves that current LLMs fail to extract relevant referential signals from massive document contexts.
- Raw Sentential Context Outperforms Structured Summaries: APEexp is token-efficient (1.6ร overhead), but abstracting discourse into structured JSON summaries deprives models of concrete lexical and syntactic anchors, causing it to underperform APEseq.
- Severe Metric Insensitivity in High-DDP Regimes: Automatic reference-based and QE metrics exhibit flat trajectories across the DDP axis. In segments with \(DDP \ge 15\), where human raters decisively prefer professional human translations (\(+0.35\) preference gap, \(p = 3.0 \times 10^{-5}\)), xCOMET registers an insignificant difference (mean gap \(-0.005\), \(p = 0.40\)).
Highlights & Insights¶
- Metric-Free, Source-Side Difficulty Lower Bound: DDP depends strictly on the source document, providing a certified mathematical lower bound on contextual demand without being contaminated by candidate translation errors or model biases.
- Deconstruction of the "Domain" Illusion: DDP exposes that conversational social data can be self-contained (62.3% \(ddp=0\)) while news can contain deep coreferential dependencies, proving that domain labels conflate discourse reality.
- Distinguishing "Input Capacity" from "Referential Reach": The findings illustrate that million-token context windows do not translate into discourse competence; LLMs struggle severely with entity consistency and pronoun resolution once antecedents exceed 15 sentences.
Limitations & Future Work¶
- Unmodeled Discourse Phenomena: DDP currently tracks only named entities and pronouns, omitting lexical cohesion, bridging anaphora, tense consistency, and ellipsis.
- Target Language Scope: Experiments primarily focus on English-to-Korean, with preliminary En-Zh metrics confirming metric flatness. Broader human evaluation across morphologically rich and low-resource pairs is left for future work.
- Potential for Contextual Attention Steering: Future research can exploit DDP anchor chains to develop sparse or targeted context retrieval mechanisms that selectively inject distant antecedents into LLM prompts.
Related Work & Insights¶
- vs Zouhar et al. (2025, 2026) & Proietti et al. (2025): Prior difficulty estimation uses post hoc translation quality metrics; DDP operates entirely source-side prior to translation generation.
- vs Post & Junczys-Dowmunt (2024): Validates their hypothesis that document-level gains concentrate in discourse-dense subsets, providing the first continuous quantitative metric to formalize this density.
- vs Fernandes et al. (2021, 2023): Moves beyond discrete, phenomenon-specific challenge sets to a continuous, document-level distance metric applicable across datasets.
Rating¶
- Novelty: โญโญโญโญโญ Formulates a pioneering, metric-free source-side continuous criterion for translation difficulty.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 6 leading LLMs, 5 context strategies, 7 metrics, and 1,000 human-annotated segments.
- Writing Quality: โญโญโญโญโญ Rigorous linguistic grounding, lucid exposition, and comprehensive ablation analyses.
- Value: โญโญโญโญโญ Essential foundation for the next generation of discourse-aware MT benchmarks and LLM context auditing.