The Blindness of Document-Level Translation Evaluation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/trotacodigos/esa-counterfactual
Area: Multilingual & Translation
Keywords: machine translation evaluation, document-level MT, discourse coherence, counterfactual evaluation, Error Span Annotation
TL;DR¶
By introducing a counterfactual evaluation condition (MIX) that recombines segments from multiple systems to disrupt discourse coherence while keeping document layout intact, this paper demonstrates that standard human evaluation protocols (Error Span Annotation) and 14 automatic metrics produce statistically indistinguishable scores, system rankings, and error annotations between coherent and incoherent documents, proving that current evaluation frameworks suffer from protocol blindness rather than annotator perceptual failure.
Background & Motivation¶
Document-level machine translation (MT) has long been considered the critical missing piece for closing the final gap toward genuine human-level translation parity. Early claims of human parity based on isolated sentence-level evaluations quickly collapsed under scrutiny once document context was restored. Yet, despite widespread adoption of document-level benchmarking, a persistent series of anomalous findings has accumulated across recent evaluation campaigns: system rankings fluctuate wildly across evaluation setups, human scores cluster unnervingly near the top of evaluation scales, surface-level string metrics such as chrF compete effectively with heavy LLM-based evaluators, and literary translationโtheoretically the most context-dependent domainโparadoxically emerges as the easiest to satisfy. These persistent anomalies raise an urgent concern regarding the construct validity of current document-level evaluation methodologies.
Cognitive science distinguishes between content-driven blindness, where familiar or expected narratives suppress critical attention, and presentation-driven blindness, where attention is bounded by the visual structure and affordances of an interface. In current MT evaluation standards, a core assumption has gone largely unchallenged: that providing document-level presentation will naturally and inevitably elicit document-level judgment. Current protocols like Error Span Annotation (ESA) present the full document as surrounding context, but persist in eliciting evaluations and error highlights on a segment-by-segment basis. When human annotators notice global discourse breakdownsโsuch as sudden register shifts or conflicting character namesโthe interface provides no explicit channel to record cross-segment violations, forcing them to treat each segment on its own local merits.
To empirically test whether current evaluation protocols actually record document coherence, this paper designs an adversarial counterfactual condition (MIX). The counterfactual preserves identical document presentation and sentence ordering while drawing contiguous segments from distinct MT systems, deliberately destroying inter-sentential discourse consistency while holding local fluency intact. Core idea: by constructing a controlled counterfactual document condition that breaks cross-segment coherence while preserving document-level layout, this work tests whether human protocols and automatic metrics genuinely record discourse coherence, revealing that current evaluation failures stem from protocol blindness rather than human perceptual insensitivity.
Method¶
Overall Architecture¶
The experimental framework establishes an end-to-end evaluation pipeline comparing how human evaluators and automatic metrics behave under varying context conditions. Given a source document split into segments, translations are elicited under three distinct conditions: isolated segments without surrounding context (SEG), a coherent full document produced entirely by a single translation system (DOC), and an incoherent counterfactual document assembled by sampling segments across multiple distinct translation systems (MIX). The MIX construction ensures that each system in the pool contributes at least one segment, and that adjacent segments differ in originating system to disrupt register, honorifics, and entity continuity. Annotations are collected via the Error Span Annotation (ESA) protocol on the Pearmut platform with professional bilingual translators, while 14 automated metrics spanning surface, learned, quality estimation, and LLM-as-judge paradigms are evaluated synchronously using rigorous equivalence testing procedures.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Source Documents and Multi-System Candidate Pool"] --> B["Counterfactual Document Construction<br/>Isolated SEG / Coherent DOC / Recombined MIX"]
B --> C["Dual Cognitive Manipulation Checks<br/>Pairwise System Discriminability and Coherence Blind Test"]
C --> D["Tripartite Equivalence Hypothesis Testing<br/>Score TOST H1 / Ranking Agreement H2 / Error TOST H3"]
D --> E["Evaluation Attribution and Behavioral Diagnostics<br/>Score Recoverability Regression and Metric Convergence"]
E --> F["Output: Construct Validity Diagnosis of Document MT Protocols"]
Key Designs¶
1. Counterfactual Document Construction: Targeted Disruption of Discourse Coherence Under Controlled Presentation To isolate whether protocols measure discourse coherence independently of visual document presentation, this design constructs counterfactual documents \(d_{\text{mix}}\) that preserve external presentation while breaking contextual consistency. Formally, for a document of \(n\) segments, DOC presents all segments translated by a single system \(s\). The MIX condition preserves source content and segment ordering but draws each segment translation from the pool of competing MT systems, enforcing the constraints that every system contributes at least once and that each position avoids the system used in DOC. Under the theoretical additive decomposition of segment score \(e_{s,k}\): $\(e_{s,k} = e^{\text{seg}}_{s,k} + e^{\text{ctx}}_{s,k}\)$ a valid document-level protocol assumes that access to context introduces non-zero adjustment \(e^{\text{ctx}}_{s,k}\). Under the counterfactual condition \(e^{\text{mix}}_{s,k} = e^{\text{seg}}_{s,k} + e^{\text{ctx,mix}}_{s,k}\), if the protocol registers discourse incoherence, \(e^{\text{ctx,mix}}_{s,k}\) must diverge significantly from \(e^{\text{ctx}}_{s,k}\), penalizing the scrambled document.
2. Dual Cognitive Manipulation Checks: Disentangling System Discriminability from Evaluator Perception To preempt alternative explanations that a null result between DOC and MIX might stem from imperceptible differences between systems or annotator insensitivity to incoherence, two positive control studies were conducted. The first tested pairwise discriminability across the 3 most overlapping system pairs on the WMT leaderboard (e.g., REF vs. GEMINI-2.5-PRO, GPT-4.1 vs. CLAUDE-4, GEMTRANS vs. WENYIIL); human judges distinguished translation quality in 85%โ94% of trials. The second presented 102 quality-matched passage pairs (DOC vs. MIX) in a randomized, blinded setup, asking raters to identify which passage was translated by a single consistent author. Judges selected DOC in 87.3% of trials (Krippendorff's \(\alpha = 0.73, p < 0.001\)). This confirms that the disruption introduced by MIX is readily perceptible to humans, ensuring that any statistical equivalence observed during evaluation reflects protocol failure rather than perceptual blindness.
3. Tripartite Equivalence Hypothesis Testing: Statistical Formalization of Protocol Insensitivity Conventional significance testing cannot establish equivalence, as failing to reject a difference does not prove identity. This design formulates three explicit hypotheses evaluated through Two One-Sided Tests (TOST) with predefined equivalence margins: - H1 (Score Equivalence): Tests \(| \mu_{\text{DOC}} - \mu_{\text{MIX}} | < \epsilon\), where \(\epsilon = 3.0\) points on the 100-point ESA scale (corresponding to \(\approx 0.2\) standard deviations and well below inter-annotator variability). - H2 (Ranking Agreement): Evaluates Kendall's \(\tau\) on induced system rankings against \(\tau_{\min} = 0.667\), supplemented by pairwise bootstrap clustering (\(B = 1{,}000, \alpha = 0.05\)) to handle closely tied systems. - H3 (Error Equivalence): Tests per-annotation error count differences against \(\epsilon_{\text{err}} = 0.5\) errors (roughly 20% of baseline frequency), while assessing whether error spans localize within the initial or terminal 20% of segments where transitions occur.
4. Evaluation Attribution and Behavioral Diagnostics: Uncovering the Segment-Level Fallback Mechanism To explain why annotators perceive incoherence yet fail to reflect it in scores, linear regression models evaluate score recoverability from isolated segments to document conditions: $\(E_{\text{condition}}(d) = \beta_0 + \beta_1 E_{\text{SEG}}(d) + \epsilon\)$ SEG scores explain 52% of the variance in MIX scores (\(R^2 = 0.52, r = 0.72\)), but only 20% of DOC scores (\(R^2 = 0.20, r = 0.45\)). This establishes that when cross-segment coherence is broken, both human annotators and automatic metrics fall back to evaluating segments in isolation. Furthermore, active per-segment annotation time decreases from 76 seconds in SEG to 61 seconds in DOC and 56 seconds in MIX, demonstrating that surrounding document context acts primarily as a cognitive reading shortcut to accelerate segment comprehension rather than as an evaluative signal that penalizes inconsistency.
Key Experimental Results¶
Main Results¶
The primary evaluation was conducted on the WMT 2025 EnglishโKorean benchmark across news, social media, and literary domains, utilizing 10 professional translators across 18,420 annotations alongside 14 automatic metrics. Table 1 reports the paired TOST equivalence testing for score differences (H1) between DOC and MIX.
| Evaluator / Metric | Granularity / Scale | Sample Size \(n\) | Margin \(\epsilon\) | Mean Difference (DOC - MIX) | 90% Confidence Interval | TOST \(p\)-value |
|---|---|---|---|---|---|---|
| Human ESA | Human (0โ100) | 308 | 3.00 | -1.00 | [-2.05, +0.04] | < 0.001 |
| chrF | Surface (0โ100) | 230 | 3.00 | +0.0187 | [-0.515, +0.552] | < 0.001 |
| xCOMET-XXL | Learned Ref (0โ1) | 230 | 0.05 | +0.0002 | [-0.016, +0.016] | < 0.001 |
| METRICX-24-XXL | Learned Error (0โ25)* | 230 | 0.50 | +0.0037 | [-0.113, +0.121] | < 0.001 |
| COMETKIWI-XXL | Learned QE (0โ1) | 230 | 0.05 | +0.0001 | [-0.006, +0.007] | < 0.001 |
| GEMBA V2 | LLM-as-judge (0โ100) | 230 | 3.00 | -0.1538 | [-0.892, +0.584] | < 0.001 |
| d-BLEU | Document Surface (0โ100) | 230 | 3.00 | -0.2795 | [-0.837, +0.278] | < 0.001 |
| doc-COMET | Document Learned (0โ1) | 230 | 0.05 | -0.0008 | [-0.003, +0.001] | < 0.001 |
| SLIDE | Document Window QE (0โ1) | 230 | 0.05 | +0.0016 | [-0.007, +0.010] | < 0.001 |
| FALCON | Document LLM (0โ100) | 230 | 3.00 | +0.0421 | [-0.687, +0.771] | < 0.001 |
*Note: METRICX represents error scores where lower is better. All evaluators decisively reject non-equivalence (\(p < 0.001\)).
Table 2 details system-level scores and bootstrap statistical clusters (\(B = 1{,}000, \alpha = 0.05\)) across the 9 evaluated MT systems and human reference (REF) under SEG, DOC, and MIX.
| System | Human SEG (Score/Clust) | Human DOC (Score/Clust) | Human MIX (Score/Clust) | chrF (DOC/MIX) | xCOMET-XXL (DOC/MIX) | GEMBA V2 (DOC/MIX) |
|---|---|---|---|---|---|---|
| GEMINI-2.5-PRO | 89.63 (1) | 91.97 (1) | 91.05 (2) | 33.18 / 33.21 (1/1) | .791 / .792 (1/1) | 87.95 / 87.91 (1/1) |
| REF (Human) | 89.20 (1) | 91.78 (1) | 91.75 (1) | โ / โ | โ / โ | โ / โ |
| WENYIIL | 89.34 (1) | 89.11 (2) | 89.63 (2) | 30.72 / 30.79 (3/3) | .790 / .787 (1/1) | 86.74 / 86.71 (2/2) |
| GPT-4.1 | 87.45 (2) | 89.16 (2) | 88.90 (2) | 30.35 / 30.25 (3/3) | .782 / .782 (1/1) | 86.18 / 86.22 (2/2) |
| CLAUDE-4 | 86.98 (2) | 86.32 (3) | 88.45 (2) | 32.21 / 32.29 (2/2) | .735 / .734 (3/3) | 85.43 / 85.39 (3/3) |
| COMMANDA-MT | 86.59 (2) | 87.66 (3) | 87.77 (3) | 29.14 / 29.11 (4/4) | .770 / .772 (2/2) | 85.07 / 85.12 (3/3) |
| GEMTRANS | 86.40 (2) | 82.80 (5) | 86.47 (3) | 28.67 / 28.65 (4/4) | .790 / .791 (1/1) | 83.21 / 83.26 (4/4) |
| IRB-MT | 81.45 (3) | 85.13 (4) | 82.23 (4) | 25.95 / 25.97 (6/6) | .716 / .717 (4/3) | 81.34 / 81.29 (5/5) |
| YOLU | 80.93 (3) | 79.25 (6) | 82.77 (4) | 27.64 / 27.58 (5/5) | .739 / .738 (3/3) | 82.86 / 82.90 (4/4) |
| LANIQO | 77.63 (4) | 77.90 (6) | 80.27 (5) | 23.36 / 23.41 (7/7) | .719 / .721 (3/3) | 79.52 / 79.55 (5/5) |
| Total Clusters | 4 | 6 | 5 | 7 / 7 | 4 / 3 | 5 / 5 |
| Kendall's \(\tau\) (DOC vs MIX) | โ | โ | 0.778 | 1.00 | 1.00 | 1.00 |
Ablation & Error Analysis¶
Table 3 analyzes the error counts per annotation, severity breakdown, and spatial boundary localization for Hypothesis H3 across conditions.
| Condition / Comparison | Sample Size \(n\) | Total Errors / Seg | Major Errors / Seg | Zero-Error Rate (%) | Boundary Spans (%) | Paired Mean Diff (90% CI) | TOST \(p\)-value (\(\epsilon_{\text{err}} = 0.5\)) |
|---|---|---|---|---|---|---|---|
| SEG (Isolated) | 5,920 | 2.36 | 0.63 | 12.8% | 50.3% | โ | โ |
| DOC (Coherent) | 5,920 | 2.31 | 0.55 | 15.5% | 50.1% | โ | โ |
| MIX (Counterfactual) | 5,920 | 2.37 | 0.53 | 15.7% | 50.0% | โ | โ |
| DOC vs MIX (Paired) | 1,157 pairs | โ | โ | โ | โ | -0.15 [-0.25, -0.05] | < 0.001 |
| DOC vs SEG (Paired) | 1,184 pairs | โ | โ | โ | โ | +0.02 [-0.06, +0.11] | < 0.001 |
| SEG vs MIX (Paired) | 1,185 pairs | โ | โ | โ | โ | -0.08 [-0.18, +0.01] | < 0.001 |
Key Findings¶
- Unpenalized and paradoxically inverted scores: Not only are DOC and MIX scores statistically equivalent, but raw differences frequently exhibit a minor reversal where MIX receives marginally higher scores (human mean 86.93 vs. 86.11; METRICX-24-QE-doc difference -0.69) and slightly fewer Major errors (0.53 vs. 0.55), directly opposing the prediction that discourse disruption degrades quality.
- Perfect rank invariance across automated metrics: All 14 automatic metrics yield an exact rank correlation of \(\tau = 1.00\) between DOC and MIX, confirming that document-level extensions (e.g., d-BLEU, doc-COMET, FALCON) fail to register the complete loss of inter-sentential coherence.
- Metric-human rank agreement peaks under MIX: Across all evaluated metrics (from surface chrF to LLM-as-judge GEMBA V2), Kendall's \(\tau\) with human rankings is strictly higher under MIX than under DOC. This indicates that metrics align with human evaluators not through mutual appreciation of discourse nuance, but through a shared fallback to segment-level heuristics.
- Boundary localization of errors remains flat: Only 50.0% of error spans fall within the outer 20% boundaries of segments under MIX, identical to DOC (50.1%) and SEG (50.3%), proving that annotators do not mark errors at the seams where system switches occur.
Highlights & Insights¶
- Adversarial counterfactual methodology: The MIX protocol offers a rigorous diagnostic tool that disentangles layout presentation from contextual coherence by preserving sentence syntax and page layout while injecting controlled inconsistencies across register, honorifics, and entity names.
- Protocol blindness over annotator blindness: The dual manipulation check definitively proves that human annotators readily perceive document incoherence (87.3% identification accuracy), but the standard ESA interface forces evaluators into a segment-bound bottleneck where cross-sentence incoherence has no formal feedback channel.
- Goodhart's Law in MT evaluation: Much of the empirical progress claimed by recent document-level MT architectures and metrics appears to reflect segment-level quality gains aggregated over larger token windows rather than authentic discourse-level competence.
Limitations & Future Work¶
- Language pair specificity: The empirical evaluation focuses in depth on EnglishโKorean. While Korean's grammaticalized honorifics and pro-drop properties provide a rigorous stress test, findings should be replicated on language families with different discourse features (e.g., grammatical gender and mood consistency in Romance or Slavic languages).
- Upper-bound nature of the counterfactual: Combining distinct systems creates prominent register and terminology shifts, but does not capture intra-system document-level generation failures such as hallucinated anaphora or subtle discourse drift.
- Protocol redesign remains open: The paper conclusively diagnoses the structural blindness of current protocols, but developing a practical, high-agreement evaluation interface that captures holistic narrative coherence without inflating annotation costs remains an open challenge.
Related Work & Insights¶
- vs MQM and ESA (Freitag et al., 2021; Kocmi et al., 2024b): While contemporary frameworks incorporate full-document display into human evaluation workflows, this work proves that passive context presentation fails to elicit active discourse-level error attribution.
- vs Context-Extended Automatic Metrics (doc-COMET, d-BLEU, SLIDE): Metric extensions that concatenate context sentences or apply sliding windows yield identical system rankings on coherent and incoherent documents (\(\tau = 1.00\)), showing that extended receptive fields do not equate to discourse awareness.
- vs LLM-as-a-Judge Evaluators (GEMBA V2, FALCON): Despite prompts explicitly requesting evaluations of overall discourse coherence and fluency, LLM evaluators display identical blindness under counterfactual conditions, mirroring the segment-level fallback of human evaluators.
Rating¶
- Novelty: โญโญโญโญโญ Formulates an incisive counterfactual probe that directly challenges a foundational assumption in document-level MT evaluation.
- Experimental Thoroughness: โญโญโญโญโญ Backed by 18,420 expert annotations, 14 automatic metrics, dual manipulation controls, and rigorous TOST equivalence testing.
- Writing Quality: โญโญโญโญโญ Lucid, compelling, and methodologically grounded synthesis of cognitive theory and statistical evaluation.
- Value: โญโญโญโญโญ Provides a pivotal wake-up call for the NLP evaluation community, establishing critical safeguards against pseudo-progress in document-level generation.