Summarization Is Not Dead Yet¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Natural Language Generation
Keywords: Text Summarization, LLM Capabilities, Multidimensional Evaluation, Factuality Verification, Linguistic Analysis
TL;DR¶
Challenging the widespread narrative that large language models have rendered summarization research obsolete, this paper introduces a rigorous multi-track evaluation across 5 diverse domains and 5 frontier LLMs, demonstrating that human summaries maintain decisive advantages in informativeness and faithfulness while LLM outputs are preferred primarily for surface-level fluency.
Background & Motivation¶
The rapid advancement of instruction-tuned large language models (LLMs) has sparked widespread claims across the natural language processing community that machine-generated summaries have matched or even surpassed human-authored references (notably Pu et al., 2023, "Summarization is (almost) dead"). This narrative has prompted scholars to question whether summarization remains an open research problem or whether the core challenges have been largely addressed by general-purpose foundation models. This viewpoint typically leans on three recurring empirical observations: model summaries regularly secure higher win rates in crowd-sourced human preference evaluations, LLM judges systematically score model outputs above human references, and surface-level consistency checks often report lower hallucination rates for LLMs than for humans.
However, a critical examination reveals that these conclusions stem from profound methodological flaws and evaluation illusions. First, holistic human preference judgments inevitably conflate superficial fluency and stylistic polish with true information density; annotators presented with a single composite choice routinely favor well-formed, articulate outputs even when critical facts are omitted or diluted. Second, automated LLM-as-Judge setups frequently suffer from unchecked position bias and severe self-preference bias, artificially elevating familiar model writing styles. Most crucially, conventional factuality verification protocols adhere strictly to source-only matching: any extrinsic proposition absent from the source document is automatically penalized as an unsupported hallucination. Because human writers deliberately supply background context and world knowledge to explain complex scientific or legal concepts to lay audiences, source-only checkers systematically penalize legitimate human contextualization while letting superficial model omissions pass undetected.
This work directly dismantles these methodological artifacts by decoupling evaluation dimensions, eliminating data contamination, incorporating external knowledge into factuality auditing, and analyzing the underlying lexical, syntactic, and discourse structures of the generated texts. Core idea: by establishing a decoupled multi-dimensional human and self-excluding judge evaluation protocol, introducing dual-stream factuality verification backed by external web evidence, and conducting comprehensive corpus-level linguistic analyses across five diverse domains, this paper systematically refutes the "summarization is solved" hypothesis, proving that while current LLMs have raised the floor of summarization quality, their performance ceiling remains well below human capabilities.
Method¶
Overall Architecture¶
The paper develops an extensive, multi-track evaluation methodology spanning five heterogeneous benchmark domains: serialized Chinese web novels (CNNSum), popular scientific reporting (SciNews), multi-document English news (DiverseSumm), academic talk video transcripts (VISTA), and European Union legal legislation across 24 official languages (EurLexSum). Five frontier LLMs are benchmarked under identical generation settings: GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro Preview, Qwen3.5-397B-A17B, and Kimi-K2.5. The evaluation pipeline progresses through four interconnected tracks designed to isolate superficial presentation from substantive content: decoupled multi-dimensional human evaluation, self-exclusion bias-mitigated LLM-as-Judge evaluation, dual-stream external knowledge factuality verification, and multi-level computational linguistic profiling.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: 5 Diverse Multi-Domain Datasets<br/>Novels / Science / Multi-Doc / Video / Multilingual Law"] --> B["Decoupled Multi-Dimensional Evaluation<br/>Orthogonal scoring on Info / Faith / Coherence / Conciseness"]
B --> C["Self-Exclusion Bias-Mitigated Protocol<br/>Cross-model scoring excluding self-judgments + randomized order"]
C --> D["Dual-Stream Factuality Verification<br/>Atomic claim decomposition + Source union External Search validation"]
D --> E["Three-Level Deep Linguistic Profiling<br/>Lexical diversity / Syntactic dependency depth / Discourse flow"]
E --> F["Output: Comprehensive Benchmark & Human-LLM Gap Characterization"]
Key Designs¶
1. Decoupled Multi-Dimensional Evaluation: Conflating Surface Fluency with Content -> Four Orthogonal Dimensions and Blind Listwise Ranking -> Disentangling Quality Trade-offs and Exposing Content Deficits To address the primary limitation of holistic evaluations that conflate surface coherence with factual substance, the evaluation decouples summary quality into four orthogonal Likert dimensions scored from 1 to 5: Informativeness, Faithfulness, Coherence, and Conciseness. Candidate summaries (one human reference and five model outputs) are evaluated via a blind listwise ranking protocol where system identities are anonymized and presentation order is independently randomized for each evaluator. Each instance is assessed by three qualified crowd annotators, with inter-annotator agreement rigorously monitored via ordinal Krippendorff's \(\alpha \ge 0.70\). Furthermore, to counter concerns that human references might benefit from training set contamination, professional domain experts authored completely new, contamination-free human summaries under controlled offline conditions without AI assistance, verifying that human superiority is not an artifact of pretraining memory.
2. Self-Exclusion Bias-Mitigated Protocol: LLM Judge Self-Preference and Position Skew -> Cross-Evaluation Excluding Self-Judgments and Order Shuffling -> Establishing a Robust Automated Judgment Benchmark LLM judges exhibit marked self-preference bias (consistently favoring outputs generated by their own model family) and vulnerability to candidate ordering. In this protocol, all five evaluated frontier LLMs act simultaneously as candidate generators and evaluation judges across all five datasets. When calculating the scores and ranking for an output generated by model \(M\), model \(M\)'s own score and rank are strictly excluded, and the final rating is computed as the arithmetic mean of the remaining four independent models. Presentation orders are re-randomized for each judge call, and pairwise win rates are derived from the aggregated listwise rankings. A pre-evaluation pilot confirmed strong alignment between judge rankings and human consensus (Kendall's \(\tau = 0.61 \sim 0.78\)), verifying that the bias-mitigated judge panel provides an objective, reproducible benchmark.
3. Dual-Stream Factuality Verification: Source-Only Penalization of Legitimate Contextualization -> Decomposing Atomic Claims with Joint Source-Web Verification -> Accurately Distinguishing True Hallucinations from Explanatory Enrichment Standard factual consistency tools treat any assertion not verbatim supported by the source document as an extrinsic hallucination, disproportionately penalizing human writers who introduce legitimate world knowledge to contextualize specialized topics. To resolve this provenance-blind bias, the framework standardizes four state-of-the-art claim-level factuality verification frameworks (FaStFact, SAFE, FActScore, VeriScore) on a unified GPT-5.4 backend. Summaries are segmented into atomic claims and decontextualized. Crucially, the verification evidence pool is expanded from the source document alone to the union of the source text and external web evidence retrieved via search APIs (Jina Reader and Serper API). A proposition is marked as an actual hallucination only when it cannot be substantiated by either the source document or authoritative external knowledge bases, ensuring fair evaluation of explanatory context.
4. Three-Level Deep Linguistic Profiling: Surface Metric Blindness to Generation Homogeneity -> Lexical, Syntactic, and Discourse Multi-Scale Metrics -> Uncovering Model Structural Shallowness and Linear Source Dependence To provide a structural, mechanistic explanation for why LLM summaries read smoothly yet lack informational depth, the study conducts corpus-level linguistic profiling across three complementary linguistic tiers using Stanza and Qwen3-Embedding-8B. At the lexical level, Type-Token Ratio (TTR) and Moving-Average TTR (MATTR with a window of 50 tokens) quantify vocabulary diversity. At the syntactic level, dependency parse tree depth (Tree Depth) and noun phrase modifier counts (NP Modifiers) assess structural complexity and information packaging density. At the discourse level, topic progression (cosine similarity between consecutive sentence embeddings), information ordering (Kendall's \(\tau\) against source document chunk positions), and compression ratios characterize how models organize narrative flow.
Evaluation Protocol and Task Formalization¶
- Dataset Allocation: Evaluates 1,820 human annotation instances and 11,499 automated evaluation instances across 5 datasets: CNNSum (695 Chinese novel instances), SciNews (4,188 English science journalism instances), DiverseSumm (245 multi-document news instances), VISTA (1,859 video-to-text academic talk instances), and EurLexSum (4,512 EU legal instances across 24 official languages, 188 per language).
- Generation Settings: All models are accessed via official APIs using greedy decoding (\(T = 0\)), a maximum completion budget of 1,024 new tokens, default API top-p/top-k parameters, and uniform domain-specific instruction templates per dataset.
- Statistical Significance: Differences between human and model scores are validated via the two-sided Wilcoxon signed-rank test pooled across datasets, with the Benjamini-Hochberg False Discovery Rate (FDR) procedure applied at \(\alpha = 0.05\).
Key Experimental Results¶
Main Results: Human vs. LLM Multi-Dimensional Comparison¶
In human evaluation, pairwise win rates demonstrate that crowd annotators consistently prefer human-authored summaries over every LLM across all datasets, with win rates ranging from 0.66 to 0.90. The Likert score differences (\(\Delta = \text{Human} - \text{Model Average}\)) illustrate that this preference is driven by clear content superiority.
| Dataset | Human vs GPT Win Rate | Human vs Claude Win Rate | Human vs Gemini Win Rate | Human vs Qwen Win Rate | Human vs Kimi Win Rate | Info. \(\Delta\) | Faith. \(\Delta\) | Coher. \(\Delta\) | Conci. \(\Delta\) |
|---|---|---|---|---|---|---|---|---|---|
| CNNSum (Novel) | 0.74 | 0.74 | 0.88 | 0.90 | 0.72 | +0.40 | +0.34 | -0.10 | -0.06 |
| SciNews (Lay Science) | 0.71 | 0.84 | 0.77 | 0.90 | 0.82 | +0.34 | +0.37 | -0.11 | -0.09 |
| DiverseSumm (Multi-Doc) | 0.78 | 0.82 | 0.85 | 0.81 | 0.84 | +0.38 | +0.32 | -0.06 | +0.07 |
| VISTA (Video Talk) | 0.74 | 0.85 | 0.82 | 0.90 | 0.78 | +0.35 | +0.29 | -0.12 | -0.05 |
| EurLexSum (24 EU Langs) | 0.66 | 0.82 | 0.86 | 0.90 | 0.84 | +0.45 | +0.38 | -0.05 | -0.10 |
When tested on the newly authored Contamination-Free references, human margins expand significantly across every benchmark: on CNNSum, Informativeness \(\Delta\) grows to \(+0.45\) and Faithfulness to \(+0.38\); on SciNews, Informativeness \(\Delta\) reaches \(+0.43\) and Faithfulness \(+0.45\); on EurLexSum, Informativeness \(\Delta\) widens to \(+0.51\) and Faithfulness to \(+0.44\). This confirms that human superiority is not due to memorization of legacy test sets.
Ablation Study: Source-Length Stratification & Source-Only Factuality Inversion¶
To analyze how information integration load impacts performance, the SciNews benchmark was stratified into source-length tertiles. In addition, an ablation on factuality verification evaluated the effect of restricting evidence solely to the source text (disabling external web verification).
Table 1: Human-Model Differences on SciNews Stratified by Source Length (\(\Delta = \text{Human} - \text{Model Average}\))
| Document Length Tertile | Info. \(\Delta\) | Faith. \(\Delta\) | Coher. \(\Delta\) | Conci. \(\Delta\) | Key Observation |
|---|---|---|---|---|---|
| Short (0โ33%) | +0.18 | +0.26 | -0.03 | +0.04 | Models hold a slight edge in coherence; content gap is narrow |
| Medium (34โ66%) | +0.29 | +0.35 | +0.05 | +0.06 | Coherence gap reverses in favor of humans as context expands |
| Long (67โ100%) | +0.54 | +0.57 | +0.13 | +0.18 | Human dominance expands drastically on ultra-long integration |
Table 2: Factuality Scores Under Source-Only Verification (Averaged across FaStFact, SAFE, FActScore, VeriScore)
| Dataset | Human Score | GPT-5.4 | Claude Opus | Gemini 3.1 | Qwen3.5 | Kimi-K2.5 | Gap \(\Delta\) (Human - Models) |
|---|---|---|---|---|---|---|---|
| CNNSum | 0.843 | 0.789 | 0.782 | 0.771 | 0.776 | 0.781 | +0.063 |
| SciNews | 0.689 | 0.752 | 0.748 | 0.751 | 0.729 | 0.724 | -0.052 |
| DiverseSumm | 0.816 | 0.763 | 0.758 | 0.745 | 0.753 | 0.737 | +0.065 |
| VISTA | 0.798 | 0.748 | 0.743 | 0.731 | 0.726 | 0.721 | +0.064 |
| EurLexSum | 0.850 | 0.783 | 0.803 | 0.787 | 0.784 | 0.779 | +0.063 |
Key Findings¶
- The Fluency-Fidelity Illusion: In human Likert evaluations, humans hold massive, statistically significant advantages on Informativeness (\(+0.17\) to \(+0.47\)) and Faithfulness (\(+0.31\) to \(+0.51\)). However, models match or exceed humans on Coherence (GPT at \(-0.13\), Claude at \(-0.25\)). Holistic assessments mask this dichotomy, allowing fluent prose to overshadow significant omissions and hallucinations.
- Debunking the "Human Hallucination" Myth: Under source-only checking on SciNews, human scores plummet to 0.689, trailing all five models (model average 0.741, \(\Delta = -0.052\)). But when external search is enabled, human accuracy rebounds and surpasses all models. This proves that prior reports of humans "hallucinating more than LLMs" were an artifact of penalizing valid background contextualization.
- Structural Homogeneity in LLM Writing: Linguistic profiling reveals that human summaries exhibit substantially higher vocabulary diversity (TTR \(\Delta = +0.08 \sim +0.28\)), deeper dependency trees (\(\Delta = +0.05 \sim +0.24\)), and denser noun modification (\(\Delta = +0.04 \sim +0.19\)). LLMs exhibit significantly higher information ordering (\(\Delta = -0.03 \sim -0.19\)), demonstrating that models adhere rigidly to source document linear sequence rather than synthesizing content according to narrative logic.
Highlights & Insights¶
- Empirical Refutation of Premature Convergence: By demonstrating that the gap between human references and LLMs widens on long-context synthesis and contamination-free references, this study firmly establishes that summarization is far from solved.
- Gold-Standard Benchmarking Protocols: The integration of self-exclusion in LLM-as-Judge evaluation and external-knowledge-augmented factuality checking provides a reproducible, bias-resistant blueprint for evaluating generative models.
- Critical Implications for Downstream Pipelines: Flaws in summarization propagate directly into Retrieval-Augmented Generation (RAG), search indexing, and professional decision systems, where omission of single key facts or stylistic flattening degrades entire workflows.
Limitations & Future Work¶
- Static Evaluation Snapshot: The study benchmarks five frontier models in a zero-shot regime using greedy decoding. While sensitivity checks confirm stability under varied temperatures and chain-of-thought prompts, task-specific fine-tuning or RL-aligned summarizers warrant future investigation.
- Domain Scope: While spanning novels, science, news, video talks, and multilingual law, the study does not evaluate clinical medical encounters, multi-turn conversational transcripts, or low-resource non-European languages.
- Search Engine Coverage Bottlenecks: Open-domain factuality verification depends on web crawler and search engine indices; specialized internal legal databases or niche literary content may suffer from incomplete search recall.
Related Work & Insights¶
- vs. Pu et al. (2023) "Summarization is (almost) dead": Pu et al. relied on non-expert holistic preference ratings to argue that summarization research was finished. This paper proves that holistic preference is confounded by surface fluency and that multidimensional scoring reveals persistent human superiority.
- vs. Zhang et al. (2024) "Benchmarking LLMs for News Summarization": While Zhang et al. introduced freelance human references, they focused narrowly on single-document news. This paper scales across five diverse genres, introducing syntactic depth and discourse organization analyses to identify fundamental structural limits in LLM generation.
Rating¶
- Novelty: โญโญโญโญโญ [Addresses a central community controversy with a rigorous, multi-track decoupled evaluation framework]
- Experimental Thoroughness: โญโญโญโญโญ [Evaluates 5 frontier models across 5 datasets, encompassing 1,820 human annotations, 11,499 automated evaluations, and comprehensive statistical significance testing]
- Writing Quality: โญโญโญโญโญ [Exceptionally clear argumentation, flawless structural progression, and highly informative empirical visualizations]
- Value: โญโญโญโญโญ [Sets a new benchmark for text generation evaluation and refocuses the NLP community on faithful reasoning and discourse synthesis]