Skip to content

LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/BDML-lab/LongNovel
Area: LLM Reasoning
Keywords: Long-context evaluation, hallucination detection, novel summarization, multi-scale benchmark, logical reasoning

TL;DR

To tackle the elusive challenge of detecting factual hallucinations in massive narrative contexts, this paper introduces LongNovel, a bilingual benchmark spanning 16k to 100k tokens across eight fine-grained hallucination categories constructed via multi-model arbitration and entity-referenced generation, uncovering severe performance collapse in long-range temporal and causal reasoning even among frontier LLMs.

Background & Motivation

Recent breakthroughs in extending the context windows of Large Language Models (LLMs) to 100k tokens and beyond have enabled end-to-end processing of book-length literary texts and extensive multi-document corpora. However, the brute-force expansion of context capacity does not inherently resolve the persistent vulnerability of factual hallucinations in long-form generation. While shallow contexts such as news articles or technical papers primarily hinge on isolated explicit entities, long-form novels exhibit intricate narrative structures, complex character interpersonal dynamics, and latent plot interdependencies. Evaluating hallucinations in book-length narratives requires models to conduct multi-hop inference across distant chapters rather than relying on local sentence-level overlap. Conventional NLI-based detectors (e.g., SummaC, AlignScore) and surface-level lexical metrics fail in this regime due to their narrow context windows, underscoring an urgent demand for dedicated long-context evaluation benchmarks.

Constructing a reliable benchmark for long-context novel hallucination detection faces two formidable roadblocks. First, the prohibitive cost of human annotation poses an insurmountable barrier: comprehensively verifying novel-length texts requires exhaustive reading and cross-referencing, making human-annotated datasets like NoCha and StorySumm difficult to scale, whereas existing synthetic datasets like LCHD remain constrained to shorter context windows below 100k tokens. Second, the evolutionary dynamics of hallucination patterns as context length scales up remain largely unexamined. Prevailing long-context benchmarks (such as Fables) are restricted to single-length evaluations, precluding a rigorous understanding of how model sensitivity, error types, and instruction degradation shift across progressive context lengths.

To bridge these critical gaps, this work introduces LongNovel. Built upon 29 complete Chinese novels and chapter-level English texts from BookSum, the benchmark establishes a standardized testbed across four context lengths (16k, 32k, 64k, and 100k tokens). By marrying multi-model arbitration with entity-referenced hallucination perturbation, complemented by high-agreement human validation, the benchmark ensures both real-world authenticity and comprehensive category coverage. Core idea: construct LongNovel, a multi-scale bilingual benchmark spanning 16k, 32k, 64k, and 100k tokens covering eight fine-grained hallucination types, combining multi-model arbitration of natural model failures with entity-referenced targeted synthesis to reveal fundamental reasoning bottlenecks in long-context summarization.

Method

Overall Architecture

The construction pipeline of LongNovel progresses through four sequential stages: source corpus collection and textual unit segmentation, multi-scale sliding-window context extension, dual-track data synthesis (multi-model arbitration and entity-referenced hallucination generation), and rigorous pairwise human revision. Chinese novels and English chapters are first divided into coherent textual units \(u_i\) of 2k–6k tokens, each paired with high-quality human summaries. A sliding-window protocol then concatenates units into four target length intervals: S(16k), M(32k), L(64k), and XL(100k). During synthesis, natural failure modes generated by multiple LLMs are verified via cross-model arbitration, while prompt-guided perturbations inject eight fine-grained hallucination types based on extracted entities, followed by minimal-pair human verification.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Corpus Collection & Chapter-Level Unit Segmentation<br/>29 Chinese Novels + BookSum Chapters (2k-6k Tokens)"] --> B["Dynamic Sliding-Window Length Extension"]
    B --> C["Multi-Model Arbitration for Natural Failure Extraction"]
    B --> D["Entity-Referenced Fine-Grained Hallucination Synthesis"]
    C --> E["Two-Tier Sliding-Window RAG & SFT Extrapolation Evaluation"]
    D --> E
    E --> F["Pairwise Human Consensus Review & Benchmark Release<br/>600 Test Instances / 8 Error Types / 4 Length Scales"]

Key Designs

1. Dynamic Sliding-Window Length Extension: Preserving narrative continuity and preventing unnatural plot truncation Arbitrary length truncation across arbitrary sentence boundaries disrupts chronological flow and plot coherence, reducing hallucination verification to trivial local lexical matching rather than long-range narrative tracking. LongNovel defines four target ranges \(R(T) = [T - 2\text{k}, T + 4\text{k}]\) for \(T \in \{16\text{k}, 32\text{k}, 64\text{k}, 100\text{k}\}\) measured by the Qwen3 tokenizer. For each book represented as a sequence of textual units \(B = \{u_1, u_2, \dots, u_n\}\), a dynamic sliding window \(w = [u_j, \dots, u_k]\) expands unit-by-unit until the cumulative token count enters \(R(T)\). The text sequence within \(w\) is then extracted alongside its concatenated human summaries \(S_w = [s_j, \dots, s_k]\), and the window reset to \(w = [u_{m+1}]\) where \(m = \lfloor(j+k)/2\rfloor\). This overlapping progression preserves cross-chapter narrative dependencies, requiring evaluation models to track character trajectories and storyline progressions across extensive contexts.

2. Multi-Model Arbitration for Natural Failure Extraction: Capturing real-world model failure modes while eliminating single-evaluator bias Relying entirely on manual drafting of negative long summaries is economically unviable, while using a single LLM introduces idiosyncratic generation artifacts. To mirror real-world error distributions, LongNovel adopts a Multi-Model Arbitration (MMA) framework. Initial long-context summaries are generated by GLM-4-9B-chat, Qwen3-32B, and GPT-4o. Rather than relying on fragile binary labeling, two independent judges (GPT-4.1 and Claude-3.5-Sonnet) assign continuous consistency scores from 0 to 5 accompanied by detailed rationales. These evaluations are subsequently aggregated and submitted to Gemini-1.5-Flash as the final arbitrator, ensuring that complex, multi-event hallucinations reflect authentic model failure modes without penalizing valid abstractive compression.

3. Entity-Referenced Fine-Grained Hallucination Synthesis: Eliminating category skew and guaranteeing balanced 8-class coverage Because LLMs naturally exhibit strong category bias when generating unconstrained summaries, LongNovel develops an Entity-Referenced Hallucination Generation (ERHG) strategy to enforce balanced coverage across fine-grained error types. Key narrative entities including characters, locations, organizations, numbers, and core actions are systematically extracted from human-written ground-truth summaries \(s_i\). Using these anchors, GPT-4.1, Gemini-1.5-Flash, and Claude-3.5-Sonnet perturb \(s_i\) across eight dedicated prompt templates representing distinct error categories: Entity, Numerical, Relation, Logical Inversion, Event, Temporal, Causal, and Event Fabrication. Perturbations introducing 1–2 error types receive a consistency score of 2, while 3 or more errors receive a score of 1, establishing controlled severity levels across all context scales.

4. Two-Tier Sliding-Window RAG & SFT Extrapolation Evaluation: Balancing local evidence grounding with full-context hallucination prevention Evaluating full 100k-token texts through direct LLM prompting incurs immense token overhead and suffers from attention dilution, whereas standard RAG fragments narrative context. LongNovel implements a narrative-constrained sliding-window RAG protocol. Source documents are segmented into 3,500-character blocks \(D = \{D_0, \dots, D_{N-1}\}\) and summaries into 75-character spans \(S = \{S_0, \dots, S_{M-1}\}\). Employing BGE-M3 embeddings, summary spans are anchored to document blocks under a monotonic chronological constraint (\(k_j \ge k_{j-1}\)). To mitigate false positives from localized retrieval, a Two-Tier Review mechanism evaluates each summary span within its dynamic local window (\(R_1 = \lfloor N/3 \rfloor - 1\)); if the local consistency score falls to \(\le 2\), the system triggers a fallback evaluation over the complete document context. Coupled with progressive context-length SFT for open-source backbones, this setup reduces evaluation token consumption by nearly 40% while providing a principled inference baseline.

Benchmark & Evaluation Protocol

The evaluation protocol employs a binary classification boundary: instances with consistency scores between 3 and 5 are designated non-hallucinated, whereas scores from 0 to 2 indicate hallucinated content. Model performance is measured via Balanced Accuracy (BAcc), calculated as the arithmetic mean of class-specific recalls to counter distribution skew, and the Matthews Correlation Coefficient (MCC) to evaluate true vs. false positives and negatives across the confusion matrix. The test set comprises 600 rigorously curated instances evenly distributed across S (200), M (200), L (100), and XL (100) scales with a 50:50 positive-to-negative ratio. Extensive evaluations cover leading open-source models (InternLM2.5-20B, GLM-4-9B, Llama3.1-8B-Instruct, Qwen3-8B/14B/32B) and proprietary frontier models (Claude-3.5-Sonnet, DeepSeek-v3, DeepSeek-r1, DeepSeek-v4-flash, GPT-5.2-chat) across diverse prompting paradigms (Zero-shot Prompt-E/Prompt-B, CoT, RAG, and Voting Ensembles).

Key Experimental Results

Main Results

The main benchmark evaluates open-source and commercial models across all four context scales in terms of Balanced Accuracy (BAcc) and MCC.

Model S (16k) BAcc S (16k) MCC M (32k) BAcc M (32k) MCC L (64k) BAcc L (64k) MCC XL (100k) BAcc XL (100k) MCC
Open-Source Models
InternLM2.5-20B 0.545 0.217 0.505 0.024 0.450 -0.196 0.480 -0.143
GLM-4-9B-chat 0.505 0.014 0.510 0.059 0.500 0.000 0.500 0.000
Minicheck-7B 0.500 0.000 0.392 -0.335 0.420 -0.266 0.310 -0.484
Llama3.1-8B-instruct 0.525 0.052 0.077 -0.837 0.000 -1.000 0.000 -1.000
Qwen3-8B 0.568 0.233 0.608 0.237 0.533 0.075 0.480 -0.050
Qwen3-14B 0.588 0.285 0.598 0.219 0.547 0.105 0.517 0.036
Qwen3-32B 0.590 0.292 0.575 0.180 0.597 0.253 0.580 0.187
Qwen3-32B-SFT 0.775 0.587 0.795 0.625 0.750 0.551 0.720 0.490
Commercial Models
Claude-3.5-Sonnet 0.755 0.510 0.665 0.332 0.680 0.363 0.670 0.340
DeepSeek-v3 0.755 0.544 0.735 0.521 0.680 0.450 0.670 0.433
DeepSeek-r1 0.775 0.592 0.735 0.510 0.640 0.382 0.680 0.450
DeepSeek-v4-flash 0.785 0.573 0.810 0.625 0.830 0.660 0.830 0.660
GPT-5.2-chat 0.785 0.570 0.770 0.541 0.810 0.620 0.740 0.482
Voting Ensemble (DSV4/GPT/C) 0.820 0.641 0.815 0.630 0.830 0.660 0.820 0.641

Ablation Study

The ablation experiments investigate the impact of training data composition (MMA alone vs. MMA combined with ERHG) on model fine-tuning across different context lengths.

Model & Variant S (16k) BAcc S (16k) MCC M (32k) BAcc M (32k) MCC L (64k) BAcc L (64k) MCC XL (100k) BAcc XL (100k) MCC
Qwen3-8B (Zero-shot Base) 0.568 0.233 0.608 0.237 0.533 0.075 0.480 -0.050
Qwen3-8B-MMA (MMA-only fine-tuning) 0.577 0.248 0.577 0.171 0.610 0.261 0.710 0.447
Qwen3-8B-FULL (MMA + ERHG full fine-tuning) 0.773 0.596 0.770 0.563 0.800 0.643 0.760 0.579
Llama3.1-8B-instruct (Zero-shot Base) 0.525 0.052 0.077 -0.837 0.000 -1.000 0.000 -1.000
Llama3.1-8B-MMA (MMA-only fine-tuning) 0.585 0.290 0.502 0.008 0.020 -0.961 0.000 -1.000
Llama3.1-8B-FULL (MMA + ERHG full fine-tuning) 0.715 0.508 0.542 0.173 0.210 -0.583 0.310 -0.583
Qwen3-32B (Zero-shot Base) 0.590 0.292 0.575 0.180 0.597 0.253 0.580 0.187
Qwen3-32B-MMA (MMA-only fine-tuning) 0.605 0.343 0.587 0.281 0.610 0.352 0.680 0.469
Qwen3-32B-FULL (MMA + ERHG full fine-tuning) 0.775 0.587 0.795 0.625 0.750 0.551 0.720 0.490

Key Findings

  • Systematic performance degradation under extreme context lengths: Across both open-source and commercial tiers, accuracy steadily decreases as context expands from 16k to 100k tokens. Smaller open-source models collapse entirely: Llama3.1-8B-instruct plummets to 0.000 BAcc and -1.000 MCC at 64k and 100k scales. Manual case analysis demonstrates this is not a JSON parsing issue, but catastrophic instruction collapse where the model degenerates into repetitive loops or re-summarizes the novel instead of executing hallucination evaluation.
  • Chain-of-Thought (CoT) acts as a double-edged sword: The utility of CoT varies sharply by architecture. While CoT improves Claude-3.5-Sonnet's 32k accuracy from 0.665 to 0.715, it degrades DeepSeek-v3 from 0.735 to 0.685 at 32k and from 0.670 to 0.610 at 100k, demonstrating that ungrounded reasoning steps in long-context scenarios tend to amplify hallucination cascades.
  • Hierarchical detection difficulty across error types: Fine-grained error analysis reveals that Event and Numerical hallucinations are the easiest to detect (Claude-3.5-Sonnet reaches 78.71% recall on Event; GPT-5.2-chat achieves 82.81% on Numerical). In stark contrast, Temporal and Causal hallucinations are the hardest, yielding 0.00% recall for multiple open-source models, and only 36.25% recall for GPT-5.2-chat in the temporal category, demonstrating widespread blindness to chronological and causal reversals.
  • Targeted synthesis activates robust extrapolation: Incorporating ERHG negative samples during supervised fine-tuning enables Qwen3-32B-FULL to maintain 0.720 BAcc and 0.490 MCC at the 100k scale (compared to 0.580 BAcc for base zero-shot), proving that balanced fine-tuning on fine-grained negative patterns is crucial for long-context robustness.

Highlights & Insights

  • Dual-track complementary synthesis architecture: By combining multi-model arbitration of natural model degradation with entity-referenced controllable perturbation, LongNovel circumvents the multi-thousand-dollar expense of manual long-context annotation while maintaining a strictly balanced distribution across 8 error types.
  • Monotonic chronological RAG with two-tier verification: Constraining segment alignment to adhere to the narrative timeline alongside an automatic full-document fallback mechanism reduces retrieval token overhead by nearly 40% while preventing localized false positives.
  • Severe context-positional attention bias in long-range prompts: Systematic empirical comparison between Prompt-B (summary at beginning) and Prompt-E (summary at end) reveals that placing the long novel text first allows the model to construct a comprehensive scene context, resulting in vastly superior detection stability when evaluating the target summary at the end.

Limitations & Future Work

  • Model parameter scale restricted to 32B in open-source evaluation: The open-source benchmarking stops at 32B parameters; investigating length extrapolation dynamics in 70B+ models (e.g., Llama-3.1-70B, Qwen-72B) remains an open direction.
  • Domain limitation to fictional narrative structures: LongNovel focuses on literary novels with intricate plotlines. Validating whether the observed failure modes translate identically to non-fictional domains such as multi-source financial filings or legal transcripts warrants further exploration.
  • Lack of specialized algorithmic mitigations for causal/temporal errors: While the benchmark demonstrates that models fail catastrophically on causal and temporal inversions, developing architectural mechanisms (e.g., explicit event-graph tracking) to fix these vulnerabilities is left for future work.
  • vs. NoCha / StorySumm: While NoCha and StorySumm established human-labeled narrative faithfulness evaluation, their high annotation costs constrained dataset scale and context lengths to below 16k tokens. LongNovel automates synthesis and pushes context evaluation to the 100k frontier across 4 multi-scale tiers.
  • vs. Fables / Clipper: Fables covers 100k contexts but only evaluates at a single static scale without examining multi-scale decay; Clipper generates synthetic true/false claims rather than holistic summary hallucination detection. LongNovel offers the first comprehensive multi-scale benchmark specifically targeting long-context summary hallucinations.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering multi-scale 16k–100k bilingual benchmark for novel summarization hallucination detection with an elegant dual-track synthesis strategy.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across open-source and frontier commercial models, 4 context scales, 8 error categories, rigorous ablations, and bootstrap confidence intervals.]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear presentation with precise mathematical formulation of sliding-window algorithms and insightful qualitative error taxonomy.]
  • Value: ⭐⭐⭐⭐⭐ [Provides an indispensable, highly challenging evaluation testbed for overcoming hallucination bottlenecks in ultra-long context LLM applications.]