Towards AI-Assisted Research Writing: Benchmarking Citation Verification and Claim Grounding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://kgarg8.github.io/emnlp26-sciig/
Area: Information Retrieval & RAG
Keywords: scientific introduction generation, citation verification, claim grounding, retrieval-augmented generation, multi-dimensional evaluation
TL;DR¶
This paper introduces the first large-scale benchmark and evaluation framework for Scientific Introduction Generation (SciIG), comprising 3,900 top-tier papers and a 30-metric multi-channel suite that systematically diagnoses modern LLMs across citation correctness, factuality, and rhetorical argumentation.
Background & Motivation¶
The explosive growth of scholarly production across artificial intelligence and computer science has strained the peer review and publication ecosystem to an unprecedented scale. Recent conference cycles witnessed staggering volumes of research submissionsโsurpassing 50,000 at AAAI 2027, 17,000 at EMNLP 2026, 19,000 at ICLR 2026, and over 40,000 at NeurIPS 2026. In this hyper-productive scholarly environment, maintaining clarity of motivation, conceptual rigor, and systematic literature contextualization places a daunting burden on researchers. The introduction section serves as the intellectual gateway and rhetorical foundation of every academic paper, tasked with establishing the research territory, pinpointing open technical gaps, and articulating the precise contributions under John Swales' Create-a-Research-Space (CARS) framework. Consequently, building reliable, AI-assisted authoring systems for drafting research paper introductions has evolved into both an urgent practical necessity and a formidable natural language generation challenge.
However, existing explorations into LLM-assisted academic writing suffer from critical structural shortcomings. Current multi-document summarization (MDS) approaches (such as PRIMERA) and broad automated survey frameworks (such as AutoSurvey and SurveyGen) emphasize extensive topic coverage across large document corpora, lacking the sharp argumentation, focused territory-to-gap transitions, and precise contribution positioning demanded by individual research paper introductions. Furthermore, existing scientific text corpora were compiled years prior and have almost certainly contaminated the pretraining corpora of modern frontier LLMs, inflating evaluation metrics and masking severe model deficiencies. When deployed to weave multiple related works into a cohesive narrative, frontier models frequently suffer from fabricated citations, attribution mismatch, factual drift, and rhetorical disjointednessโweaknesses that remain largely unquantified due to the lack of dedicated, multi-faceted benchmarking datasets and grounded evaluation suites.
To bridge these gaps, this work establishes the first standardized benchmark for Scientific Introduction Generation (SciIG), formalizing it as a conditional generation problem strictly grounded in contemporary research inputs that post-date model pretraining cutoffs. The core idea is to curate an un-contaminated benchmark of 3,900 papers from NAACL 2025 and ICLR 2025 with an automated metadata-enrichment pipeline, and propose a 30-metric multi-channel evaluation framework spanning lexical overlap, semantic alignment, content coverage, faithfulness, consistency, citation correctness, and narrative fluency to uncover the true capabilities and failure modes of frontier LLMs in scholarly prose generation.
Method¶
Overall Architecture¶
The Scientific Introduction Generation (SciIG) task is formalized as a conditional mapping \(f: (T, A, R) \mapsto I\), where the input comprises the target paper's Title \(T\), Abstract \(A\), and a set of \(k\) cited Related Papers \(R = \{r_1, r_2, \dots, r_k\}\) (each encompassing a resolved title, abstract, and author list). The system outputs an academically styled, rhetorically sound Introduction \(I\). The methodology encompasses a five-stage end-to-end dataset curation and enrichment pipeline, a spectrum of instruction-level and few-shot prompting paradigms, and a 30-metric hybrid evaluation suite combining automated formulations, LLM-as-a-judge scorers, and blinded expert human evaluations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Target Paper (Title, Abstract) and Related Papers R"] --> B["End-to-End Dataset Construction Pipeline<br/>PDF acquisition, grobid2json parsing, LLaMA-4 mapping & Semantic Scholar enrichment"]
B --> C["Instruction & Few-Shot Prompting Paradigms<br/>Short, Medium, Elaborate, Few-Shot exemplars & AutoCoT cognitive planning"]
C --> D["Claim Grounding & Citation Correctness Suite<br/>Keyphrase extraction, Longformer Contextual Relevance & APA citation precision/recall"]
D --> E["Multi-Dimensional Holistic Evaluation Suite<br/>Formula-based automated metrics, structured LLM-as-a-judge & blinded expert audits"]
E --> F["Output: Benchmarked Introductions & Granular Diagnostic Reports"]
Key Designs¶
1. End-to-End Dataset Construction Pipeline: Resolving and Enriching Multi-Source Scholarly Metadata Extracting structured citation links from unstructured academic PDFs without noisy artifact leakage represents a major technical obstacle. The authors developed a five-phase ingestion pipeline: (i) collecting official PDF proceedings from NAACL 2025 (800 papers) and ICLR 2025 (3,100 papers); (ii) parsing raw PDFs into structured TEI-XML and JSON trees via S2ORC's grobid2json; (iii) extracting in-text introductory citation strings using regular expression patterns capturing "Author et al., Year" conventions; (iv) deploying LLaMA-4-Maverick under a strict JSON-schema prompt to resolve extracted citation strings to full bibliographic reference titles, avoiding heuristics that fail on complex multi-author truncations; and (v) querying Semantic Scholar across 59,619 citation lookups via a dual-stage resolution strategy (unofficial endpoint followed by the official Graph API with a strict title cosine similarity threshold \(\ge 0.85\)). This pipeline achieves an impressive 94.0% resolution rate across 3,900 papers, supplying clean, rich contextual abstracts for cited works without pretraining cutoff contamination.
2. Instruction & Few-Shot Prompting Paradigms: Decoupling Rhetorical Planning from Surface Generation To systematically investigate model sensitivity to guidance depth, the benchmark designs seven distinct prompting configurations partitioned into two paradigms: instruction-level prompting and few-shot in-context learning. The instruction spectrum ranges from Short (minimal title/abstract/citation context) and Medium (explicit research-gap framing) to Elaborate, which enforces a rigid four-paragraph academic structure totaling ~600 words (Paragraph 1: broad area and contextual significance; Paragraph 2: specific problem and gap grounded in prior citations; Paragraph 3: novel contributions contrasting with cited work; Paragraph 4: broader impact and purpose). To test whether explicit reasoning scaffolds mitigate hallucination, the authors implement an AutoCoT strategy that instructs the model to autonomously formulate intermediate reasoning steps (territory definition, gap isolation, contribution mapping) in a single pass before synthesizing the text. Furthermore, the few-shot paradigm leverages held-out isolation sets (54 NAACL and 52 ICLR exemplars) to assess performance trajectories across 0-shot, 1-shot, 2-shot, and 3-shot configurations.
3. Claim Grounding & Citation Correctness Suite: Mathematical Grounding across Heterogeneous Contexts Standard n-gram metrics (ROUGE/BLEU) cannot distinguish valid scientific claims from unsubstantiated hallucinations. To resolve this, the framework introduces rigorous claim grounding and citation correctness metrics. KeyBERT with an all-MiniLM-L6-v2 backbone extracts top-5 keyphrases from the generated text (\(K_{gen}\)), ground truth introduction (\(K_{GT}\)), and input context (\(K_{ctx}\)), matched via Porter stemming. Reference-based Coverage (RBC) and Reference-free Coverage (RFC) are formulated as: $\(RBC = \frac{|\{k \in K_{GT} \mid \exists k' \in K_{gen}, \text{stem}(k) = \text{stem}(k')\}|}{|K_{GT}|}, \quad RFC = \frac{|\{k \in K_{ctx} \mid \exists k' \in K_{gen}, \text{stem}(k) = \text{stem}(k')\}|}{|K_{ctx}|}\)$ Alongside keyphrase-based Faithfulness and Consistency, the authors formulate Contextual Relevance using Longformer-base-4096 representations, computing mean cosine similarities between the generated introduction vector \(g\) and the set of heterogeneous context component vectors \(\{c_1, \dots, c_n\}\) (title, abstract, and each cited paper abstract): $\(\text{Contextual Relevance} = \frac{1}{n} \sum_{i=1}^n \frac{g \cdot c_i}{|g| |c_i|} + \epsilon\)$ For citation attribution, in-text citation keys are extracted from generated outputs using LLaMA-4-Maverick and matched against the gold Related Works bibliography to compute Citation Precision (penalizing fabricated or off-target citations) and Citation Recall (measuring recovery of the author's intended foundational literature).
4. Multi-Dimensional Holistic Evaluation Suite: Bridging Surface Overlap and Semantic Adjudication The evaluation framework consolidates 30 distinct metrics spanning seven core categories: Lexical Overlap (ROUGE-1/2/L, BLEU, METEOR), Semantic Similarity (BERTScore, BLEURT, Contextual Relevance), Content Coverage (RBC, RFC, Judge), Faithfulness (Keyphrase Faithfulness, Judge), Consistency (Keyphrase Consistency, Judge), Citation Correctness (Precision, Recall, Citation Contextual Quality Judge), and Narrative Quality (GPT-2 Perplexity, Judge Narrative Quality). To complement formula-based scores that over-penalize valid stylistic paraphrasing, the system deploys GPT-4o as an impartial evaluator querying structured JSON outputs with bulleted rationales, complemented by a rigorous, blinded human expert evaluation across 10 qualitative criteria.
Key Experimental Results¶
Main Results¶
The benchmark evaluates five state-of-the-art language models with pretraining knowledge cutoffs strictly preceding the 2025 dataset horizon: DeepSeek-V3-0324 (671B MoE open weights), Gemma-3-12B-it, LLaMA-4-Maverick (17B active parameters, 1M context window), Mistral-Small-3.1-24B, and closed-source commercial baseline GPT-4o. All models were evaluated under identical sampling configurations (\(T=0.7, p=0.95\)) with the Elaborate prompt.
The quantitative results across both benchmark splits are summarized below:
| Dataset | Metric Category & Name | DeepSeek-V3 | Gemma-3-12B | LLaMA-4-Mav | Mistral-Small | GPT-4o |
|---|---|---|---|---|---|---|
| NAACL 2025 | ROUGE-1 | 0.4161 | 0.4361 | 0.4402 | 0.4388 | 0.4282 |
| ROUGE-2 | 0.1232 | 0.1304 | 0.1509 | 0.1340 | 0.1310 | |
| ROUGE-L | 0.1649 | 0.1725 | 0.1765 | 0.1754 | 0.1531 | |
| BLEU | 0.1175 | 0.1232 | 0.1198 | 0.1043 | 0.0781 | |
| BERTScore | 0.8415 | 0.8475 | 0.8448 | 0.8467 | 0.8422 | |
| Contextual Relevance | 0.9691 | 0.9684 | 0.9694 | 0.9696 | 0.9687 | |
| Ref-based Coverage (RBC) | 0.4435 | 0.4513 | 0.4845 | 0.4692 | 0.4713 | |
| Ref-free Coverage (RFC) | 0.3913 | 0.3666 | 0.4367 | 0.3959 | 0.4035 | |
| Keyphrase Faithfulness | 0.3138 | 0.2933 | 0.3230 | 0.3263 | 0.2892 | |
| Citation Recall | 0.5158 | 0.5763 | 0.5574 | 0.4914 | 0.4826 | |
| Citation Precision | 0.8819 | 0.9049 | 0.8889 | 0.9097 | 0.8686 | |
| Perplexity (PPL) \(\downarrow\) | 28.5031 | 32.3897 | 18.4870 | 25.6458 | 21.8494 | |
| Judge Narrative Quality | 0.8892 | 0.8888 | 0.9001 | 0.8928 | 0.9004 | |
| ICLR 2025 | ROUGE-1 | 0.3863 | 0.4342 | 0.4298 | 0.4326 | 0.4154 |
| ROUGE-2 | 0.1195 | 0.1340 | 0.1537 | 0.1344 | 0.1289 | |
| ROUGE-L | 0.1700 | 0.1715 | 0.1689 | 0.1894 | 0.1830 | |
| Contextual Relevance | 0.9696 | 0.9700 | 0.9707 | 0.9713 | 0.9729 | |
| Ref-based Coverage (RBC) | 0.4461 | 0.4732 | 0.4794 | 0.4725 | 0.4697 | |
| Citation Recall | 0.4954 | 0.5413 | 0.5505 | 0.4671 | 0.4657 | |
| Citation Precision | 0.8965 | 0.8822 | 0.8942 | 0.9357 | 0.9238 | |
| Perplexity (PPL) \(\downarrow\) | 36.9546 | 29.1004 | 21.1194 | 29.2585 | 22.0046 |
Input Context Ablation Study¶
Focusing on the top-performing LLaMA-4-Maverick, an extensive ablation assesses the impact of progressive input context expansion: Title-only, Title + Abstract, and Title + Abstract + Related Work, evaluated across both Elaborate and Medium prompting strategies:
| Prompt Strategy | Input Configuration | ROUGE-1 | ROUGE-2 | BERTScore | Citation Precision | Citation Recall | Judge Citation Quality | PPL \(\downarrow\) |
|---|---|---|---|---|---|---|---|---|
| Elaborate | Title-only | 0.3545 | 0.0772 | 0.8324 | 0.2399 | 0.0918 | 0.5174 | 14.5978 |
| Title+Abstract | 0.4093 | 0.1202 | 0.8408 | 0.3610 | 0.1199 | 0.5184 | 19.9132 | |
| T+A+Related | 0.4402 | 0.1509 | 0.8448 | 0.9357 | 0.5574 | 0.8051 | 18.4870 | |
| Medium | Title-only | 0.3356 | 0.0660 | 0.8235 | 0.0012 | 0.0001 | 0.3696 | 16.6799 |
| Title+Abstract | 0.3949 | 0.1084 | 0.8320 | 0.0175 | 0.0039 | 0.5184 | 20.7426 | |
| T+A+Related | 0.4513 | 0.1625 | 0.8471 | 0.8548 | 0.6193 | 0.8388 | 26.2687 |
In human expert evaluation (20 source papers, 80 blind outputs, 1โ5 scale), LLaMA-4-Maverick (3-shot) secured the strongest qualitative ratings across content coverage (4.50), idea flow (4.60), motivation clarity (4.40), and contributions summary (4.20). Concurrently, GPT-4o (0-shot) achieved a flawless 5.00 in Hallucination Resistance, confirming its conservative stance against fabricated facts.
Key Findings¶
- Related Works are an absolute prerequisite for non-hallucinatory citation: Without explicit Related Works inputs (Title-only or Title+Abstract), model citation precision collapses to 0.0012โ0.3610 and recall drops near zero (0.0001โ0.1199), forcing models to invent phantom citations. Supplying rich related-work metadata causes precision to soar to 0.8548โ0.9357 and recall to 0.5574โ0.6193, proving that LLMs excel at contextual literature synthesis but cannot reliably retrieve citations from parametric memory alone.
- Few-shot exemplars establish critical structural anchors: Progressing from 0-Shot to 3-Shot yields continuous improvements in lexical overlap (ROUGE-1: 0.4402 \(\to\) 0.4460, ROUGE-2: 0.1509 \(\to\) 0.1647) and reference-free coverage (0.4367 \(\to\) 0.4755), with human experts scoring 3-shot outputs substantially higher across all rhetorical dimensions.
- Automated-Human metric divergence highlights evaluation subtleties: Spearman rank correlations reveal that LLM-as-a-Judge aligns moderately with formula-based metrics on faithfulness (\(\rho=0.35\)) and consistency (\(\rho=0.45\)). However, citation quality exhibits an inverse correlation (\(\rho=-0.19\)) with overlap metrics, as strict string matching unfairly penalizes contextually valid, paraphrased references.
Highlights & Insights¶
- Scalable Realism via Deliberately Constrained Inputs: By rejecting the unrealistic luxury of feeding post-hoc Methodology and Results sections into the prompt, the SciIG benchmark models the authentic epistemic state of a researcher drafting an introduction, offering a genuinely scalable and reproducible benchmark.
- Architectural Divergence Between Open and Proprietary Systems: LLaMA-4-Maverick's massive context handling and instruction tuning achieve superior narrative fluency (PPL 18.4870) and balanced coverage, while GPT-4o delivers robust conservatism against hallucinated citations, offering practical architectural guidelines for hybrid academic copilot pipelines.
- The Cognitive Burden of Chain-of-Thought in Rhetorical Writing: While AutoCoT improves factual keyphrase precision (Keyphrase Faithfulness reaches 0.3610), the internal reasoning traces disrupt surface-level narrative flow, elevating perplexity and trailing direct structured instructions (Elaborate 0-shot) in expert human assessments.
Limitations & Future Work¶
- Omission of Multimodal Scientific Diagrams: Modern high-impact AI introductions frequently rely on architectural teaser diagrams or illustrative motivational figures; the current text-only benchmark leaves multimodal generation out of scope.
- Systemic Deficiencies in Recovering Granular Research Contributions: Constrained to Title, Abstract, and Related Works, all evaluated models struggled to synthesize concrete, bulleted contributions, frequently defaulting to generic, conclusion-like summaries that omitted fine-grained experimental findings.
- Future Trajectories: Enriching the input space with lightweight intermediate representations (such as distilled pseudocode or key empirical metric summaries) and exploring interactive, human-in-the-loop citation recommendation interfaces.
Related Work & Insights¶
- vs PRIMERA & Graph MDS: Traditional multi-document summarizers rely on heavy AMR graph parsing or explicit tree encoders, focusing broadly on multi-source news synthesis. SciIG tackles scholarly rhetoric directly, enforcing territory-gap-contribution positioning without external parser dependencies.
- vs AutoSurvey & SurveyGen: Broad survey generation systems prioritize sweeping breadth across dozens of pages, whereas SciIG targets localized, high-density rhetorical argumentation where unsupported claims instantly compromise scholarly integrity.
- vs Liebling et al. (2025) Interactive Copilot: Liebling et al. examine interactive human-LLM drafting loops on a small curated set; this work formulates an offline, single-pass, standardized benchmark over 3,900 papers, establishing an objective, reproducible baseline for future research.
Rating¶
- Novelty: โญโญโญโญโ Pioneers a contamination-free, standardized benchmark for scientific introduction generation grounded in contemporary literature.
- Experimental Thoroughness: โญโญโญโญโญ Rigorously spans 3,900 papers, 5 foundation models, 7 prompting paradigms, and a 30-metric multi-channel evaluation suite.
- Writing Quality: โญโญโญโญโญ Exceptionally clear mathematical formulations, meticulous data cleaning protocols, and transparent case study analyses.
- Value: โญโญโญโญโญ Establishes a foundational testbed for academic writing assistants and demystifies the actual boundaries of LLMs in literature-grounded claim formulation.