Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://kgarg8.github.io/emnlp26-sciig/
Area: LLM Reasoning
Keywords: Scientific Introduction Generation, Academic Writing Benchmark, Multi-Input Generation, Citation Grounding, LLM Evaluation
TL;DR¶
This paper proposes SciIG, the first large-scale systematic benchmark for evaluating large language models on generating scientific paper introductions, curating 3,900 post-cutoff papers from NAACL 2025 and ICLR 2025 and employing a 30-metric multi-channel framework combining automated formula metrics, LLM judges, and expert human evaluations.
Background & Motivation¶
The exponential growth of generative artificial intelligence, particularly large language models (LLMs), has sparked profound interest in their deployment as collaborative assistants across complex cognitive and specialized tasks. In the computer science and machine learning communities, the volume of scholarly submissions has reached unprecedented historical peaks: recent conference cycles report over 50,000 paper submissions at AAAI 2027, over 17,000 at EMNLP 2026, approximately 19,000 at ICLR 2026, and more than 40,000 at NeurIPS 2026. Under such immense publication velocity, researchers bear an exhausting intellectual burden in maintaining motivation clarity, contextual rigor, and high rhetorical standards when drafting introductions—a critical section that establishes the scholarly territory, defines the core technical gap, and articulates paper contributions.
Despite the remarkable fluency of current general-purpose LLMs, generating rigorous scientific introductions remains fundamentally challenging. Unlike generic summarization or conversational prose, scientific introductions demand strict adherence to academic genre conventions, precise factual grounding in existing literature, and balanced positioning of novel contributions against prior work. Prior scholarly text generation efforts—spanning multi-document summarization benchmarks like Multi-XScience, automated survey generators like SurveyGen and AutoSurvey, or interactive writing tools—either rely on outdated datasets whose papers are heavily memorized in LLM pretraining corpora, or focus on broad, unconstrained literature reviews rather than the tightly structured rhetoric required by conference introductions. Furthermore, practitioners and online tutorials frequently deploy ad-hoc prompt templates without systematic empirical validation or rigorous multi-faceted quality auditing.
To bridge this gap, the community requires an authoritative, post-cutoff benchmarking platform with clean input-output boundaries and multi-dimensional evaluation criteria. The core idea is to formulate Scientific Introduction Generation (SciIG) as a multi-input conditional mapping from Title, Abstract, and structured Related Works to a complete academic introduction, curating a 3,900-sample benchmark from NAACL 2025 and ICLR 2025 to systematically evaluate frontier LLMs across formula-based, LLM-as-a-judge, and human evaluation axes.
Method¶
Overall Architecture¶
The Scientific Introduction Generation (SciIG) task is formally framed as a conditional generation problem: given a target research paper's Title \(T\), Abstract \(A\), and a set of \(k\) cited Related Papers \(R = \{r_1, r_2, \dots, r_k\}\) (where each \(r_i\) comprises a paper title, abstract, and author list), the objective is to learn a mapping \(f: (T, A, R) \mapsto I\) to produce a coherent, academically styled Introduction \(I\). This tripartite input design directly maps onto Swales' CARS (Create a Research Space) rhetorical model: the related work \(R\) establishes the research territory and identifies existing gaps, the abstract \(A\) provides the primary technical contributions, and the title \(T\) defines the conceptual framing. The benchmark operates via an end-to-end dataset creation pipeline, a spectrum of instruction and few-shot prompting paradigms, and a 30-metric comprehensive evaluation suite.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
subgraph DataPipeline["Three-Phase Dataset Construction Pipeline"]
direction TB
A["Conference Paper Acquisition<br/>NAACL 2025 & ICLR 2025 PDFs"] --> B["Grobid Parsing & Citation Extraction<br/>TEI-XML parsing + regex identification"]
B --> C["Citation Mapping & Two-Stage API Enrichment<br/>LLaMA4 bib matching + S2 API metadata fetch"]
end
DataPipeline --> D["Structured Triplet Input (T, A, R)<br/>Title + Abstract + JSON Related Papers"]
subgraph PromptingEng["Hierarchical Prompting Strategy Space"]
direction TB
E["Instruction-Level Strategies<br/>Short / Medium / Elaborate / AutoCoT"]
F["Few-Shot Exemplar Strategies<br/>Zero-Shot / 1-Shot / 2-Shot / 3-Shot"]
end
D --> PromptingEng
PromptingEng --> G["Frontier Model Inference<br/>LLaMA4-Maverick / Gemma-3 / Mistral / DeepSeek / GPT-4o"]
G --> H["Tripartite Comprehensive Evaluation Suite<br/>Formula Metrics + LLM-as-a-Judge + Human Review"]
H --> I["Empirical Writing Quality & Boundary Insights"]
Key Designs¶
1. Three-Phase Dataset Construction Pipeline: Automated extraction, mapping, and API enrichment of citation contexts
Extracting citation-grounded scholarly introductions from raw conference PDFs is heavily hindered by layout heterogeneity and incomplete bibliographic metadata. The authors engineer a five-step, three-phase automated pipeline: Phase I acquires official and arXiv PDFs and processes them via the S2ORC grobid2json parser into structured TEI-XML representations. Phase II identifies in-text citations using regex targeting "Author et al., Year" patterns and leverages LLaMA 4-Maverick to match these citation keys against the bibliography section to retrieve clean reference titles. Phase III queries the Semantic Scholar API via a two-stage retrieval strategy: first querying an unofficial endpoint with title normalization (lowercased, punctuation-collapsed), and falling back to the official Graph API with a strict title similarity matching threshold (\(\ge 0.85\)). Across 59,619 lookup queries, the pipeline achieves an exceptional 94.0% citation resolution rate, constructing 3,900 benchmark instances that post-date the training cutoffs of all evaluated models.
2. Hierarchical Prompting Strategy Space: Disentangling structural constraints from in-context demonstration scaling
To investigate how LLMs respond to varying levels of rhetorical scaffolding, the framework benchmarks seven prompt configurations across two orthogonal paradigms. Under the instruction-level paradigm, the Short prompt provides minimal context; Medium emphasizes gap articulation and citation synthesis; Elaborate enforces a rigid four-paragraph academic structure (100–150 words per paragraph: domain overview, literature gap, novel contributions, and impact summary); and AutoCoT elicits autonomous intermediate reasoning steps (territory summary, unresolved challenges, and contribution outline) within a single generation pass without external scaffolding. Under the few-shot paradigm, the framework tests Zero-Shot, One-Shot, Two-Shot, and Three-Shot prompting using randomly sampled exemplars from dedicated held-out subsets (54 papers for NAACL, 52 for ICLR), allowing precise isolation of the marginal gains from in-context examples versus explicit rubric instructions.
3. Tripartite Multi-Channel Evaluation Framework: Unifying formulaic metrics, LLM-as-a-judge, and blind human auditing
Relying solely on lexical overlap fails to capture scholarly argumentation quality, factual grounding, and rhetorical flow. The proposed suite integrates 30 metrics across seven dimensions. In addition to classical lexical overlap (ROUGE-1/2/L, BLEU, METEOR), semantic similarity incorporates Longformer-based Contextual Relevance:
where \(\mathbf{g}\) is the mean-pooled embedding of the generated introduction, and \(\mathbf{c}_i\) are embeddings of the title, abstract, and each cited paper. Content coverage is disentangled into Reference-based Coverage (RBC, matching ground-truth introduction keyphrases via Porter-stemmed KeyBERT) and Reference-free Coverage (RFC, matching Title+Abstract keyphrases). Faithfulness and consistency are measured both through keyphrase grounding ratios and GPT-4o structured JSON scoring rubrics. Citation correctness tracks identifier precision and recall alongside judge-evaluated contextual alignment. Finally, fluency is assessed via GPT-2 Perplexity and human evaluation across ten qualitative criteria on 80 model outputs.
Benchmark & Evaluation Protocol¶
The benchmark evaluates five state-of-the-art models released prior to or near January 2025: LLaMA 4-Maverick (17B active parameters, 1M context), DeepSeek-V3-0324 (671B MoE), Mistral-Small-3.1-24B, Gemma-3-12B-it, and GPT-4o. Generation across all models uses fixed nucleus decoding hyperparameters: temperature \(T=0.7\), top-\(p=0.95\), and fixed random seeds. For human evaluation, 20 source papers were annotated across four model outputs (LLaMA 4-Maverick 0-shot, 3-shot, AutoCoT, and GPT-4o 0-shot), totaling 80 samples rated on a 1–5 scale across ten fine-grained criteria.
Key Experimental Results¶
Main Results¶
Performance of the five evaluated models on NAACL 2025 (800 samples) and ICLR 2025 (3,100 samples) under the Elaborate prompting strategy (lower Perplexity indicates superior fluency):
| Dataset | Model | ROUGE-1 | ROUGE-L | Contextual Relevance | RBC (Coverage) | Citation Precision | Perplexity (↓) | Judge Narrative |
|---|---|---|---|---|---|---|---|---|
| NAACL 2025 | DeepSeek-V3 | 0.4161 | 0.1649 | 0.9691 | 0.4435 | 0.8819 | 28.5031 | 0.8892 |
| Gemma-3-12B | 0.4361 | 0.1725 | 0.9684 | 0.4513 | 0.9049 | 32.3897 | 0.8888 | |
| LLaMA 4-Maverick | 0.4402 | 0.1765 | 0.9694 | 0.4845 | 0.8889 | 18.4870 | 0.9001 | |
| Mistral-Small-3.1 | 0.4388 | 0.1754 | 0.9696 | 0.4692 | 0.9097 | 25.6458 | 0.8928 | |
| GPT-4o | 0.4282 | 0.1531 | 0.9687 | 0.4713 | 0.8686 | 21.8494 | 0.9004 | |
| ICLR 2025 | DeepSeek-V3 | 0.3863 | 0.1700 | 0.9696 | 0.4461 | 0.8965 | 36.9546 | 0.8835 |
| Gemma-3-12B | 0.4342 | 0.1715 | 0.9700 | 0.4732 | 0.8822 | 29.1004 | 0.8899 | |
| LLaMA 4-Maverick | 0.4298 | 0.1689 | 0.9707 | 0.4794 | 0.8942 | 21.1194 | 0.8986 | |
| Mistral-Small-3.1 | 0.4326 | 0.1894 | 0.9713 | 0.4725 | 0.9357 | 29.2585 | 0.8907 | |
| GPT-4o | 0.4154 | 0.1830 | 0.9729 | 0.4697 | 0.9238 | 22.0046 | 0.9006 |
Ablation Study¶
Ablation on input context configurations and prompting strategies using LLaMA 4-Maverick on the NAACL 2025 dataset:
| Prompt Strategy | Context Input | ROUGE-1 | BERTScore | Contextual Relevance | RBC (Coverage) | Citation Recall | Citation Precision | Perplexity (↓) |
|---|---|---|---|---|---|---|---|---|
| Elaborate | Title-only | 0.3545 | 0.8324 | 0.9651 | 0.3918 | 0.0918 | 0.2399 | 14.5978 |
| Title + Abstract | 0.4093 | 0.8408 | 0.9657 | 0.4805 | 0.1199 | 0.3610 | 19.9132 | |
| T + A + Related | 0.4402 | 0.8448 | 0.9684 | 0.4845 | 0.5574 | 0.9357 | 18.4870 | |
| Medium | Title-only | 0.3356 | 0.8235 | 0.9633 | 0.3878 | 0.0001 | 0.0012 | 16.6799 |
| Title + Abstract | 0.3949 | 0.8320 | 0.9650 | 0.4715 | 0.0039 | 0.0175 | 20.7426 | |
| T + A + Related | 0.4513 | 0.8471 | 0.9690 | 0.4697 | 0.6193 | 0.8548 | 26.2687 | |
| Few-Shot (T+A+R) | Zero-Shot (Elaborate) | 0.4402 | 0.8448 | 0.9684 | 0.4845 | 0.5574 | 0.9357 | 18.4870 |
| One-Shot | 0.4418 | 0.8486 | 0.9677 | 0.4935 | 0.5930 | 0.9506 | 19.1426 | |
| Two-Shot | 0.4440 | 0.8496 | 0.9674 | 0.4913 | 0.6083 | 0.9470 | 23.0976 | |
| Three-Shot | 0.4460 | 0.8492 | 0.9681 | 0.4985 | 0.6076 | 0.9515 | 19.6214 |
Key Findings¶
- Essentiality of Related Works Context: Omitting related works collapses citation precision and recall (recall drops from 0.5574 to 0.1199 under Elaborate; precision drops from 0.9357 to 0.3610), proving that LLMs cannot accurately anticipate or cite external scholarship without explicit structured bibliographic inputs.
- Superiority and Diminishing Returns of Few-Shot Prompting: Three-Shot prompting achieves the best performance across lexical overlap and human qualitative dimensions (e.g., flow of ideas 4.60 vs. 4.30 in zero-shot, methods summary 4.25 vs. 3.65). However, gains plateau sharply after Two-Shot, indicating that 2–3 demonstrations provide optimal in-context stylistic priming without incurring excessive token costs.
- The Reasoning–Fluency Dilemma in AutoCoT: While AutoCoT significantly boosts keyphrase faithfulness (0.3610 vs. 0.2393 in Elaborate), it increases GPT-2 perplexity and degrades human evaluation ratings (overall faithfulness 4.11 vs. 4.70 in 3-shot). Unconstrained internal reasoning steps disrupt the seamless narrative progression demanded by academic prose.
- Contribution Summaries Reverting to Conclusions: Qualitative human auditing of 80 outputs reveals that without access to full paper methodology and experimental sections, models consistently produce generic, conclusion-like summaries rather than precise, bullet-style contribution breakdowns, failing to recover nuanced research questions present in author-written ground truths.
Highlights & Insights¶
- Clean Post-Cutoff Benchmarking Design: By deliberately selecting accepted papers from NAACL 2025 and ICLR 2025, the authors eliminate the pervasive risk of training set contamination that invalidates older scientific summarization datasets.
- Theory-Grounded Input Minimization: Grounding input selection in Swales' CARS rhetorical model allows minimal input tokens \((T, A, R)\) to effectively capture full introduction semantics, ensuring the benchmark remains highly reproducible and cost-efficient.
- Cross-Metric Correlation Revelations: Spearman rank correlation analysis demonstrates moderate alignment between LLM-as-a-judge and formulaic metrics (\(\rho=0.30\) to \(0.45\)), but reveals a negative correlation in citation quality (\(\rho=-0.19\)), emphasizing that rigid n-gram citation matching over-penalizes valid alternative citations that the LLM judge correctly recognizes as contextually appropriate.
Limitations & Future Work¶
- Absence of Deeper Methodological and Experimental Context: Restricting inputs to Title, Abstract, and Related Works deliberately excludes full Methods and Results sections to maintain scalability, preventing models from articulating fine-grained numerical findings or specific algorithmic nuances.
- Omission of Scientific Figure Generation: The benchmark focuses exclusively on text, leaving out the generation of architectural diagrams, graphical abstracts, and motivation figures, which represent essential non-natural image reasoning challenges in academic writing.
- Privacy Trade-Offs in Proprietary Model Usage: While commercial models like GPT-4o achieve competitive fluency, transmitting unpublished academic manuscripts to third-party endpoints poses confidentiality and intellectual property risks, underscoring the urgent need to advance performant local open-weight models like LLaMA 4-Maverick.
Related Work & Insights¶
- vs Multi-Document Summarization (PRIMERA, Nandy & Bandyopadhyay 2025): While MDS models distill multi-source documents into unbiased, symmetric summaries, SciIG requires argumentative asymmetry—framing prior work solely to establish research gaps that the target paper's novel contributions explicitly resolve.
- vs Automated Scientific Survey Generation (AutoSurvey, SurveyGen): Survey generation prioritizes broad thematic coverage and taxonomical categorization across dozens of papers, whereas SciIG operates under strict word budgets (600–800 words), demanding tightly focused rhetorical progression from broad background to concrete contributions.
- vs Interactive AI Authoring Systems (Liebling et al. 2025): Interactive systems focus on citation recommendation and in-line suggestions within a human-in-the-loop workflow, whereas SciIG establishes a standardized, reproducible, single-pass generation benchmark with fixed contextual inputs.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Pioneering benchmark for standalone scientific introduction generation with clean post-cutoff data]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluation across 5 models, 7 prompting strategies, and 30 automated/LLM/human metrics]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary academic structure, comprehensive mathematical formulations, and thorough error analysis]
- Value: ⭐⭐⭐⭐⭐ [Provides indispensable practical guidelines and realistic capability boundaries for AI-assisted research writing]