Skip to content

ClaimFlow: Tracing the Evolution of Scientific Claims in NLP

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/UKPLab/emnlp2026-claim-flow
Area: LLM (Other)
Keywords: scientific claim evolution / citation analysis / claim relation classification / scholarly document mining / scientific knowledge graph

TL;DR

This paper introduces ClaimFlow, the first claim-centric benchmark and dataset tracing cross-paper scientific interactions across 1,617 expert-annotated NLP papers, formalizes the Claim Relation Classification task, and constructs an automated evolution graph spanning ~13k ACL Anthology papers to quantitatively reveal how scientific ideas propagate, face early vetting, and mature primarily through extension and qualification.

Background & Motivation

Scientific research progresses through the continuous formulation, empirical testing, and accumulation of claims. Across computational linguistics and natural language processing (NLP), paradigms have undergone dramatic transformations over recent decadesโ€”transitioning from rule-based symbolic parsing to feature-engineered statistical models, contextualized representations, and modern large language models (LLMs). Yet, existing bibliometric and scholarly mining frameworks predominantly operate at the paper-level citation network or rely on surface-level keyword trends and coarse citation intent categories (such as background, methodology use, or result comparison). These coarse abstractions treat entire publications as atomic units, recording that an interaction took place without capturing whether subsequent research validated, expanded, constrained, or overturned a specific empirical assertion.

This granularity mismatch has left the community without systematic tools to trace how scientific knowledge shifts and matures. Philosophers of science, including Karl Popper and Imre Lakatos, have long emphasized that epistemic progress is propelled by the continuous scrutiny and refinement of testable hypotheses. In academic literature, later authors explicitly position their contributions relative to earlier findings; however, the necessary evidence is distributed across localized citation contexts and the novel assertions made within the citing manuscript. Conventional citation intent analysis lacks proposition-level alignment and cannot distinguish between merely adopting a baseline model versus validating or contesting its fundamental claims. Conversely, existing scientific fact-checking and argument mining datasets largely evaluate isolated sentences or within-document argumentation, omitting the dynamic, cross-paper temporal dialogue through which scientific claims evolve.

To bridge this fundamental gap, this work advocates shifting the analytical primitive from papers to scientific claims. Core Idea: explicitly model cross-paper scientific discourse as a directed graph of empirical claims interconnected by epistemic stances (support, extend, qualify, refute, background), and construct an expert-annotated benchmark alongside a tripartite classification framework to examine the macro-level propagation and maturation of NLP research over four decades.

Method

Overall Architecture

ClaimFlow formalizes, benchmarks, and automatically recovers cross-paper claim interactions across the scholarly literature. The framework consists of four sequential stages: first, defining scientific claims as canonical, testable empirical assertions and establishing a five-class taxonomy of cross-paper epistemic relations; second, framing Claim Relation Classification as a supervised multi-class problem over a structured tripartite input; third, assembling an end-to-end automated pipeline (incorporating DeBERTa claim identification, greedy semantic canonicalization, and few-shot relation inference) to construct ClaimFlow-AutoGraph over 13,358 ACL Anthology papers; and finally, evaluating structural topology, survival curves, and exposure-normalized influence metrics across publication decades.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["ACL Anthology Corpus<br/>(1979โ€“2025 Parsed Text)"] --> B["Stage 1: Claim Identification & Normalization<br/>Extract testable assertions & merge paper-level variants"]
    B --> C["Stage 2: Tripartite Citation Evidence Alignment<br/>(Cited Claim, Citation Context, Citing Claim)"]
    C --> D["Stage 3: Epistemic Relation Classification & Graph Construction<br/>Infer support/extend/qualify/refute/background"]
    D --> E["Stage 4: Longitudinal Topological Evolution Analysis<br/>Density, modularity & exposure-normalized survival"]

Key Designs

1. Scientific Claim Ontology and Five-Class Epistemic Taxonomy: Delineating Fine-Grained Engagement Rather than cataloging rhetorical citation habits, ClaimFlow anchors its ontology on empirical testability. A scientific claim is defined as a central assertion regarding methods, datasets, or empirical phenomena advanced as a primary contribution of a paper. To distinguish core propositions from surface linguistic variations, textual statements across abstract, introduction, and conclusion sections are normalized into canonical claims. Cross-paper interactions are categorized into five mutually exclusive epistemic relations: - SUPPORT: The citing claim provides empirical evidence or findings that directly reinforce and are consistent with the cited claim; - EXTEND: The citing claim applies the cited claim to a new task, domain, dataset, or architectural variant while preserving its foundational assertion; - QUALIFY: The citing claim identifies boundary conditions, performance limitations, or failure modes that narrow the scope of the cited claim without invalidating it; - REFUTE: The citing claim offers contradicting empirical evidence, failure cases, or counter-arguments that directly challenge the cited claim; - BACKGROUND: The cited claim is referenced for context, comparative positioning, or historical lineage without empirical evaluation or modification. This taxonomy decouples scientific stance from mere rhetorical polarity, establishing a rigorous foundation for tracking idea maturation.

2. Tripartite Evidence Alignment Paradigm: Grounding Discursive Context and Empirical Assertion Accurately determining epistemic stance cannot rely on isolated claim pairs or citation sentences alone. Authors frequently express critique or incremental adaptation using subtle linguistic hedging and indirect discourse within local citation sentences; meanwhile, the overarching empirical novelty of the citing paper resides outside the citation sentence itself. To resolve this ambiguity, the task is formulated over a unified tripartite input \((h_{\text{cited}}, s, h_{\text{citing}})\), where \(h_{\text{cited}}\) is the cited claim from the earlier paper, \(h_{\text{citing}}\) is the novel claim asserted by the citing paper, and \(s\) is the local citation context comprising the citation sentence along with its immediately preceding and succeeding sentences. Concatenating and jointly encoding these components enables the model to leverage local rhetorical markers while simultaneously comparing the empirical scopes of the two scientific propositions.

3. Cascaded ClaimFlow-AutoGraph Construction Pipeline: Mitigating Redundancy and Error Compounding To scale from the gold benchmark of 1,617 papers (5,689 claims and 4,871 relations) to the entire corpus spanning 46 years, the authors engineered a robust three-stage automated synthesis pipeline: - Sentence-Level Identification: A DeBERTa-v3 model fine-tuned on ClaimFlow extracts candidate claim spans from unannotated manuscripts; - Lightweight Semantic Canonicalization: Papers frequently restate core contributions across abstracts, introductions, and conclusions. Candidate claims within the same document are clustered greedily based on pairwise sentence embeddings using a cosine similarity threshold of \(\tau = 0.90\). The longest textual realization is retained as the cluster centroid, compressing claim nodes by 11% with a high cluster purity of 0.92, thus preventing spurious multi-edge inflation; - Few-Shot Relation Inference: Candidate citingโ€“cited claim pairs connected by explicit citation links are classified using GPT-4.1 under a 4-shot prompt with an explicit irrelevant fallback option to filter out non-substantive co-citations.

4. Topological Dynamics and Exposure-Normalized Influence Modeling: Correcting for Citation Maturation Bias To evaluate the true historical imprint of scientific assertions independently of chronological age, the authors formulated normalized influence metrics that correct for temporal publication accumulation: $\(\text{AgeRank}(h) = \frac{|\{h' \in \mathcal{H} : y(h') < y(h)\}|}{|\mathcal{H}|}\)$ $\(\text{NormInfluence}(h) = \frac{\#\{\text{papers citing } h\}}{\#\{\text{papers published after } y(h)\}}\)$ where \(y(h)\) denotes the publication year of claim \(h\). NormInfluence computes the ratio of unique citing papers to the entire post-publication literature volume, neutralizing the cumulative advantage enjoyed by early publications. Coupled with global edge density \(\delta(G_t) = \frac{|E_t|}{|V_t|(|V_t|-1)}\) and community modularity \(Q(G_t)\), this quantitative lens tracks the macro-level transition of the NLP field from task-specific divergence to foundation-model convergence.

A Worked Example

Consider the evolutionary trajectory of the landmark evaluation claim for BLEU (Papineni et al., 2002): 1. Tripartite Input Configuration: - Cited Claim \(h_{\text{cited}}\): "BLEU provides an automatic machine translation metric that correlates highly with human evaluation." (2002) - Citing Claim \(h_{\text{citing}}\): "Existing standard metrics such as BLEU may fail to adequately flag numerical translation errors." (2021) - Citation Context \(s\): "In our behavioral testing, slight perturbations in numerals alter sentence semantics drastically without severely penalizing n-gram overlap. While BLEU remains widely utilized for general MT evaluation [Papineni et al., 2002], it exhibits substantial blind spots when critical numerical accuracy is required." 2. Inference Flow: - The encoder identifies the concessive discourse marker ("While... it exhibits substantial blind spots"); - Cross-comparing \(h_{\text{cited}}\) and \(h_{\text{citing}}\) demonstrates that the citing authors do not refute BLEU's general utility in standard MT, but restrict its validity in numerical translation settings; - Output Label: QUALIFY.

Key Experimental Results

Main Results

Evaluated on the stratified, paper-level partitioned ClaimFlow test set (70-15-15 split to prevent citation cross-leakage), the benchmark compares supervised fine-tuned encoders and few-shot prompted LLMs:

Model Setting Architecture / Paradigm Precision Recall Macro-F1
Majority Baseline Class frequency prior 0.18 0.20 0.18
Fine-Tuned Encoder BERT 0.51 0.50 0.50
Fine-Tuned Encoder RoBERTa 0.73 0.71 0.70
Fine-Tuned Encoder DeBERTa-v3 0.76 0.75 0.75
Fine-Tuned Encoder SciBERT 0.77 0.76 0.76
Prompted (4-shot) LLaMA-3-70B-Instruct 0.71 0.70 0.70
Prompted (4-shot) Mixtral-8x7B-Instruct 0.73 0.74 0.73
Prompted (4-shot) GPT-3.5-turbo 0.75 0.75 0.75
Prompted (4-shot) GPT-4o 0.78 0.78 0.78
Prompted (4-shot) GPT-4.1 0.82 0.81 0.81

Ablation Study

The evaluation examines the contribution of individual input elements in the tripartite tuple, alongside pipeline-stage validation on ClaimFlow-AutoGraph:

Input Component Ablation BERT (F1) RoBERTa (F1) SciBERT (F1) DeBERTa (F1) Observation
Citation Context Only (\(s\)) 0.44 0.46 0.51 0.54 Misses core propositional assertions
Cited Claim + Context (\((h_{\text{cited}}, s)\)) 0.45 0.50 0.56 0.60 Omits citing paper's empirical boundary
Full Tripartite Tuple (\((h_{\text{cited}}, s, h_{\text{citing}})\)) 0.50 0.70 0.75 0.76 Jointly captures claim comparison & discourse
Pipeline Stage Validation (AutoGraph) Precision Recall Macro-F1 Observation
Claim Identification 0.83 0.82 0.82 Sentence-level claim extraction
Relation Classification 0.82 0.81 0.81 Evaluated on gold claim-context pairs
End-to-End Graph Recovery 0.81 0.79 0.80 Full cascade retains high fidelity (drop of 0.01 F1)

Key Findings

  1. Critical Role of Tripartite Grounding: Omitting the citing claim causes DeBERTa's macro-F1 to plummet from 0.76 to 0.60; relying exclusively on the citation context further degrades performance to 0.54. This empirical drop confirms that sentence-level citations cannot substitute for explicit claim representations.
  2. Pervasive Claim Isolation and Front-Loaded Vetting: Within the 13k-paper AutoGraph, 63.5% of claims remain completely isolated (never reused by later work), with a modest mean propagation count of 1.37 across the corpus. Furthermore, only 11.1% of all claims ever face challenges (8.3% via qualification and a mere 2.8% via direct refutation). Survival analysis demonstrates that the median time to first challenge is just 2 years. Once a claim survives this initial inspection window, its cumulative uncertainty steadily declines as it becomes embedded as standard background knowledge.
  3. Cumulative, Non-Adversarial Maturation: Following an initial challenge, 46.3% of claims receive no subsequent substantive engagement, and 13.2% revert to purely neutral background citations. Among claims that continue to be actively engaged, further extension (13.8%) and support (11.1%) substantially outnumber repeated qualification (10.4%) or continued refutation (5.2%). Influential trajectories such as BERT and BLEU exemplify how foundational claims mature through progressive scope qualification and domain extension rather than direct replacement.
  4. Graph Modularity Dynamics Reflect Paradigm Shifts: Global edge density consistently remains around \(10^{-3}\). Graph modularity steadily rose through the 1990s and 2000s, reflecting intense specialization into sub-communities (parsing, MT, information extraction). However, modularity peaked and noticeably declined in recent years, demonstrating that universal pre-trained representations and foundation models have cross-connected previously fragmented sub-fields.

Highlights & Insights

  • Fine-Grained Claim-Centric Perspective: Moves beyond paper-as-a-unit bibliometrics, delivering the first long-horizon, proposition-level knowledge graph spanning 46 years of NLP history.
  • High-Quality Expert Dual-Annotation: Developed through exhaustive double-blind annotation by domain experts, achieving a solid inter-annotator agreement (Cohen's \(\kappa = 0.77\)) on subtle epistemic relations.
  • Empirical Refutation of Perpetual Scientific Revolutions: Provides large-scale quantitative evidence that NLP development is predominantly characterized by early selection and progressive, non-confrontational consolidation rather than sustained Popperian falsification debates.

Limitations & Future Work

  • Corpus and Venue Boundary: The study is restricted to the ACL Anthology (ACL, EMNLP, NAACL, EACL, etc.), omitting multidisciplinary AI venues (NeurIPS, ICML, ICLR) and preprint archives (arXiv), which may influence observed propagation speeds.
  • Section-Level Claim Truncation: Annotations and extractions were confined to paper abstracts, introductions, and conclusions, potentially overlooking granular technical assertions embedded within method and experimental bodies.
  • Boundary Ambiguity Between Extension and Qualification: Qualitative error analysis reveals that ~30% of model misclassifications involve confounding EXTEND and QUALIFY, particularly when an empirical study broadens an application domain while introducing task-specific constraints.
  • vs. Citation Intent Classification (e.g., SciCite, Cohan et al., 2019): Traditional citation intent schemes categorize rhetorical functions (e.g., Method vs. Background). ClaimFlow delves deeper into epistemic stance, determining whether a prior factual finding was validated, extended, qualified, or refuted.
  • vs. Scientific Fact-Checking (e.g., SciFact, Wadden et al., 2020): Fact-checking benchmarks typically perform static binary truth assessments on isolated sentences against external evidence. ClaimFlow tracks dynamic, temporally situated interactions among assertions produced by the evolving research community.

Rating

  • Novelty: โญโญโญโญโญ Pioneering claim-centric conceptualization and formalization of cross-paper epistemic relations.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous expert annotation, comprehensive encoder and LLM evaluation, cascaded validation, and rich macro-scale longitudinal case studies.
  • Writing Quality: โญโญโญโญโญ Exceptionally articulated, conceptually mature, and deeply grounded in the philosophy of science.
  • Value: โญโญโญโญโญ Provides invaluable infrastructure for literature discovery, autonomous scientific agents, research synthesis, and scientometrics.