Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ryabhmd/SciFactCheck
Area: NLP Understanding
Keywords: Scientific Factuality, Hallucination Evaluation, Domain Fine-Tuning, Overclaiming, Attribution Hallucination
TL;DR¶
Introduces the SCIFACTCHECK benchmark spanning 2,500 concepts across five scientific domains and a fine-grained evaluation framework targeting unverifiability, overclaiming, and attribution, empirically demonstrating via controlled minimal-pairing that scientific fine-tuning degrades factual reliability and exacerbates a dissociation where models exhibit higher linguistic certainty despite lower internal parametric confidence.
Background & Motivation¶
Large language models are increasingly deployed to explain complex scientific concepts, assist literature exploration, and automate scientific discoveries, making factual accuracy critical in these high-stakes applications. However, generative LLMs are notoriously prone to hallucinations, particularly on long-tail specialized knowledge and in long-form open-ended generations. While hallucination detection has evolved rapidly in general NLP, existing scientific evaluation benchmarks remain largely confined to the biomedical domain and treat factuality as a binary, monolithic classification task, ignoring the rich diversity of error types and the epistemic differences across scientific disciplines.
A widespread assumption in the field has been that adapting general-purpose foundation models on large-scale scientific corpora—via continual pre-training or supervised fine-tuning—naturally injects domain knowledge and mitigates factual confabulations. This assumption has fueled the release of specialized scientific LLMs such as SciLitLLM, SciTulu, and S1-Base. Yet, this premise has never been rigorously scrutinized under controlled experimental conditions where scientific models are directly benchmarked against their exact base models across multiple distinct failure modes and academic domains.
This paper addresses this fundamental gap by establishing an open-ended closed-book benchmark derived from community-validated scientific consensus and systematically decoupling content correctness, rhetorical stance, and citation provenance. Core idea: evaluate 18 language models across five scientific disciplines on the newly constructed 2,500-concept benchmark SCIFACTCHECK using a controlled minimal-pairing design, systematically decoupling unverifiability (VER), overclaim (OVR), and attribution (ATR) errors to uncover how domain-specific scientific fine-tuning alters factual reliability and epistemic uncertainty.
Method¶
Overall Architecture¶
The evaluation framework comprises four primary components: benchmark concept extraction grounded in peer-reviewed literature across five scientific domains; a two-stage closed-book generation protocol instructing models to produce a 200-word explanatory paragraph and subsequent APA-formatted citations; an automatic evaluation framework deploying dedicated metrics for unverifiability (OpenFActScore and VeriScore), overclaiming (sentence-level linguistic certainty and length-normalized parametric log probabilities), and attribution hallucinations (CiteVal); and an expert human validation pilot study auditing atomic claim check-worthiness and citation validity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Curated Scholarly Knowledge Bases<br/>S2ORC Highly Cited Reviews (≥100 Citations) + IEP"] --> D1["Multi-Domain Benchmark Construction<br/>2,500 Concepts across 5 Fields with Two-Stage Closed-Book Prompting"]
D1 --> D2["Modular Factuality Hallucination Framework<br/>Decoupling Unverifiability (VER), Overclaim (OVR), and Attribution (ATR)"]
D2 --> D3["Dual-Faceted Uncertainty Analysis<br/>Measuring Sentence Linguistic Certainty vs. Sequence Parametric Logprobs"]
D3 --> D4["Expert Check-Worthiness and Attribution Audit<br/>Bilingual Human Validation across Domains and Matching Thresholds"]
D4 --> Output["18-Model Empirical Findings<br/>Factual Reliability Degradation and Epistemic Dissociation"]
Key Designs¶
1. Multi-Domain Benchmark Construction: Domain Narrowness and Subjectivity in Prior Datasets -> Grounding 2,500 Concepts in Highly Cited Survey Papers and Encyclopedias -> Enabling Balanced Cross-Disciplinary Closed-Book Evaluation Prior hallucination datasets focus overwhelmingly on biomedicine or Wikipedia biographies, failing to capture the full spectrum of scholarly knowledge. The authors map disciplines using the Open Research Knowledge Graph (ORKG) taxonomy into five domains: Engineering, Life Sciences, Physical Sciences & Mathematics, Social & Behavioural Sciences, and Arts & Humanities. For the first four empirical domains, review articles with at least 100 citations are retrieved from the Semantic Scholar Open Research Corpus (S2ORC) to ensure established consensus, and concepts are semi-automatically extracted using Gemini-2.5-Flash with manual verification. For Arts & Humanities, 500 philosophical concepts are sampled from the Internet Encyclopedia of Philosophy (IEP). To evaluate generation without confounding retrieval biases, models are queried with a zero-shot, two-stage closed-book prompt: first producing a strictly factual 200-word paragraph about the concept (with study names and dates forbidden to control claim volume), followed by generating APA-formatted citations supporting the claims.
2. Modular Factuality Hallucination Framework: Conflation of Distinct Factual Error Modes -> Decoupling Verification, Rhetorical Overstatement, and Attribution into Dedicated Metrics -> Providing Granular Diagnosis of Model Failure Modes Rather than collapsing all factual mistakes into a single accuracy score, the framework isolates three content verifiability error types. Unverifiability (VER) captures claims unsupported by trustworthy literature, evaluated through OpenFActScore (OFS, retrieving paragraphs from the source survey paper via GTR-T5 and verifying via Gemma-7B-it with a length penalty \(\gamma=10\)) and VeriScore (extracting claims and querying web documents via Serper API, scored using an \(F_1@K\) metric anchored to the median claim count \(K=19\)). Attribution (ATR) evaluates fabricated bibliographic citations via CiteVal, an automated pipeline that parses APA references and verifies DOIs and titles against the CrossRef database using fuzzy string matching (threshold \(\alpha=0.9\)). To prevent models from artificially inflating scores by hallucinating hundreds of references, CiteVal incorporates an \(F_1@M\) formulation anchored to the cross-model median citation count \(M=5\): $\(\text{Precision} = \frac{v}{n}, \quad \text{Recall@}M = \min\left(\frac{v}{M}, 1\right)\)$ $\(\text{CiteVal} = F_1@M = \frac{2 \cdot \text{Precision} \cdot \text{Recall@}M}{\text{Precision} + \text{Recall@}M}\)$ where \(n\) is total generated citations and \(v\) is verified valid citations, enforcing equal coverage expectations across all models.
3. Dual-Faceted Uncertainty Analysis: Disconnection Between Model Confidence and Textual Tone -> Pairing Sentence-Level Linguistic Assertion with Token-Level Parametric Logprobs -> Exposing Rhetorical Masking of Internal Epistemic Uncertainty Overclaiming (OVR) occurs when scientific statements exaggerate the certainty, exclusivity, or generalizability of underlying findings. Because dedicated automatic metrics for overclaiming in open-ended text do not exist, the framework operationalizes this dimension via two complementary uncertainty proxies. Linguistic Certainty is calculated using an established neural certainty estimator that rates each sentence on a 1–6 scale based on lexical hedging versus confident assertion. Parametric Certainty evaluates the model's internal statistical confidence via length-normalized token log probabilities: $\(\text{Parametric Certainty} = \frac{1}{|Y|} \sum_{t=1}^{|Y|} \log P(y_t \mid y_{<t}, x)\)$ By contrasting these two axes across matched model pairs, the evaluation explicitly reveals whether models learn genuine epistemic grounding or merely acquire the superficial assertive tone typical of academic writing.
4. Expert Check-Worthiness and Attribution Audit: Imperfect Tool Alignment with Domain Expertise -> Multi-Domain Human Pilot Study Validating Metric Reliability and Claim Check-Worthiness -> Establishing Empirical Grounding and Exposing Infrastructure Limits To benchmark the validity of the automated pipeline, the authors conduct a human validation pilot study with 10 annotators holding scientific degrees across all five target domains. For VER, 958 atomic claims sampled from GPT-4o-mini and Sonar are annotated under a two-step protocol: first judging whether each claim is scientifically check-worthy, and second judging factual correctness against reference survey papers and web search snippets. For ATR, 115 citations extracted from SciTulu-7b and Llama-3.1-8B are manually validated against CrossRef, and fuzzy-matching thresholds (\(\alpha \in \{0.8, 0.9, 0.95\}\)) are audited. The threshold \(\alpha=0.9\) achieves an optimal balance between precision (0.79) and recall (0.91), yielding an overall accuracy of 0.90 and an \(F_1\) score of 0.85 against expert ground truth.
Experimental Setup¶
The study evaluates 18 LLMs comprising 7 tightly controlled general-versus-scientific pairs spanning four open-source families—Qwen2.5 (7B/14B) vs. SciLitLLM1.5, Tulu-2 (7B/70B) vs. SciTulu, Qwen3 (8B/32B) vs. S1-Base, and Llama-3.1-8B vs. OpenScholar-8B (a RAG-tuned model)—alongside proprietary baselines (GPT-4o-mini and Perplexity Sonar). Inference is standardized with sampling temperature \(\tau=0.3\), repetition penalty 1.1, and maximum output length 1024 tokens. Group comparisons employ two-sided Mann-Whitney U tests, while prompt-level paired evaluations across all 2,500 concepts employ Wilcoxon signed-rank tests with Cohen's \(d\) effect sizes.
Key Experimental Results¶
Main Results¶
The main evaluation results across all 18 models and error dimensions (aggregated across all five scientific domains) are presented below:
| Model Family & Variant | OFS (↑) | VeriScore (K=19, ↑) | Ling. Cert. (1-6) | Param. Cert. (logprob) | CiteVal (M=5, α=0.9, ↑) |
|---|---|---|---|---|---|
| Proprietary Models | |||||
| gpt-4o-mini | 0.661 ± .22 | 0.728 ± .23 | 4.801 ± .16 | – | 0.257 ± .26 |
| perplexity-sonar | 0.580 ± .22 | 0.548 ± .24 | 4.901 ± .08 | – | 0.451 ± .30 |
| Llama-2 & Tulu-2 Family | |||||
| Llama-2-7b-chat | 0.614 ± .24 | – | 4.747 ± .17 | -0.080 ± .02 | 0.173 ± .24 |
| └ tulu-2-7b | 0.627 ± .23 | – | 4.765 ± .19 | -0.166 ± .05 | 0.059 ± .15 |
| └ scitulu-7b | 0.632 ± .23 | – | 4.821 ± .16 | -0.228 ± .05 | 0.013 ± .07 |
| Llama-2-70b-chat | 0.622 ± .21 | – | 4.776 ± .16 | -0.098 ± .02 | 0.227 ± .27 |
| └ tulu-2-70b | 0.656 ± .21 | 0.662 ± .22 | 4.793 ± .17 | -0.205 ± .04 | 0.300 ± .28 |
| └ scitulu-70b | 0.655 ± .22 | 0.653 ± .22 | 4.802 ± .18 | -0.169 ± .04 | 0.177 ± .21 |
| Llama-3.1 Family | |||||
| Llama-3.1-8B-Instruct | 0.650 ± .22 | 0.604 ± .23 | 4.787 ± .17 | -0.360 ± .06 | 0.400 ± .32 |
| └ OpenScholar-8B | 0.628 ± .23 | 0.629 ± .23 | 4.793 ± .17 | -0.374 ± .06 | 0.327 ± .31 |
| Qwen2.5 Family | |||||
| Qwen2.5-7B-Instruct | 0.652 ± .22 | 0.594 ± .22 | 4.782 ± .19 | -0.116 ± .03 | 0.250 ± .28 |
| └ SciLitLLM1.5-7B | 0.598 ± .23 | 0.549 ± .23 | 4.747 ± .21 | -0.173 ± .05 | 0.212 ± .28 |
| Qwen2.5-14B-Instruct | 0.667 ± .21 | – | 4.807 ± .16 | -0.114 ± .03 | 0.323 ± .31 |
| └ SciLitLLM1.5-14B | 0.604 ± .23 | – | 4.791 ± .20 | -0.105 ± .04 | 0.278 ± .29 |
| Qwen3 Family | |||||
| Qwen3-8B | 0.671 ± .21 | – | 4.850 ± .14 | -0.097 ± .03 | 0.274 ± .26 |
| └ S1-Base-8B | 0.605 ± .22 | – | 4.912 ± .10 | -0.141 ± .02 | 0.164 ± .20 |
| Qwen3-32B | 0.660 ± .20 | – | 4.833 ± .14 | -0.149 ± .03 | 0.373 ± .29 |
| └ S1-Base-32B | 0.608 ± .21 | – | 4.853 ± .13 | -0.108 ± .02 | 0.231 ± .27 |
Controlled Statistical Comparisons¶
Mann-Whitney U group tests (General vs. Scientific) and prompt-level paired Wilcoxon signed-rank tests (\(N = 2,500\)) confirm systematic factuality degradation:
| Comparison & Evaluation Metric | General Median | Scientific Median | Test Statistic (\(U\) or \(W\)) | \(p\)-value | Cohen's \(d\) | Statistical Takeaway |
|---|---|---|---|---|---|---|
| Global Group: General vs. Scientific | ||||||
| OpenFActScore (OFS) | 0.667 | 0.632 | \(U = 183,667,700.5\) | \(<0.001\) | \(+0.15\) | General models significantly more factual |
| VeriScore (\(F_1@19\)) | 0.647 | 0.600 | \(U = 20,178,132.0\) | \(<0.001\) | \(+0.13\) | General models significantly more factual |
| Linguistic Certainty | 4.837 | 4.869 | \(U = 147,495,607.0\) | \(<0.001\) | \(-0.15\) | Scientific models write more assertively |
| Parametric Certainty | -0.125 | -0.154 | \(U = 208,606,795.0\) | \(<0.001\) | \(+0.36\) | Scientific models are internally less confident |
| CiteVal (\(F_1@5\)) | 0.182 | 0.000 | \(U = 190,622,048.5\) | \(<0.001\) | \(+0.22\) | Scientific models fabricate more citations |
| Paired Test: Qwen2.5-7B vs SciLitLLM-7B | ||||||
| OFS / VeriScore | 0.684 / 0.611 | 0.600 / 0.556 | \(W = 1,053,141.5\) | \(<0.001\) | \(+0.24\) / \(+0.20\) | Substantial drop in factual claim accuracy |
| Parametric Certainty | -0.114 | -0.168 | \(W = 163,798.0\) | \(<0.001\) | \(+1.50\) | Severe deterioration in model token certainty |
| CiteVal | 0.167 | 0.000 | \(W = 668,373.0\) | \(<0.001\) | \(+0.14\) | Valid citation rate plummets to median zero |
| Paired Test: Qwen3-8B vs S1-Base-8B | ||||||
| OFS / CiteVal | 0.706 / 0.200 | 0.630 / 0.118 | \(W = 866,699.5\) | \(<0.001\) | \(+0.30\) / \(+0.47\) | Base model significantly dominates on both |
| Ling. Cert. vs. Param. Cert. | 4.892 / -0.095 | 4.940 / -0.141 | \(W = 704,687 / 77,100\) | \(<0.001\) | \(-0.53\) / \(+1.80\) | Sharp dissociation of linguistic tone and logprobs |
Key Findings¶
- Scientific fine-tuning systematically impairs factuality across dimensions: Across all evaluated pairs, scientific models underperform their general-purpose counterparts in OFS, VeriScore, and CiteVal (all \(p < 0.001\)). Citation validity experiences the steepest degradation: median CiteVal drops to zero in several scientific models (e.g., SciTulu-70b falls from 0.300 to 0.177, SciLitLLM-7B drops from 0.167 to 0.0).
- Severe epistemic dissociation between linguistic tone and internal confidence: Scientific models exhibit higher linguistic certainty alongside substantially lower parametric certainty (\(d = +0.36\)). Fine-tuning appears to teach models the assertive stylistic conventions of academic prose without grounding the underlying representations, leading to pronounced overclaiming behavior.
- Orthogonality of hallucination dimensions: Pairwise Pearson correlations between VER, OVR, and ATR are near zero (CiteVal vs. OFS: \(r = -0.13\); CiteVal vs. Parametric Certainty: \(r = -0.19\); Linguistic Certainty vs. all others: \(|r| \le 0.04\)). Even within VER, OFS and VeriScore correlate weakly (\(r = 0.22\)), proving that evaluation conclusions are heavily shaped by evidence sources and that optimizing for one hallucination type does not mitigate others.
- Scientific check-worthiness remains fundamentally contested: In the human pilot study, annotator agreement on whether an atomic statement is scientifically check-worthy is only fair (\(\kappa = 0.29\), dropping to \(\kappa = 0.17\) in Arts & Humanities), driven by scope ambiguity, disciplinary disagreement, and blurry boundaries. Automated fact-checking tools achieve modest agreement with domain experts (VeriScore \(\kappa = 0.27\), OFS \(\kappa = 0.05\)), highlighting an urgent need for domain-specific verification infrastructure.
Highlights & Insights¶
- Coverage-capped CiteVal metric prevents volume exploitation: By combining fuzzy CrossRef DOI/title validation with an \(F_1@M\) formulation anchored to median citation count, CiteVal prevents models from gaming precision or recall by generating either an empty list or dozens of hallucinated references.
- Strictly controlled minimal pairing debunks domain fine-tuning: By isolating model architecture and comparing each scientific checkpoint directly against its base on 2,500 identical concepts, the study disproves the common belief that domain-specific fine-tuning naturally improves factual accuracy.
- Identification of rhetorical overclaiming risks: Demonstrates empirically that fine-tuning produces models that sound more scholarly and confident while being statistically less certain internally—a combination that presents severe deception risks for non-expert users.
Limitations & Future Work¶
- Authors' stated limitations: Subject classification inherits S2ORC's automated clustering, which cannot fully decouple multidisciplinary papers; CiteVal validates citation existence but not evidential support for specific assertions due to paywalled full texts; overclaim detection relies on uncertainty proxies rather than direct semantic overstatement classifiers.
- Unaddressed gaps: The closed-book evaluation does not examine the interaction between scientific fine-tuning and retrieval-augmented generation (RAG) at scale; annotator qualifications were set at the bachelor's level rather than doctoral peer-reviewers, which may limit validation depth for highly niche concepts.
- Future directions: Developing calibrated scientific fine-tuning objectives that penalize ungrounded lexical certainty, constructing full-text open-access citation verification graphs, and formulating domain-specific operational guidelines for scientific check-worthiness.
Related Work & Insights¶
- vs. FActScore / VeriScore: FActScore established atomic fact decomposition for biographies against Wikipedia, while VeriScore generalized verification to web search. However, web retrieval frequently retrieves non-authoritative sources for niche scientific claims (e.g., social media for humanities). SCIFACTCHECK grounds verification directly in peer-reviewed survey literature and scholarly registries.
- vs. SciFact / SciFact-open: Earlier scientific fact-checking datasets evaluate binary stance classification over provided abstracts (NLI). In contrast, this paper tackles open-ended, closed-book long-form text generation and bibliographic citation fabrication in modern LLMs.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering multi-domain evaluation framework decoupling three hallucination types and uncovering epistemic dissociation in scientific LLMs.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ 18 models, 2,500 concepts across 5 domains, robust non-parametric tests, and human verification across 958 claims and 115 citations.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, lucid exposition, and compelling narrative supported by thorough empirical evidence.
- Value: ⭐⭐⭐⭐⭐ Essential warning for the AI for Science community, providing critical guidance for fine-tuning methodologies and factuality benchmarking.