Advancing Fact-Checking for Large Language Models via Argumentative Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZyraChen/ArgCheck
Area: LLM (Other)
Keywords: LLM Fact-Checking, Argumentative Reasoning, Abstract Argumentation Framework, Adversarial Retrieval, MAVEN Benchmark
TL;DR¶
Addressing confirmation bias in retrieval and multi-source evidence conflict in real-world claim verification, this paper introduces ArgCheck, a neuro-symbolic framework coupling multi-round adversarial Pro-Con evidence collection with formal Dung abstract argumentation under grounded semantics, improving Macro-F1 by 5.39 and 11.10 points on MAVEN and AVeriTeC, respectively.
Background & Motivation¶
Large language models (LLMs) are increasingly deployed across safety-critical domains such as medicine, finance, and legal intelligence, yet their susceptibility to factual hallucinations and unfaithful generations introduces severe risks that obstruct trustworthy real-world adoption. Because static parametric knowledge bases cannot exhaustively track fast-evolving real-world occurrences, prevailing automated fact-checking pipelines adopt retrieval-augmented paradigms to pull candidate evidence from heterogeneous web sources. However, real-world open-web evidence is inherently fragmented and fraught with severe contradictions, arising either from disparate reporting stances across platforms or from temporal latency where superseded historical accounts clash with ongoing event updates.
Despite this operational reality, existing benchmarks and verification methodologies systematically fall short in resolving evidential conflicts. At the benchmark level, classical datasets such as FEVER and HoVer generate synthetic claims from clean encyclopedic sources, lacking authentic evidential contradictions; meanwhile, contemporary real-world benchmarks like CHEF and AVeriTeC rarely supply claims whose verification explicitly pivots on resolving acute multi-source conflicts, while typically providing superficial labels that fail to convey ruling-style justifications or fine-grained error categorizations. Methodologically, current LLM verifiers exhibit two crippling bottlenecks: single-agent verification suffers from confirmation bias by retrieving evidence confirming initial intuitions while ignoring counter-arguments, whereas unstructured multi-agent debate (MAD) exchanges free-form text that lacks a rigorous, principled logic mechanism for conflict arbitration, frequently collapsing into fragmented rationales, circular arguments, and ungrounded verdicts.
To overcome both benchmark and methodological limitations, the authors introduce MAVEN (Multi-source Argumentation-based VErificatioN dataset), comprising 1,185 real-world claims across ten domains annotated with ruling-style justifications and a two-level error taxonomy, and develop ArgCheck, an argumentation-grounded fact-checking architecture. Core idea: decompose multi-source fact-checking into adversarial dual-agent cross-examination, formal abstract argumentation graph construction guided by asymmetric authority and timeliness, and axiomatic conflict resolution via grounded semantics, steering the judge agent with skeptically accepted arguments to generate reliable, evidence-grounded verdicts.
Method¶
Overall Architecture¶
ArgCheck formalizes the fact-checking task as a mapping \(f: \mathcal{C} \times \mathcal{W} \to \mathcal{V} \times \mathcal{E}\), where a target claim \(c \in \mathcal{C}\) is evaluated against the web \(\mathcal{W}\) to output a verdict \(v \in \{\text{Supported}, \text{Refuted}, \text{NEE}\}\) (NEE denoting Not Enough Evidence) alongside an evidence-grounded ruling justification \(e \in \mathcal{E}\). The framework operates through three sequential neuro-symbolic stages: (1) claim decomposition breaks complex assertions into atomic, verifiable sub-claims; (2) adversarial Pro and Con agents iteratively query opposing evidence to assemble a balanced multi-source evidence pool; and (3) an abstract argumentation framework is built over the evidence, where attack edges are determined by semantic contradiction weighted by authority and timeliness, and grounded semantics computes the unique skeptically accepted argument extension that guides the final judge agent.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input Complex Claim<br/>Input Claim"] --> D1["Atomic Sub-Claim Decomposition & Adversarial Dual Retrieval<br/>Pro/Con multi-round cross-examination via stance-conditioned queries"]
D1 --> D2["Authority & Timeliness Oriented Argumentation Graph Construction<br/>Pairwise semantic conflict detection with asymmetric priority mapping"]
D2 --> D3["Grounded Semantics Conflict Resolution & Judge Adjudication<br/>Least fixed-point defense computation to ground ruling justification"]
D3 --> Out["Output Three-Class Verdict & Structured Justification<br/>Supported / Refuted / NEE"]
Key Designs¶
1. Atomic Sub-Claim Decomposition & Adversarial Dual Retrieval: Confirmation Bias & Missing Counter-Evidence -> Pro/Con multi-round cross-examination via stance-conditioned queries -> Assembling a balanced, diverse evidence pool free from confirmation bias
Real-world claims frequently conflate multiple factual predicates that cannot be resolved in a single step. ArgCheck first invokes an LLM to decompose the input claim \(c\) into atomic sub-claims \(\mathcal{C}_{sub} = \{c_1, c_2, \dots, c_n\}\) strictly adhering to atomicity, verifiability, completeness, and non-redundancy.
To prevent single-perspective confirmation bias, ArgCheck introduces two adversarial agents with divergent objectives: a Pro Agent \(M_P\) dedicated to locating supporting evidence, and a Con Agent \(M_C\) tasked with finding refuting counter-evidence. In round \(r=1\), each agent \(M_k\) (\(k \in \{P, C\}\)) generates a stance-conditioned keyword query \(Q_{k, i}^{(1)} = G_{ini}(c_i, k)\) for each atomic sub-claim \(c_i\), fetching initial evidence summaries via search API \(S(\cdot)\) into a shared evidence pool \(\mathcal{D}^{(1)}\). Across subsequent rounds \(r \in \{2, \dots, R\}\) (\(R=5\)), the agents conduct iterative cross-examination: agent \(M_k\) examines the opposing agent's evidence \(\mathcal{D}_{\neg k}^{(r-1)}\) from the preceding round, identifies its most vulnerable point, and formulates a targeted rebuttal query: $\(Q_k^{(r)} = G_{adv}(\mathcal{D}_{\neg k}^{(r-1)}, k)\)$ The shared pool updates via union aggregation: $\(\mathcal{D}^{(r)} = \mathcal{D}^{(r-1)} \cup \left( \bigcup_{k \in \{P, C\}} S(Q_k^{(r)}) \right)\)$ This adversarial dynamic neutralizes model bias, actively searching for refutations and uncovering contradictory viewpoints across diverse web domains.
2. Authority & Timeliness Oriented Argumentation Graph Construction: Unstructured Evidential Conflicts -> Pairwise semantic conflict detection with asymmetric priority mapping -> Establishing an axiomatic abstract argumentation attack network
The enriched evidence pool \(\mathcal{D}^{(R)}\) inevitably gathers contradictory statements. ArgCheck maps the evidence pool into an Abstract Argumentation Framework (AAF) \(\mathcal{F} = (\mathcal{A}, \mathcal{R})\), where each evidence summary serves as an argument \(a \in \mathcal{A}\) associated with its target sub-claim.
To orient attacks objectively, ArgCheck scores each argument along two dimensions: - Authority Score \(\alpha(a) \in \{1, 2, 3\}\): Automatically classified via domain whitelists. Official government bodies, international organizations (e.g., WHO), and peer-reviewed academic repositories receive high authority (\(\alpha = 3\)); established mainstream journalism and industry reports receive medium authority (\(\alpha = 2\)); personal blogs and unverified social media platforms receive low authority (\(\alpha = 1\)). - Timeliness Score \(\tau(a) \in (0, 1]\): Captures information recency for time-sensitive claims. Given publication timestamp \(t(a)\) and reference retrieval date \(t_{ref}\), the elapsed day difference is \(\Delta t(a) = t_{ref} - t(a)\). The score is computed using an exponential decay function: $\(\tau(a) = \exp\left(-\lambda \cdot \Delta t(a)\right)\)$ where \(\lambda = \frac{\ln 2}{365}\), establishing a one-year half-life. Undated evidence is assigned the lowest timeliness.
For any argument pair \((a, b)\) sharing a sub-claim, a symmetric conflict predicate \(cf(a, b)\) is evaluated via prompted LLM to detect substantive logical, numerical, or factual contradiction. A directed attack edge \((a, b) \in \mathcal{R}\) is instantiated if and only if semantic conflict holds in conjunction with the priority predicate \(\pi(a, b)\): $\((a, b) \in \mathcal{R} \iff cf(a, b) \land \pi(a, b)\)$ where the priority predicate is formulated as: $\(\pi(a, b) \iff \alpha(a) > \alpha(b) \lor \tau(a) > \tau(b) \lor \left(\alpha(a) = \alpha(b) \land \tau(a) = \tau(b)\right)\)$ This non-parametric formulation handles the authority-recency trade-off gracefully: if argument \(a\) is strictly superior on one criterion and not inferior on the other, it unidirectionally attacks \(b\); if \(a\) and \(b\) conflict across criteria (e.g., \(a\) has higher authority but \(b\) is more recent) or tie on both, mutual attacks \(a \leftrightarrow b\) are added, leaving conflict resolution to the surrounding topological graph structure.
3. Grounded Semantics Conflict Resolution & Judge Adjudication: Fragile Free-form Consensus -> Least fixed-point defense computation to ground ruling justification -> Synthesizing faithful, evidence-grounded adjudications
To identify accepted evidence without heuristic guesswork, ArgCheck relies on Dung's grounded semantics. A set \(\mathcal{B} \subseteq \mathcal{A}\) is conflict-free if no arguments within \(\mathcal{B}\) attack each other, and it defends an argument \(a\) if \(\mathcal{B}\) attacks every attacker of \(a\). The grounded extension \(E_g\) is the unique minimal complete extension, characterized as the least fixed point of the defense operator \(\Gamma_{\mathcal{F}}\): $\(\Gamma_{\mathcal{F}}(\mathcal{B}) = \{a \in \mathcal{A} \mid \mathcal{B} \text{ defends } a\}\)$ $\(E_g = \mathrm{lfp}(\Gamma_{\mathcal{F}})\)$ Starting from \(X_0 = \emptyset\), the extension expands iteratively via \(X_{t+1} = \Gamma_{\mathcal{F}}(X_t)\) until convergence \(X_{t+1} = X_t = E_g\). Unattacked arguments are accepted immediately; their outgoing attacks defeat opposing arguments, which in turn reinstates defended arguments in subsequent iterations. Arguments trapped in unresolved mutual attacks remain outside \(E_g\) as disputed context.
Finally, the accepted argument set \(\mathcal{A}_{acc} = E_g\) and the complete graph \(\mathcal{F}\) are fed to the Judge Agent. The Judge is constrained to adjudicate the overall claim based primarily on \(\mathcal{A}_{acc}\), while utilizing defeated and disputed relations across \(\mathcal{F}\) to generate a structured, ruling-style justification explaining why particular counter-arguments were rejected.
A Worked Example¶
Consider the real-world case study from Section 6.6: - Claim: "Lewis Hamilton started from 18th position in the 2025 F1 Qatar Grand Prix Sprint." (Ground Truth: Refuted) - Constructed Arguments: - \(A_1\): Qualifying report stating Hamilton qualified in P18 (Authority: High, Timeliness: Older); - \(A_2\): Post-qualifying report stating Hamilton received a penalty requiring a pit lane start (Authority: High, Timeliness: Newer); - \(A_3\): Social media blog claiming the penalty was rescinded (Authority: Low); - \(A_4\): Official FIA timing document confirming pit lane start (Authority: High, Timeliness: Latest); - \(A_5\): Secondary news summary reporting P18 start (Authority: Medium, Outdated). - Argumentation Conflict Resolution: - Due to newer timeliness, penalty report \(A_2\) attacks qualifying report \(A_1\) (\(A_2 \to A_1\)). - Blog post \(A_3\) asserts penalty cancellation, attacking \(A_2\) (\(A_3 \to A_2\)). - Official timing document \(A_4\) holds highest authority and recency, attacking \(A_3\) (\(A_4 \to A_3\)) and \(A_5\) (\(A_4 \to A_5\)). - Grounded Extension Computation: \(A_4\) is unattacked, entering \(E_g\) at step 1. Because \(A_4 \in E_g\) attacks \(A_3\), it successfully defends \(A_2\). Consequently, \(A_2\) is accepted into \(E_g\) at step 2, which in turn attacks and defeats \(A_1\). The final grounded extension converges to \(E_g = \{A_2, A_4\}\). - Adjudication: Based on the accepted authoritative facts in \(E_g\), the Judge Agent correctly outputs Refuted and explains how official penalties superseded initial qualifying results, whereas baseline LLMs without argumentation falsely concluded Supported based on surface mentions of "P18".
Loss & Training¶
ArgCheck is an entirely training-free neuro-symbolic framework, requiring no parameter fine-tuning and offering immediate compatibility with general-purpose LLMs. The primary implementation adopts qwen3-max as the backbone (sampling temperature 0.1, max search strategy) coupled with the Jina Search API. The adversarial retrieval budget is set to \(R = 5\) iterations. For latency-sensitive production environments, ablation studies demonstrate that setting \(R = 2\) rounds captures the vast majority of performance gains while cutting API calls by 56% (from 47.0 to 20.7), token usage by 65% (from 188.7K to 65.5K), and wall-clock latency to 44.0 seconds per claim.
Key Experimental Results¶
Main Results¶
ArgCheck was evaluated on the MAVEN benchmark (1,185 claims) and the AVeriTeC development set (500 claims mapped to 3-class format), using qwen3-max across all methods:
| Method | MAVEN Acc. (%) | MAVEN Macro-P (%) | MAVEN Macro-R (%) | MAVEN Macro-F1 (%) | AVeriTeC Acc. (%) | AVeriTeC Macro-P (%) | AVeriTeC Macro-R (%) | AVeriTeC Macro-F1 (%) |
|---|---|---|---|---|---|---|---|---|
| Zero-Shot (ZS) | 62.70 | 57.30 | 58.42 | 57.50 | 58.80 | 60.24 | 54.86 | 53.49 |
| Zero-Shot + Thinking (ZS-T) | 64.56 | 59.94 | 61.89 | 60.36 | 62.40 | 59.19 | 54.89 | 54.84 |
| Chain-of-Thought (CoT) | 61.27 | 54.36 | 54.29 | 54.28 | 64.60 | 59.60 | 53.88 | 54.69 |
| Self-Refine (SR) | 64.30 | 60.09 | 59.39 | 59.58 | 65.00 | 61.49 | 55.85 | 55.84 |
| Multi-Agent Debate (MAD) | 67.17 | 67.42 | 60.65 | 62.46 | 62.20 | 57.25 | 49.39 | 50.00 |
| ArgCheck (Ours) | 73.25 | 69.32 | 66.92 | 67.85 | 75.40 | 67.84 | 66.62 | 66.94 |
In LLM-based explanation quality evaluation using gemini-3-flash-preview scored against human-annotated references (1–5 scale):
| Method | Correctness | Completeness | Faithfulness | Coherence | Clarity | Overall Score |
|---|---|---|---|---|---|---|
| Zero-Shot | 3.31 | 3.13 | 3.54 | 4.23 | 4.50 | 3.74 |
| Zero-Shot + Thinking | 3.35 | 3.16 | 3.48 | 4.27 | 4.54 | 3.76 |
| Chain-of-Thought | 3.24 | 3.07 | 3.33 | 4.18 | 4.43 | 3.65 |
| Self-Refine | 3.18 | 3.11 | 3.30 | 4.12 | 4.39 | 3.62 |
| Multi-Agent Debate | 3.11 | 2.90 | 3.17 | 4.22 | 4.49 | 3.58 |
| ArgCheck (Ours) | 3.72 | 3.71 | 4.03 | 4.37 | 4.55 | 4.07 |
Cross-backbone validation on a label-stratified 200-claim subset of AVeriTeC using deepseek-v4-pro:
| Method (Backbone: DeepSeek-V4-Pro) | Accuracy (%) | Macro-P (%) | Macro-R (%) | Macro-F1 (%) |
|---|---|---|---|---|
| Zero-Shot | 51.0 | 55.4 | 48.3 | 46.7 |
| Zero-Shot + Thinking | 53.0 | 56.8 | 49.8 | 48.4 |
| Chain-of-Thought | 53.5 | 56.2 | 51.1 | 48.2 |
| Self-Refine | 62.5 | 58.4 | 53.7 | 53.3 |
| Multi-Agent Debate | 60.0 | 57.6 | 54.8 | 53.7 |
| ArgCheck (Ours) | 75.5 | 64.2 | 61.2 | 62.1 |
Ablation Study¶
Systematic ablations isolate the complementary roles of adversarial retrieval and formal argumentation:
| Configuration | MAVEN Acc. (%) | MAVEN Macro-F1 (%) | AVeriTeC Acc. (%) | AVeriTeC Macro-F1 (%) | Analysis |
|---|---|---|---|---|---|
| ArgCheck (Full Framework) | 73.25 | 67.85 | 75.40 | 66.94 | Optimal neuro-symbolic configuration |
| w/o Adv. (Single-shot retrieval) | 64.73 (-8.52) | 61.64 (-6.21) | 63.60 (-11.80) | 58.39 (-8.55) | Severe retrieval bias; degrades to single-agent levels |
| w/o Arg (No argumentation graph) | 71.05 (-2.20) | 64.16 (-3.69) | 72.80 (-2.60) | 64.03 (-2.91) | Unresolved conflicts degrade recall and faithfulness |
Computational cost trade-offs across inference configurations on AVeriTeC dev instances:
| Configuration (10 Stratified Claims) | Avg. API Calls | Avg. Tokens (k) | Latency (s) | Efficiency Profile |
|---|---|---|---|---|
| Zero-Shot | 1.0 | 1.1 | 2.0 | Minimal latency but severe conflict vulnerability |
| Multi-Agent Debate | 14.0 | 70.2 | 59.6 | Unstructured dialogue with high overhead |
| ArgCheck (\(R=2\), Fast/Budget) | 20.7 | 65.5 | 44.0 | -56% calls vs R=5, retaining primary accuracy gains |
| ArgCheck (\(R=5\), Full) | 47.0 | 188.7 | 100.7 | Top-tier verification accuracy |
Key Findings¶
- Adversarial Retrieval Drives Recall Expansion: Removing adversarial multi-round retrieval collapses MAVEN accuracy by 8.52 points and AVeriTeC accuracy by 11.80 points, proving that multi-perspective counter-query generation is indispensable for overcoming single-agent confirmation bias.
- Formal Argumentation Enforces Explanatory Faithfulness: Reasoning-verdict coherence analysis reveals that ArgCheck achieves a Faithful Verification Rate (FVR) of 67.8% (9.0 points above ZS-T) and suppresses Compound Error Rate (CER) to 14.6% (compared to 24.3%–26.4% across baselines). Grounding verdicts in accepted extensions eliminates ungrounded correct guesses and hallucinated rationales.
- Multi-Agent Debate Suffers from Reasoning Fragmentation: Despite competitive verdict accuracy, MAD yields the lowest explanation completeness (2.90) and faithfulness (3.17), indicating that free-form conversational debates destabilize reasoning coherence compared to formal graph-theoretic extensions.
Highlights & Insights¶
- Principled Neuro-Symbolic Synergy: Combines the broad semantic parsing capabilities of LLMs (sub-claim extraction, semantic conflict identification) with the mathematical guarantees of Dung's abstract argumentation theory (tractable fixed-point convergence, unique grounded extension).
- Asymmetric Priority Ordering: Instead of arbitrary scalar scoring, authority and timeliness are integrated through a non-parametric partial order rule, deferring trade-off arbitration to the surrounding graph topology.
- Diagnostic Benchmark MAVEN: Introduces a high-value benchmark specifically designed around multi-source contradictions, featuring ruling-style explanations, credibility ratings, and a two-level error hierarchy spanning factual, temporal, data, semantic, evidence, and unverifiability dimensions.
Limitations & Future Work¶
- Inference Latency & API Overhead: Performing up to 5 rounds of adversarial search and pairwise LLM-based conflict detection increases wall-clock latency and token consumption relative to single-shot verifiers. Future iterations should explore dynamic early-stopping criteria or offline distillation into compact single-model verifiers.
- Sensitivity to LLM Conflict Predication: The symbolic argumentation graph relies on LLM-prompted semantic conflict assessments \(cf(a, b)\), where human audits indicate an 80.5% agreement rate with 25% over-detection and 14% under-detection, introducing noise into the attack topology.
- Expressive Bounds of AAF: The framework currently employs Dung's atomic attack framework; expanding toward bipolar argumentation frameworks (BAFs) to model explicit support relations and quantitative degrees of belief represents a promising research avenue.
Related Work & Insights¶
- vs Single-Agent Prompting (CoT / Self-Refine / FacTool): Standard tool-augmented approaches treat retrieval as a passive lookup prone to confirmation bias; ArgCheck implements active adversarial cross-examination to balance evidence pools.
- vs Multi-Agent Debate (MAD / Debating Truth): Free-form conversational debate suffers from position drift and rhetorical dominance; ArgCheck structures conflicting arguments into a formal graph, computing mathematically sound acceptances.
- vs Argumentative Claim Verification (ArgRAG / QBAF): Prior works operate over static top-k evidence sets using numerical strength propagation under gradual semantics; ArgCheck unites dynamic adversarial search with skeptically accepted extensions under grounded semantics.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of Dung's abstract argumentation theory with adversarial dual-agent retrieval for LLM fact-checking]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across two LLM backbones, comprehensive human conflict audits, detailed cost benchmarking, and multi-dimensional explanation analysis]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally rigorous formulation of argumentation semantics, transparent case studies, and clear conceptual motivation]
- Value: ⭐⭐⭐⭐⭐ [Provides a robust, reproducible blueprint for verifiable, hallucination-resistant LLM deployment in mission-critical applications]