Advancing Fact-Checking for Large Language Models via Argumentative Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZyraChen/ArgCheck
Area: LLM Reasoning / Fact-Checking / Neuro-Symbolic Reasoning
Keywords: Fact-Checking, Argumentative Reasoning, Abstract Argumentation Framework, Adversarial Retrieval, Evidence Conflict Resolution
TL;DR¶
Addressing confirmation bias in single-agent retrieval and the lack of principled conflict resolution over heterogeneous web sources, this paper presents MAVEN, a real-world benchmark with fine-grained error taxonomy and ruling-style justifications, alongside ArgCheck, a neuro-symbolic framework coupling multi-round adversarial retrieval with Dung's abstract argumentation grounded semantics, boosting Macro-F1 over the strongest baselines by 5.39 on MAVEN and 11.10 on AVeriTeC.
Background & Motivation¶
Large language models are rapidly emerging as pivotal backbones across safety-critical domains such as healthcare, finance, and legal adjudication, yet their persistent hallucinations and unfaithful assertions pose severe societal risks that impede trustworthy adoption. Because the breadth of real-world claims precludes exhaustive pre-compilation into static knowledge bases, automated fact-checking systems increasingly rely on open-web search to retrieve external candidate evidence. However, uncurated online information inevitably introduces evidence conflicts stemming from divergent ideological viewpoints across platforms, outdated statistics, and dynamically evolving events.
Existing evaluation benchmarks and verification methodologies fall drastically short in realistic evidential conflict scenarios. On the benchmark front, standard datasets (e.g., FEVER, HoVer) predominantly rely on algorithmic generation over structured encyclopedias without real-world conflicting accounts, while open-domain benchmarks often provide only coarse classification labels devoid of ruling-style justifications or systematic error taxonomies. On the algorithmic front, current fact-checking pipelines suffer from two fundamental bottlenecks: single-agent verification exhibits severe confirmation bias by retrieving evidence aligned with initial predispositions while ignoring contradictory accounts, whereas multi-agent debate frameworks lack formal, mathematically grounded mechanisms to arbitrate evidential disagreements, frequently causing reasoning chains to fragment during open-ended dialogue.
To establish a principled foundation for multi-source conflict resolution, this work bridges iterative dialectical retrieval with formal argumentation theory. The authors introduce MAVEN, a curated benchmark of authentic claims explicitly featuring multi-source evidential conflicts and structured two-level error taxonomies, and formulate the ArgCheck verification architecture. Core idea: neutralize retrieval bias via multi-round Pro/Con cross-examination, model conflicting web evidence as an abstract argumentation framework with attack relations oriented by source authority and timeliness, compute the unique skeptical grounded extension via fixed-point semantics, and adjudicate verdicts and justifications strictly anchored in accepted arguments.
Method¶
Overall Architecture¶
ArgCheck formalizes the fact-checking process as a function \(f : \mathcal{C} \times \mathcal{W} \rightarrow \mathcal{V} \times \mathcal{E}\), mapping an input claim \(c\) and open web \(\mathcal{W}\) to a verdict \(v \in \{\text{Supported}, \text{Refuted}, \text{NEE}\}\) and an evidence-grounded explanation \(e \in \mathcal{E}\). The framework executes across three pipeline stages: first, the input claim is broken down into verifiable atomic sub-claims; second, adversarial Pro and Con agents iteratively generate counter-queries against each other's evidence over \(R\) rounds to amass a balanced evidence pool; third, evidence items are organized into an Abstract Argumentation Framework (AAF) where attack edges are determined by pairwise semantic conflicts combined with authority and timeliness scores, followed by grounded extension computation to isolate the accepted arguments that guide final adjudication by the Judge Agent.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Complex Input Claim c"] --> B["Atomic Claim Decomposition<br/>Four Core Principles: Atomicity·Verifiability·Completeness·Non-redundancy"]
B --> C["Adversarial Multi-round Evidence Collection<br/>Pro/Con Agents Iteratively Target Opposing Weak Points"]
C --> D["Authority & Timeliness-Oriented Argumentation Graph Construction<br/>Semantic Conflict Detection cf(a,b) with Priority Predicate Orienting Attack Edges"]
D --> E["Grounded Semantics Extension Computation<br/>Least Fixed Point of Defense Operator Derives Accepted Set Eg"]
E --> F["Judge Agent Adjudication & Justification<br/>Anchored on Accepted Eg while Explaining Defeated/Disputed Evidence"]
Key Designs¶
1. Atomic Claim Decomposition: Bundled multi-fact assertions -> Four-principle constrained extraction -> Eliminating multi-hop verification ambiguity
Real-world claims often intertwine multiple factual premises, entity attributions, quantitative numbers, and restrictive scopes. Verifying the compound claim in a single monolithic query causes search dilution and missed counter-evidence. ArgCheck enforces a structured decomposition pipeline adhering to four formal principles to derive sub-claims \(C_{\text{sub}} = \{c_1, c_2, \dots, c_n\}\): (1) Atomicity, ensuring each sub-claim conveys exactly one verifiable assertion; (2) Verifiability, requiring each proposition to be assessable via external records; (3) Completeness, guaranteeing that the joint set covers all factual content in the original claim; and (4) Non-redundancy, avoiding overlapping propositions. Categorizing facts into entities, events, temporal anchors, quantities, qualifiers, and relational predicates ensures unambiguous search targets for subsequent adversarial rounds.
2. Adversarial Multi-round Evidence Collection: Single-agent confirmation bias -> Pro/Con dialectical cross-examination -> Assembling a balanced multi-source evidence pool
Single-perspective retrieval inevitably suffers from confirmation bias, collecting corroborating evidence while systematically omitting refuting clues. ArgCheck deploys two competing agents with divergent objectives: a Pro Agent (\(M_P\)) searching for supporting evidence and a Con Agent (\(M_C\)) actively uncovering refuting information. In round \(r=1\), each agent \(M_k\) generates initial queries \(Q_{k,i}^{(1)} = G_{\text{ini}}(c_i, k)\) for each sub-claim \(c_i\), querying the web API \(S(\cdot)\) to form the base evidence pool \(D^{(1)}\). In subsequent adversarial iterations \(r \in \{2, \dots, R\}\), each agent scrutinizes the opposing agent's evidence summaries from the preceding round \(D_{\neg k}^{(r-1)}\) and generates a focused counter-query to exploit identified evidentiary weaknesses: $\(Q_k^{(r)} = G_{\text{adv}}(D_{\neg k}^{(r-1)}, k)\)$ The shared evidence pool updates incrementally as \(D^{(r)} = D^{(r-1)} \cup \bigcup_{k \in \{P,C\}} S(Q_k^{(r)})\). Running for \(R=5\) rounds systematically surfaces obscure counter-evidence, neutralizing retrieval bias and providing balanced multi-source evidence for downstream conflict resolution.
3. Authority & Timeliness-Oriented Argumentation Graph Construction: Flat evidential contradictions -> Semantic conflict detection with dual-priority predicates -> Formalizing directed attack relations
Conflicting web accounts cannot be resolved through majority voting due to variations in source credibility and temporal evolution. ArgCheck maps each evidence summary to an argument node \(a \in A\) within an AAF \(F=(A, R)\), parameterized by an authority level \(\alpha(a) \in \{1, 2, 3\}\) (derived from domain classification: high for government/academic, medium for mainstream news, low for social media/self-media) and a timeliness score computed via exponential decay: $\(\tau(a) = e^{-\lambda \cdot \Delta t(a)}, \quad \lambda = \frac{\ln 2}{365}, \quad \Delta t(a) = t_{\text{ref}} - t(a)\)$ For any pair of arguments \((a, b)\) addressing the same sub-claim, an LLM-based symmetric semantic conflict predicate \(cf(a, b)\) is first queried. An ordered attack \((a, b) \in R\) is established if and only if semantic conflict holds in conjunction with the priority predicate \(\pi(a, b)\): $\((a, b) \in R \iff cf(a, b) \land \pi(a, b)\)$ $\(\pi(a, b) \iff \alpha(a) > \alpha(b) \lor \tau(a) > \tau(b) \lor (\alpha(a) = \alpha(b) \land \tau(a) = \tau(b))\)$ When two conflicting arguments exhibit opposing advantages across authority and timeliness (or tie on both), mutual attacks \((a, b)\) and \((b, a)\) are added. This preserves the trade-off without arbitrary manual weighting, deferring resolution to graph-level defense structures.
4. Grounded Semantics Extension Computation: Subjective heuristics in evidential disputes -> Dung's skeptical defense fixed-point computation -> Deriving a unique conflict-free accepted subset
To evaluate which arguments survive evidential conflicts, ArgCheck applies Dung's (1995) grounded semantics. A candidate subset \(B \subseteq A\) is conflict-free if it contains no internal attacks, and \(B\) defends an argument \(a\) if every attacker of \(a\) is counter-attacked by some argument in \(B\). The grounded extension \(E_g\) corresponds to the unique least fixed point of the defense operator \(\Gamma_F\): $\(\Gamma_F(B) = \{a \in A \mid B \text{ defends } a\}, \quad E_g = \text{lfp}(\Gamma_F)\)$ Starting from \(X_0 = \emptyset\), unattacked arguments (high-authority, recent, and uncontested) are accepted at step 1. Their outgoing attacks eliminate lower-priority contradictory claims, thereby defending and reinstating other well-founded arguments in a cascading propagation until \(X_{t+1} = X_t\). Conflicting arguments locked in mutual attack without external defense remain outside \(E_g\) and are preserved as disputed context. This skeptical property prevents overcommitting to weakly defended claims.
5. Judge Agent Adjudication & Justification: Multi-subclaim global composition -> Dual-track context prompting -> Generating faithful verdicts and ruling explanations
Because individual accepted arguments address separate sub-claims, global verdict determination requires holistic claim-level reasoning. The Judge Agent receives the original claim \(c\), the accepted arguments \(A_{\text{acc}} = E_g\), and the full argumentation framework \(F=(A, R)\). The final verdict (Supported, Refuted, or NEE) is adjudicated primarily against \(A_{\text{acc}}\). Crucially, defeated and disputed arguments from the full graph provide the exact counterfactual context necessary to generate ruling-style explanations explaining why superseded rumors or outdated reports were rejected. Furthermore, boundary rules stipulate that if any critical sub-claim is refuted, the entire claim is marked Refuted; if an essential sub-claim lacks conclusive evidence, it is designated Not Enough Evidence.
A Worked Example: Adjudicating the 2025 F1 Qatar Grand Prix Sprint Starting Grid¶
Consider verifying the claim: "Lewis Hamilton started from 18th position in the 2025 F1 Qatar Grand Prix Sprint" (Ground Truth: Refuted). 1. Adversarial Retrieval: Five argument nodes are extracted: \(A_1\) (qualifying result P18, high authority), \(A_2\) (stewards issued penalty requiring pit lane start, high authority), \(A_3\) (social rumor claiming penalty was rescinded, low authority), \(A_4\) (official FIA timing document confirming pit lane start, high authority), and \(A_5\) (blog post asserting P18 start, medium authority). 2. Argumentation Framework Construction: Timeliness gives \(A_2\) an attack on earlier qualifying data \(A_1\) (\(A_2 \rightarrow A_1\)). High-authority official document \(A_4\) attacks low-authority rumor \(A_3\) (\(A_4 \rightarrow A_3\)) and medium-authority blog \(A_5\) (\(A_4 \rightarrow A_5\)). Rumor \(A_3\) initially attacked penalty report \(A_2\) (\(A_3 \rightarrow A_2\)). 3. Fixed-Point Evaluation: \(A_4\) is completely unattacked and enters \(E_g\) at step 1. \(A_4\)'s attack on \(A_3\) neutralizes the sole attacker of \(A_2\), effectively defending and reinstating \(A_2\) into \(E_g\). In turn, \(A_2\) defeats \(A_1\). 4. Adjudication: The grounded extension yields \(E_g = \{A_2, A_4\}\). The Judge Agent concludes that the pit lane start is conclusively accepted, correctly assigning the verdict Refuted, whereas unstructured baselines mistakenly predicted Supported after latching onto mentions of "P18".
Key Experimental Results¶
Main Results¶
Experiments were conducted on MAVEN (1,185 authentic claims) and the public AVeriTeC development set (500 real-world claims). All methods utilized Qwen3-Max as the backbone with a sampling temperature of 0.1 and a maximum search limit of \(R=5\) rounds. Performance is evaluated across Accuracy, Macro-Precision, Macro-Recall, and Macro-F1.
| Dataset | Method | Accuracy (%) | Macro-P (%) | Macro-R (%) | Macro-F1 (%) |
|---|---|---|---|---|---|
| MAVEN | Zero-Shot | 62.70 | 57.30 | 58.42 | 57.50 |
| MAVEN | Zero-Shot + Thinking | 64.56 | 59.94 | 61.89 | 60.36 |
| MAVEN | Chain-of-Thought (CoT) | 61.27 | 54.36 | 54.29 | 54.28 |
| MAVEN | Self-Refine | 64.30 | 60.09 | 59.39 | 59.58 |
| MAVEN | Multi-Agent Debate (MAD) | 67.17 | 67.42 | 60.65 | 62.46 |
| MAVEN | ArgCheck (Ours) | 73.25 | 69.32 | 66.92 | 67.85 |
| AVeriTeC (dev) | Zero-Shot | 58.80 | 60.24 | 54.86 | 53.49 |
| AVeriTeC (dev) | Zero-Shot + Thinking | 62.40 | 59.19 | 54.89 | 54.84 |
| AVeriTeC (dev) | Chain-of-Thought (CoT) | 64.60 | 59.60 | 53.88 | 54.69 |
| AVeriTeC (dev) | Self-Refine | 65.00 | 61.49 | 55.85 | 55.84 |
| AVeriTeC (dev) | Multi-Agent Debate (MAD) | 62.20 | 57.25 | 49.39 | 50.00 |
| AVeriTeC (dev) | ArgCheck (Ours) | 75.40 | 67.84 | 66.62 | 66.94 |
Ablation Study & In-Depth Diagnostics¶
The ablation study validates the isolation of adversarial multi-round retrieval (w/o Adv., replaced with single-round retrieval) and the abstract argumentation framework (w/o Arg, retaining adversarial retrieval but bypassing the formal graph). In addition, LLM-based explanation quality evaluation (scored 1–5 by Gemini-3-Flash-Preview on MAVEN) and reasoning-verdict coherence distributions (Faithful Verification Rate FVR vs. Compound Error Rate CER) are reported.
| Evaluation Category | Variant / Metric Dimension | MAVEN Metrics | AVeriTeC (dev) Metrics | Findings / Key Observations |
|---|---|---|---|---|
| Ablation Study | ArgCheck (Full Model) | 73.25 Acc / 67.85 F1 | 75.40 Acc / 66.94 F1 | Full adversarial search + Dung grounded semantics |
| Ablation Study | ArgCheck w/o Arg | 71.05 Acc / 64.16 F1 | 72.80 Acc / 64.03 F1 | Removing argumentation degrades Acc by 2.20–2.60 |
| Ablation Study | ArgCheck w/o Adv. | 64.73 Acc / 61.64 F1 | 63.60 Acc / 58.39 F1 | Single-round retrieval collapses to single-agent levels |
| Explanation Quality (1–5) | Correctness | 3.72 (Best Baseline 3.35) | — | Substantial alignment with ground-truth justifications |
| Explanation Quality (1–5) | Completeness | 3.71 (Best Baseline 3.16) | — | Covers both refutations and necessary counter-evidence |
| Explanation Quality (1–5) | Faithfulness | 4.03 (Best Baseline 3.54) | — | Explanations anchored strictly on accepted arguments |
| Explanation Quality (1–5) | Overall Quality | 4.07 (Best Baseline 3.76) | — | Outperforms the strongest baseline by +0.31 |
| Reasoning Coherence | Faithful Verification Rate (FVR) | 67.8% | — | Correct verdict combined with high-quality justification |
| Reasoning Coherence | Compound Error Rate (CER) | 14.6% | — | Simultaneous failure in verdict and reasoning (baselines >24%) |
Key Findings¶
- Complementarity of Adversarial Retrieval and Formal Argumentation: Eliminating adversarial retrieval causes a dramatic drop of 8.52 points in MAVEN accuracy, proving that multi-round adversarial search is indispensable for overcoming confirmation bias. Furthermore, removing the argumentation layer degrades accuracy by 2.20–2.60 points and Macro-F1 by 2.91–3.69 points, demonstrating that formal conflict resolution provides a substantial boost even over an identical, rich evidence pool.
- Suppression of Compound Failures and Elevated Faithfulness: In reasoning-verdict coherence analysis, ArgCheck achieves an FVR of 67.8% (exceeding ZS-T by 9.0 points) while slashing CER to 14.6% (at least 9.7 points lower than all baselines). Grounding verdicts in formal extensions ensures accuracy gains originate from verified evidence rather than lucky guesses.
- Diminishing Returns Across Retrieval Rounds: Evaluating rounds \(R \in \{1, \dots, 5\}\) reveals that accuracy surges from 68.19% at \(R=1\) to 72.15% at \(R=2\), followed by a plateau through \(R=5\) (73.25%). In production environments, configuring \(R=2\) provides a Pareto-optimal operating point, slashing API calls by 56% and token overhead by 65% while preserving over 90% of the maximum performance gain.
- Cross-Backbone Generalization: When tested on a 200-claim stratified subset using DeepSeek-V4-Pro, ArgCheck reached 75.5% accuracy and 62.1 Macro-F1, outperforming Self-Refine and MAD by 13.0 and 8.4 percentage points respectively, proving the framework does not overfit to Qwen3-Max.
Highlights & Insights¶
- Grounded Semantics as the Optimal Epistemological Choice for Factuality: Unlike preferred or credulous semantics that yield multiple acceptable sets, Dung's grounded semantics provides a unique, skeptical fixed point, strictly enforcing the principle that an assertion must withstand every attack to be accepted as truth.
- Mutual-Attack Design for Authority-Timeliness Trade-Offs: When authority and timeliness yield divergent preferences, creating bidirectional attacks preserves the conflict without imposing arbitrary domain-dependent priority weights, allowing topological graph reinstatement to resolve the impasse naturally.
- Diagnostic Benchmark MAVEN with Two-Level Error Taxonomy: Spanning 1,185 instances across 10 domains, MAVEN provides a structured benchmark with 6 first-level and 22 second-level error categories, explicitly enabling error diagnosis for unverifiable, temporal, and semantic failures.
Limitations & Future Work¶
- Computational Overhead of Iterative Inference: Under \(R=5\), ArgCheck averages 47.0 API calls, 188.7K tokens, and 100.7 seconds of latency per claim. Developing adaptive stopping criteria based on graph convergence and distilling argumentation traces into a unified dense verifier represent vital next steps.
- Residual Reliance on LLM Prompting in Symbolic Construction: Constructing the symbolic framework relies on LLMs for atomic claim decomposition and pairwise conflict detection \(cf(a, b)\). A human audit of 200 conflict pairs revealed an 80.5% agreement rate, indicating non-trivial rates of over-detection (perspectives falsely marked conflicting) and under-detection (implicit contradictions missed).
- Scale and Language Imbalance: MAVEN contains 1,185 claims, of which 1,008 are Chinese and 177 are English. Expanding to broader multilingual corpora and larger pre-training/fine-tuning sets remains an open avenue.
Related Work & Insights¶
- vs Multi-Agent Debate (MAD / Liang et al., 2024): MAD uses open-ended free-form debate that tends to oscillate positions and fragment reasoning chains (evidenced by MAD's lowest completeness score of 2.90 on MAVEN); ArgCheck replaces unstructured debate with formalized graph semantics, yielding superior verdict recall and grounded explanations (completeness 3.71).
- vs Quantitative Bipolar Argumentation (ArgRAG / Zhu et al., 2025): ArgRAG constructs QBAFs over fixed top-k retrieval and relies on gradual semantics to compute continuous claim scores; ArgCheck dynamically expands the evidence pool via adversarial cross-examination and extracts an exact extension of skeptically accepted arguments under grounded semantics.
- vs Tool-Augmented Fact-Checkers (FactTool / FactLLaMA / MiniCheck): Traditional single-agent retrieval frameworks suffer from severe confirmation bias; ArgCheck's dialectical Pro/Con agents enforce adversarial cross-checks that uncover concealed refutations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Seamlessly unifies formal abstract argumentation theory with multi-agent adversarial retrieval for robust conflict resolution.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation covering MAVEN and AVeriTeC, complete ablations, 5-dimensional explanation audits, four-quadrant coherence analysis, and cross-backbone testing.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formalization, clean conceptual narrative, and intuitive illustrative case studies.
- Value: ⭐⭐⭐⭐⭐ Establishes a definitive neuro-symbolic standard for fact-checking under realistic, conflicting open-web evidence.