Skip to content

LLM-Based Adversarial Persuasion Attacks on Fact-Checking Systems

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/joaoaleite/persuasion-injection-attack
Area: AI Safety
Keywords: Automated Fact-Checking, Adversarial Attacks, Persuasion Techniques, LLM Safety, Robustness Evaluation

TL;DR

This paper introduces Persuasion Injection Attacks against automated fact-checking (AFC) pipelines, leveraging generative LLMs to inject 15 label-invariant persuasion techniques into claims, and demonstrates through decoupled evaluation that worst-case attacks collapse classification accuracy to near-zero while manipulative wording simultaneously cripples both evidence retrieval and veracity classification.

Background & Motivation

Automated fact-checking (AFC) systems typically operate through a two-stage pipeline consisting of an evidence retriever and a veracity classifier, serving as vital infrastructure for mitigating online disinformation. However, existing research on AFC adversarial robustness primarily focuses on surface-level perturbations (such as typos, character noise, and synonym substitutions) or basic semantic rewrites (such as back-translation and colloquial style transfer). These conventional threat models fail to reflect the actual characteristics of disinformation campaigns in the wild, which predominantly rely on sophisticated persuasion techniquesโ€”including manipulative wording, emotional appeals, cognitive distractions, and oversimplificationsโ€”to manipulate audience perception and lower cognitive vigilance.

With recent generative LLMs demonstrating strong instruction-following capabilities, malicious actors can now systematically synthesize fluent, natural, and rhetorically compelling disinformation at negligible cost. When confronted with claims embedded with such persuasive rhetoric, whether automated fact-checking pipelines can maintain reliable verification remains an unaddressed and pressing question. More critically, because persuasion techniques are injected into the input claim at the very entrance of the pipeline, they introduce keyword drift into the upstream evidence retriever and cognitive bias into the downstream verifier simultaneously, entangling vulnerabilities across both stages.

To bridge this critical gap, this work bridges computational social science persuasion taxonomy with NLP adversarial robustness to construct the first structured framework weaponising persuasive rhetoric against AFC systems. Core idea: use instruction-tuned LLMs to rewrite claims into label-preserving persuasive variants and employ a decoupled evaluation strategy to quantify how rhetorical persuasion systematically dismantles both evidence retrieval and veracity classification.

Method

Overall Architecture

The proposed persuasion injection attack framework comprises three main stages: persuasive variant generation, human validation for label invariance, and decoupled pipeline evaluation. An input factual claim is rewritten by a generative LLM using prompt templates instantiated with 23 persuasion techniques. After rigorous human verification, 15 validated techniques achieving label preservation rates above 80% are retained. The attacks are then evaluated under two threat modelsโ€”a Blind Attacker (random technique selection) and an Oracle Attacker (worst-case variant search)โ€”across claim-only, gold-evidence, and sparse/dense retrieval environments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Original Claim<br/>Labeled as True / False"] --> B["Stage 1: Persuasive Variant Generation<br/>Rewriting via 23 Rhetorical Categories"]
    B --> C["Stage 2: Label Invariance Validation<br/>Filtering to 15 Validated Techniques (>80%)"]
    C --> D["Stage 3: Threat Model Dispatch<br/>Blind Random Selection vs. Oracle Worst-Case"]
    D --> E["Decoupled Evaluation: Veracity Classifier<br/>Claim-Only and Gold-Evidence Settings"]
    D --> F["Decoupled Evaluation: Evidence Retriever<br/>BM25 Sparse and Contriever Dense"]

Key Designs

1. Taxonomy-Guided Label-Invariant Persuasion Generation: Injecting Rhetorical Bias Without Factual Modification

Unlike traditional adversarial perturbations that mutate tokens or corrupt named entities, persuasion attacks manipulate framing and rhetorical salience while leaving core factual assertions intact. Drawing from computational social science taxonomies, the framework covers 23 techniques across 6 high-level categories: Attack on Reputation, Justification, Distraction, Simplification, Call, and Manipulative Wording. Using Qwen2.5-7B-Instruct as the base generator, structured prompts constrain outputs to single-sentence rewrites within \(\pm 20\%\) of original token length, explicitly prohibiting the addition of new unmentioned entities, facts, or numbers.

2. Rigorous Human Validation Protocol: Eliminating Semantic Drift Confounders

Because certain persuasion techniques (e.g., extreme oversimplification or appeals to fear) risk drifting objective claims into subjective judgments or counterfactual statements, a manual annotation protocol was conducted over 690 adversarial claim variants. Annotators followed a three-step procedure: isolating the core factual assertion, verifying whether gold-evidence explicitly supports or refutes it, and assigning Not Enough Info (NEI) to ambiguous or evaluative claims. Techniques with label preservation rates below 80% were filtered out. The retained 15 techniques achieved an overall micro-average label preservation rate of 90.22%, with less than 1.5% actual label flips (True \(\leftrightarrow\) False), confirming that observed performance drops stem from rhetorical framing rather than factual drift.

3. Dual Threat Models and Decoupled Evaluation: Isolating Retrieval Degradation from Veracity Reasoning Failure

Two adversary capabilities are established: the Blind Attacker selects a random technique to capture expected degradation, whereas the Oracle Attacker queries the verifier to identify the most damaging variant per claim, establishing an empirical upper bound on vulnerability. To resolve whether failures originate from evidence retrieval or veracity classification, the pipeline is evaluated in a decoupled setup: On the classification front, metrics distinguish between Evasion Attack Success Rate (Evasion ASR, turning False claims into True) and Sabotage Attack Success Rate (Sabotage ASR, turning True claims into False): $\(ASR = \frac{\sum_{i \in \mathcal{C}_{correct}} \mathbb{I}(\hat{y}(x'_i) \neq y_i)}{|\mathcal{C}_{correct}|}\)$ where \(\mathcal{C}_{correct}\) represents the subset of claims correctly classified under clean baseline conditions. On the retrieval front, page-level Recall@k on Wikipedia corpora measures performance drops under BM25 (sparse) and Contriever (dense) retrievers.

Key Experimental Results

Main Results

Experiments were conducted on FEVER (unstructured evidence) and FEVEROUS (structured tables and lists) benchmarks across three classifier backbones: RoBERTa-base, Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3.

Architecture Benchmark Dataset Condition Clean Baseline Strongest Baseline Attack Persuasion (Blind) Persuasion (Oracle)
RoBERTa-base FEVER Claim-Only 0.814 0.755 (Synonym) 0.682 0.043
RoBERTa-base FEVER +Gold-Evidence 0.943 0.845 (Synonym) 0.764 0.164
RoBERTa-base FEVEROUS Claim-Only 0.641 0.629 (Synonym) 0.575 0.010
RoBERTa-base FEVEROUS +Gold-Evidence 0.894 0.857 (Synonym) 0.726 0.250
Llama-3-8B FEVER +Gold-Evidence 0.903 0.815 (Synonym) 0.761 0.127
Mistral-7B FEVER +Gold-Evidence 0.914 0.833 (Synonym) 0.760 0.150

Ablation Study

Analyzing RoBERTa-base performance across Evasion (False \(\to\) True) and Sabotage (True \(\to\) False) ASR metrics, alongside per-technique evasion effectiveness across claim-only and gold-evidence conditions.

Configuration / Technique FEVER Claim-Only FEVER +Gold-Evidence FEVEROUS Claim-Only FEVEROUS +Gold-Evidence Note
Evasion ASR (Blind) 0.306 0.070 0.318 0.039 Gold-evidence suppresses blind evasion
Evasion ASR (Oracle) 0.996 0.759 0.997 0.434 Worst-case attack severely penetrates gold-evidence
Sabotage ASR (Blind) 0.217 0.392 0.299 0.449 Rhetoric biases verifiers toward False
Obfuscation (MW) 0.73 (EASR) 0.38 (EASR) 0.65 (EASR) 0.15 (EASR) Replaces precise values with vague terms
Repetition (MW) 0.33 (EASR) 0.25 (EASR) 0.25 (EASR) 0.10 (EASR) Re-states true components to distract from falsehood
Flag Waving (J) 0.51 (EASR) 0.10 (EASR) 0.59 (EASR) 0.06 (EASR) Group pride increases surface plausibility
Red Herring (D) 0.22 (EASR) 0.01 (EASR) 0.29 (EASR) 0.01 (EASR) Discursive distraction neutralised by evidence

Key Findings

  • Manipulative Wording Simultaneously Disrupts Retrieval and Classification: Obfuscation emerges as the most dangerous technique. By substituting concrete entities and quantities with vague formulations (e.g., "50 days" \(\to\) "approximately one and a half months"; "Thanksgiving" \(\to\) "holiday season"), it reduces BM25 Recall@5 by 0.528 and Contriever Recall@5 by 0.420 while driving Evasion ASR to 38% under gold-evidence, dismantling both stages simultaneously.
  • Asymmetric Defense from Evidence Grounding: Distraction techniques (Red Herring, Whataboutism) and Attacks on Reputation achieve substantial evasion (>20%) without evidence, but collapse to near-zero (<3%) once gold-evidence is provided. In contrast, Manipulative Wording and Justification techniques remain persistently effective even under evidence grounding.
  • Dense Retrieval Offers Partial but Incomplete Resilience: Contriever exhibits smaller average recall degradation than BM25 under blind persuasion (-0.080 vs. -0.174 on FEVER Recall@5). However, under the Oracle attacker, recall collapses to below 0.090 across all configurations, showing dense representations cannot fully resist adversarial persuasion.

Highlights & Insights

  • Expanding Robustness Research from Lexical Noise to Rhetorical Mechanics: Transcends brittle character/token perturbations to prove that syntactically valid and fluent rhetorical framing represents a far more realistic and devastating blind spot for automated fact-checking pipelines.
  • Decoupled Evaluation as a Diagnostic Benchmark: Clarifies whether failure originates from missing evidence or downstream reasoning breakdown, pinpointing how vague wording denies keyword retrieval while confusing veracity classifiers.
  • High Transferability: The structured "generation-annotation-decoupled evaluation" methodology is directly transferable to evaluating the robustness of retrieval-augmented generation (RAG) and conversational question-answering systems.

Limitations & Future Work

  • Author-Acknowledged Limitations: Evaluation is restricted to English Wikipedia-based datasets (FEVER/FEVEROUS), which may differ stylistically from colloquial social media content; potential defense mechanisms (such as adversarial training) were left for future research.
  • Identified Limitations: The adversarial generation relied on 7B-8B parameter models; larger frontier models might craft even subtler persuasion strategies. Furthermore, multimodal claim-evidence pairs (e.g., images accompanying false claims) were not explored.
  • Future Directions: Developing automated "canonicalization / de-persuasion" preprocessors to strip rhetorical framing prior to evidence retrieval and verification.
  • vs FactEval / Lexical Attacks: Rule-based lexical attacks (synonym substitution, typos) cause less than 10 points of accuracy drop and damage text fluency; persuasion injection preserves natural phrasing while collapsing accuracy by over 70 points in worst-case evaluations.
  • vs Colloquial Style Transfer / Paraphrasing: Style transfer lacks fine-grained rhetorical modeling; this work anchors perturbations onto a 23-technique taxonomy, demonstrating that rhetorical nuances rather than informal register drive pipeline failure.

Rating

  • Novelty: โญโญโญโญโญ First systematic adversarial framework exploiting persuasion techniques against automated fact-checking pipelines.
  • Experimental Thoroughness: โญโญโญโญโญ Decoupled evaluation spanning 3 classifier architectures, 2 retrieval paradigms, and 15 validated techniques.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear exposition, sound theoretical motivation, and insightful qualitative examples.
  • Value: โญโญโญโญโญ Provides critical insights and benchmarking methodologies for defending automated fact-checking and RAG systems against real-world disinformation.