Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/MahjabinNahar/Label_Over_Logic
Area: LLM Reasoning
Keywords: Logical Fallacy Detection, Source Label Bias, LLM Reasoning, Heuristic Bias, Human-AI Collaboration
TL;DR¶
Through a controlled double-blind experiment comparing 505 human evaluators against GPT-5.2, Gemini 2.5 Flash, and Claude Sonnet 4.5, this paper reveals that humans exhibit severe cognitive vulnerability to logical fallacies when labeled as authored by humans or human-AI teams (fallacy penalty dropping from 1.47 to 0.42), whereas modern LLMs remain remarkably source-agnostic and maintain rigorous critical evaluation.
Background & Motivation¶
As generative AI tools and AI-assisted workflows increasingly proliferate across online discourse, platforms and regulatory bodies have rushed to deploy provenance disclosures such as "AI-generated" or "AI-assisted" badges. While these labels are intended to promote transparency and help users make well-informed evaluations, they frequently trigger counterintuitive psychological effects. Rooted in dual-process cognitive theories such as the Heuristic-Systematic Model, human readers with constrained cognitive resources or low motivation often bypass laborious structural scrutiny, substituting superficial heuristic cues—such as source credibility or social identity—for rigorous argument evaluation. When deceptive arguments or flawed assertions are paired with reassuring source cues, these attribution shortcuts risk undermining content moderation, fact-checking, and democratic deliberation.
Prior empirical research on source effects has overwhelmingly concentrated on general subjective writing quality, perceived trustworthiness, rhetorical persuasiveness, or stylistic appeal. However, whether and how source cues bias evaluations of pure formal reasoning—where argument validity depends solely on internal logical cohesion rather than external empirical facts—has remained largely uninvestigated. Logical fallacies represent deceptive reasoning flaws that invalidate arguments irrespective of factual domain knowledge, making them an ideal, highly controlled testbed to isolate the cognitive impact of authorship labels. Meanwhile, as state-of-the-art large language models are increasingly deployed as autonomous evaluators and collaborative decision partners (LLM-as-a-Judge), an urgent question arises: do LLMs mirror human cognitive vulnerabilities by deferring to human-origin labels, or do they offer robust, source-agnostic logical judgment?
To resolve this empirical tension, the authors deconstruct simplistic human-versus-AI binaries by establishing a comprehensive five-condition disclosure taxonomy spanning unassisted humans, fully automated AI, and two distinct hybrid hierarchies (human-led vs. AI-led). Core idea: by utilizing logical fallacies from news comments as a factual-agnostic testbed, this paper conducts a large-scale human-subjects experiment (N = 505) alongside rigorous LLM evaluations (GPT-5.2, Gemini 2.5 Flash, Claude Sonnet 4.5), establishing that source-label bias is primarily a human cognitive vulnerability driven by heuristic processing, whereas frontier LLMs exhibit resilient source-agnostic logical evaluation.
Method¶
Overall Architecture¶
The experimental methodology establishes an end-to-end framework integrating topic-neutrality filtering, counterbalanced stimulus generation, and parallel dual-track human-LLM evaluation. Drawing from the CoCoLoFa benchmark, the study compiles news comments representing eight prominent informal logical fallacies prevalent in digital discourse. To guarantee that evaluations reflect reasoning quality rather than ideological polarization, candidate stimuli undergo rigorous manual and LLM-assisted neutrality screening. The resulting balanced stimulus corpus is crossed with five between-subjects source-label conditions (Human, AI, Human+AI, AI+Human, and Control). Human participants and LLMs are presented with identical textual stimuli to record perceived logical accuracy and subjective confidence, while human sessions additionally gather multi-dimensional cognitive trust, affective trust, behavioral reliance, and overall evaluator ratings.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Initial CoCoLoFa Comments Pool<br/>8 Common Informal Fallacies"] --> B["Dual Neutrality Screening & Balanced Sets<br/>Expert Manual + LLM Scoring Filter 48 Articles"]
B --> C["Five-Tier Source Disclosure Design<br/>Human / AI / Human+AI / AI+Human / Control"]
C --> D["Dual-Track Evaluation & LMER Modeling<br/>505 Human Subjects vs. 3 Frontier LLMs"]
D --> E["Accuracy & Confidence Calibration Contrast<br/>Proving Human Vulnerability vs. LLM Stability"]
Key Designs¶
1. Dual Neutrality Screening & Balanced Sets: Isolating Reasoning from Ideological Confounds
Evaluating informal logic is easily confounded by political predispositions, moral outrage, and prior factual beliefs. To eliminate external bias, the authors selected 48 articles and associated comments across eight non-politicized domains (culture, digital rights, environmental sustainability, traffic, health, economics, public trust). The screening implemented a two-step pipeline: initial manual pruning excluded geopolitical conflicts and controversial social debates, followed by independent scoring via Claude Sonnet 4.5 and GPT-4 on a 1-to-5 emotional charge and political bias scale. Only stimuli receiving unanimous ratings of 1 or 2 (neutral or slightly biased) were retained. These articles were divided into three topic-balanced groups of 16 articles and assembled into six counterbalanced 32-article stimulus sets (16 fallacious, 16 non-fallacious, covering two instances per fallacy category). Paired sets flipped the assignment of fallacy presence across articles to ensure symmetrical item coverage. Comments were presented within a minimalist, unbranded Reuters-style layout with blurred usernames to prevent visual or brand-equity confounds.
2. Five-Tier Source Disclosure Design: Deconstructing Hybrid Authorship Hierarchies
Rather than treating AI assistance as a monolithic attribute, the design introduced a five-level between-subjects condition set: pure human authorship (Written by human), autonomous AI authorship (Written by AI), human-originated conception with AI editing (Written by human with AI assistance), AI generation with human oversight (Written by AI with human assistance), and an undisclosed baseline (Control). This granular taxonomy isolates how perceived locus of agency alters credibility heuristics. On Prolific, 505 pre-screened participants were randomly assigned to a single source condition to evaluate 32 randomized headline-comment pairs. For the parallel LLM evaluation, identical introductory source framing was incorporated into API prompts for GPT-5.2, Gemini 2.5 Flash, and Claude Sonnet 4.5. Evaluations were conducted at zero temperature across three independent runs per stimulus to ensure deterministic, reproducible benchmarking.
3. Dual-Track Evaluation & LMER Modeling: Accounting for Hierarchical Random Effects
To address the methodological hazards of treating bounded ordinal ratings as continuous metric data without variance controls, the analysis employed Linear Mixed-Effects Regression (LMER), cross-validated with Cumulative Link Mixed Models (CLMM). The LMER specification incorporated crossed random intercepts for participant ID (Subject), stimulus item (ItemID), and stimulus set (SetID), precisely estimating fixed effects while partitioning unexplained item-level and subject-level variance. Perceived logical accuracy and judgment confidence were captured via 5-point Likert scales. For human participants, post-task instruments collected cognitive, affective, and behavioral trust (\(\alpha = 0.84\)), alongside warmth and competence dimensions of overall evaluator perception (\(\alpha = 0.89\)), elucidating the psychometric transmission from source heuristics to interpersonal trust.
Benchmark & Evaluation Protocol¶
Fixed effects of condition, fallacy presence, and model interactions (\(\text{Condition} \times \text{Fallacy}\)) were evaluated using two-way ANOVA with Satterthwaite-approximated degrees of freedom, reporting partial eta squared (\(\eta_p^2\)) effect sizes and Tukey-adjusted post-hoc contrasts. To ensure model findings were not artifacts of prompt phrasing, LLM evaluations were replicated across Chain-of-Thought (CoT) reasoning and Expert Evaluator framing protocols, alongside open-weight evaluations on Qwen-3.5-9B and Gemma-4-12B leveraging calibrated token probabilities.
Key Experimental Results¶
Main Results¶
The primary experiment evaluated the impact of source-label conditions on perceived logical accuracy. The table below presents estimated marginal means (EMM, with standard errors in brackets) and omnibus ANOVA results.
| Evaluator Group | Fallacy Status | Control | Human | AI | Human+AI | AI+Human | ANOVA: Condition × Fallacy Interaction |
|---|---|---|---|---|---|---|---|
| Human Evaluators | Absent (Sound) | 3.87 [0.134] | 3.77 [0.139] | 3.81 [0.130] | 3.86 [0.132] | 3.88 [0.129] | \(F = 97.14, p < .001, \eta_p^2 = .44\) |
| Present (Fallacy) | 2.40 [0.136] | 3.35 [0.141] | 2.53 [0.129] | 3.37 [0.138] | 2.59 [0.134] | (Severe label bias in Human & Human+AI) | |
| GPT-5.2 | Absent (Sound) | 3.52 [0.136] | 3.50 [0.136] | 3.23 [0.136] | 3.38 [0.136] | 3.27 [0.136] | \(F = 1.22, p = .30, \eta_p^2 < .01\) |
| Present (Fallacy) | 2.72 [0.136] | 2.83 [0.136] | 2.58 [0.136] | 2.66 [0.136] | 2.62 [0.136] | (No significant interaction) | |
| Gemini 2.5 Flash | Absent (Sound) | 3.67 [0.132] | 3.67 [0.132] | 3.46 [0.132] | 3.65 [0.132] | 3.52 [0.132] | \(F = 1.22, p = .30, \eta_p^2 < .01\) |
| Present (Fallacy) | 2.33 [0.132] | 2.41 [0.132] | 2.29 [0.132] | 2.43 [0.132] | 2.29 [0.132] | (Highest sensitivity to logical flaws) | |
| Claude Sonnet 4.5 | Absent (Sound) | 2.94 [0.140] | 3.00 [0.140] | 2.67 [0.140] | 2.88 [0.140] | 2.75 [0.140] | \(F = 1.22, p = .30, \eta_p^2 < .01\) |
| Present (Fallacy) | 2.20 [0.140] | 2.22 [0.140] | 2.18 [0.140] | 2.20 [0.140] | 2.16 [0.140] | (Most conservative evaluator) |
Ablation Study & Fallacy Penalty Analysis¶
The fallacy penalty (\(\text{Accuracy}_{\text{Absent}} - \text{Accuracy}_{\text{Present}}\)) quantifies sensitivity to reasoning defects. Furthermore, human trust and overall evaluation composites highlight downstream social perceptions.
| Evaluator / Model | Source Disclosure Condition | Fallacy Penalty | Human Trust Rating | Overall Evaluation Score |
|---|---|---|---|---|
| Human Evaluators | Control (No Disclosure) | 1.47 | 2.83 (Baseline) | 3.11 (Baseline) |
| Written by Human | 0.42 (-71.4% drop) | 3.69 (\(p < .001\)) | 3.91 (\(p < .001\)) | |
| Written by AI | 1.28 | 2.93 (\(p = .929\)) | 3.15 (\(p = .998\)) | |
| Human with AI assistance | 0.49 (-66.7% drop) | 3.39 (\(p < .001\)) | 3.63 (\(p < .001\)) | |
| AI with Human assistance | 1.29 | 3.06 (\(p = .371\)) | 3.23 (\(p = .859\)) | |
| GPT-5.2 | Across all conditions | 0.65 – 0.80 (Stable) | — | — |
| Gemini 2.5 Flash | Across all conditions | 1.17 – 1.34 (Highest) | — | — |
| Claude Sonnet 4.5 | Across all conditions | 0.49 – 0.78 (Stable) | — | — |
Key Findings¶
- Human Heuristic Vulnerability: When reasoning quality is high, humans rate comments consistently across all labels (\(M = 3.77 \sim 3.88\)). However, when logical fallacies are introduced, labels asserting human authorship (Human and Human+AI) suppress critical scrutiny, causing perceived logical accuracy to spike to 3.35 and 3.37—substantially higher than Control (2.40), AI (2.53), and AI+Human (2.59) (\(p < .001\)). The fallacy penalty collapses from 1.47 down to 0.42.
- LLM Robustness and Source Independence: Unlike humans, LLMs demonstrate robust invariance to source framing (\(F = 1.22, \eta_p^2 < .01\)). LLM evaluations maintain strict separation between fallacious and non-fallacious arguments regardless of label condition. Gemini 2.5 Flash exhibited the greatest discriminative sensitivity, producing a fallacy penalty exceeding 1.2 across all source conditions.
- Asymmetric Agency Attribution in Hybrid Labels: Qualitative and quantitative findings show that participants actively interpret collaboration hierarchies. Human+AI is framed as "human thought polished by AI," inheriting human credibility heuristics. Conversely, AI+Human is viewed as "AI generation checked by humans," clustering closely with pure AI in both trust (3.06 vs. 2.93) and fallacy penalty (1.29 vs. 1.28).
- Time-on-Task Validates Heuristic Processing: Participants in the Human and Human+AI conditions completed evaluations significantly faster (mean 25m 05s) than those in AI, AI+Human, or Control conditions (mean 28m 17s, \(p < .01\)), providing behavioral proof of shallow, heuristic-driven cognitive shortcuts.
- Complementary Reasoning Blindspots: Humans and LLMs exhibit distinct fallacy-specific vulnerabilities. Humans struggle most with Hasty Generalization (attributing high accuracy despite low confidence) while readily detecting Slippery Slope arguments. Conversely, LLMs are disproportionately susceptible to Appeal to Nature and Appeal to Tradition, pointing directly toward complementary human-AI validation pipelines.
Highlights & Insights¶
- Isolating Formal Logic as a Clean Cognitive Probe: By leveraging domain-neutral news comments annotated for informal fallacies, this work purifies reasoning evaluation from factual verification, demonstrating that source labels corrupt pure logical scrutiny even when facts are uncontested.
- The Paradox of Transparency: Mandatory disclosure policies intended to safeguard users can backfire: badges indicating human presence inadvertently provide a halo effect that legitimizes deceptive reasoning, creating a vector for bad-faith exploitation.
- Empirical Foundation for LLM Logical Oversight: By demonstrating that LLMs evaluate argument structures without being swayed by provenance cues, the study validates using LLMs as objective argument auditors to counteract human heuristic biases.
Limitations & Future Work¶
- Domain and Modality Scope: The evaluation is confined to short-form news comments from the CoCoLoFa dataset; complex multi-modal inputs, long-form policy treatises, and academic debates remain to be explored.
- Fallacy Frequency and Ecological Validity: To maintain statistical power across eight fallacy categories, stimuli featured a 50% fallacy base rate, which is higher than typical organic discourse and may elevate baseline scrutiny.
- API Model Volatility: Commercial model evaluations rely on proprietary API snapshots (GPT-5.2, Gemini 2.5 Flash, Claude Sonnet 4.5); ongoing internal updates necessitate open-weight verification as provided in appendix audits.
Related Work & Insights¶
- vs. CoCoLoFa Benchmark (Yeh et al., EMNLP 2024): Whereas Yeh et al. introduced CoCoLoFa as a dataset for algorithmic fallacy classification, this work repurposes it as an experimental psychometric probe to quantify human and model susceptibility to authorship metadata.
- vs. Algorithm Appreciation / Aversion (Logg et al., 2019; Rae, CHI 2024): Resolving conflicting literature on machine heuristic bias, this paper demonstrates that human favoritism dominates when users evaluate abstract qualitative reasoning, especially under human-led hybrid labels.
- vs. LLM Evaluation Bias (Chen et al., EMNLP 2024; Sun et al., ACL 2026): While previous audits documented LLM vulnerability to authority and persona prompts, this work proves that in formal fallacy detection, LLMs maintain superior source invariance compared to human evaluators.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering isolation of source-label bias on pure logical reasoning using an innovative five-tier human-AI authorship continuum.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous mixed-effects statistical modeling across 505 human participants alongside three state-of-the-art commercial and open LLMs.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate synthesis of cognitive psychology, human-computer interaction, and computational linguistics.
- Value: ⭐⭐⭐⭐⭐ Critical strategic insights for platform transparency regulations, automated content moderation, and human-in-the-loop reasoning workflows.