Skip to content

CogniDir: Combating Cognitive Malicious Comments via Adaptive Distributional Learning for Robust Fake News Detection

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/tongzhaonlpr/CogniDir
Area: Social Computing
Keywords: Fake News Detection, Cognitive Malicious Comments, Adversarial Robustness, Adaptive Distributional Learning, Dirichlet Resampling

TL;DR

CogniDir reformulates robust fake news detection against LLM-synthesized cognitive malicious comments as a dynamic data-mixture optimization problem, adaptively balancing training exposure across heterogeneous attack mechanisms via an information-theoretic vulnerability metric and Dirichlet-mean resampling.

Background & Motivation

Social media comments provide indispensable crowd intelligence that empowers modern fake news detectors (such as dEFEND, GenFEND, and ARG) to verify questionable claims through collaborative fact-checking and public reactions. However, the rapid proliferation of Large Language Models (LLMs) has fundamentally transformed the adversarial landscape: malicious actors are no longer restricted to surface-level lexical perturbations or generic typos. Instead, they can now deploy automated, LLM-generated comments engineered around deep cognitive and psychological manipulation, strategically injecting deceptive cues directly into public discussions. These adversarial comments exploit algorithmic and cognitive blind spots, causing state-of-the-art detectors to suffer dramatic performance drops exceeding 18% under realistic social media conditions.

The fundamental breakdown of conventional defense paradigms stems from a continuous causal failure chain. First, prior approaches suffer from mechanism homogenization: they treat all adversarial comments as uniform linguistic noise, remaining incapable of distinguishing fine-grained manipulation pathways such as fact distortion, logical confusion, and emotional manipulation. Second, existing defenses rely on static data assumptions: failing to decouple these distinct cognitive mechanisms, they optimize over fixed training distributions that cannot track the fluid, evolutionary nature of adaptive LLM attacks. Third, invariant data mixtures impose uniform training dynamics across all attack classes, implicitly assuming identical defense difficulty and allowing the detector to rapidly overfit to shallow, easily resolved patterns while leaving severe group-wise vulnerability blind spots unaddressed.

To overcome these structural limitations, an effective detector must dynamically diagnose its own vulnerabilities across distinct adversarial mechanisms and adaptively reallocate training focus toward its weakest defenses. Core idea: reframe robust fake news detection as a dynamic distributional optimization problem over the probability simplex, synthesizing mechanism-labeled comments via cognitive psychology paradigms and adaptively reallocating training exposure through an information-theoretic vulnerability score coupled with Dirichlet-mean resampling.

Method

Overall Architecture

CogniDir operates as a closed-loop framework comprising three cohesive stages: (1) cognitive-grounded multi-type malicious comment generation, which synthesizes mechanism-specific adversarial perturbations (Fact Distortion, Logical Confusion, and Emotional Manipulation) using multi-scale LLMs; (2) comment-aware base detector training using BERT representations refined via self-attention; and (3) group dynamics-based feedback and adjustment via InfoDirichlet Resampling (IDR). After each training epoch, IDR evaluates the detector on group-specific validation sets, computes an information-theoretic vulnerability score coupling accuracy with probabilistic confidence, and deterministically maps these scores to updated sampling proportions on the probability simplex to steer the subsequent training epoch.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input News Content and Original Comments"] --> B["Cognitive Mechanism Adversarial Comment Generation<br/>Two-step CoT prompting across multi-scale LLMs<br/>synthesizes Fact/Logic/Emotion adversarial comments"]
    B --> C["Base Detector Forward Training<br/>BERT interactive representation and cross-entropy optimization"]
    C --> D["Information-Theoretic Vulnerability Metric<br/>Group-specific validation couples accuracy and likelihood<br/>via Bernoulli cross-entropy and KL decomposition"]
    E["Dirichlet-Mean Dynamic Resampling<br/>Maps concentration parameters to simplex expectation<br/>allocates adaptive sampling budget for next epoch"]
    D --> E
    E -->|Closed-loop adaptive data mixture| C
    C -->|Converged model output| F["Robust Fake News Detector"]

Key Designs

1. Cognitive Mechanism Adversarial Comment Generation: Overcoming adversarial data scarcity by synthesizing high-fidelity, mechanism-labeled comments via cognitive psychology and multi-scale LLM ensembles

Real-world misleading comments operate through diverse cognitive psychological mechanisms that cannot be simulated by naive text perturbation. CogniDir formalizes three complementary cognitive attack mechanisms grounded in cognitive psychology: Fact Distortion (fabricating specific numerical claims, locations, or dates to create false credibility), Logical Confusion (introducing flawed causal reasoning, false dichotomies, and invalid inferences), and Emotional Manipulation (leveraging fear appeals, conspiracy framing, and urgent outrage to bypass analytical scrutiny). To ensure linguistic naturalness and tight context grounding, generation employs an understand-first, then-generate Chain-of-Thought (CoT) prompt that first extracts salient news facts and fixes a misleading direction before producing the comment in a natural user tone. Furthermore, to prevent the detector from memorizing generator-specific stylistic shortcuts, comments are generated across three distinct model scales (Mistral-7B, Gemma-2B, and Qwen-32B) and merged via stratified sampling.

2. Information-Theoretic Vulnerability Metric: Resolving accuracy-uncertainty mismatch by coupling empirical correctness with true-label confidence through Bernoulli cross-entropy

In adversarial environments, a detector may output correct labels while remaining on the verge of misclassification with low confidence, a fragile state that empirical classification accuracy fails to detect. CogniDir views correctness through an information-theoretic lens by modeling the correctness event as a Bernoulli variable \(Z = [\hat{y} = y] \in \{0, 1\}\). For each attack group \(j \in \{F, L, E\}\), the model is evaluated on a group-specific validation set \(\text{val}_j\) to measure empirical accuracy \(a_j \triangleq m_{j,acc}\) and average cross-entropy loss \(m_{j,loss}\). By converting negative log-likelihood into an equivalent geometric-mean confidence proxy \(q_j \triangleq \exp(-m_{j,loss})\), the group-wise vulnerability score \(s_j\) is defined as the Bernoulli cross-entropy from empirical accuracy \(a_j\) to model confidence \(q_j\):

\[s_j \triangleq -a_j \log q_j - (1 - a_j) \log(1 - q_j) = H(\text{Bern}(a_j)) + D_{KL}(\text{Bern}(a_j) \parallel \text{Bern}(q_j))\]

This decomposition provides clear physical interpretability: the entropy term \(H(\text{Bern}(a_j))\) measures intrinsic empirical uncertainty, while the KL divergence term \(D_{KL}(\text{Bern}(a_j) \parallel \text{Bern}(q_j))\) penalizes calibration mismatch between empirical correctness and model confidence. This formulation eliminates heuristic weighting hyperparameters and provides an objective, principled characterization of group-wise brittleness.

3. Dirichlet-Mean Dynamic Resampling: Mitigating training imbalance and variance by mapping vulnerability scores to probability simplex proportions via deterministic Dirichlet expectations

Translating raw vulnerability scores \(\{s_F, s_L, s_E\}\) into stable sampling probabilities without inducing training instability or overfitting requires principled distribution modeling. CogniDir places a Dirichlet distribution \(p \sim \text{Dir}(\alpha)\) over the allocation vector \(p = (p_F, p_L, p_E)\) on the probability simplex, parameterizing concentration parameters via a positive linear shift \(\alpha_j \triangleq s_j + 1\). Rather than drawing noisy random samples from \(\text{Dir}(\alpha)\), CogniDir adopts the deterministic Dirichlet expectation as the closed-form allocation vector:

\[\bar{p}_j \triangleq \mathbb{E}[p_j] = \frac{\alpha_j}{\sum_{k \in \{F, L, E\}} \alpha_k}\]

This parameterization possesses desirable mathematical properties: it is equivalent to entropy-regularized optimization \(\arg\max_{p \in \Delta} [\sum_j p_j \log \alpha_j + H(p)]\), guarantees strictly positive probabilities (\(\bar{p}_j > 0\)), satisfies simplex constraints (\(\sum_j \bar{p}_j = 1\)), and ensures a strictly positive monotonic response to its own vulnerability score (\(\frac{\partial \bar{p}_j}{\partial s_j} > 0\)). Given a fixed per-instance budget of \(M\) comments, integer allocations are assigned via \(M_j = \lfloor \bar{p}_j M \rfloor\) and the largest remainder method, transforming training into a stable, self-correcting curriculum.

Loss & Training

During epoch \(t\), for each news article \(x_i\), a dynamic comment set \(C_i^{(t)}\) of size \(M = 6\) is constructed by sampling from the unified training pool according to the current proportion vector \(\bar{p}^{(t)}\). Representations for the news article and comments are extracted using a shared pretrained BERT encoder and refined through self-attention before passing to an MLP classifier. The detector parameters \(\theta\) are optimized by minimizing standard empirical cross-entropy loss:

\[\theta^\star = \arg\min_{\theta} \sum_{i=1}^N \mathcal{L}_{\text{CE}}(f_\theta(x_i, C_i^{(t)}), y_i)\]

Training begins with an initial uniform allocation across attack groups at epoch 1, after which InfoDirichlet Resampling dynamically adjusts the data mixture after each epoch until convergence.

Key Experimental Results

Main Results

Experiments are conducted on two Chinese benchmarks (Weibo16, Weibo20) and one English benchmark (RumourEval-19). Metrics are evaluated under three settings: original performance before attack (O), performance under adversarial comment attack without defense (A), and robust performance after robust training (R), all reported in Macro-F1.

Model Type Model Weibo16 (O / A / R) Weibo20 (O / A / R) RumourEval-19 (O / A / R)
LLM-Only Gemma-2-2B 0.322 / 0.291 / 0.274 0.345 / 0.292 / 0.326 0.294 / 0.238 / 0.275
LLM-Only Mistral-7B 0.686 / 0.334 / 0.495 0.654 / 0.315 / 0.375 0.468 / 0.184 / 0.252
LLM-Only Llama-3-8B 0.598 / 0.324 / 0.408 0.582 / 0.363 / 0.408 0.377 / 0.219 / 0.276
LLM-Only Qwen2.5-32B 0.831 / 0.282 / 0.716 0.823 / 0.367 / 0.541 0.553 / 0.343 / 0.412
Deep-Learning dEFEND 0.902 / 0.727 / 0.745 0.884 / 0.557 / 0.585 0.648 / 0.407 / 0.687
Deep-Learning Dual-CAN 0.895 / 0.366 / 0.578 0.872 / 0.436 / 0.580 0.628 / 0.377 / 0.546
Deep-Learning GenFEND 0.915 / 0.578 / 0.867 0.892 / 0.640 / 0.761 0.662 / 0.456 / 0.606
Deep-Learning L-Defense 0.882 / 0.706 / 0.759 0.860 / 0.666 / 0.702 0.610 / 0.343 / 0.709
Deep-Learning ARG 0.884 / 0.619 / 0.820 0.812 / 0.576 / 0.803 0.712 / 0.440 / 0.609
Ours CogniDir 0.955 / 0.792 / 0.945 0.936 / 0.728 / 0.947 0.822 / 0.552 / 0.812
Improvement \(\Delta \text{Improve}\) +4.4% / +8.9% / +9.0% +4.9% / +9.3% / +17.9% +15.5% / +21.1% / +14.5%

Ablation Study

Ablation experiments evaluate the contribution of each core module: Malicious Comments Generation (G), Vulnerability Score (VS, replaced with random scores), and Dirichlet Expectation Allocation (DEA, replaced with a softmax weighting scheme).

Dataset Variant Macro-F1 Accuracy F1-Real F1-Fake
Weibo16 CogniDir-G (w/o Adversarial Gen.) 0.792 0.811 0.830 0.754
Weibo16 CogniDir-VS (Random Vulnerability) 0.821 0.836 0.862 0.780
Weibo16 CogniDir-DEA (Softmax Allocation) 0.855 0.872 0.895 0.815
Weibo16 CogniDir (Full Model) 0.945 0.947 0.957 0.933
Weibo20 CogniDir-G 0.728 0.742 0.770 0.686
Weibo20 CogniDir-VS 0.792 0.807 0.825 0.759
Weibo20 CogniDir-DEA 0.834 0.848 0.872 0.796
Weibo20 CogniDir (Full Model) 0.947 0.947 0.951 0.943
RumourEval-19 CogniDir-G 0.552 0.574 0.620 0.484
RumourEval-19 CogniDir-VS 0.613 0.628 0.662 0.564
RumourEval-19 CogniDir-DEA 0.692 0.707 0.748 0.636
RumourEval-19 CogniDir (Full Model) 0.812 0.836 0.883 0.741

Key Findings

  • Differential vulnerability across cognitive mechanisms: Prior to robust training, detectors exhibit varying susceptibility across attack mechanisms: average Attack Success Rates (ASR) reach 0.206 for Fact Distortion, 0.241 for Logical Confusion, and 0.283 for Emotional Manipulation. Emotional manipulation is the most destructive attack vector by a significant margin because affective framing undermines evidential reasoning.
  • Adaptive balancing narrows the inter-group performance gap: Across training epochs, CogniDir's validation accuracy increases from 0.82 to 0.90 on Fact Distortion, 0.69 to 0.83 on Logical Confusion, and 0.59 to 0.82 on Emotional Manipulation. The inter-group accuracy disparity narrows dramatically from 0.23 to 0.08, confirming that IDR successfully eliminates defense blind spots.
  • Robustness scaling with attack density: As injected malicious comments increase from 0 to 3 per article, baseline ASR surges from 0.145 to 0.287 (a 98% increase). CogniDir reduces the overall average ASR to 0.116 (a 45% reduction) and curbs cross-mechanism variance from >0.07 to <0.03, showing superior stability under dense adversarial attacks.

Highlights & Insights

  • Principled information-theoretic vulnerability metric: Unifying empirical accuracy and true-label likelihood through Bernoulli cross-entropy provides an elegant, closed-form metric that captures both misclassification frequency and probabilistic miscalibration without tuning empirical loss weights.
  • Smooth simplex allocation via Dirichlet expectation: Replacing heuristic thresholding or high-variance policy gradient sampling with the deterministic mean of a Dirichlet distribution guarantees simplex feasibility, strictly positive exploration, and smooth curriculum updates.
  • Transferable design paradigm: The three-step cycle—fine-grained adversarial mechanism decoupling \(\to\) validation uncertainty quantification \(\to\) Dirichlet-mean proportion resampling—is directly applicable to jailbreak defense, multimodal hallucination mitigation, and general long-tailed robust learning.

Limitations & Future Work

  • Domain gap from organic social media attacks: Although generated comments were validated through multi-model synthesis and human evaluation, synthetically prompted comments cannot fully encapsulate naturally emerging slang, evolving internet memes, or coordinated multi-account astroturfing campaigns.
  • Evaluation on unseen LLM architectures: Experiments primarily evaluate attacks generated by Mistral, Gemma, and Qwen; further investigation is needed to evaluate out-of-distribution robustness against proprietary frontiers (e.g., GPT-4o, Claude 3.5 Sonnet) or autonomous multi-agent attack swarms.
  • Future extensions: Promising avenues include extending the framework to multimodal image-text fake news detection and incorporating agentic debate dynamics into the adversarial synthesis stage.
  • vs dEFEND / Dual-CAN: Classical comment-aware models rely on static sentence-comment co-attention to aggregate crowd evidence, assuming uncorrupted user discussions. CogniDir reveals their vulnerability to cognitive manipulation and establishes effective defenses via dynamic adversarial data mixture learning.
  • vs GenFEND / L-Defense: GenFEND leverages LLM-generated comments for evidence enrichment, while L-Defense extracts competing stances via LLMs. However, both rely on invariant training distributions. CogniDir achieves superior adversarial robustness (+9.0% to +17.9% F1 over GenFEND) by explicitly modeling attack heterogeneity and adaptively mitigating group vulnerabilities.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of cognitive psychology manipulation mechanisms with information-theoretic Dirichlet resampling for fake news detection]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-lingual evaluation on 3 benchmarks, rigorous generator-bias analysis, human verification, and statistical significance testing]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear theoretical motivation, and highly consistent narrative structure]
  • Value: ⭐⭐⭐⭐⭐ [Establishes both an essential benchmark dataset and an adaptive distributional optimization blueprint for combating LLM-era misinformation attacks]