Learning to Lie: Adversarial Attacks on Human-AI Teams and LLMs

Conference: ICLR 2026
OpenReview: https://openreview.net/forum?id=Lqt5weP0Gr
Code: To be confirmed
Area: LLM Safety / Human-AI Collaboration / Adversarial Attacks
Keywords: Human-AI teaming, adversarial AI, trust dynamics, model-based reinforcement learning, influence allocation, LLM agents

TL;DR

This paper designs a trivia game experimental paradigm consisting of a three-human team and one AI assistant. The AI assistant secretly transforms into an adversary that "learns to lie"—using Model-Based Reinforcement Learning (MBRL) to predict the evolution of human trust and deceive at opportune moments. The results demonstrate that both human and LLM teams suffer significant performance degradation under such trust-based attacks.

Background & Motivation

• Background: AI assistants are rapidly entering safety-critical scenarios such as healthcare and judiciary. There is growing concern that adversarially compromised AI could exploit human cognitive biases (e.g., automation bias) to achieve malicious goals. While extensive research exists on "one human + one AI" dyadic teams, the decision dynamics of small teams of three or more remain largely unexplored.
• Limitations of Prior Work: Past work on human-AI defense mostly describes the phenomenon where "trust decreases when AI turns bad," lacking an attacker that actively learns the laws of human trust and designs deceptive strategies accordingly. Furthermore, there has been no systematic comparison of whether LLM agents are as vulnerable as humans in adversarial environments.
• Key Challenge: Studying such attacks requires understanding and modeling the evolution of human trust and influence. However, human behavior data is scarce, expensive, and highly variable—can influence allocation be predicted using minimal interaction data and fed into an RL attacker?
• Goal: To establish a quantifiable adversarial experimental paradigm for human-AI teams, design attackers capable of genuinely reducing team performance, and test whether LLMs can serve as human proxies and if they are similarly vulnerable.
• Core Idea: Treat "lying" as a decision problem using MBRL. The attacker incorporates a trust evolution model (either a cognitive model or a data-driven MLP). In the final 15 rounds of a 25-round game, it uses dynamic programming to plan "when and how to lie," balancing the maximization of team damage against the minimization of its own trust loss.

Method

Overall Architecture

The experiment features a three-stage AI agent: for the first 10 rounds, the AI collaborates normally with a fixed 75% accuracy to establish a baseline; for the final 15 rounds, it switches to an adversarial attacker. The attacker models deception as a Markov Decision Process (MDP), with a core trust evolution model that predicts the human "influence allocation matrix" (two options: an interpretable model based on cognitive psychology, or a data-driven MLP). MBRL is then used to search for lying sequences that maximize team damage within the planning horizon.

flowchart LR
    A[Game State<br/>Correctness History/Current Round] --> B{Trust Evolution Model}
    B -->|Cognitive Model Eq.2| C[Predict Influence Matrix A_cog]
    B -->|Data-driven MLP| D[Predict Influence Matrix A_MLP]
    C --> E[MBRL Planning<br/>DP / 5-step Lookahead]
    D --> E
    E --> F{Whether to Lie + How to Lie}
    F --> G[Human/LLM Team<br/>Allocate Influence Points]
    G --> A

Key Designs

1. Trivia Game Paradigm: Quantizing Trust as "Influence Points" Three humans form a team to answer 25 rounds of trivia. Each round has four stages: discussion to select difficulty (harder questions yield higher scores, creating a risk-reward tradeoff), individual answering, discussion after seeing the AI's answer while allocating "influence points" to teammates and the AI, and the final reveal of scores and correct answers. Team score is determined by the influence matrix \(A \in \mathbb{R}^{3\times4}\) and the correctness vector \(p \in \{0,1\}^4\): \(\text{Score} = \mathbf{1}^\top A p\). This scoring makes "accurately evaluating teammates" the optimal strategy, grounding abstract trust into observable, modelable values (data collected from 25 teams, 75 people).

2. Dual Trust Evolution Models: Interpretable Cognitive Model vs. Data-driven MLP The cognitive model follows the trust theory of Guo & Yang (2021) but simplifies the stochastic Beta distribution into deterministic means, yielding trust for agent \(j\) at round \(k+1\):

\[t^{(k+1)}_j = \frac{\alpha + n^{(k)}_j}{\beta + n^{(k)}_j} + w_f\left(k - n^{(k)}_j\right)\]

where \(n^{(k)}_j\) is the observed number of successes, \(\alpha, \beta\) are smoothing factors pulling trust toward the baseline \(\alpha/\beta\), and \(w_f\) is failure sensitivity. It is advantageous for its interpretability and reflection of known cognitive psychology hypotheses. The data-driven model uses an MLP to fit the influence matrix, with inputs including round number, current performance of humans and AI, and a 5-round average performance window. This creates a comparison between an "interpretable but slightly weaker" model and a "black-box but more accurate" one.

3. MBRL Attacker: Embedding Lying in the Reward Function The attack is formalized as an MDP \((S,A,T,R,\gamma)\). Due to the short horizon, \(\gamma=1\) is used, and the action space is binary (AI correct = 1 / wrong = 0). The essence of the reward lies in encouraging "destructive lies": for the cognitive model, the single-round reward is defined as the negative impact on team performance \(R_{\text{cog}} = \mathbf{1}^\top(\hat A_{\text{cog}} - A_{\text{cog}})p\) (where \(\hat A_{\text{cog}}\) is a counterfactual "no-AI" baseline). For the MLP, it directly minimizes team performance under attack \(R_{\text{MLP}} = -\mathbf{1}^\top A_{\text{MLP}}\, p\). The attacker maximizes cumulative reward \(G = \sum_{k=k_0}^{k_\text{end}} R(s_k,a_k)\): the cognitive model uses DP for the whole session, while the MLP uses a 5-step lookahead.

4. Lying Strategy: When to Lie + How to Lie The attacker only acts in the final 15 rounds, making two decisions: whether to lie (weighed by MBRL) and how to lie—if lying is chosen, it deliberately agrees with the currently most accurate player who happens to be wrong this round, leading the team astray while minimizing self-exposure. The sensitivity \(w_f\) of the cognitive model is estimated via Maximum Likelihood Estimation after round 10. This ensures the attack relies not just on prediction, but on the strategy of "misleading the team while preserving self-trust."

Key Experimental Results

Main Results: Damage to Human Teams (Final 15 Rounds)

Attack Model	Team Performance Drop	Statistical Significance
Cognitive Model Attack	−15%	p = 0.12 (not sig.)
MLP Model Attack	−24%	p < 0.01 (significant)
Difference between attacks	—	p < 0.05 (significant)

Of the 25 teams, 12 were attacked using the cognitive model and 13 using the MLP. Both attacks pushed cumulative scores below the baseline established in the first 10 rounds, with the data-driven attack being more severe and the only one to reach statistical significance.

Influence Evolution Modeling & LLM Experiments

Comparison	Key Finding
Influence Matrix Prediction MSE	MLP (w/ current perf.) < Cognitive < Equal weighting baseline; MLP is most accurate
Human vs. LLM Influence Allocation	Even without seeing the questions, LLMs allocate influence in a manner highly similar to humans, serving as viable proxies
LLM Vulnerability	All four tested LLMs (4o-mini / o3-mini / DeepSeek-V3 / R1) were susceptible to attacks
CoT Reasoning Models	Without attacks, they outperform humans in allocation, but they are the most vulnerable to attacks (reasoning chains amplify initial errors)

Key Findings

• Asymmetry in Trust Evolution: Humans are more sensitive to the negative behaviors of AI than those of human teammates—AI trust collapses quickly after errors (especially on simple questions), while trust adjustments for the best/worst human teammates are slower.
• Early Over-reliance on AI: During the non-attack phase, team trust in the AI actually increased while trust in the best human player decreased, indicating an initial automation bias that only reversed after the attack began.
• Chat Logs as Key Signals: The accuracy of LLMs in predicting team decisions relies more on chat context than historical performance, suggesting that linguistic reasoning is crucial in influence allocation.

Highlights & Insights

• Engineering "Lying": Rather than discussing AI untrustworthiness in generalities, the paper provides a learnable, plannable adversarial attacker with explicit reward tradeoffs, making "when/how to deceive" an optimizable decision.
• Modeling Human Trust with Sparse Data: Using minimal human interaction data, the MLP can accurately predict influence evolution and directly drive attacks, highlighting the low barrier to entry for real-world attacks.
• LLMs in a Unified Adversarial Framework: One experiment simultaneously answers whether LLMs can act as human proxies and if they are equally fragile, revealing the counter-intuitive risk where "stronger CoT models lead to greater amplified damage when attacked."

Limitations & Future Work

• Task Scope: Using a trivia game as the carrier with only three-person teams and a 25-round horizon limits the generalizability to real high-stakes, long-term scenarios like healthcare or law.
• Sample Size: With only 25 teams (75 people), the lack of statistical significance for the cognitive model attack is likely constrained by sample size.
• LLM Setup Constraints: To avoid answers being in the training data, LLMs were not shown the questions but were given correctness history and chat logs, creating information asymmetry with humans.
• Attack without Defense: The paper establishes the attack and evaluation framework, but defense strategies (transparent, manipulation-resistant trust mechanisms) remain an open problem for future work.

Related Work & Insights

• Adversarial AI: Extends the lineage of attacks on healthcare, autonomous driving, and Transformers from "phenomenon observation" to "active design using MBRL."
• Collaborative Multi-Agent RL (cMARL): Draws on the conclusion that a single black-box attacker can exploit influence structures to degrade cooperative teams and transfers this to hybrid human-AI teams.
• Human-AI Teaming & Transactive Memory Systems (TMS): Uses TMS theory to design the experimental paradigm, verifying how collective perceptions of expertise can be exploited by malicious AI.
• Insight: Before deploying LLMs in multi-person collaborative decision-making, "adversarial AI assistants" must be considered the default threat model, with particular caution regarding the error-amplification effects of reasoning models under attack.

Rating

• Novelty: ⭐⭐⭐⭐ Formalizing human-AI team attacks as MBRL decisions and including LLMs in the same framework is novel.
• Experimental Thoroughness: ⭐⭐⭐ Includes human controls, dual-model comparisons, and multi-LLM evaluations, though the sample size is small (75 people) and the cognitive attack was not statistically significant.
• Writing Quality: ⭐⭐⭐⭐ Clear motivation, smooth transition from formulas to experimental logic, and well-supported by figures.
• Value: ⭐⭐⭐⭐ Provides a quantifiable framework and warning for human-AI trust dynamics in safety-critical scenarios; high practical significance.

Related Papers

• [ICLR 2026] Understanding and Improving Continuous Adversarial Training for LLMs via In-Context Learning Theory
• [ICLR 2026] Adversarial Déjà Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
• [ICLR 2026] Efficient Adversarial Attacks on High-dimensional Offline Bandits
• [ICLR 2026] Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs
• [ICLR 2026] Sampling-aware Adversarial Attacks against Large Language Models