Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Tecitomous/MASP-Proactive-Dialogue
Area: LLM Safety
Keywords: Proactive Dialogue, Mentalized Adversarial Self-Play, Soft User Bias, Belief-Desire-Intention (BDI), Rationality-Constrained Success Rate (RCSR)
TL;DR¶
Proposes MASP, formulating proactive dialogue as a partially observable game over a BDI latent mental state space. By decoupling reward-side frozen teacher mentalization from policy-side student estimation, introducing user-relative progress rewards, and optimizing both agents via rationality-constrained alternating GRPO self-play, MASP eliminates soft user bias and drastically improves task success against resistant, realistic users.
Background & Motivation¶
Proactive dialogue tasks—including charity persuasion, emotional support, empathetic conversation, and buyer-seller price bargaining—demand far more than generating contextually fluent responses. An intelligent agent must strategically guide an extended multi-turn interaction toward a global goal while continuously anticipating how its utterances alter the user's underlying beliefs, desires, and behavioral decisions. Standard large language models, while highly capable in reactive conversation, struggle with multi-turn proactive control because next-token likelihood maximization provides no long-range strategic foresight. To address this limitation, recent literature has explored test-time Monte Carlo tree search, trainable policy planners, and user world modeling.
However, existing frameworks suffer from a fundamental training asymmetry: the dialogue system is optimized as the sole strategic learner, while the interacting user is simulated by a static rule set, prompted role-play, or an unadapted neural generator. Consequently, these simulators exhibit an inherent "soft user bias," readily succumbing to conversational pressure and conceding without realistic hesitation. Policies trained and evaluated against such overly compliant simulators achieve deceptively high success rates by learning brittle shortcuts, which collapse completely when deployed against genuine users who harbor skepticism, practical constraints, or rational resistance. While recent works attempt to harden test environments via dedicated user LMs or iterative adversarial refinement, they do not integrate rational user resistance into the core optimization objective of the dialogue policy itself.
The critical insight is that user resistance should not serve merely as a hardened test benchmark, but as an adaptive training partner in a dynamic multi-agent game with explicit mental state evolution. Core idea: formulate proactive dialogue as a partially observable game over an evolving BDI (Belief-Desire-Intention) latent mental state geometry, decoupling frozen teacher mentalization for reward attribution from student inference for policy conditioning, and alternating GRPO self-play updates with user-relative progress and rationality-constrained resistance rewards.
Method¶
Overall Architecture¶
MASP models the multi-turn interaction as a partially observable Markov game between a system policy \(\pi_\theta\) and a user policy \(\pi_\psi\). At each conversational turn \(t\), the user's hidden internal condition is captured by a latent mental state \(z_t\) combining natural language descriptions of belief, desire, and intention with continuous affective variables. The user policy \(\pi_\psi(\cdot \mid h_t, z_t, p_u, g)\) conditions directly on this true hidden state and user profile \(p_u\) to formulate rational responses. In contrast, the system policy \(\pi_\theta(\cdot \mid h_t, \hat{z}_t, g)\) observes only an estimated mental state \(\hat{z}_t\) predicted from dialogue history \(h_t\) by a student mentalizer. The complete pipeline executes in three stages: Phase 0 pre-trains teacher and student mentalizers on silver BDI annotations and freezes the teacher; Phase 1 warm-starts both policies via behavior cloning on human transcripts to serve as supervised references; Phase 2 executes alternating GRPO self-play updates with frozen reward estimators, driving both the system and the user to co-evolve against each other.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Initial Context h0 & Task Goal g"] --> B["BDI Hybrid Latent Mental State Representation<br/>(Bt, Dt, It text embeddings + affective scalars)"]
B --> C["Teacher-Student Mentalization Decoupling<br/>(Frozen teacher Fω for reward / Student Mϕ for policy)"]
C --> D["User-Relative Progress in Latent Space<br/>(Normalized projection from initial z0 to target z*)"]
D --> E["Bidirectional Rational Adversarial Rewards<br/>(System progress + inference vs User rational resistance)"]
E --> F["Alternating GRPO Self-Play Optimization<br/>(Grouped advantages + KL anchors to BC references)"]
F --> G["Output: Robust Proactive System Policy πθ"]
Key Designs¶
1. BDI Hybrid Latent Mental State Representation: semantic interpretability versus metric tractability Pure hidden-state dialogue embeddings lack explicit semantic grounding, making it impossible to establish standardized target states across different domains, whereas rigid dialogue-act slot schemas fail to transfer across diverse tasks such as persuasion, therapy, and negotiation. MASP resolves this dichotomy by adopting a Belief-Desire-Intention (BDI) hybrid representation. At turn \(t\), the user state is expressed as a 6-tuple \(m_t = (B_t, D_t, I_t, \rho_t, c_t, v_t)\), where \(B_t, D_t, I_t\) represent free-form natural language strings capturing what the user currently believes, desires, and intends to do, while scalar coordinates \(\rho_t \in [0, 1]\), \(c_t \in [0, 1]\), and \(v_t \in [-1, 1]\) track receptivity, commitment/confidence, and emotional valence, respectively. Each text component is encoded via a frozen text backbone \(E(\cdot)\) into mean-pooled, unit-normalized embeddings \(e_B, e_D, e_I \in \mathbb{R}^d\). The unified metric vector is concatenated as: $\(z_t = [e_B; e_D; e_I; \sqrt{\alpha_\rho}\rho_t; \sqrt{\alpha_c}c_t; \sqrt{\alpha_v}v_t]\)$ This hybrid parameterization pairs the expressiveness of natural language mentalization with the metric stability of continuous vectors, allowing precise geometric distance calculations.
2. Teacher-Student Mentalization Decoupling: preventing policy exploitation of adaptive reward estimators When a single mental-state estimator is shared between policy conditioning and reward computation, reinforcement learning agents routinely discover degenerate policies that exploit estimator blind spots, causing reward hacking. MASP addresses this via an asymmetric teacher-student decoupling. An offline teacher mentalizer \(F_\omega(h_t, p_u, g)\), pre-trained on silver annotations generated by an LLM-based Oracle Belief Updater (OBU), possesses privileged observation of the conversation history, task goal, and private user profile \(p_u\). Once trained, \(F_\omega\) is strictly frozen and used exclusively on the reward side. In contrast, the system policy only observes state estimates produced by a student mentalizer \(\hat{z}_t = M_\phi(h_t)\), which relies strictly on observable dialogue history. Continuous student predictions are rendered into textual control prompts via cosine nearest-neighbor retrieval over a pre-computed BDI prior cache. This division prevents the system policy from gaming the reward function and forces it to actively steer dialogue to deduce genuine user mental dynamics.
3. User-Relative Progress in Latent Space: non-uniform initial states and sparse outcome rewards In proactive interaction, different users start from vastly differing initial attitudes: a naturally sympathetic user begins close to target acceptance, while a staunch skeptic starts far away. Relying on absolute Euclidean distance to a global target \(z^\star\) disproportionately rewards trivial initial states and penalizes hard-fought partial improvements. MASP establishes user-relative progress by projecting the current mental displacement onto the vector connecting the user's specific initial state \(z_0\) to the task-level target \(z^\star\): $\(s_t = \frac{(z_t - z_0)^\top A (z^\star - z_0)}{(z^\star - z_0)^\top A (z^\star - z_0) + \varepsilon_{\text{prog}}}\)$ where \(A = \text{diag}(\frac{1}{3d} I_{3d}, 1, 1, 1)\) balances high-dimensional textual embeddings against the three scalar metrics. The step-wise incremental reward is \(\Delta s_t = s_{t+1} - s_t\). This formulation measures directional velocity rather than absolute position, converting sparse end-of-dialogue goals into a dense, calibrated per-turn trajectory signal.
4. Bidirectional Rational Adversarial Rewards: avoiding degenerate unconstrained rejection and sycophantic concession Training a user simulator purely as an unconstrained adversary causes optimization collapse: the user policy quickly learns to blindly reject every proposal regardless of system logic. Conversely, rewarding concession recreates the soft user pathology. MASP resolves this via a dual-criteria reward structure incorporating an external LLM rationality judge \(R_{\text{rat}}\) producing feedback \(q_t \in \{-1, 0, 1\}\) (where \(1\) denotes a rational response given internal mental state, \(-1\) denotes irrationality, and \(0\) indicates parsing failure). The reward functions for the system and user are formulated as: $\(r^S_t = w_{\text{prog}} \Delta s_t - w_{\text{inf}} \|\hat{z}_t - z_t\|_A^2\)$ $\(r^U_t = -w_{\text{prog}} \Delta s_t + w_{\text{rat}} q_t\)$ The system is incentivized to maximize task progress (\(\Delta s_t > 0\)) while minimizing its mentalization error (\(\|\hat{z}_t - z_t\|_A^2\)). Meanwhile, the user policy is rewarded for resisting premature concessions (\(-w_{\text{prog}}\Delta s_t\)) yet penalized whenever its resistance becomes irrational. As a result, the user evolves into an authentic sparring partner that concedes only when supplied with convincing justification.
Loss & Training¶
During Phase 2, MASP alternates utterance-level Grouped Relative Policy Optimization (GRPO) between system and user agents across \(K\) outer iterations. In the system update phase, the user policy \(\pi_\psi\) and reward estimators are frozen; dialogue rollouts are grouped into bundles \(G\), discounted returns are normalized into relative advantage estimates \(\hat{A}_i\), and the system LoRA parameters are optimized using clipped ratio loss regularized by a token-level KL divergence penalty against the Phase 1 supervised warm-start policy \(\pi_{\theta 0}\). In the user update phase, the system policy \(\pi_\theta\) is frozen while the user LoRA parameters are updated with identical GRPO objectives anchored to \(\pi_{\psi 0}\). Following each self-play cycle, the student mentalizer \(M_\phi\) is refreshed via mean squared error regression on accumulated self-play prefixes mixed with offline real dialogue data, tracking frozen teacher targets under a stop-gradient barrier.
Key Experimental Results¶
Main Results¶
MASP is evaluated across four multi-turn proactive dialogue benchmarks: Persuasion for Good (P4G), Emotional Support Conversation (ESConv), EmpatheticDialogues, and CraigslistBargain. Baselines include Supervised Fine-Tuning (SFT) and Planner-only scaffolds across Qwen3-4B and Qwen3-0.6B backbones. Evaluations report task Success Rate within 8 turns (SR@8), Average Turns (AT, lower is better), and Rationality-Constrained Success Rate (RCSR@8), which counts a dialogue as successful only when the prefix rationality score satisfies \(\bar{q}_{\tau_s} \ge 0.70\).
| Dataset | Backbone | Method | Soft User SR↑ | Soft User AT↓ | Soft User RCSR↑ | Hard User SR↑ | Hard User AT↓ | Hard User RCSR↑ |
|---|---|---|---|---|---|---|---|---|
| P4G | Qwen3-4B | SFT | 0.616 | 6.08 | 0.384 | 0.576 | 6.23 | 0.455 |
| Qwen3-4B | Planner-only | 0.677 | 4.88 | 0.434 | 0.616 | 5.55 | 0.384 | |
| Qwen3-4B | MASP (Ours) | 0.859 | 4.08 | 0.556 | 0.788 | 4.56 | 0.566 | |
| Qwen3-0.6B | SFT | 0.414 | 6.99 | 0.253 | 0.354 | 7.35 | 0.242 | |
| Qwen3-0.6B | Planner-only | 0.424 | 6.32 | 0.232 | 0.364 | 6.84 | 0.202 | |
| Qwen3-0.6B | MASP (Ours) | 0.707 | 5.14 | 0.465 | 0.626 | 5.76 | 0.434 | |
| ESConv | Qwen3-4B | SFT | 0.467 | 7.17 | 0.410 | 0.390 | 7.27 | 0.333 |
| Qwen3-4B | Planner-only | 0.969 | 2.38 | 0.923 | 0.980 | 2.08 | 0.918 | |
| Qwen3-4B | MASP (Ours) | 0.990 | 1.96 | 0.949 | 1.000 | 1.88 | 0.964 | |
| Qwen3-0.6B | SFT | 0.297 | 7.56 | 0.226 | 0.323 | 7.32 | 0.241 | |
| Qwen3-0.6B | Planner-only | 0.954 | 2.77 | 0.897 | 0.990 | 1.84 | 0.903 | |
| Qwen3-0.6B | MASP (Ours) | 1.000 | 2.11 | 0.949 | 1.000 | 1.65 | 0.928 | |
| Craigslist | Qwen3-4B | SFT | 0.646 | 5.38 | 0.045 | 0.558 | 6.28 | 0.234 |
| Qwen3-4B | Planner-only | 0.431 | 5.67 | 0.020 | 0.315 | 6.86 | 0.091 | |
| Qwen3-4B | MASP (Ours) | 0.689 | 4.60 | 0.073 | 0.621 | 5.61 | 0.259 | |
| Qwen3-0.6B | SFT | 0.726 | 4.84 | 0.057 | 0.547 | 6.47 | 0.191 | |
| Qwen3-0.6B | Planner-only | 0.172 | 7.07 | 0.011 | 0.484 | 6.41 | 0.129 | |
| Qwen3-0.6B | MASP (Ours) | 0.728 | 4.24 | 0.030 | 0.641 | 5.50 | 0.211 | |
| Empathetic | Qwen3-4B | SFT | 0.030 | 7.89 | 0.017 | 0.022 | 7.91 | 0.013 |
| Qwen3-4B | Planner-only | 0.042 | 7.75 | 0.027 | 0.037 | 7.78 | 0.024 | |
| Qwen3-4B | MASP (Ours) | 0.092 | 7.46 | 0.059 | 0.072 | 7.57 | 0.048 | |
| Qwen3-0.6B | SFT | 0.013 | 7.95 | 0.007 | 0.002 | 7.99 | 0.001 | |
| Qwen3-0.6B | Planner-only | 0.032 | 7.82 | 0.023 | 0.021 | 7.88 | 0.013 | |
| Qwen3-0.6B | MASP (Ours) | 0.063 | 7.64 | 0.044 | 0.036 | 7.78 | 0.024 |
Ablation Study & Robustness Analysis¶
A comprehensive component ablation on the P4G benchmark (Qwen3-4B backbone, 99 test dialogues, budget \(T=8\), threshold \(\tau_q=0.70\)) isolates the distinct contributions of each module.
| Variant | Removed or Modified Component | Soft SR↑ | Soft AT↓ | Soft RCSR↑ | Hard SR↑ | Hard AT↓ | Hard RCSR↑ |
|---|---|---|---|---|---|---|---|
| Full MASP | None (Complete framework) | 0.859 | 4.08 | 0.556 | 0.788 | 4.56 | 0.566 |
| No adaptive close | Remove adaptive close injection | 0.707 | 4.86 | 0.556 | 0.707 | 4.95 | 0.495 |
| No warmup | Remove P4G warmup policy | 0.828 | 3.83 | 0.283 | 0.727 | 4.55 | 0.283 |
| No eval scaffold | Remove warmup policy & adaptive close | 0.606 | 6.12 | 0.404 | 0.626 | 6.28 | 0.394 |
| No adversarial self play | Planner-only scaffold (no self-play) | 0.677 | 4.88 | 0.434 | 0.616 | 5.55 | 0.384 |
| No GRPO self play | Supervised Phase 1 policy only | 0.616 | 6.08 | 0.384 | 0.576 | 6.23 | 0.455 |
In addition, evaluating across an ensemble of 3 distinct user simulators \(\mathcal{U}\) on P4G demonstrates robust generalization: - Worst-case SR: MASP achieves 0.737, substantially outperforming Planner-only (0.616) and SFT (0.545). - Worst-case RCSR: MASP retains 0.556, whereas Planner-only and SFT drop to 0.384. - Turn-level Trajectory Analysis (Table 3): Against the hard user, MASP exhibits superior rationality-constrained performance from early stages, achieving RCSR@3 of 0.444 and RCSR@4 of 0.485 (compared to Planner-only's 0.212 / 0.273 and SFT's 0.071 / 0.141), proving that MASP's performance stems from early strategic alignment rather than late lucky guesses.
Key Findings¶
- The "No Warmup" Shortcut Trap: Removing the warmup policy reduces average turns on soft users to a blazing 3.83 while maintaining 0.828 SR. However, its RCSR collapses from 0.556 down to 0.283. This demonstrates that raw SR/AT metrics reward aggressive, premature closing tactics that exploit compliant simulators but violate genuine human conversational rationality.
- Self-Play Overcomes Hard User Resistance: On the hard user setting, MASP outperforms Planner-only in RCSR by 18.2 points (0.566 vs 0.384 on P4G), proving that co-training against an evolving, rationally resistant user forces the system to develop grounded persuasive logic rather than relying on static prompt scripts.
Highlights & Insights¶
- Internalizing Theory-of-Mind into the Optimization Loss: Instead of treating user mental states as frozen prompt context or downstream evaluation features, MASP incorporates the mental state estimation error \(\|\hat{z}_t - z_t\|^2_A\) directly into the policy reward, compelling the agent to balance task persuasion with active state discovery.
- Rationality-Gated Adversarial Formulation: By formulating the adversary reward as \(r^U_t = -w_{\text{prog}}\Delta s_t + w_{\text{rat}}q_t\), the framework elegantly sidesteps both degenerate adversarial rejection and sycophantic concession, establishing a robust paradigm for LLM multi-agent self-play.
- RCSR Metric Exposes Evaluator Exploitation: The introduction of Rationality-Constrained Success Rate (\(\bar{q}_{\tau_s} \ge 0.70\)) reveals pervasive shortcut exploitation in task-oriented dialogues, providing a rigorous blueprint for future multi-agent benchmark construction.
Limitations & Future Work¶
- Dependence on Synthetic Annotations and LLM Judges: Phase 0 mentalization pre-training and Phase 2 rationality scoring rely on commercial LLM judges (GPT-4o and GPT-3.5); systematic biases or parsing errors in these upstream modules could propagate into policy behavior.
- Computational Overhead of Bilateral Self-Play: Alternating GRPO across two separate LoRA policy networks (\(\pi_\theta, \pi_\psi\)) while refreshing student mentalizers requires significantly higher rollout and gradient compute than single-agent training.
- Reduced Privileged Gap Without User Profiles: When task datasets lack explicit user personas (\(p_u = \emptyset\)), the observational divergence between teacher and student narrows, reducing the teacher's role to a stabilizing variance regularizer rather than a rich privileged observer.
Related Work & Insights¶
- vs PPDPP & GDP-Zero (Deng et al., 2023; Yu et al., 2023): Prior planning and MCTS approaches optimize system action selection against fixed, non-adaptive user simulators; MASP introduces dynamic co-evolutionary self-play, immunizing the policy against soft simulator biases.
- vs Dial & Flipping the Dialogue (Zhu et al., 2025; Naous et al., 2025): While recent user simulator works improve evaluation realism by refining static test environments, MASP directly incorporates the adversary into the inner loop of policy reinforcement learning.
- vs GAIA (Wang et al., 2025a): GAIA uses zero-sum self-play with Chain-of-Mind planning for non-collaborative dialogue; MASP extends this to partially observable games with continuous BDI mental geometries and calibrated user-relative progress shaping.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulating proactive dialogue as a partially observable BDI game with rationality-constrained adversarial self-play is highly original.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 4 benchmarks, featuring comprehensive ablation, multi-simulator stress tests, and extensive human-AI validation.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid mathematical formalism, compelling narrative on the soft user dilemma, and transparent disclosure of failure modes.
- Value: ⭐⭐⭐⭐⭐ Directly addresses the critical vulnerability of LLM dialogue systems exploiting simulator sycophancy, offering high practical and methodological impact.