Skip to content

Learning User Simulators with Turing Rewards

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/SusanWYS/turing-rl
Area: LLM Reasoning
Keywords: User Simulation, Turing Rewards, Reinforcement Learning, GRPO, Personalized Dialogue

TL;DR

This paper introduces Turing-RL, a reinforcement learning approach for training personalized user simulators that replaces conventional ground-truth response matching with a discriminative Turing reward (scoring how indistinguishable a generated response is from real human behavior conditioned on user history). Optimized via GRPO with chain-of-thought reasoning, Turing-RL consistently outperforms baseline methods on both LLM-as-a-judge and human evaluations across multi-turn chat and Reddit forum discussions.

Background & Motivation

Building large language model (LLM) agents that reliably simulate human users in interactive environments is a crucial stepping stone for constructing interactive training testbeds, evaluating personalized assistant systems, and simulating collective behaviors in computational social science. However, simulating an individual is fundamentally challenging because human identity and conversational behavior resist clean demographic categorization: two individuals sharing identical background traits can express opposing perspectives, and fine-grained personal preferences cannot be recovered from coarse group-level descriptions alone. In open-ended conversations, the space of plausible responses to a given context is immense—the same user facing the same prompt could express their intent in dozens of distinct, valid ways.

Existing user simulation paradigms (such as supervised fine-tuning, HumanLM-style semantic similarity scoring, or log-probability maximization over target tokens) all rely on a common foundational assumption: they derive their learning signals from matching a single ground-truth response. This "response matching" formulation exhibits severe structural limitations. If a model generates a completely plausible, highly in-character reply that happens to diverge in phrasing or topical focus from the single recorded ground truth, it suffers heavy log-probability penalties or low similarity scores. Conversely, optimizing explicitly for ground-truth overlap often encourages models to adopt verbose, cautious, or teacher-like explanations that read like helpful AI assistants rather than authentic, idiosyncratic human users.

To move beyond the confines of point-to-point response matching, the classical Turing Test provides a principled operationalization of human-likeness: true human-likeness is defined by indistinguishability, where an external evaluator cannot tell machine-generated utterances from human ones. Core idea: train LLM user simulators via reinforcement learning on a discriminative Turing reward—where an LLM judge evaluates how indistinguishable a model generation is from the real user's response conditioned on behavioral history and context—and optimize the policy with chain-of-thought reasoning using Group Relative Policy Optimization (GRPO).

Method

Overall Architecture

The Turing-RL pipeline encompasses three primary phases: user persona induction from behavioral history, supervised fine-tuning (SFT) warm-start with latent chain-of-thought (CoT) traces, and reinforcement learning with discriminative Turing rewards via Group Relative Policy Optimization (GRPO). Given a target user's disjoint historical interactions \(h\), an induced stable persona \(\rho\), and the current dialogue context \(x\), the policy model \(\pi_\theta\) first autoregressively generates an internal reasoning trace \(z\) capturing user intent and style, followed by the candidate user response \(\hat{y}\). A discriminative LLM judge evaluates the candidate against the ground-truth response \(y^\star\) under randomized blind presentation, scoring the candidate on immediate target alignment, human goal plausibility, and communication style while enforcing strict anti-gaming penalty checks. The resulting bounded Turing reward updates the policy parameters using group-relative advantages.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: User History h + Current Context x"] --> B["User Persona Induction<br/>Extract stable traits & stylistic quirks ρ"]
    B --> C["Chain-of-Thought Autoregressive Generation<br/>Sample intent trace z + candidate response y"]
    C --> D["Discriminative Turing Judge<br/>Pairwise comparison against ground truth y*"]
    D --> E["Turing Reward Capping & Length Penalty<br/>Prevent reward hacking & length drift"]
    E --> F["GRPO Policy Optimization<br/>Group-relative advantage updates LoRA weights"]

Key Designs

1. User Persona Induction: Behavioral History Distillation without Target Contamination
Directly prompting an LLM with raw historical conversational blocks is inefficient and can overwhelm the model with transient conversational noise. Turing-RL pairs the raw behavior block \(h\) with a structured persona \(\rho\) synthesized by a lightweight auxiliary model (such as GPT-5.4-nano). The induction prompt instructs the auxiliary model to write in the target user's first-person voice across five specific fields: values, observable verbal quirks, expression style, reply length prior, and recurring personal background cues. Critically, \(h\) and \(\rho\) are fixed per user and kept strictly disjoint from both the target evaluation context \(x\) and the ground-truth response \(y^\star\), ensuring zero data leakage into the simulation environment.

2. Chain-of-Thought Autoregressive Generation: Reverse-Engineered Intent Warm-Start
Directly training a policy to output human responses without internal reasoning frequently leads to superficial lexical imitation or generic assistant-like outputs. Turing-RL incorporates an SFT warm-start stage before RL training. For each training instance, a teacher model analyzes the user's history, persona, context, and ground truth \(y^\star\) to reverse-engineer a hidden reasoning trace \(z\) explaining what motivated the user to take that specific conversational action. The student policy is trained with completion-only loss to generate <reasoning>z</reasoning> followed by [HUMAN]: y. During subsequent RL exploration and inference, the policy autoregressively generates its own reasoning trace before producing its utterance, establishing coherent causal grounding for its conversational choices.

3. Discriminative Turing Judge: Multi-Dimensional Scoring with Anti-Gaming Penalties
Instead of measuring surface overlap, the training judge evaluates how indistinguishable the generated response is from the human response. The judge inspects the user history and current context alongside a randomized pair of candidate responses (one human \(y^\star\), one model \(y\)). It evaluates three positive criteria (each scored \(0.0\) to \(1.0\)): immediate target (whether the response reacts to a plausible local point or executes an authentic topic pivot), human goal (whether the underlying motive matches the user's personality rather than an assistant-like service intent), and communication style and length (whether it matches idiosyncratic capitalization, punctuation, informality, and brevity). Furthermore, the judge applies four strict penalty deductions: source-copy penalties for unquoted history regurgitation, wrong target or speaker role errors, unsupported adversarial reframing (inventing fake disputes), and assistant-like output penalties (penalizing structured lists, coaching tone, or boilerplate transitions). The net gap maps into a standardized 1–7 Likert rating \(s(y, y^\star)\).

4. Turing Reward Capping and Asymmetric Length Penalty: Defending Against Reward Exploitation
In preliminary RL experiments, unconstrained policy optimization caused models to over-index on dramatic or stylized prose that judges rated as "more human than human"—a classic manifestation of reward hacking. Turing-RL mitigates this vulnerability by capping the raw judge score at 5 before normalizing into \([0, 1]\): $\(r_{\text{turing}}(y, y^\star) = \frac{\min\{s(y, y^\star), 5\} - 1}{6}\)$ In addition, to prevent conversational models from gaming the evaluation via verbosity inflation, the framework applies an asymmetric length penalty whenever response length \(\ell\) falls outside a tolerance band \([r_{\min}, r_{\max}]\) relative to the ground-truth length \(\ell_{\text{gt}}\): $\(p = \min\left\{\lambda_{\text{short}} \max\left(\frac{r_{\min} - r}{r_{\min}}, 0\right) + \lambda_{\text{long}} \max\left(\frac{r - r_{\max}}{r_{\max}}, 0\right), p_{\text{cap}}\right\}\)$ where \(r = \ell / \ell_{\text{gt}}\), assigning heavier weight to overly terse responses (\(\lambda_{\text{short}} > \lambda_{\text{long}}\)) to preserve the user's natural conversational volume.

Loss & Training

During the RL phase, the policy is optimized using Group Relative Policy Optimization (GRPO). For each prompt, the policy samples \(G=4\) outputs \(o_i = (z_i, \hat{y}_i)\). Rather than training an auxiliary critic model, advantages are computed by standardizing rewards across the sampled group: \(\hat{A}_{i,t} = (r_i - \text{mean}(r)) / \text{std}(r)\), where token-level updates across both reasoning traces and response tokens share the same group-normalized advantage. The objective maximizes PPO-clipped surrogate gains (\(\varepsilon = 0.2\)) alongside a KL divergence penalty (\(\beta = 1 \times 10^{-3}\)) against the reference SFT model. The base policy uses Qwen3-8B (with native thinking mode disabled) and is trained with PEFT LoRA (rank \(r=64\), \(\alpha=32\)) on veRL across 3 epochs.

Key Experimental Results

Main Results

The framework was evaluated on two structurally distinct domains: PRISM (multi-turn human-LLM conversational chat; 128 held-out users, 880 target turns) and ConvoKit (Reddit forum threads from r/tifu and r/worldnews; 102 held-out users, 267 target comments). Automatic evaluations were conducted using Claude Sonnet 4.6 (a judge disjoint from the training judge Qwen3.5-397B), complemented by a 360-participant forced-choice human Turing test on Prolific.

Dataset / Dimension Evaluation Metric Qwen3-8B Base SFT-Init Logprob-RL Sim-RL Turing-RL (Ours)
Chat (PRISM) Turing Distinguishability (1–7) ~3.30 3.65 ~3.45 3.75 4.31 ± 0.08
Response Similarity Sim (%) ~2.5% 3.8% ~2.0% 5.2% 5.3 ± 0.8%
Grounded Specificity [0, 1] ~0.44 0.46 ~0.45 0.49 0.512 ± 0.009
Human Win Rate WR (vs GT) — 0.49 ± 0.061 — 0.50 ± 0.055 0.57 ± 0.050
Reddit (ConvoKit) Turing Distinguishability (1–7) ~3.20 3.52 ~3.30 3.58 3.68 ± 0.18
Response Similarity Sim (%) ~1.8% 2.6% ~1.5% 3.4% 3.0 ± 0.9%
Grounded Specificity [0, 1] ~0.33 0.34 ~0.33 0.36 0.367 ± 0.020
Human Win Rate WR (vs GT) — 0.41 ± 0.045 — 0.52 ± 0.049 0.50 ± 0.051

In an out-of-domain transfer evaluation on the first 100 users of the task-oriented benchmark MS-TOD, the Chat-trained Turing-RL simulator demonstrated robust zero-shot generalization:

Model Variant Turing Score (1–7) Ground-Truth Similarity Sim ([0, 1])
SFT-Init 3.265 0.1791
Sim-RL 3.645 0.3133
Turing-RL (Ours) 3.857 0.3190

Ablation Study

The authors ablated user representations \(u\) across three distinct configurations: full input \(u = (h, \rho)\) combining raw history and persona, history only \(u = h\), and persona only \(u = \rho\) (Turing score on 1–7 scale, Sim in %, Specificity in \([0, 1]\)):

Representation Setting Domain Turing Score Similarity Sim (%) Specificity
Full \(u = (h, \rho)\) (History + Persona) Chat 4.31 ± 0.08 5.3 ± 0.8 0.512 ± 0.009
History only \(u = h\) Chat 4.26 ± 0.08 4.7 ± 0.8 0.497 ± 0.010
Persona only \(u = \rho\) Chat 4.23 ± 0.08 4.1 ± 0.7 0.509 ± 0.009
Full \(u = (h, \rho)\) (History + Persona) Reddit 3.68 ± 0.18 3.0 ± 0.9 0.367 ± 0.020
History only \(u = h\) Reddit 3.78 ± 0.18 3.3 ± 1.0 0.364 ± 0.019
Persona only \(u = \rho\) Reddit 3.84 ± 0.19 2.7 ± 0.7 0.329 ± 0.018

Ablating the auxiliary model used for persona induction (comparing GPT-5.4-nano, Qwen3-8B with thinking, and Claude Opus 4.8) revealed that scaling the persona inducer yields negligible differences in downstream simulation performance, confirming that compact behavioral traits are sufficient.

Key Findings

  • Content Matching and Indistinguishability Diverge: While optimizing for response similarity (Sim-RL) successfully increases keyword and semantic overlap with the single recorded target, it fails to improve the Turing score over the baseline SFT checkpoint. In contrast, Turing-RL dramatically enhances human indistinguishability while simultaneously preserving high content similarity.
  • Frontier LLMs Struggle with Organic Human Persona: Untrained frontier models (such as GPT-5 and Qwen3.5-397B) perform surprisingly poorly on the Turing metric. Their outputs tend to be overly verbose, structured, and hedged, immediately revealing an assistant-like conversational stance rather than organic human user behavior.
  • Human vs. LLM Evaluator Dynamics: On Chat, Turing-RL achieved a human win rate of \(0.57\) against ground truth, significantly outperforming SFT-Init (\(p = 0.044\)) and Sim-RL (\(p = 0.022\)) in paired permutation tests. However, human judges experienced greater cognitive fatigue on Reddit due to high text volume, where Sonnet 4.6 demonstrated superior consistency and discrimination accuracy.

Highlights & Insights

  • Conceptual Shift in Training Paradigms: Reframing user simulation as a distribution indistinguishability challenge rather than point-to-point text regression provides a principled mathematical formulation for human behavior modeling.
  • Multi-Factor Anti-Gaming Judge Formulation: Combining positive criteria (target, motive, style) with explicit penalty checks for verbatim copying, adversarial tangents, and assistant-like phrasing prevents RL policies from degenerating into degenerate local optima.
  • Principled Reward Bounding and Length Regularization: Capping the maximum reward at 5 prevents the policy from chasing exaggerated persona caricatures, while the asymmetric length penalty prevents length exploitation without stifling conversational variety.

Limitations & Future Work

  • Reliance on Massive Evaluator Models: Training-time reward generation relies on a 397B-parameter MoE model (Qwen3.5-397B-A17B), incurring substantial computational cost and potentially inheriting the judge's systemic blind spots.
  • Constrained Interaction Modalities: The current empirical validation is focused on linear multi-turn chat and public forum threads; broader settings such as multi-party negotiations, collaborative task execution, and goal-directed agent interactions remain unexplored.
  • Dual-Use and Impersonation Risks: Highly authentic user simulators can potentially be repurposed for non-consensual impersonation, spear-phishing, or automated astroturfing, underscoring the urgent necessity of watermarking frameworks and synthetic text detectors.
  • vs HumanLM (Wu et al., 2026): HumanLM trains simulators via psychological state alignment and multi-attribute similarity rewards. Turing-RL shows that explicitly optimizing for similarity can constrain generation diversity, whereas a discriminative Turing reward fosters natural human-like variation.
  • vs Logprob-RL (Gandhi et al., 2026): Gandhi et al. optimize dialogue simulators by maximizing the log-probability of ground-truth responses under latent CoT. However, log-probability heavily penalizes valid alternative lexical formulations, whereas Turing-RL assesses global plausibility.
  • vs PersonaLLM / Odyssim (Jiang et al., 2024; Zhou et al., 2026a): While earlier efforts rely on demographic prompting or verbal feedback on static profiles, Turing-RL combines behavioral history conditioning with RL-driven indistinguishability optimization, preventing persona collapse.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Directly operationalizes the classic Turing Test into an RL reward signal for user simulation, breaking the bottleneck of ground-truth response matching]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans multi-turn chat and forum discussions, comprehensive ablations, out-of-domain transfer to MS-TOD, and an extensive 360-participant human study]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous, coherent narrative flow, clear mathematical formalization, and exhaustive appendix documentation of all prompts and training dynamics]
  • Value: ⭐⭐⭐⭐⭐ [Provides a foundational methodology for conversational agent evaluation, synthetic user generation, and computational social science simulation]