Simulating Multimodal Multiparty Student Interactions with Large Language Models for Training Adaptive Pedagogical Agents¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Dialogue Systems
Keywords: user simulation; multimodal interaction; multiparty dialogue; reinforcement learning; pedagogical agents
TL;DR¶
This paper introduces an LLM-based user simulation framework for triadic robot-student collaboration that serializes gaze and gestures into inline textual stage directions, using structural phase tagging and dynamic persona reinforcement to effectively train adaptive pedagogical reinforcement learning policies.
Background & Motivation¶
In multiparty educational tutoring, pedagogical robots must not only comprehend and generate verbal dialogue but also regulate subtle social dynamics, such as turn-taking, gaze-mediated addressee selection, and non-verbal backchanneling. Reinforcement learning (RL) offers a principled paradigm for optimizing such adaptive pedagogical policies; however, its application in real classrooms is heavily bottlenecked by data scarcity and safety concerns. Directly exposing novice learners to exploratory RL policies during trial-and-error training is prohibitively expensive, time-consuming, and ethically risky. While user simulation has long served as a cost-effective surrogate, conventional rule-based or agenda-based simulators lack linguistic diversity, and modern LLM-based simulators remain largely confined to dyadic, text-only interactions, disregarding critical non-verbal channels like gaze and gesture.
This omission is especially detrimental in multiparty tutoring: who looks at whom is frequently as decisive as what is being articulated. A simulator oblivious to non-verbal cues prevents the agent from learning when to intervene, how to establish joint attention, or how to rebalance asymmetric student engagement. Furthermore, deploying end-to-end multimodal foundation models incurs substantial latency and compute overhead, while multi-turn role-playing in LLMs regularly suffers from "persona drift" and mode collapse, causing individual characters to regress toward a bland, homogeneous conversational baseline.
This work addresses the problem by recognizing that rich non-verbal social signals can be effectively discretized and serialized into textual inline "stage directions" within the prompt context, allowing standard text-only LLMs to simulate complex social dynamics without requiring custom multimodal architectures. Core idea: encode gaze and gesture directly as inline textual stage directions within dialogue history, combined with two-phase discourse tagging and per-turn dynamic instruction reinforcement (DIR), constructing a controllable, high-fidelity multimodal multiparty simulator that successfully trains adaptive pedagogical policies across online asynchronous and offline RL paradigms.
Method¶
Overall Architecture¶
The proposed simulation framework couples three primary modules in a closed-loop environment: an AIML-based dialogue manager generating deterministic base verbal tutoring content, an RL agent governing the social delivery (factorized gaze and verbal reference actions), and an off-the-shelf text-only LLM (such as Qwen3-14B-Chat or Mistral-24B-Instruct) functioning as the student simulator. The collaborative math problem-solving session between a robot tutor (Emma) and two students is modeled as a Partially Observable Markov Decision Process (POMDP). The state representation consists of a sliding window of tokenized dialogue history incorporating inline stage directions, and the agent outputs discrete communicative actions; in turn, the LLM generates a structured two-phase multimodal response reflecting both peer debate and student-to-tutor explanation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Dialogue State Input<br/>Sliding-window history with stage directions"] --> B["Factorized Action Selection<br/>Verbal reference R & gaze direction G"]
B --> C["Response Augmentation Module<br/>Inject stage directions into AIML base text"]
C --> D["Structural Phase-Tagged Generation<br/>Hard constraints on Phase 1 & Phase 2"]
D --> E["Dynamic Instruction Reinforcement DIR<br/>Per-turn persona prompt injection"]
E --> F["Simulated Response & State Update<br/>Generated text and student gaze tokens"]
Key Designs¶
1. Textual Encoding of Multimodal Signals: Bidirectional Factorized Social Signaling
Rather than relying on resource-intensive multimodal generative architectures, the framework establishes a bidirectional textual serialization mechanism that casts non-verbal behaviors into parenthetical "stage directions." In the Robot-to-Sim input direction, the RL agent's discrete choices are translated into descriptors appended to the robot's line, e.g., Emma: "What do you think is next?" (Emma looks at the monitor and shrugs), explicitly directing joint visual attention. In the Sim-to-Robot output direction, the LLM is instructed to embed parenthetical student actions (e.g., (Student A looks at Student B)), which the environment parses directly into the agent's observation space. The agent's action space is factorized into independent components \(\mathcal{A} = \mathcal{R} \times \mathcal{G}\) (\(|\mathcal{A}| = 12\)), where verbal reference \(\mathcal{R} \in \{\text{Student A}, \text{Student B}, \text{None}\}\) and gaze direction \(\mathcal{G} \in \{\text{Student A}, \text{Student B}, \text{Monitor}, \text{Thinking}\}\). Keeping \(\mathcal{R}\) and \(\mathcal{G}\) decoupled allows the policy to execute functionally necessary deictic gestures, such as addressing a student verbally while looking at the shared problem workspace.
2. Structural Phase-Tagged Prompting: Enforcing Collaborative Discourse Norms
Natural peer learning interactions follow a well-defined two-stage sequence: students first deliberate collaboratively (Peer Discussion) before formulating a consensus explanation to the tutor (Explanation). When prompted naively, LLMs routinely bypass peer interaction to immediately address the tutor. To prevent this structural breakdown, the framework enforces a rigid Phase Tagging schema:
- [Phase 1: Peer Discussion]: Students A and B discuss the mathematical problem strictly among themselves, and are explicitly forbidden from addressing Emma.
- [Phase 2: Explanation to the Robot]: Exactly one student presents the agreed-upon procedural step to the robot tutor.
This structural partitioning imposes hard boundaries on the conversational flow, ensuring the simulated interaction faithfully reflects authentic collaborative pedagogical protocols.
3. Student Primitives and Dynamic Instruction Reinforcement (DIR): Mitigating Persona Drift
Over prolonged multi-turn horizons, LLMs typically suffer from context dilution, causing distinct participant profiles to regress toward an average persona. The framework introduces "Student Primitives" covering five discrete confidence tiers (\(P \in \{1, \dots, 5\}\), from Extremely Shy to Very Confident) as control knobs to manipulate initiative-taking propensities and floor-holding tendencies. To sustain these behavioral traits across full episodes, Dynamic Instruction Reinforcement (DIR) appends a concise, high-priority instruction solely to the current prompt context at every turn (e.g., specifying that "Student A is shy and rarely speaks first, while Student B is dominant"). This dynamic system note is strictly excluded from the persisted dialogue history, ensuring enduring behavioral divergence across imbalance tiers without bloating the long-term context buffer.
4. ICAP-Grounded Multi-Objective Reward & Asynchronous Policy Optimization
To guide the agent toward meaningful pedagogical outcomes, the reward function synthesizes participation equity and collaborative knowledge construction grounded in the ICAP framework:
with \(\alpha_E = \alpha_D = 0.5\). Here, \(R_E(t)\) denotes Word Count Entropy (WCE), penalizing verbal participation skew, while \(R_D(t)\) represents the Dialogue Expressivity Ratio, quantifying lexical alignment and shared unique vocabulary between peers. To circumvent the 30-second-per-step latency bottleneck of sequential LLM inference, the environment employs Asynchronous PPO (APPO) via the IMPALA architecture, decoupling parallelized vLLM simulation actors from the central policy learner. V-trace off-policy corrections reconcile behavioral lag, boosting system throughput to 9.97 aggregate steps/sec compared to 1.46 steps/sec under synchronous PPO.
A Worked Example¶
Consider a ratio problem session involving Student A (Shy, \(P=2\)) and Student B (Extremely Shy, \(P=1\)). Robot Emma introduces the task: Emma: So to make the punch... I may need to make up to eight pitchers. (Emma nods).
In response, the simulator generates Phase 1: (Student A glances at Student B, then back at the paper) Student A: Hmm, maybe we need to figure out how long it takes to make one pitcher first? (Student B looks up briefly, then back down) Student B: (Nods slowly) Maybe.
Next, Emma selects an action with Gaze directed at Student A. The LLM then executes Phase 2, prompting only the cued student to address Emma: (Emma looks at Student A) Student A: Letโs start with the smallest amount you might need... If you can figure out how long it takes to make one pitcher...
This interaction demonstrates the joint operation of inline stage directions, structural phase boundaries, and stable personality expressions within a single decision step.
Key Experimental Results¶
Main Results¶
The framework was validated against an IRB-approved corpus of 12 undergraduate pairs (71 conversations, 737 turns, 2,770 multimodal utterances) engaging with a physical NAO robot. The table below presents the semantic fidelity (BERTScore F1) against human transcripts, conversational compliance (Phase Coherence Score, PCS), and Off-Policy Evaluation (OPE) via Doubly Robust estimation (\(V_{\text{DR}}\)) on real-world interaction logs.
| Evaluation Dimension / Task | Metric | Ours (Mistral-Full / Offline-Sim) | Baseline (Qwen-Full / PPO Online) | Real Reference / Ceiling (Human / Offline-Real) |
|---|---|---|---|---|
| Linguistic Realism (Overall) | BERTScore F1 \(\uparrow\) | 0.8626 | 0.8741 | 0.9019 (Inter-Speaker Consistency) |
| Linguistic Realism (Phase 1) | BERTScore F1 \(\uparrow\) | 0.8600 | 0.8943 | 0.9223 |
| Linguistic Realism (Phase 2) | BERTScore F1 \(\uparrow\) | 0.8747 | 0.8728 | 0.9093 |
| Social Turn Obedience | Phase Coherence Score (PCS) | 0.443 (95% CI: [0.381, 0.504]) | 0.237 (95% CI: [0.207, 0.268]) | 0.366 (95% CI: [0.300, 0.433]) |
| Real-World OPE Evaluation | Doubly Robust \(V_{\text{DR}} \uparrow\) | 1.009 (Offline-Simulated DQN) | 0.921 (PPO Online) / 0.990 (Random) | 1.126 (Offline-Real Upper Bound) |
Ablation Study¶
The table below contrasts policy variants across in-simulation reward, sampling curriculum, and real-data OPE estimators (Direct Method \(V_{\text{DM}}\), Doubly Robust \(V_{\text{DR}}\), and Importance Sampling \(V_{\text{IS}}\)):
| Policy / Configuration | Sampling & Algorithm | In-Sim Reward (Mean ยฑ SD) | \(V_{\text{DR}}\) (Doubly Robust) | \(V_{\text{IS}}\) (Importance Sampling) | Evaluation Stability / Note |
|---|---|---|---|---|---|
| Random (Baseline) | N/A | 4.8713 ยฑ 2.6707 | 0.990 (0.945โ1.036) | 0.999 (0.967โ1.026) | Uniformly random actions |
| PPO (Online) | Uniform / PPO | 4.3401 ยฑ 2.8202 | 0.921 (0.673โ1.223) | 0.914 (0.674โ1.172) | Low throughput hurts sample efficiency |
| Uniform (Online) | Uniform / APPO | 4.7386 ยฑ 3.1636 | 0.933 (0.583โ1.214) | 0.966 (0.823โ1.147) | Uniform sampling across pairs |
| ImbalanceFocused (Online) | Imbalanced / APPO | 5.1612 ยฑ 3.1446 | 1.012 (0.489โ1.456) | 0.953 (0.709โ1.166) | Learns robust intervention on skewed dyads |
| Offline-Simulated (DQN) | Uniform / DQN | 4.7927 ยฑ 3.1943 | 1.009 (0.936โ1.117) | 1.002 (0.979โ1.052) | Trained solely on synthetic simulator logs |
| No Gaze/Gesture (Ablation) | Imbalanced / APPO | 5.1299 ยฑ 2.7334 | Unreliable (> 10โถ) | Unreliable (> 10โถ) | Ablating non-verbal cues causes severe distribution shift |
| Offline-Mixed (50/50) | Uniform / DQN | 5.5675 ยฑ 2.9760 | 1.056 (0.931โ1.162) | 0.988 (0.906โ1.075) | Reference augmented policy |
| Offline-Real (Reference) | Uniform / DQN | 4.8101 ยฑ 3.3901 | 1.126 (0.899โ1.414) | 0.914 (0.707โ1.103) | Theoretical ceiling memorized on real logs |
Key Findings¶
- Crucial Role of Multimodal Grounding: When non-verbal cues are omitted (
No Gaze/Gestureablation), the agent achieves a high in-simulation reward of 5.13, but OPE on real human transcripts suffers catastrophic failure: standard importance sampling weights explode (\(> 10^6\)), and effective sample size collapses (\(ESS < 2.0\)). This empirical failure diagnoses extreme distributional shift, proving that inline gaze and gestures provide essential structural constraints aligning simulator state dynamics with reality. - DIR Successfully Controls Personality Imbalance: Word Count Entropy decreases monotonically as assigned imbalance rises from Level 0 to Level 4 (mean WCE drops from 0.94 to 0.58, Kruskal-Wallis \(H(4) = 77.8, p < 0.001\)). Ablating DIR causes context dilution over long episodes, erasing behavioral divergence between shy and dominant profiles.
- Offline Learning on Synthetic Logs Scales Robustly: A DQN policy trained purely on 6 million synthetic transitions generated by the simulator achieves \(V_{\text{DR}} = 1.009\), matching live online APPO policies (\(V_{\text{DR}} = 1.012\)). This validates an offline paradigm that bypasses LLM inference latency during policy optimization.
- Emergence of Natural Pedagogical Strategies: The trained agent autonomously learns "Congruent Cueing" (pairing direct eye gaze with verbal student nomination during problem introductions) and learns to suppress intervention attempts during repetitive AIML dialog system glitches.
Highlights & Insights¶
- Framing multimodal interaction as textual inline stage directions bypasses the high compute and latency bottlenecks of end-to-end multimodal architectures, turning off-the-shelf LLMs into viable multi-party simulators.
- Curriculum training on heavily skewed dyads (
ImbalanceFocused) compels the agent to master proactive intervention strategies that generalize effectively to balanced collaborative settings. - The failure of off-policy evaluation on verbal-only agents illustrates how non-verbal grounding acts as a necessary regularizer preventing policies from collapsing into unrealistically narrow action distributions.
Limitations & Future Work¶
- The real-world reference corpus is relatively compact (\(N = 71\) dialogues), resulting in overlapping confidence intervals among competent policies during OPE and limiting fine-grained ranking resolution.
- A noticeable stylistic Sim-to-Real gap remains: instruction-tuned LLMs exhibit verbosity bias, producing longer sentences (Average Sentence Length, \(p < 0.001\)) and elevated syntactic complexity (Flesch-Kincaid Grade Level of 6.2 vs. 4.1 for real undergraduates).
- Discrete text-based stage directions cannot capture fine-grained continuous spatial dynamics, making direct transfer to physical manipulation or continuous gesture coordination non-trivial.
Related Work & Insights¶
- vs. DuetSim and Dyadic Simulators: Most prior LLM user simulators are restricted to dyadic, text-only task-oriented dialogues; this work extends simulation to triadic human-robot interactions featuring explicit non-verbal gaze coordination.
- vs. Agenda-Based Simulators: Rule-based POMDP simulators lack natural linguistic variance; the proposed approach marries structural turn governance with the rich generative diversity of modern LLMs.
- vs. Maidment et al. (2022): Previous work derived static POMDPs directly from scarce human logs; this study leverages generative LLMs to synthesize millions of interaction transitions for scalable offline RL.
Rating¶
- Novelty: โญโญโญโญโญ Elegant textual encoding of non-verbal social signals enabling full-pipeline multiparty RL policy training with off-the-shelf LLMs.
- Experimental Thoroughness: โญโญโญโญโ Thorough multi-dimensional evaluation covering linguistic fidelity, phase coherence, persona entropy, and four OPE estimators, bounded only by reference corpus sample size.
- Writing Quality: โญโญโญโญโญ Clear conceptual formulation, disciplined terminology, and rigorous analysis of the Sim-to-Real gap and offline RL pitfalls.
- Value: โญโญโญโญโญ Provides a foundational, highly reproducible blueprint for prototyping socially grounded pedagogical agents and collaborative AI systems.