Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ebm-rl/RP-See-Think-Speak-RL
Area: Video Understanding
Keywords: Video Multimodal, Role-Playing Agent, Reinforcement Learning, Decoupled Rewards, GRPO
TL;DR¶
EBM-RL decouples video role-playing generation into an explicit See-Think-Speak pipeline and optimizes each stage via four complementary rewards under GRPO, eliminating "atmosphere-blind" responses and achieving situational persona consistency.
Background & Motivation¶
The rapid advancement of large language model-driven anthropomorphic agents has catalyzed role-playing language agents (RPAs) that emulate character personas, styles, and values from literary works and movie scripts. However, virtually all existing role-playing agents operate exclusively on text-only or static image-text inputs, leaving them largely "atmosphere-blind." In real-world cinematic interactions, dialogue cannot be judged solely from text history; when facing mortal danger or high-stakes crises, a humorous character joking out of context breaks immersion, whereas displaying nervous restraint or courage reflects genuine situational fidelity.
While general vision-language models (VLMs) have attained remarkable perceptual and question-answering capabilities across long video benchmarks, they predominantly behave as neutral, objective narrators. They lack persistent, distinctive expressions of character personality, values, and emotional tone. More fundamentally, standard multimodal models tightly couple "seeing" and "thinking" within a single black-box reasoning chain, making it impossible to identify whether an inappropriate response stems from visual misperception or faulty cognitive reasoning. Furthermore, traditional role-playing benchmarks prioritize rigid adherence to static personas, ignoring how humans dynamically adapt their behavior under situational stress.
To overcome these fundamental limitations, this paper introduces video-grounded visual perception into role-playing dialogue, explicitly organizing the agent's generative process into an interpretable "See-Think-Speak" cognitive flow. Core idea: propose EBM-RL (Eye-Brain-Mouth Reinforcement Learning), a decoupled GRPO framework that partitions video role-playing into structured perception, cognitive reasoning, and utterance generation, optimized via stage-specific CLIP scene-text alignment and perceptual-cognitive gain rewards to enforce situational persona consistency.
Method¶
Overall Architecture¶
EBM-RL operationalizes the human cognitive loop of observing the environment, reasoning over state transitions, and delivering situated dialogue. Given an input tuple \(x = (V, P_u, P_a, H)\) comprising video clip \(V\), user profile \(P_u\), assistant profile \(P_a\), and dialogue history \(H\), the policy model \(\pi_\theta\) generates a strictly formatted completion: $\(y = \langle\text{perception}\rangle y_v \langle/\text{perception}\rangle \langle\text{think}\rangle y_t \langle/\text{think}\rangle \langle\text{answer}\rangle y_a \langle/\text{answer}\rangle\)$ During training, EBM-RL deploys Group Relative Policy Optimization (GRPO) without a critic model. For each input \(x\), it samples a group of candidate trajectories \(\{y_g\}_{g=1}^G \sim \pi_\theta(\cdot \mid x)\) and computes a 4D reward vector capturing visual grounding, cognitive utility, semantic answer faithfulness, and structural format integrity. Per-dimension group Z-score normalization produces stable scalar advantages that update the policy end-to-end.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Inputs<br/>Video Clip V + Character Profiles + Dialogue History"] --> B["Stage 1: Eye (Perception)<br/>Objective observation of actions, micro-expressions, key objects & atmosphere"]
B --> C["Stage 2: Brain (Think)<br/>Synthesize visual evidence with persona to deduce stakes & dialogue direction"]
C --> D["Stage 3: Mouth (Answer)<br/>Generate concise spoken line aligned with situation & persona"]
B -.->|CLIP similarity matching| R1["Scene-Text Alignment Reward r_vis"]
B & C -.->|Lexical cleaning + likelihood gain| R2["Perceptual-Cognitive Gain Reward r_pcg"]
D -.->|BERTScore token alignment| R3["Expressive Faithfulness Reward r_sem"]
B & C & D -.->|XML structural constraints| R4["Dense Format Reward r_fmt"]
R1 & R2 & R3 & R4 --> E["Group Z-score Normalized Advantage<br/>GRPO Policy Update"]
Key Designs¶
1. Structured See-Think-Speak Decoupled Trajectory: Disentangling Perception, Cognition, and Utterance
Standard end-to-end VLMs conflate visual perception and dialogic reasoning into an opaque representation, compounding hallucinations and hindering error attribution. EBM-RL enforces strict separation across three modular functional blocks:
- \(y_v\) (<perception>) serves as the "Eye," recording only observable visual facts in camera mode, including character physical interactions, facial micro-expressions, plot-driving objects, and lighting tension (safe vs. dangerous), while strictly prohibiting subjective narrative conjecture or figure naming;
- \(y_t\) (<think>) acts as the "Brain," synthesizing objective visual cues with dialogue history \(H\) and persona profiles \((P_u, P_a)\) to determine conversational direction (topic continuation vs. inquiry/reflection) and applying a high-stakes personality filter (e.g., humor transitioning to bravery in imminent danger);
- \(y_a\) (<answer>) functions as the "Mouth," outputting a single concise line of spoken colloquial dialogue that respects immediate situational tension without bookish narration or AI-sounding preambles.
2. CLIP-based Scene-Text Alignment Reward: Anchoring Visual Evidence Against Hallucinations
To ensure that the <perception> block faithfully reflects the visual frames rather than producing generic hallucinations, EBM-RL establishes a multimodal grounding reward. Uniformly sampling \(N\) representative video frames \(\{f_j\}_{j=1}^N\), normalized CLIP visual embeddings \(v_j \in \mathbb{R}^d\) are extracted offline; the generated perception segment \(y_v\) is encoded into normalized text embedding \(u \in \mathbb{R}^d\). Recognizing that cinematic role-play is often triggered by sudden, sparse visual climaxes (such as an unsheathed weapon or a monster's sudden lunge), the framework adopts the CLIP-Max variant:
$\(r_{\text{vis}}^{\max}(V, y_v) = \max_{j \in \{1,\dots,N\}} v_j^\top u\)$
This formulation prioritizes peak visual relevance over uniform temporal averaging, preventing critical instantaneous signals from being diluted (with an alternative sentence-level Top-\(K\) pooling variant \(r_{\text{vis}}^{\text{TopK}}\) also evaluated for comparison). Empty or malformed blocks immediately receive zero reward.
3. Lexically Sanitized Perceptual-Cognitive Gain Reward: Quantifying Thought Utility for Answer Prediction In open-ended reinforcement learning with chain-of-thought, reasoning steps frequently succumb to "answer leakage" or trivial copying, where the thinking block simply quotes the planned response to artificially inflate likelihood scores. To enforce genuine cognitive utility, EBM-RL computes the Perceptual-Cognitive Gain (PCG) on a lexically sanitized representation. Using longest common subsequences (LCS) and token-level \(n\)-gram overlap, answer-cue sentences are stripped from \(y_t\) to yield \(\bar{z}(y) = y_v \oplus \text{Clean}_{\text{lex}}(y_t, y_a)\), accompanied by an explicit copying penalty \(L_{\text{copy}}(y_t, y_a)\). The reward measures the log-likelihood gain of the ground-truth utterance \(d_{\text{gt}}\) evaluated under a frozen reference policy \(\pi_{\text{ref}}\): $\(r_{\text{pcg}}(x, y_v, y_t) = \text{GT-LL}(d_{\text{gt}} \mid x \oplus \bar{z}(y)) - \text{GT-LL}(d_{\text{gt}} \mid x) - L_{\text{copy}}(y_t, y_a)\)$ where \(\text{GT-LL}\) denotes the average token log-likelihood. By measuring informational gain from visual perception and intermediate reasoning, this reward strictly penalizes vacuous thinking and forces the model to extract actionable situational intelligence.
4. Dense Format and Expressive Faithfulness Rewards: Stabilizing the GRPO Advantage Space
Unconstrained free-form generation under RL easily triggers format collapse. EBM-RL defines a dense format reward \(r_{\text{fmt}}(y) = S_{\text{tag}}(y) + S_{\text{ord}}(y) + S_{\text{bd}}(y)\), combining tag existence (\(\pm 0.5\) per tag), chronological stage ordering (\(+1.0\)), and boundary constraints (\(+0.5\) for starting with <perception> and ending with </answer>, \(-1.0\) otherwise), providing smooth, non-binary gradient signals. For final response fidelity, \(r_{\text{sem}}(y_a, a^*)\) evaluates clipped DeBERTa-based BERTScore-F1 against the reference utterance \(a^*\). To resolve scale disparities across the 4D reward vector \(r_g = [r_{\text{fmt}}, r_{\text{sem}}, r_{\text{vis}}, r_{\text{pcg}}]\), group-wise Z-score normalization is applied independently per dimension before computing the weighted advantage \(A_g = \sum_k w_k \tilde{r}_{g,k}\) with weights \(w = [1.0, 1.0, 0.8, 0.8]\).
Loss & Training¶
The framework adopts Qwen2.5-VL-7B-Instruct as its backbone. Training proceeds in two stages: Stage 1 performs warm-up supervised fine-tuning (SFT) for 3 epochs on 2.3k high-quality CoT exemplars generated by Gemini 3 Pro to establish strong XML format priors; Stage 2 conducts GRPO fine-tuning for 1 epoch on 8 NVIDIA A100 GPUs using the constructed movie benchmark (37 films across 13 franchises, 34k samples, strictly partitioned by session ID and with subtitles masked to eliminate target leakage). GRPO optimizes a clipped surrogate objective with a token-level KL divergence penalty against reference policy \(\pi_{\text{ref}}\): $\(\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \frac{1}{G}\sum_{g=1}^G \frac{1}{T_g}\sum_{t=1}^{T_g} \min\left( r_{g,t}(\theta) A_g, \, \text{clip}(r_{g,t}(\theta), 1-\epsilon, 1+\epsilon) A_g \right) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right]\)$
Key Experimental Results¶
Main Results¶
Evaluation employs an anonymized GPT-5-mini judge across three dimensions aligned with See-Think-Speak: Visual Evidence Grounding (VEG, perception fidelity), Situational Persona Compatibility (SPC, high-stakes personality modulation), and Conversational Naturalism (CN, colloquial spoken fluency).
| Model | Scale / Category | VEG โ | SPC โ | CN โ | Avg. โ |
|---|---|---|---|---|---|
| GPT-5 (Proprietary Frontier) | Closed API | 79.11 | 79.01 | 81.22 | 79.78 |
| Gemini-3 (Proprietary Frontier) | Closed API | 80.08 | 77.37 | 83.96 | 80.47 |
| InternVL3-38B-Instruct | 38B Open VLM | 74.95 | 70.82 | 74.43 | 73.40 |
| Qwen2.5-VL-32B-Instruct | 32B Open VLM | 74.61 | 71.01 | 73.32 | 72.98 |
| InternVL3-14B-Instruct | 14B Open VLM | 73.87 | 67.75 | 72.78 | 71.47 |
| InternVL3-8B-Instruct | 8B Open VLM | 71.85 | 65.53 | 72.76 | 70.05 |
| Qwen2.5-VL-7B-Instruct | 7B Base Model | 70.75 | 65.05 | 71.71 | 69.17 |
| RoleMRC (Text-only RP) | 7B RP Baseline | 70.51 | 66.77 | 73.14 | 70.14 |
| Crab (Text-only RP) | 7B RP Baseline | 67.86 | 66.87 | 75.23 | 69.99 |
| Haruhi (Text-only RP) | 7B RP Baseline | 67.09 | 62.09 | 72.38 | 67.19 |
| Qwen2.5-VL-7B-SFT | 7B Stage-1 Warmup | 70.82 | 66.46 | 72.08 | 69.79 |
| Char-EBM-CLIP-TopK (Ours) | 7B Full Model | 73.47 | 69.94 | 73.69 | 72.37 |
| Char-EBM-CLIP-Max (Ours) | 7B Full Model | 74.25 | 70.37 | 74.78 | 73.13 |
Among comparable 7B-scale models, Char-EBM-CLIP-Max achieves top performance across all metrics, improving the average score by +3.96 over its base model Qwen2.5-VL-7B. Crucially, its VEG score (74.25) rivals 32B/38B-class foundation VLMs (Qwen2.5-VL-32B at 74.61), while SPC surges by +5.32 points (65.05 to 70.37), proving that text-only role-playing models fail because they cannot observe situational tension.
Ablation Study¶
The ablation investigates the individual impact of reward components and reward weight balancing on the test set (\(w = [w_{\text{fmt}}, w_{\text{sem}}, w_{\text{vis}}, w_{\text{pcg}}] = [1.0, 1.0, 0.8, 0.8]\)):
| Configuration | Reward Weights \(w\) | VEG โ | SPC โ | CN โ | Avg. โ | Note |
|---|---|---|---|---|---|---|
| Char-EBM-CLIP-Max (Full) | [1.0, 1.0, 0.8, 0.8] |
74.25 | 70.37 | 74.78 | 73.13 | Full 4D decoupled reward pipeline |
| w/o \(r_{\text{vis}}\) (No Visual Alignment) | [1.0, 1.0, 0.0, 0.8] |
73.03 | 69.71 | 74.57 | 72.44 | Drops visual grounding & hurts downstream SPC |
| w/o \(r_{\text{pcg}}\) (No Cognitive Gain) | [1.0, 1.0, 0.8, 0.0] |
70.92 | 66.53 | 72.24 | 69.90 | Severe drop in reasoning and naturalness |
| Equal Weight on Vision & PCG | [1.0, 1.0, 1.0, 1.0] |
74.10 | 70.54 | 74.69 | 73.11 | Higher SPC but slightly dominates VEG & CN |
| Over-emphasizing Semantic Reward | [1.0, 1.2, 1.0, 1.0] |
71.39 | 67.29 | 72.11 | 70.26 | Biases toward surface matching, harming synergy |
Key Findings¶
- \(r_{\text{pcg}}\) is the linchpin for cross-stage reasoning: Removing \(r_{\text{pcg}}\) triggers a dramatic 3.23 point drop in overall average score and a 3.84 point collapse in SPC. While \(r_{\text{vis}}\) provides raw visual anchors, \(r_{\text{pcg}}\) forces the cognitive reasoning stage to transform visual cues into character-specific action logic.
- Sparse peak pooling outperforms temporal smoothing: CLIP-Max substantially outperforms CLIP-SentTopK across VEG (74.25 vs 73.47) and SPC (70.37 vs 69.94). Cinematic interactions are predominantly driven by sudden emotional shifts or critical plot actions, which are diluted by sentence-level Top-\(K\) averaging.
- Robust out-of-domain and cross-task generalization: On unseen held-out films and characters (How to Train Your Dragon, The Matrix, Twilight Saga), Char-EBM-CLIP-Max reaches an average score of 72.73 (+3.37 over base). On the egocentric real-world embodied dataset HoloAssist, EBM outperforms its 7B base by +4.66/+2.14/+5.43/+4.08 across VEG/SPC/CN/Avg. Moreover, on general zero-shot VideoQA benchmarks without any fine-tuning, EBM yields gains of +1.93 on NExT-QA, +2.60 on PororoQA, and +2.50 on ActivityNet-QA.
Highlights & Insights¶
- Explicit See-Think-Speak Decomposition: Disentangling video-grounded generation into verifiable observation, situation-aware cognitive synthesis, and concise spoken dialogue eliminates multimodal hallucination and establishes transparent causal credit assignment during reinforcement learning.
- Anti-Cheating Lexical Sanitization for PCG: Purging verbatim overlap between thought chains and ground-truth utterances via longest common subsequence analysis prevents reward hacking and ensures true information-theoretic reasoning gains.
- Situational Consistency Over Static Caricatures: Redefining character fidelity from rigid catchphrase repetition to situational emotional modulation (e.g., humor transitioning into resolute bravery under mortal peril) establishes a new benchmark for immersive interactive agents.
Limitations & Future Work¶
- Reliance on Edited Cinematic Narrative: Datasets are derived from scripted Hollywood feature films, where camera cuts, cinematic lighting, and scripted pacing differ from continuous first-person embodied environments.
- Offline Batch Processing Gap: Current evaluations operate on pre-segmented video clips with offline histories, lacking real-time, bi-directional closed-loop execution within 3D VR and game engines.
- Future Directions: Extending EBM-RL to online streaming multimodal feeds and integrating persistent episodic character memory graphs with dynamic embodied state machines.
Related Work & Insights¶
- vs. Text-Only Role-Playing Agents (RoleMRC / Crab / Haruhi): Traditional RP models exhibit strong stylistic imitation but remain blind to physical environments, generating tone-deaf jokes during deadly crises; supplying external textual video descriptions fails to yield robust integration. EBM-RL natively unifies visual perception with character personality.
- vs. Foundation Video LLMs (Qwen2.5-VL / InternVL3): General VLMs behave as neutral third-person narrators, lacking personality immersion, values, and colloquial brevity. EBM-RL matches the visual grounding of 32B/38B models at 7B scale while achieving significantly higher situational character fidelity.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering decoupled See-Think-Speak RL framework that introduces situational consistency to video role-playing.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous validation across 37 films (34k samples), multi-judge blind pairwise preference, expert human studies, cross-domain HoloAssist, and external VideoQA transfer.
- Writing Quality: โญโญโญโญโญ Exemplary mathematical clarity, well-structured exposition, and rich implementation details in appendices.
- Value: โญโญโญโญโญ Highly impactful architecture for next-generation interactive VR NPCs, conversational embodied AI, and interpretable multimodal reasoning.