Skip to content

PersonaDual: Balancing Personalization and Objectivity via Adaptive Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/XiaoyouLiu/personadual
Area: dialogue
Keywords: personalized dialogue, adaptive reasoning, dual-process theory, reinforcement learning, factual objectivity

TL;DR

To resolve the double-edged sword dilemma where personalized user personas enhance subjective alignment but degrade factual objectivity and logical reasoning, PersonaDual unifies general objective reasoning and personalized reasoning within a single LLM, adaptively switching between modes via a novel DualGRPO reinforcement learning algorithm.

Background & Motivation

As Large Language Models increasingly power interactive agents and conversational assistants, mainstream architectures routinely integrate persistent user personas and cross-session memory mechanisms. While aligning model outputs with individual user preferences substantially improves engagement and user satisfaction, it introduces a severe downside: personalized information frequently acts as an overwhelming prior bias during text generation. This bias triggers sycophancy, hallucinations, and stereotypical responses, often overriding the model's parametric factual knowledge. In rigorous, knowledge-intensive scenarios such as biomedical question answering, scientific reasoning, and mathematical problem-solving, irrelevant or unaligned user personas cause factual accuracy drops of up to 8%.

Existing personalization alignment methodologiesโ€”primarily realized via in-context learning, parameter-efficient fine-tuning, or preference-based reinforcement learningโ€”concentrate almost exclusively on maximizing subjective preference compliance, leaving their adverse repercussions on logical reasoning and objective factuality largely neglected. Although recent studies have recognized this "alignment tax", proposed solutions rely primarily on brittle manual triggers, rigid heuristic prompts, or expensive manually annotated datasets. Empirical evaluations uncover that the utility of persona information is strictly context-dependent: while domain-aligned personas can marginally assist problem-solving, unaligned or irrelevant personas act as disruptive noise that severely degrades reasoning fidelity.

Drawing inspiration from the dual-process theory in cognitive psychology, where human cognition flexibly coordinates between distinct reasoning systems depending on situational demands, developing an intrinsic mechanism for LLMs to decide when to engage personalization is paramount. Core idea: integrate general-purpose objective reasoning and personalized reasoning within a single model via explicit control prefixes, and train the model using a novel DualGRPO reinforcement learning algorithm featuring prefix-forced sampling and dual-mode advantage decomposition to adaptively govern mode selection based on context.

Method

Overall Architecture

PersonaDual equips an LLM with the autonomous capability to determine whether and how to leverage persona signals when answering a query. Formally, given a user query \(q\) and a persona description \(p\), the model executes a two-step adaptive inference process: first, an internal mode-selection policy \(\sigma(m \mid q, p)\) samples a reasoning mode \(m \in \mathcal{M} = \{g_m, p_m\}\) (where \(g_m\) represents the general objective mode and \(p_m\) represents the personalized mode); second, the corresponding learnable control prefix \(pfx_m \in \{[General\_mode], [Personalized\_mode]\}\) is prepended to the input sequence, conditioning the generation of the final response \(y \sim \pi_\theta(\cdot \mid q, p, pfx_m)\).

To jointly optimize the mode selector and conditional generator without external router modules, PersonaDual follows a two-stage training paradigm. In Stage 1, supervised fine-tuning (SFT) on a curated mixed-mode dataset instills robust reasoning chains for each distinct mode. In Stage 2, DualGRPO reinforcement learning refines adaptive mode switching conditioned on the query and persona, balancing intra-mode generation quality with inter-mode credit assignment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input: User Query q + Persona p"] --> B["Prefix-Forced Balanced Sampling<br/>Symmetrically rollout n trajectories per mode"]
    B --> C["Dual-Mode Response & Reward Evaluation<br/>Rule-based accuracy + Format regularization"]
    C --> D["Dual-Mode Advantage Decomposition<br/>Intra-mode relative reward + Inter-mode zero-sum signal"]
    D --> E["Prefix-Strengthened Policy Gradient Update<br/>Scale prefix advantage by factor ฮฒ"]
    E --> F["Inference Deployment: Autonomous prefix selection & response generation"]

Key Designs

1. Mixed-Mode Integration SFT: Disentangling Objective and Personalized Reasoning Chains To enable a single model parameter space to accommodate mutually contradictory reasoning behaviors, the authors construct the PersonaDualData-SFT dataset (8,000 samples) by distilling high-quality reasoning traces from DeepSeek-R1. For the general objective mode conditioned on \([General\_mode]\), the persona \(p\) is deliberately omitted from the prompt, forcing the model to generate de-personalized reasoning grounded purely in factual evidence and verifiable logic. For the personalized mode activated by \([Personalized\_mode]\), the complete context \((q, p)\) is supplied, guiding the model to dissect persona attributes (such as occupation, interests, and emotional needs) and actively weave them into the reasoning chain. Crucially, on objective tasks, SFT trajectories are filtered through a "gain-based selection": an instance retains the personalized mode only if incorporating the persona helps the model reach the correct answer; otherwise, it is assigned to the general objective mode, establishing the initial capability to filter out deceptive persona cues.

2. Prefix-Forced Balanced Sampling: Overcoming Mode Collapse and Exploration Skew Following SFT warm-up, the model frequently exhibits strong prior biases toward a single mode prefix for a given input \((q, p)\), where \(\pi_\theta(pfx_{gm} \mid q, p) \gg \pi_\theta(pfx_{pm} \mid q, p)\) or vice versa. Unconstrained exploratory rollouts during reinforcement learning would rapidly collapse into the dominant mode, resulting in an extreme scarcity of counter-mode trajectories and preventing fair, pairwise quality comparison between the two modes under identical conditions. DualGRPO resolves this exploration bottleneck via prefix-forced sampling: for each rollout instance, the algorithm explicitly forces the generation of an equal number of trajectories (\(n\) under each mode, with \(n=4\) yielding a balanced batch of 8 rollouts per prompt): $$ {y^{(i)}{gm}}}^n \sim \pi_\theta(\cdot \mid q, p, pfx_{gm}), \quad {y^{(i){pm}}) $$ This design guarantees an unbiased empirical reference for evaluating the consequences of engaging versus bypassing personalization across every training example.}^n \sim \pi_\theta(\cdot \mid q, p, pfx_{pm

3. Dual-Mode Advantage Decomposition: Harmonizing Intra-Mode Optimization and Inter-Mode Routing Standard GRPO computes relative advantages exclusively within a single sampled group, which evaluates intra-mode response variance but remains blind to macro-level quality disparities between modes. DualGRPO decomposes the advantage into intra-mode and inter-mode components. Denoting the mean rewards under the general and personalized modes as \(\mu_{gm}\) and \(\mu_{pm}\) respectively, the advantage for sample \(i\) in mode \(m \in \{gm, pm\}\) is defined as: $$ A^{intra}{m,i} = r^{(i)}m - \mu_m $$ $$ A^{inter} = \mu_m - \mu \neq m $$ The composed advantage is }}, \quad \bar{m\(A_{m,i} = A^{intra}_{m,i} + A^{inter}_{m,i}\). When one mode outperforms the other on average (\(\mu_m > \mu_{\bar{m}}\)), all candidates from that mode receive a positive credit shift, while the competing mode is penalized by an equal zero-sum amount. This composite signal simultaneously drives intra-mode response perfection and steers the mode selection policy toward the higher-reward mode.

4. Prefix-Strengthened Policy Gradient Updates: Mitigating Token-Level Gradient Dilution In reasoning tasks involving long chain-of-thought outputs, the mode selection token resides at the very beginning of the sequence (first 1-2 tokens), followed by hundreds of intermediate reasoning steps. During standard policy gradient backpropagation, the learning signal on these early critical decision tokens tends to vanish or dilute over extended trajectories. DualGRPO introduces a prefix amplification coefficient \(\beta > 1\) (empirically set to \(\beta=2.0\)), explicitly scaling the advantage of prefix tokens during policy gradient computation: $$ A^{pfx}{m,i} = \beta \cdot A $$ This amplification accelerates the acquisition of context-dependent routing rules, ensuring that the model reliably binds input query semantics and persona relevance to the appropriate initial mode token.

Loss & Training

The two-stage training paradigm is executed on 8 NVIDIA A800 GPUs using DeepSpeed ZeRO-3 in bfloat16 precision. Stage 1 (SFT) fine-tunes the base model for 1 epoch with a global batch size of 16, a peak learning rate of 5e-5, and a cosine scheduler. Stage 2 (DualGRPO RL) runs for 5 epochs with a global batch size of 64, a linear learning rate of 1e-6, a KL divergence penalty coefficient of 0.04, and prefix weighting \(\beta=2.0\). The reward function comprises a rule-based correctness reward (exact match or GPT-4o-mini automated verification against ground truth) combined with a format regularization penalty ensuring adherence to the designated prefix and response structure.

Key Experimental Results

Main Results

Experiments are primarily benchmarked on Qwen3-8B-Instruct, with generalization confirmed on LLaMA3.1-8B-Instruct. Objective tasks cover biomedical reasoning (PubMedQA), open-domain factual retrieval (TriviaQA), professional multi-discipline QA (MMLU-Pro), graduate-level reasoning (SuperGPQA), and competition mathematics (MATH500). Subjective personalization tasks comprise PersonaFeedback and FSPO-roleplay. Objective benchmarks are evaluated under both Unaligned and Aligned persona conditions.

Model PubMedQA TriviaQA MMLU-Pro SuperGPQA MATH500 Objective Avg. PersonaFeedback FSPO-roleplay Personalized Avg.
Qwen3-8B-Instruct 0.377/0.440 0.316/0.429 0.631/0.682 0.310/0.327 0.664/0.775 0.460/0.531 0.733 0.686 0.710
CoT* 0.447/0.537 0.437/0.500 0.601/0.636 0.318/0.313 0.764/0.775 0.513/0.552 0.739 0.785 0.762
G-SFT-RL* (Objective Expert) 0.489/0.539 0.434/0.509 0.616/0.644 0.320/0.332 0.771/0.777 0.526/0.560 0.658 0.717 0.688
P-SFT-RL* (Personalized Expert) 0.346/0.388 0.354/0.448 0.576/0.651 0.307/0.326 0.669/0.695 0.450/0.502 0.738 0.762 0.750
ALIGNXPERT-PBA* 0.373/0.436 0.321/0.428 0.589/0.655 0.321/0.331 0.657/0.767 0.452/0.523 0.733 0.684 0.709
PersonaDual-Prompt* 0.348/0.437 0.334/0.431 0.578/0.629 0.297/0.332 0.730/0.752 0.457/0.516 0.746 0.736 0.741
PersonaDual-Router* 0.472/0.417 0.438/0.503 0.599/0.644 0.296/0.329 0.775/0.784 0.516/0.535 0.733 0.767 0.750
PersonaDual (Ours)* 0.509/0.549 0.450/0.517 0.621/0.659 0.331/0.349 0.789/0.810 0.540/0.577 0.748 0.799 0.774
No-Persona Upper Bound 0.526 0.450 0.633 0.338 0.791 0.548 โ€“ โ€“ โ€“

Note: Objective metrics report "Unaligned Accuracy / Aligned Accuracy". Personalized tasks are evaluated under Aligned personas. Models marked with * are built on Qwen3-8B-Instruct.

Ablation Study

The ablation experiments examine the progression from SFT to RL as well as the necessity of each constituent in DualGRPO:

Stage & Configuration Objective Unaligned Acc (%) Objective Aligned Acc (%) Personalized Aligned Acc (%) Note
Stage 1: PersonaDual-SFT 52.3 55.4 74.4 Dual-mode SFT warmup without adaptive RL mode switching
Stage 2: PersonaDual (Full) 54.0 57.7 77.4 Full model: SFT + DualGRPO adaptive optimization
w/o DualAdv 53.4 55.8 75.5 Removes inter-mode advantage decomposition; lacks cross-mode steering
w/o DualAdv + PfxSmp 53.6 56.0 76.7 Removes both inter-mode advantage and forced sampling; causes mode collapse

Mode distribution diagnostics reveal that under "Objective Task + Unaligned Persona", prompt-based and router-based baselines erroneously engage personalized reasoning in 62.0% and 34.3% of cases respectively, whereas PersonaDual reliably selects the general objective mode 100.0% of the time. Similarly, on subjective tasks, PersonaDual achieves 100.0% personalized mode activation.

Key Findings

  • Eliminating Unaligned Persona Degradation: Under unaligned persona inputs, specialized personalization models (e.g., ALIGNXPERT-PBA) degrade to 45.2% average objective accuracy, whereas PersonaDual achieves 54.0%, virtually matching the clean no-persona upper bound (54.8%) by filtering out irrelevant context.
  • Harvesting Aligned Persona Gains: When persona attributes are semantically aligned with the query topic, PersonaDual achieves 57.7% objective accuracy, surpassing unaligned performance by +3.7% and beating the no-persona upper bound (54.8%) by 2.9%, validating that adaptive reasoning leverages helpful user background to improve factual problem-solving.
  • Tightly Coupled RL Components: Eliminating forced sampling (PfxSmp) induces policy collapse into a single mode, preventing cross-trajectory exploration; removing dual advantage (DualAdv) deprives balanced trajectories of inter-mode contrastive guidance. Both components are indispensable.
  • Asymmetric Multi-Turn Shift Resilience: Multi-turn dialogue evaluation shows that transitioning from "Personalized \(\to\) Objective" in turn 2 retains 100% mode alignment with single-turn baselines. However, transitioning from "Objective \(\to\) Personalized" drops alignment to 84.3%, indicating that shifting from formal analytical deduction to subjective persona styling carries higher cognitive inertia.

Highlights & Insights

  • Internalizing Dual-Process Cognition: Systematically unifies Kahneman's dual-process cognitive framework within a single LLM through explicit prefix-conditioned mode routing, avoiding complex multi-agent cascading overhead.
  • DualGRPO Multi-Tier Credit Assignment: Elegantly combines intra-mode relative advantage and inter-mode zero-sum reward differences, concurrently optimizing generation quality within a mode and routing decisions across modes in a single policy update step.
  • Prefix Advantage Scaling: Tackles the fundamental RL problem of token-level gradient dilution in long CoT trajectories by scaling the advantage of early mode-decision tokens by factor \(\beta\), providing a clean recipe for early-token policy steering.

Limitations & Future Work

  • Subjective Benchmark Breadth: Current personalization evaluation is restricted to multiple-choice benchmarks (PersonaFeedback, FSPO-roleplay); testing on open-ended creative interaction remains desirable.
  • Multilingual Scope: Experiments are restricted to English; cross-lingual persona dynamics and cultural biases have not yet been evaluated.
  • Multi-Turn Context Adaptation: The observed drop to 84.3% mode alignment when switching from objective to personalized turns suggests that multi-turn conversational trajectories should be explicitly incorporated into future RL training curricula.
  • vs ALIGNXPERT (Li et al., 2025a): While ALIGNXPERT maximizes preference compliance via preference-based alignment, it incurs heavy factuality penalties on objective tasks; PersonaDual introduces dual-mode reasoning to prevent factual degradation without sacrificing personalization quality.
  • vs PrefDisco (Li et al., 2026): PrefDisco benchmarks persona-induced factuality degradation but provides no generative solution; PersonaDual introduces an end-to-end model architecture and RL optimization paradigm that solves the issue.
  • vs AdaCoT / Thinkless (Lou et al., 2025; Fang et al., 2025): While adaptive CoT literature focuses on balancing latency and reasoning depth to save inference tokens, PersonaDual generalizes adaptive routing to the orthogonal dimension of objective factuality versus subjective user personalization.

Rating

  • Novelty: โญโญโญโญโญ First principled framework to balance factuality and personalization through dual-process reasoning and DualGRPO reinforcement learning.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously tested across 7 objective and subjective benchmarks, dual base model backbones, RL component ablations, multi-turn dynamics, and attention visualizations.
  • Writing Quality: โญโญโญโญโญ Extremely lucid narrative structure, precise mathematical formulations, and thorough empirical analysis.
  • Value: โญโญโญโญโญ Provides an essential architectural blueprint for production conversational assistants and memory-enabled agents.