Skip to content

Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generation

  • Conference: ACL 2025
  • arXiv: 2502.11423
  • Code: GitHub
  • Area: NLP Generation / Dialogue Systems / Persona
  • Keywords: Personalized Dialogue, Persona Sentiment, Polarity Sensitivity, LLM Robustness, ConvAI2

TL;DR

Large-scale analysis reveals that the quality of personalized dialogues generated by LLMs is highly sensitive to the sentiment polarity of user personas—negative personas lead to an overemphasis on persona traits that triggers contradictions, whereas positive personas generate higher-quality dialogues through selective persona integration. Based on these insights, mitigation strategies combining turn-by-turn generation, persona ranking, and sentiment-aware prompting are proposed.

Background & Motivation

  • Limitations of Prior Work: Personalized dialogue systems generate customized responses by embedding the user's persona into the system prompt. However, the impact of the persona's sentiment polarity (positive/negative/neutral) on dialogue quality has been largely neglected. Previous research has focused on better integrating personas without considering how the inherent sentiment characteristics of the persona affect LLM behavior.

  • Key Challenge: LLMs exhibit high sensitivity to the sentiment polarity of the context (as validated by previous studies). However, when mapping this characteristic to persona dialogue scenarios, a counterintuitive phenomenon occurs: dialogues with negative personas yield higher Consistency (C) scores, but also a higher contradiction rate—meaning that LLMs create more self-contradictions when forcibly integrating negative personas. The fundamental issue is that LLMs do not "selectively ignore" persona information that is detrimental to coherence.

  • Goal: (1) RQ1: Are LLMs sensitive to the sentiment polarity of users? Quantify the dialogue quality differences across different polarity pairings; (2) RQ2: If they are sensitive, how can LLMs be made more robust to polarity? Design methods that improve dialogue quality without modifying the persona content.

  • Key Insight: Inspired by psychological research showing that dialogue quality varies with participants' personality traits, the authors categorized ConvAI2 personas into positive (2,691), negative (1,006), and neutral (2,429) using a sentiment classifier (confidence \(> 0.99\)). Roughly 60% of personas naturally carry sentiment polarity (17% negative), indicating that this is a prevalent phenomenon rather than an edge case.

Method

Overall Architecture

The study consists of two phases: (1) Diagnosis Phase—generating and evaluating approximately 58K dialogues using 4 LLMs \(\times\) 5 pairings \(\times\) 8 metrics to comprehensively quantify polarity sensitivity; (2) Treatment Phase—proposing and validating three mitigation strategies.

Key Designs

  1. Polarized Persona Construction:

    • Function: Construct user personas with clear sentiment polarity to eliminate confounding factors.
    • Mechanism: Sentiment classification is performed on each persona in ConvAI2 using distilbert-base-uncased-finetuned-sst-2-english, keeping only those with confidence \(> 0.99\) as polarized personas. Contradictions are detected using an NLI model (nli-deberta-v3-large) to iteratively construct contradiction-free personas containing \(K\) sentences. Generating 10K personas \(\times\) 3 types (Positive/Negative/Mixed), building 5 pairing types (Original/Negative/Positive/Mixed/Opposite) with 3K pairs each.
    • Design Motivation: 60% of ConvAI2 personas naturally carry sentiment polarity, demonstrating that polarity is not an artificially constructed edge case.

  2. Multi-dimensional Evaluation System:

    • Function: Comprehensively measure dialogue consistency and coherence.
    • Mechanism: Consistency metrics: C-score (NLI entailment score), Contradiction Ratio, Perplexity Gap (PPL difference with/without persona), and G-eval. Coherence metrics: Perplexity, Q-DCE, PairEval (fine-tuned LLaMA-2 for comparative evaluation), and G-eval. Human evaluation was also conducted (3 annotators \(\times\) 40 samples per configuration).
    • Design Motivation: A single metric cannot capture the multi-faceted impact of polarity; a high C-score paired with a high contradiction rate indicates that "consistency" metrics require more nuanced interpretation.

  3. Polarity Robustness Improvement Methods:

    • Function: Enable LLMs to generate more consistently high-quality dialogues under various polarized personas.
    • Mechanism: (a) Turn-by-turn generation: Generate responses one turn at a time using only a single user persona at a time to prevent polarity conflicts between both personas, allowing the use of smaller 3B-parameter models; (b) Persona ranking: Sort persona sentences based on confidence scores, putting neutral/weak sentiment first and positive sentiment last, leveraging LLM preference for the beginning of the prompt; (c) Sentiment-aware prompting: Add brief instructions reminding the model to pay attention to negative/neutral sentiment personas.
    • Design Motivation: Modifying the persona content is undesirable; therefore, solutions are approached from the perspectives of generation strategies and prompt engineering.

Loss & Training

No model training is involved; all experiments are based on zero-shot inference (greedy decoding, temperature \(= 0\)) of open-source LLMs.

Key Experimental Results

Main Results — Impact of Sentiment Pairing on Dialogue Quality

Model Pairing C score ↑ Contd. ↓ Perp. ↓ PairEval ↑
Qwen-2.5-7B Positive 0.452 8.84 7.04 2.75
Qwen-2.5-7B Negative 0.520 13.48 7.36 2.67
Qwen-2.5-7B Mixed 0.404 12.99 7.09 2.70
Qwen-2.5-7B Opposite 0.409 12.58 7.13 2.67
Ministral-8B Positive 0.595 5.78 5.80 2.67
Ministral-8B Negative 0.778 9.93 7.27 2.61

Human Evaluation Results

Pairing Consistency (1-3) Coherence (1-3)
Original 2.36 2.01
Negative 2.40 2.12
Positive 2.51 2.30

Sentiment Intensity — U-shaped Trend

Polarity Level C score Trend PairEval Trend
Strong Negative (0.0-0.1) Medium-High Medium-High
Neutral (0.4-0.6) Lowest Lowest
Strong Positive (0.9-1.0) Highest Highest
Strong Positive vs. Weakest (LLaMA) 7x Gain —
Positive vs. Negative Coherence Diff (Qwen) — +0.3

Key Findings

  • LLMs are highly sensitive to polarity: The Positive pairing performs best across almost all models and metrics; the Negative pairing shows high C-scores but significantly higher contradiction rates.
  • Negative personas cause "over-integration": The model frequently references negative persona traits, causing both entailment and contradictions to rise simultaneously, at the expense of coherence.
  • U-shaped quality distribution: Dialogue quality for strong-polarity personas (both positive and negative) is superior to that of neutral/weak-sentiment personas, with strong positive being the best.
  • Degradation in Opposite pairings: The advantage of the positive persona completely vanishes in positive-negative persona pairings.
  • Human evaluation validation: The trends of the three groups in human evaluation align with the automatic metrics.

Highlights & Insights

  • Revealing a neglected factor in dialogue quality: Prior studies focused on "how to better integrate personas," whereas this work finds that the "sentiment characteristics of the persona itself" are equally vital. This suggests that future persona dataset construction should consider the balance of sentiment distribution.
  • U-shaped finding: The discovery that strong polarity (whether positive or negative) outperforms neutral sentiment is unexpected, indicating that LLMs are more adept at processing inputs with explicit sentiment.
  • Lightweight solutions: No model retraining is required; performance can be improved simply through prompt sorting and instruction adjustments, offering high practical value.

Limitations & Future Work

  • The experiments only utilize 7B-9B parameter models; whether larger models exhibit similar sensitivity remains unknown.
  • ConvAI2 is an English dataset; the performance of polarity sensitivity in other languages has not been explored.
  • The sentiment classifier threshold of 0.99 is relatively strict; "neutral" personas might still contain weak sentiment.
  • The comparison between joint generation vs. turn-by-turn generation is affected by differences in model size (8B vs. 3B).
  • Active verification through A/B testing in real deployment scenarios is lacking.
  • vs. PersonaChat/ConvAI2 (Zhang, 2018; Dinan et al., 2020): This work is built on ConvAI2 but is the first to systematically study the impact of persona polarity on dialogue quality.
  • vs. LLM Sentiment Sensitivity (Liu et al., 2024; Wu et al., 2024b): Prior works have demonstrated that LLMs are sensitive to contextual sentiment; this paper extends this study to persona dialogue scenarios.
  • vs. Jandaghi et al. (2023): They proposed a joint generation strategy and used NLI to avoid contradictions; this paper finds that this strategy remains insufficient under negative personas.

Rating

  • Novelty: 8/10 — The first to systematically investigate the impact of persona sentiment polarity on LLM dialogues.
  • Technical Depth: 6/10 — Primarily consists of large-scale empirical analysis; the proposed solutions are relatively straightforward.
  • Experimental Thoroughness: 9/10 — 4 models \(\times\) 5 pairings \(\times\) 8 metrics + human evaluation + sentiment intensity analysis.
  • Clarity: 8/10 — Research questions are clear, and the experimental design has rigorous logic.
  • Overall Score: 7.5/10