Skip to content

PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants

Conference: ACL 2025
arXiv: 2506.09902
Code: Yes
Area: NLP / Dialogue Systems
Keywords: Personalization, Task-Oriented Dialogue, LLM-as-a-Judge, User Simulation, Benchmark Evaluation

TL;DR

This paper introduces PersonaLens, a comprehensive evaluation benchmark for the personalization capabilities of task-oriented AI assistants. It features 1,500 rich user personas, 111 tasks across 20 domains, a user simulator agent, and a judge agent. Through large-scale automated evaluation, it reveals significant deficiencies in the personalization capabilities of current LLM assistants.

Background & Motivation

As LLM-driven AI assistants become increasingly integrated into daily life (customer service, personal assistants, educational tools), personalization—tailoring responses to user preferences—has become key to improving user satisfaction. However, systematically evaluating the personalization capabilities of AI assistant systems in task-oriented scenarios remains an under-explored area.

Existing personalization benchmarks have significant limitations:

  • PersonaChat: Focuses on open-domain chit-chat, lacks structured task goals, and cannot evaluate personalization conjointly with goal completion.
  • LaMP: Evaluates personalized language tasks (e.g., email writing style) rather than conversational scenarios.
  • PENS / Cornell-Rich: Covers only narrow domains (news/movies), lacking generalizability.
  • Traditional Task-Oriented Dialogue Benchmarks (MultiWOZ, SGD): Evaluate task completion but completely ignore personalization.
  • Moreover, the above benchmarks rely heavily on human annotation, which is costly and difficult to scale.

The design concept of PersonaLens is to achieve fully automated evaluation using LLM-based agents while covering rich user contexts (preferences, historical interactions, situational factors), testing the AI assistant's ability to adapt to personal preferences while completing tasks.

Method

Overall Architecture

PersonaLens consists of three main components:

  1. User Personas: 1,500 virtual users with demographic information, multi-domain preferences, and historical interaction summaries.
  2. Task Set: 111 tasks spanning 20 domains, including single-domain and cross-domain tasks, with situational contexts attached to each task.
  3. Two LLM Agents: A User Agent (simulating real users interacting with the assistant) and a Judge Agent (scoring based on the LLM-as-a-Judge paradigm).

Key Designs

  1. User Persona Generation:

    • Demographic Information: Real demographic data is introduced from the PRISM Alignment dataset (1,500 users covering 75 countries) to ensure diversity.
    • User Preferences: Categorical preferences (fixed options, e.g., music genres) and non-categorical preferences (open-ended, e.g., specific restaurants) are generated for each domain (20 in total), conditionally generated by an LLM based on demographic information to maintain consistency.
    • Domain Mask: A binary mask \(\mu\) is introduced to simulate "users being uninterested in certain domains", making the evaluation closer to reality.
    • Historical Interaction Summary: A summary of historical conversations between the user and the AI assistant in each domain is generated, based on preferences and demographics.
    • Design Motivation: Ensures data quality through preference distribution validation (Shannon evenness) and persona consistency checks.

  2. Task Generation:

    • Single-domain tasks (86 tasks): e.g., "booking a restaurant based on user taste preferences."
    • Cross-domain tasks (25 tasks): e.g., "booking flights + hotel + car rental", involving 3-5 domains.
    • Each user-task pair is accompanied by situational context (current location, device type, time, etc.), dynamically generated by an LLM.
    • A total of 122,133 user-task scenarios are generated.

  3. User Agent and Judge Agent:

    • User Agent (\(\mathcal{U}\)): Receives user personas, tasks, and situational context to simulate a real user conversing with the assistant under test. It uses a vanilla prompting strategy (which performs better than CoT, as the latter leads to unnatural over-reasoning).
    • Judge Agent (\(\mathcal{J}\)): After the conversation ends, evaluates the assistant's performance based on user personas and task specifications. The scoring dimensions include:
      • Task Completion (TC, binary) and Task Completion Rate (TCR)
      • Personalization (P, 1-4 scale)
      • Naturalness (1-5 scale) and Coherence (1-5 scale)
    • Claude 3 Sonnet is used as the User Agent, and Claude 3.5 Sonnet is used as the Judge Agent.

Loss & Training

This paper introduces a pure evaluation benchmark and does not involve model training.

Key Experimental Results

Main Results: Performance of Different LLM Assistants

Assistant Model Single-domain TCR↑ Single-domain P↑ Cross-domain TCR↑ Cross-domain P↑
Mistral 7B 88.52% 1.93 74.54% 1.86
Llama 3.1 8B 89.55% 2.14 77.00% 2.03
Mixtral 8x7B 91.38% 2.04 78.35% 2.00
Claude 3 Haiku 95.95% 2.20 75.65% 1.98
Llama 3.1 70B 90.80% 2.21 83.03% 2.22
Claude 3.5 Haiku 91.53% 2.32 70.85% 2.18
Claude 3 Sonnet 95.98% 2.13 77.49% 2.01

Ablation Study: Impact of Contextual Information (Claude 3 Sonnet)

Setting Single-domain TCR Single-domain P Cross-domain TCR Cross-domain P
Base (Personalization instructions only) 95.98% 2.13 77.49% 2.01
+ Demographics (D) 95.52% 2.16 77.86% 2.05
+ Historical Interaction (I) 96.83% 2.59 81.30% 2.32
+ Situational Context (S) 95.74% 2.20 77.61% 2.06
+ All (D+I+S) 96.31% 2.57 82.66% 2.31

Human Evaluation Validation

Metric Cohen's Kappa (Judge vs Human) IAA (Fleiss' Kappa)
Task Completion 0.780 0.865
Personalization 0.520 0.750
Coherence (Assistant) 0.650 0.748

Key Findings

  1. Personalization scores are generally low: Personalization scores for almost all models are around 2/4, demonstrating that current LLMs still have vast room for improvement in personalization.
  2. Historical interaction is the most critical context: I (Historical Interaction Summary) causes the personalization score to jump from 2.13 to 2.59 (single domain), contributing far more than demographic information or situational context. This suggests that future assistants should prioritize building interaction memory systems over static profiles.
  3. Trade-off between TCR and Personalization: Claude 3.5 Haiku is better than Claude 3 Haiku in personalization, naturalness, and coherence, but its TCR decreases instead, indicating a potential tension between personalization and task completion.
  4. Cross-domain tasks are significantly more difficult: From single-domain to cross-domain, TCR generally drops by 10-20 percentage points. Cross-domain preference consistency remains a core challenge.
  5. Significant domain discrepancies: Recommendation-type tasks (books, music, games) have higher personalization scores, while procedural tasks (schedule management, messaging) have lower scores—the latter's strict sequential execution limits the scope for integrating preferences.
  6. High consistency between LLM-as-a-Judge and human evaluation: Cohen's Kappa reaches 0.78 for task completion, validating the reliability of automated evaluation.

Highlights & Insights

  • Scalability of the evaluation framework: With 122k+ dialogue scenarios, fully automated generation and evaluation, it far exceeds the scale of human-annotated methods.
  • Clear hierarchy of contextual information: The ablation study clearly establishes the importance hierarchy of \(I \gg D \approx S\), which has direct guiding value for product design—prioritize recording and utilizing user interaction history.
  • Dynamic turn-level analysis of personalization: Figure 5 shows the dynamic change patterns of personalization scores during the progression of the dialogue in different domains (movies start low and end high, messaging starts high and ends low), providing fine-grained insights for dialogue strategy design.
  • Complementing PersonaBench: PersonaBench focuses on extracting personal information from unstructured documents, whereas PersonaLens focuses on using personal information in interactions—together, they map out the evaluation landscape of personalized AI.

Limitations & Future Work

  • Only supports text interactions; multi-modal personalization (voice, image) is not covered.
  • Evaluation is conducted on raw LLMs without integration with real systems (simulated actions for booking/purchasing).
  • User personas and dialogues are generated by LLMs, which may inherit model biases (demographic biases, cultural assumptions, etc.).
  • The User Agent uses a vanilla prompt, which might not be realistic enough—real users often do not express preferences explicitly and require the assistant to discover them proactively.
  • The scale of personalization scoring (1-4) remains somewhat subjective, and different Judge Agents might produce inconsistencies.

PersonaLens bridges two independent research directions: task-oriented dialogue (MultiWOZ, SGD) and personalization evaluation (LaMP, PersonaChat). Adopting LLM-as-a-Judge for large-scale automated evaluation also continues the paradigm of Zheng et al. 2023. The key insight is that personalization is not a single capability but a systemic challenge that needs to be optimized globally across a multi-dimensional space of "task completion, preference adaptation, and dialogue quality."

Rating

  • Novelty: ⭐⭐⭐⭐ — The first personalization benchmark to integrate rich user personas + task orientation + automated agent evaluation. Highly unique positioning.
  • Experimental Thoroughness: ⭐⭐⭐⭐ — Includes 7 model families, single/cross-domain comparison, ablation studies, human validation, and domain-level analysis. Highly comprehensive and in-depth.
  • Writing Quality: ⭐⭐⭐⭐ — Structure is clear, formal definitions are rigorous, and the comparison with related work (Table 2) is extremely helpful.
  • Value: ⭐⭐⭐⭐ — Provides a quantitative answer (2/4 score) to "how far we are from practical personalized AI assistants." The ablation results have direct implications for product design.