UXBench: Benchmarking User Experience in AI Assistants¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mengze-hong/UXBench
Area: LLM Reasoning
Keywords: User Experience Benchmark, Interaction Feedback Signals, Generative Reward Model, Failure Recovery, Preference Alignment
TL;DR¶
UXBench presents the first user-centric benchmark grounded in authentic human-AI interaction feedback signals, evaluating LLMs across UX judgment, response generation on failure-prone queries, and complaint recovery, while demonstrating that model capabilities scale only weakly into real-world user experience gains.
Background & Motivation¶
As modern large language models transition from research prototypes into commercial AI assistants serving hundreds of millions of active users daily, traditional evaluation paradigms have revealed a profound structural disconnection from real-world user experience (UX). Existing foundational benchmarks almost exclusively assess isolated, high-complexity capabilities such as symbolic reasoning, mathematical derivation, code generation, and strict instruction following. However, empirical telemetry from commercial deployments indicates that real-world user requests are overwhelmingly concentrated in casual information queries, routine advice, and conversational chit-chat. Consequently, substantial score jumps on conventional leaderboards frequently translate into negligible, imperceptible, or even negative utility gains for end users navigating everyday dialogues.
Constructing a principled, scalable, and automated UX benchmark has historically been hindered by three fundamental methodological roadblocks. First, conventional UX investigations remain firmly rooted in classic human-computer interaction (HCI) methodologies, relying on small cohorts of human participants, lab studies, or post-hoc questionnaires (such as the UEQ), which are prohibitively expensive and impossible to scale alongside rapid model release cycles. Second, fine-grained rubric-based schemes decompose user experience into static multi-dimensional Likert scales (e.g., evaluating seven disjoint axes of helpfulness, tone, and depth); however, real users do not deliberate over formal rubrics before clicking an "unlike" button—dissatisfaction is an immediate, holistic visceral reaction to interaction friction that defies complete manual operationalization. Third, popular crowdsourced platforms like Chatbot Arena enforce pairwise side-by-side preference ranking, creating an artificial evaluation setting that fundamentally departs from organic human interactions where users consume and judge exactly one response at a time.
To overcome these foundational dilemmas, this work introduces UXBench, bypassing subjective post-hoc rubrics by directly anchoring ground-truth labels in authentic user behavioral feedback extracted from commercial deployments. By harvesting explicit feedback (likes and dislikes) alongside implicit temporal and lexical behavioral patterns from over 70,000 real-world multi-turn dialogue sessions from a mainstream commercial AI assistant, the authors assemble 7,400 rigorously audited test instances. Core idea: Grounding user experience evaluation in authentic behavioral feedback signals turns subjective user satisfaction into scalable and observable ground truth, establishing an end-to-end tri-task framework (Judge, Eval, and Recovery) paired with a specialized generative reward model to faithfully assess user-perceived utility in AI assistants.
Method¶
Overall Architecture¶
The UXBench framework operationalizes user-perceived utility through an automated, end-to-end data curation pipeline and a three-tiered task structure. Raw dialogue logs from a commercial Chinese AI assistant first undergo strict multi-stage privacy desensitization, safety gating, explicit feedback extraction, and heuristic derivation of implicit negative signals. A multi-agent filtering system comprising Miner and Judge models curates failure-prone candidate instances and verifies quality across five dimensions before expert human auditing. The resulting benchmark evaluates models across three interconnected tasks: predicting user feedback from full context (UX Judge), generating satisfying responses on challenging, previously failed queries (UX Eval), and restoring satisfaction following explicit user complaints (UX Recovery). To eliminate self-preference and scoring calibration artifacts in commercial LLMs, the pipeline incorporates a dedicated Pointwise Generative Reward Model (GRM) trained on authentic interaction feedback.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Commercial AI Assistant Multi-Turn Logs<br/>70K+ Sessions / 400K+ Turns"] --> B["Multi-Stage Behavioral Feedback Mining<br/>Explicit Likes/Dislikes + Implicit Heuristics"]
B --> C["Multi-Agent Quality Audit Pipeline<br/>Miner Candidate Extraction + Judge 5D Scoring"]
C --> D["Three-Tier Interconnected Task Design<br/>Task 1: Judge / Task 2: Eval / Task 3: Recovery"]
D --> E["Pointwise Generative Reward Modeling<br/>Pointwise GRM as Calibrated Evaluator"]
E --> F["Systematic Evaluation of 26 Frontier LLMs"]
Key Designs¶
1. Multi-Stage Behavioral Feedback Mining: Harmonizing Explicit Sparsity and Implicit Ambiguity
Recognizing that explicit user feedback (likes/dislikes) is exceptionally sparse—occurring in only 0.40% of turns across a 400,000-turn random sample due to zero immediate utility for task completion—UXBench introduces a dual-track signal extraction mechanism. Explicit feedback provides high-confidence positive and negative ground-truth anchors. To capture the massive reservoir of unlabeled interactions, the authors introduce two rule-based temporal and lexical heuristics for implicit dissatisfaction: Skim, which identifies rapid follow-up queries without meaningful engagement by checking if the response time gap \(\Delta \tau_{t+1}\) and response character length \(|r_a^t|\) satisfy \(\frac{\Delta \tau_{t+1}}{|r_a^t|} < 0.15\) alongside \(\Delta \tau_{t+1} < 30\text{s}\); and Follow-up Dissatisfaction, which isolates subsequent turns matching regex patterns for Complain/Dismiss or exhibiting high surface lexical overlap with the prior prompt (Repeat/Rephrase). Each heuristic-matched candidate is verified by a local LLM, yielding 10,337 Repeat, 4,773 Rephrase, 756 Skim, 199 Complain, and 187 Dismiss signals across 40,000 sessions to provide a dense, robust representation of real user dissatisfaction.
2. Multi-Agent Quality Audit Pipeline: Curating High-Fidelity, Failure-Prone Test Instances
To prevent benign user queries or trivial grammatical errors from polluting the benchmark, a multi-agent auditing loop powered by Hunyuan and DeepSeek models is deployed over anonymized session logs. A Miner model scans multi-turn dialogue histories to isolate candidate interaction failures \(b \in \mathcal{C}_i\) spanning ten distinct failure taxonomies. An independent Judge agent subsequently scores each candidate on a 1–5 scale across five rigorous dimensions \(q_b \in [1, 5]^5\): completeness, credibility, representativeness, severity, and signal fidelity. Only candidates achieving a mean quality score \(\bar{q}_b \ge 4.0\) and a strict minimum score across all dimensions \(\min(q_b) \ge 3.0\) are retained. Final human expert verification over sampled splits confirms that 98.5% of BAD instances and 99.8% of GOOD instances exhibit high quality, ensuring that negative cases reflect authentic interaction friction (e.g., verbosity, emotional disconnect, incomplete execution) rather than obvious factual hallucinations.
3. Three-Tier Interconnected Task Design: Progressive Evaluation Across the Interaction Lifecycle
UXBench deconstructs user experience capabilities into three progressive challenges: Task 1 (UX Judge) assesses whether an evaluator model can ingest dialogue history \(h_i\), user query \(q_i\), and candidate response \(r_i\) to perform a binary prediction of latent user satisfaction \(\hat{y}_i \in \{0, 1\}\), tested on a class-balanced set of 1,000 GOOD and 1,000 BAD instances; Task 2 (UX Eval) tests response generation under an intentionally hard, failure-skewed distribution comprising 4,900 multi-turn queries where the original production response received an explicit user dislike, measuring the percentage of newly generated responses rated as positive (\(\text{Good\%}\)); Task 3 (UX Recovery) evaluates conversational resilience in 500 failure-critical scenarios where users voiced explicit, hostile complaints (\(c_i\)), measuring the model's ability to repair user rapport and correct prior errors before the interaction prematurely terminates.
4. Pointwise Generative Reward Modeling: Neutralizing Judge Self-Preference and Neutral Escape
Faced with evidence that LLM judges suffer from extreme positive bias, severe family-level self-preference (e.g., GPT judges preferring GPT outputs by over 80%), and "neutral escape" under multi-class Likert scales (where challenging bad responses are dumped into neutral categories), the authors develop a dedicated Pointwise Generative Reward Model (GRM). Built upon Hunyuan-3 (MoE architecture with ~20B active parameters), the GRM is trained on 8,547 positive and 8,559 negative in-the-wild interactions with explicit user feedback. Evaluating candidate responses directly against authentic user feedback patterns, the trained GRM achieves an overall accuracy of 77.2% and a bad-case recall of 72.4% on Task 1, demonstrating an 83.3% pairwise agreement with human expert annotators and serving as an objective, uninflated yardstick across all generation tasks.
Loss & Training¶
The Pointwise GRM is trained on in-the-wild interaction logs collected in May 2026, creating a strict temporal separation from the UXBench test suite (sourced between January and March 2026) to prevent test-set contamination. The training corpus consists of 8,547 positive and 8,559 negative interaction pairs balanced across real user feedback distributions. The model is trained using standard autoregressive cross-entropy loss over one epoch, generating natural language critique rationale followed by a binary satisfaction verdict. Optimization utilizes a learning rate of \(4 \times 10^{-6}\), a global batch size of 128, and a distributed compute cluster of 128 NVIDIA H20 GPUs. The training loss rapidly converges and stabilizes, confirming that user satisfaction modeling can be efficiently acquired from organic interaction telemetry without manual reward engineering.
Key Experimental Results¶
Main Results¶
The authors conduct extensive benchmarking across 26 frontier commercial and open-weight language models spanning Google Gemini, OpenAI GPT, Anthropic Claude, DeepSeek, ByteDance Doubao, Tencent Hunyuan, Zhipu GLM, Moonshot Kimi, and Alibaba Qwen. The table below presents the core performance across the three UXBench tasks, where Task 2 and Task 3 are evaluated by the calibrated Pointwise GRM:
| Model Family | Model | Task 1: Acc% | Task 1: Good-Acc% | Task 1: Bad-Acc% | Task 2: Good% | Task 3: Good% |
|---|---|---|---|---|---|---|
| Gemini 2.5 Flash | 59.4% | 97.5% | 21.2% | 36.1% | 4.6% | |
| Gemini 2.5 Pro | 62.8% | 96.8% | 28.7% | 50.8% | 4.0% | |
| Gemini 3.0 Flash | 67.4% | 97.7% | 37.1% | 52.7% | 12.7% | |
| Gemini 3.1 Pro | 70.4% | 91.6% | 49.3% | 57.1% | 9.2% | |
| OpenAI | GPT-5 mini | 65.2% | 93.4% | 36.9% | 24.0% | 3.6% |
| GPT-5 | 72.9% | 89.5% | 56.2% | 34.7% | 6.9% | |
| GPT-5.1 | 72.5% | 94.8% | 50.1% | 37.1% | 7.4% | |
| GPT-5.2 | 75.0% | 85.0% | 65.1% | 30.8% | 5.4% | |
| GPT-5.5 | 74.2% | 92.7% | 55.7% | 41.2% | 9.9% | |
| Anthropic | Claude Sonnet 4.5 | 69.3% | 89.6% | 49.0% | 36.0% | 9.0% |
| Claude Opus 4.5 | 66.7% | 96.7% | 36.7% | 37.9% | 9.4% | |
| Claude Opus 4.6 | 72.0% | 92.6% | 51.5% | 44.3% | 12.8% | |
| Claude Opus 4.7 | 75.3% | 89.1% | 61.5% | 44.5% | 12.4% | |
| DeepSeek | DeepSeek R1 | 58.5% | 98.7% | 18.3% | 39.5% | 5.8% |
| DeepSeek V3 | 55.6% | 99.7% | 11.6% | 35.9% | 3.6% | |
| DeepSeek V3.2 | 64.5% | 95.7% | 33.3% | 41.2% | 7.0% | |
| DeepSeek V4 Pro | 64.5% | 97.4% | 31.7% | 49.7% | 11.0% | |
| ByteDance | Doubao Seed 2.0 Pro | 60.8% | 98.8% | 22.9% | 48.7% | 10.8% |
| Others | Hunyuan 3 Preview | 64.3% | 95.6% | 33.1% | 48.8% | 7.6% |
| GLM-5.1 | 68.5% | 96.1% | 40.9% | 56.6% | 11.2% | |
| Kimi K2.6 | 68.7% | 96.1% | 41.2% | 52.3% | 11.4% | |
| Qwen3.6-Plus | 65.8% | 96.8% | 34.9% | 52.3% | 12.0% | |
| Trained | Pointwise GRM (Ours) | 77.2% | 82.1% | 72.4% | – | – |
Ablation Study¶
The authors examine the effects of multi-class labeling, prompt steering, and cross-benchmark transfer to understand evaluation bias and model capability boundaries:
1. Formulation Comparison: Neutral Escape in Multi-Class Judgment (Task 1) When transitioning from binary judgment to a three-class format (positive / neutral / negative), models systematically avoid negative verdicts by mapping bad instances into the neutral class: | Model | Binary BAD Acc (%) | 3-Class BAD Acc (%) | Delta (pp) | BAD \(\to\) Neutral Rate (%) | | :--- | :---: | :---: | :---: | :---: | | Claude Opus 4.7 | 61.5% | 32.7% | -28.8% | 37.8% (378/1,000) | | Claude Opus 4.6 | 51.5% | 41.8% | -9.7% | 38.5% (385/1,000) | | Claude Sonnet 4.5 | 49.0% | 31.4% | -17.6% | 26.7% (267/1,000) | | Gemini 3.1 Pro | 49.3% | 45.2% | -4.1% | 10.5% (105/1,000) | | GPT-5.5 | 55.7% | 47.7% | -8.0% | 10.3% (103/1,000) | | GPT-5.2 | 65.1% | 47.8% | -17.3% | 8.8% (88/1,000) | | Hunyuan 3 Preview | 33.1% | 25.1% | -8.0% | 6.9% (69/1,000) | | DeepSeek V4 Pro | 31.7% | 27.4% | -4.3% | 6.3% (63/1,000) |
2. Cross-Benchmark Generalization: UXBench vs. Public Instruction Benchmarks (Task 2) Evaluating responses under the trained Pointwise GRM across benchmarks reveals that failure-prone real queries present a far steeper challenge than curated instruction-following test sets: | Model | Family | UXBench (N=4,900) | Arena-Hard (N=500) | WildBench (N=1,024) | AlpacaEval (N=805) | | :--- | :--- | :---: | :---: | :---: | :---: | | Gemini 3.1 Pro | Google | 57.1% (Rank 1) | 88.7% (Rank 1) | 89.2% (Rank 1) | 76.2% (Rank 1) | | Gemini 2.5 Pro | Google | 50.8% (Rank 2) | 87.5% (Rank 3) | 82.2% (Rank 6) | 70.7% (Rank 4) | | DeepSeek V4 Pro | DeepSeek | 49.7% (Rank 3) | 81.7% (Rank 6) | 83.6% (Rank 3) | 68.6% (Rank 6) | | Hunyuan 3 Preview | Hunyuan | 48.8% (Rank 4) | 85.3% (Rank 5) | 80.4% (Rank 7) | 67.1% (Rank 7) | | Doubao Seed 2.0 Pro | Doubao | 48.7% (Rank 5) | 88.3% (Rank 2) | 85.6% (Rank 2) | 71.5% (Rank 2) | | Claude Opus 4.7 | Claude | 44.5% (Rank 6) | 87.5% (Rank 3) | 83.2% (Rank 4) | 69.1% (Rank 5) | | DeepSeek V3.2 | DeepSeek | 41.2% (Rank 7) | 81.7% (Rank 6) | 73.2% (Rank 9) | 61.6% (Rank 9) | | GPT-5.5 | OpenAI | 41.2% (Rank 7) | 79.3% (Rank 8) | 82.3% (Rank 5) | 71.6% (Rank 2) | | GPT-5.2 | OpenAI | 30.8% (Rank 9) | 77.1% (Rank 9) | 76.8% (Rank 8) | 65.5% (Rank 8) | | Benchmark Average | – | 45.9% | 84.1% | 81.8% | 69.1% |
Key Findings¶
- Finding 1: Frontier LLMs exhibit severe, systematic positive bias in zero-shot UX judgment. All 26 un-aligned zero-shot models heavily favor AI outputs, achieving high GOOD accuracy (>90%) but severely under-detecting dissatisfaction (DeepSeek V3 scored 11.6% on BAD recall; Gemini 2.5 Flash reached only 21.2%). This occurs because real-world dissatisfaction stems from subtle behavioral frictions like verbosity, weak engagement, and emotional tone mismatch rather than overt factual errors.
- Finding 2: Supervised Pointwise GRMs successfully acquire human-aligned UX sensitivity. The trained Hunyuan-3 GRM achieves 77.2% overall accuracy and 72.4% BAD accuracy on Task 1, substantially outperforming Claude Opus 4.7 (61.5% BAD Acc). It also aligns closely with human experts (83.3% agreement), confirming that predicting subjective user feedback is a learnable skill requiring explicit preference data rather than an emergent byproduct of generic pretraining.
- Finding 3: Model capability scaling yields weak real-world UX returns. Tracking model generations chronologically reveals only a moderate correlation between release date and UX generation quality (Spearman's \(\rho = 0.52, p = 0.043\)), which degrades further when excluding lightweight models (\(\rho = 0.48, p = 0.082\)). Frontier reasoning and coding flagships (such as Claude Opus 4.7 at 44.5%) underperform localized general-purpose models like DeepSeek V4 Pro (49.7%) and GLM-5.1 (56.6%) on routine Chinese QA and chit-chat.
- Finding 4: Commercial LLM judges display rampant self-preference bias. Pairwise judge experiments demonstrate extreme intra-family favoritism: GPT-5.2 awards GPT-family models win rates of 75.8%–83.2% against rival models, while Gemini 3.1 Pro favors Gemini responses with >74% win rates. Under pointwise scoring, Gemini assigns over 90% positive ratings to nearly all outputs, destroying leaderboard separability.
- Finding 5: Failure recovery (Task 3) remains an unsolved bottleneck governed by social repair strategies. All frontier models collapse on Task 3, with the highest-performing model (Claude Opus 4.6) reaching only a 12.8% recovery rate. Strategy analysis indicates that 45.98% of responses rely on generic apologies, while only 3.63% provide direct corrections. Claude models mitigate tension through humorous de-escalation and humble acknowledgment, whereas GPT models over-index on error diagnosis, and DeepSeek-V3 fails due to excessive clarification requests.
Highlights & Insights¶
- Behavioral Signals as Scalable Ground Truth: By elevating organic user clicks (likes/dislikes), reading skips (Skim), and repeated queries into benchmark annotations, UXBench bridges the divide between costly HCI user studies and automated NLP benchmarking.
- Failure Taxonomy Grounded in Commercial Reality: Auditing 70K+ interactions reveals that real user dissatisfaction is driven primarily by verbosity/redundancy (34.3%), task incompleteness (24.8%), and intent misunderstanding (11.5%), whereas positive feedback hinges on answer accuracy (17.4%) and depth (15.4%).
- Challenging the Capability-UX Monolith: Demonstrates that raw reasoning depth and benchmark-cracking performance do not automatically yield satisfying conversational interactions, establishing the need for user-centric scaling laws and dedicated generative reward modeling.
Limitations & Future Work¶
- Language and Cultural Specificity: The 70,000+ session logs are harvested exclusively from a mainstream Chinese commercial AI assistant, reflecting specific conversational norms and cultural communication patterns. Cross-cultural and multilingual validation remains an open frontier.
- Single-Turn Evaluation of Multi-Turn Dynamics: While Tasks 2 and 3 evaluate responses conditioned on rich multi-turn context, the actual generation is evaluated on a single critical turn rather than through live, interactive multi-turn dialogue rollouts.
- Binary Abstraction of Subjective Utility: Formulating UX as binary satisfaction avoids neutral-escape bias but omits fine-grained diagnostic attribution explaining why a user felt unsatisfied. Future work should combine binary supervision with interpretable multi-factor critiques.
Related Work & Insights¶
- vs MT-Bench / AlpacaEval / Chatbot Arena: Earlier benchmarks evaluate synthetic queries, static academic rubrics, or side-by-side comparative rankings. UXBench benchmarks real-world users interacting with single responses in organic, multi-turn consumer environments.
- vs WildBench: While WildBench leverages real user queries from in-the-wild logs, it relies on uncalibrated LLM judges applying complex textual checklists. UXBench introduces authentic behavioral ground-truth labels and trains a dedicated, bias-resistant Pointwise GRM.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering translation of real-world implicit and explicit user interaction signals into scalable UX evaluation ground truth.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 26 frontier models, detailed bias audits, ablation of prompt strategies, and fine-grained recovery analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear structure, rigorous mathematical formulations, and compelling narrative analysis supported by high-quality visualizations.
- Value: ⭐⭐⭐⭐⭐ Serves as a vital wake-up call for the LLM community to shift alignment objectives from synthetic benchmark optimization to authentic user satisfaction.