Skip to content

SocialArena: Evaluating Social Intelligence in Large Language Model Agents via Multi-Party Strategic Interactions

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / Social Intelligence / Agent Evaluation
Keywords: Social Intelligence, Theory of Mind, Role-Playing, LLM-as-a-Judge, Arena Evaluation

TL;DR

SocialArena establishes a unified three-tier framework defining social intelligence across role-playing, 7-dimension Theory of Mind (ToM) mental state reasoning, and contextual behavioral reasoning; backed by the orthogonal U-TRIMF scenario synthesis pipeline and a $132.4K human-annotated pairwise arena protocol across 15 frontier LLMs, it reveals that models suffer from an acute "knowing vs. acting" perception-action gap and that automated LLM judges severely misalign with human social consensus.

Background & Motivation

As large language models (LLMs) demonstrate remarkable proficiency in formal problem-solving benchmarks—such as mathematical reasoning, competitive programming, and programmatic tool utilization—they are increasingly being deployed in domains requiring deep social interaction, including mental health counseling, empathetic companionship, educational tutoring, and non-player character (NPC) simulations. However, navigating the human social fabric requires far more than cold symbolic calculation; it demands affective attunement, subtext decoding, status navigation, and the dynamic resolution of interpersonal frictions. Despite its paramount significance, how effectively modern frontier models navigate the nuanced space of social intelligence remains an open and poorly grounded question.

Existing evaluation benchmarks suffer from three fundamental structural limitations. First, capability assessment remains fragmented: most benchmarks isolate single superficial facets such as stylistic role-play consistency, static multiple-choice Theory-of-Mind (ToM) puzzles (e.g., false-belief tests), or narrow persuasive negotiation games, failing to integrate social cognition into a coherent cognitive architecture. Second, scenario coverage is heavily constrained: existing suites evaluate narrow micro-domains such as dating apps, workplace negotiations, or crime-mystery games, yielding domain-locked findings that cannot generalize across diverse sociological settings. Third, evaluation methodologies face a critical reliability crisis: multiple-choice QA strips away situated interactive dynamics, while automated LLM-as-a-Judge protocols exhibit severe misalignment when diagnosing nuanced social trade-offs, interpersonal boundaries, and ethical limits.

To systematically address these bottlenecks, this paper establishes a theoretically grounded, comprehensive, and scalable benchmark for LLM social intelligence. Grounded in sociology, graph topology, and game theory, the study models social interactions from base situational coordinates to fine-grained cognitive reasoning traces and actionable decisions. Core idea: define social intelligence across a unified three-tier hierarchy spanning role-playing, 7-dimensional Theory of Mind (ToM) mental state reasoning, and contextual behavioral reasoning; synthesize broad scenarios via the orthogonal U-TRIMF framework; and benchmark 15 frontier LLMs via an explicit-thinking, human-annotated double-blind arena protocol backed by fine-grained natural-language critiques.

Method

Overall Architecture

The SocialArena benchmark architecture is built upon three modular pillars: U-TRIMF Orthogonal Scenario Space Construction to guarantee broad and unbiased scenario distributions; Structured Role-Playing Dialogue with Explicit Character Thinking to expose the latent cognitive trajectory from social perception to decision-making; and a Human-Centered Double-Blind Arena Protocol ensuring unassailable evaluation validity and rich diagnostic feedback.

Given an interactive social scenario \(S(p, b, h, n)\)—encompassing a plot summary \(p\), background setting and character descriptions \(b\), multi-turn dialogue history \(h\), and the target speaker \(n\)—two anonymized models adopt role \(n\) to generate first-person inner mental thinking alongside their observable verbal/action responses. Trained human annotators conduct pairwise multi-dimensional reviews, yielding Elo-based leaderboard rankings and granular capability profiles across 100 evaluation rubrics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Theoretical Foundations & Social Constraints"] --> B["U-TRIMF Orthogonal Scenario Space Construction<br/>6-dimensional sampling·keyword guidance·two-stage filtering"]
    B --> C["Structured Role-Playing Dialogue with Explicit Thinking<br/>7 ToM mental states·outcome-based behavior decoupling·first-person trace"]
    C --> D["Human-Centered Double-Blind Arena Protocol<br/>3-way independent review·9-dimension scoring·natural language critiques"]
    D --> E["Diagnostic Social Intelligence Profiles & Elo Rankings<br/>global leaderboard·100-dimension rubric taxonomy·failure attribution"]

Key Designs

1. U-TRIMF Orthogonal Scenario Space Construction: Preventing Domain Collapse and Archetypal Degeneracy

Existing role-play datasets typically rely on ad-hoc seed prompts that cause models to collapse into stereotypical tropes. SocialArena introduces the U-TRIMF framework, decomposing any social dilemma into three core components across six orthogonal theoretical dimensions: - User Presence (\(U\)): Distinguishes Real-world settings (\(U_1\), where no external user exists and the LLM acts as an embedded character governed by strict environmental constraints) from User-centered settings (\(U_2\), where the LLM directly serves an active human user and possesses narrative authority to steer world dynamics); - Characters (\(T\) & \(R\)): Encompasses Topological Relations (\(T\): symmetric dyads \(T_2\) and symmetric groups \(T_3\), excluding solitary individuals \(T_1\)) and Relational Models (\(R\), grounded in Fiske's 1992 Relational Models Theory: Communal Sharing \(R_1\), Authority Ranking \(R_2\), Equality Matching \(R_3\), Market Pricing \(R_4\), and Null/Asocial \(R_5\)); - Events (\(I\), \(M\), \(F\)): Encompasses Interest Structure (\(I\), derived from Game Theory: zero-sum \(I_1\), negative-sum \(I_2\), positive-sum \(I_3\)), Motivational Type (\(M\), adapted from Maslow's hierarchy: physiological \(M_1\), safety \(M_2\), belonging \(M_3\), esteem \(M_4\), self-actualization \(M_5\)), and Formality level (\(F\): informal everyday discourse \(F_1\) vs. formal institutional settings \(F_2\)).

To avoid prototype degeneracy where LLMs generate repetitive archetypal plots for identical category combinations, the pipeline introduces Keyword-Guided Scenario Synthesis: each generation is seeded with 10 thematic keywords sampled from single chapters across 10,000 literary books in Project Gutenberg as soft narrative anchors. Subsequently, a rigorous two-stage filtering pipeline is executed. Stage one applies automated rule-based filters covering ethical/safety constraints (prohibiting life-threatening, violent, criminal, or irreversible catastrophic situations), scenario complexity (excluding trivial everyday disputes, forced all-or-nothing extremes, and asymmetric stakes), and narrative structure (enforcing immediate action pressure without free-exit loopholes). Stage two employs manual expert review over plausibility, conflict balance, and role-play affordance, curating 5,702 pristine social scenarios.

2. Structured Role-Playing Dialogue with Explicit Character Thinking: De-Blackboxing Latent Social Cognition

Evaluating models solely on surface dialogue makes it impossible to discern whether an appropriate response stems from genuine socio-cognitive comprehension or superficial pattern matching. Unconstrained free-form thinking traces, however, lead to high annotation entropy and noisy reviews. SocialArena enforces a dual-structured response paradigm: - Structured Theory of Mind (ToM) Reasoning: Models must generate a first-person cognitive reasoning trace inside <thinkcharacter> tags prior to uttering their final response. Cognitive inferences must explicitly map onto Beaudoin et al.'s 7-category ToM taxonomy by adaptively selecting 1–4 salient dimensions: Intentions (\(T_1\)), Emotions (\(T_2\)), Desires (\(T_3\)), Percepts (\(T_4\)), Knowledge (\(T_5\)), Beliefs (\(T_6\)), and Non-Literal Communication (\(T_7\), covering sarcasm, politeness, deception, and subtext). - Outcome-Only Behavior Judgment: For the subsequent behavioral output (dialogue and physical actions), annotators evaluate only the context-grounded appropriateness and effectiveness of the end behavior, agnostic of intermediate rationalization paths. This decoupling directly reflects the verifiable outcome-reward setup in reinforcement learning (RLVR), providing clean credit assignment for social perception vs. social action.

3. Human-Centered Double-Blind Arena Protocol: Eliminating Judge Biases and Extracting 100-Dimension Rubrics

Given the subtle, culturally conditioned nature of social dilemmas, automated LLM judges frequently introduce severe hallucinations and stylistic biases. The authors invested $132.4K to recruit and calibrate a bilingual cohort of native English and Chinese annotators under strict double-blind arena guidelines: - Triplicate Majority Voting & Strict Quality Gates: Each paired model battle (Model A vs. Model B) is independently annotated by at least three annotators across the label space \(l \in \{A, B, T, G\}\) (A wins, B wins, both good, both poor). Samples with complete three-way divergence are discarded, and annotators maintaining a quality pass rate below 75% in expert audit batches have their submissions purged. - Nine-Dimension Multi-Scale Scoring & Natural Language Rationales: Annotators score models across the 7 ToM dimensions, overall cognitive depth (\(T\)), and behavioral appropriateness (\(B\)), while providing concrete natural-language justifications that cite verbatim dialogue evidence. - Granular 100-Dimension Rubric Taxonomy: By running multi-stage embedding clustering on 51,203 human critique statements, the authors distill a comprehensive taxonomy of 100 fine-grained evaluation rubrics spanning Role-playing (19 rubrics), Theory-of-Mind Reasoning (22 rubrics), and Behavioral Reasoning (59 rubrics), enabling unprecedented diagnostic visibility into model-specific socio-cognitive pathologies.

Benchmark & Evaluation Protocol

The benchmark evaluates 15 state-of-the-art closed-source and open-weights models ranging from 20B parameters to 397B+ MoE architectures: - Closed-source flagships: gpt-5.1-2025-11-13, qwen3-max-2026-01-23, doubao-seed-1-8-251228, claude-sonnet-4-5-20250929, gemini-3-pro-preview, minimax-m2.1, kimi-k2.5, glm-5, gpt-4o-2024-11-20, gemini-3-flash-preview; - Open-weights representatives: qwen3.5-397b-a17b, glm-4.7-flash, qwen3-32b, gpt-oss-20b, olmo-3.1-32b-think.

Between 50 and 80 pairwise battles are conducted for each model pair across the 5,702 scenarios, totaling 17,676 verified human votes mapped to global Elo ratings. Furthermore, an expert-curated subset of 152 instances (SocialArena-RewardBench) serves to benchmark automated LLM evaluators against human expert consensus on label accuracy (ACC@Label) and critique semantic consistency (ACC@Critique with F1 threshold 0.5).

Key Experimental Results

Main Results: Frontier LLM SocialArena Leaderboard

The global leaderboard ranks all 15 evaluated models using Bradley-Terry Elo scores computed from 17,676 human pairwise battles:

Rank Model Identifier Model Class Elo Score Qualitative Socio-Behavioral Profile
#1 gpt-5.1-2025-11-13 Closed-source Flagship 1553.0 Substantial margin of leadership; exceptional conflict de-escalation and long-term interest defense
#2 qwen3-max-2026-01-23 Closed-source Flagship 1400.0 Tier 1; robust handling of organizational hierarchies and multi-party pragmatic compromises
#3 doubao-seed-1-8-251228 Closed-source Flagship 1398.5 Tier 1; highly natural conversational prosody, delicate affective calibration, and high empathy
#4 claude-sonnet-4-5-20250929 Closed-source Flagship 1321.6 Impeccable role boundaries and principled ethical restraint; slightly over-defensive action style
#5 gemini-3-pro-preview Closed-source Flagship 1283.8 Comprehensive ToM cognitive attribution; occasionally hesitant in executing decisive actions
#6 minimax-m2.1 Commercial API 1264.6 Natural role voice; strong performance in informal social companionship and interpersonal repair
#7 kimi-k2.5 Commercial API 1251.8 Consistent long-horizon intent tracking; moderate assertiveness in high-tension negotiations
#8 glm-5 Commercial API 1241.9 Balanced socio-cognitive grasp; occasional vulnerability to excessive concession under pressure
#9 qwen3.5-397b-a17b Open-weights MoE 1225.2 Top-ranked open-weights model; ToM reasoning performance on par with leading commercial APIs
#10 gpt-4o-2024-11-20 Closed-source Baseline 1189.0 Legacy multimodal flagship; falls into mid-tier when confronting complex multi-agent strategic stakes
#11 gemini-3-flash-preview Lightweight Closed 1164.7 Fluent colloquial exchanges; logic oversimplifies in complex zero-sum resource distributions
#12 glm-4.7-flash Lightweight Closed 1018.1 Stable baseline persona fidelity; struggles with adversarial or layered manipulative social cues
#13 qwen3-32b Open-weights Dense 1000.0 Baseline reference anchor; frequent misattributions of hidden subtext and non-literal intent
#14 gpt-oss-20b Open-weights Baseline 944.2 Rigid behavioral options; easily steered off-character by aggressive user pressure
#15 olmo-3.1-32b-think Open-weights Dense 828.5 Exhibits explicit reasoning traces, but suffers from acute decoupling between ToM perception and action

Evaluation Reliability: Human Expert Alignment of LLM-as-a-Judge (SocialArena-RewardBench)

On the expert-validated SocialArena-RewardBench (152 triple-verified instances), leading LLM judges were evaluated against single random human evaluators:

Evaluator System Preference Label Accuracy (ACC@Label) Critique Semantic Consistency (ACC@Critique) Error Diagnosis & Failure Modes
Human Evaluator (Baseline) 0.8355 0.7632 Displays stable perception of unwritten rules, boundaries, and long-term consequences
gpt-5.4 0.4408 0.2039 Heavily biased toward superficial politeness and length; fails to detect covert boundary erosion
claude-sonnet-4.6 0.4474 0.1579 Overly pedantic on disclaimers; misses conversational subtext; lowest critique semantic match
kimi-k2.5 0.4473 0.1839 Strong bias toward scoring ties (\(T\)); overlooks nuanced tactical deceptions and hidden aggression
qwen3-max 0.4276 0.2500 Accurately flags glaring factual slips, but inverts preference when judging complex strategic trades

Inter-Annotator Agreement & Artifact Bias Auditing

  • Inter-Annotator Agreement: Under the 4-choice label distribution \(\{A, B, T, G\}\), random guessing yields \(p_{full}=6.25\%\), \(p_{maj}=56.25\%\), and \(p_{dis}=37.50\%\). Observed human annotations demonstrate high convergence: Full agreement (\(3/3\)) reached 39.18% (over \(6\times\) random chance), majority agreement (\(2/3\)) reached 52.54%, while full disagreement (\(0/3\)) dropped to 8.29% (over \(4\times\) below chance).
  • Arena Bias Analysis:
  • Length Bias: Longer responses triumphed in only 51.05% of battles, with a negligible mean length difference of 14.0 tokens and median difference of 3.0 tokens, confirming the absence of verbosity gaming.
  • Position Bias: Response B won 43.63% vs. Response A's 39.23%, indicating an exceptionally mild position effect.
  • Format / Tone Bias: Unformatted plain dialogue utterances naturally bypassed markdown and sycophantic formatting biases.

Key Findings

  1. The Knowing-Doing Disconnect: Social Perception Does Not Equal Social Action
    Among the top 10 cluster centers extracted from human critiques, 6 cluster centers belong to Behavioral Reasoning (e.g., "Demonstrating mature social competence & relationship-building", "Resolving core practical concerns", "Driving conflict de-escalation"). Critically, human critiques on Behavioral Reasoning are overwhelmingly negative across almost all models (negative proportions exceeding 75–80% for long-horizon conflict resolution and relationship repair), whereas Role-Playing consistency displays balanced positive/negative feedback. This unmasks an insidious "capability illusion": models speak smoothly and maintain character personas, but utterly stumble when translating social perceptions into actions that resolve friction, respect boundaries, or achieve pragmatic goals.
  2. Orthogonality Between Cognitive STEM Problem-Solving and Social Intelligence
    Models that exhibit near-parity on math, coding, and instruction-following benchmarks diverge massively on SocialArena. For instance, open-weights qwen3.5-397b-a17b and commercial glm-5 share similar coding-agent rankings but exhibit nearly a 200-point Elo delta on social dynamics; compact reasoning models like olmo-3.1-32b-think fail to crack 830 Elo despite extensive CoT tuning. Pure cognitive task optimization does not organically foster social intelligence.
  3. Scenario-Specific Specialization and Fragility
    Kendall \(\tau\) rank correlation across scenario types demonstrates severe cross-scenario volatility. While conflict-heavy contexts (workplace, neighbors, business) exhibit high inter-correlation (\(\tau \approx 0.75-0.89\)), peaceful or collaborative domains (romantic, TRPG/roleplay, casual friendship) correlate poorly with conflict scenarios (\(\tau \approx 0.41-0.56\)). Model strengths do not transfer uniformly across distinct relational dynamics.

Highlights & Insights

  • The U-TRIMF Theoretical Scaffold: By operationalizing Maslow's motivations, Fiske's relational models, and game-theoretic payoffs into a 6-dimensional generative grid, the paper pioneers a principled methodology to synthesize diverse, non-degenerate social spaces.
  • Explicit ToM Mind-Reading Architecture: Structuring latent social reasoning into 7 explicit ToM mental dimensions while keeping behavior scoring purely outcome-based provides an interpretable blueprint for designing future social reward models and verifiable RLHF pipelines.
  • Debunking Automated Social Judges: By empirically proving that premier LLMs achieve less than 45% label agreement and below 25% critique alignment with human experts, the work provides conclusive evidence that human validation remains irreplaceable in evaluative domains governed by subjective social norms.

Limitations & Future Work

  • Single-Turn Interaction Horizon: To manage the substantial annotation budget ($132.4K) and ensure statistical control, evaluation centers on single-turn completions seeded by rich conversational history, leaving long-horizon multi-turn trust development and relationship decay for future expansion.
  • Cultural and Demographic Boundaries: Annotations were conducted by native English and Chinese speakers; subtle norms in high-context vs. low-context cultures, regional subtext, and diverse sociolinguistic conventions warrant broader community engagement via the ongoing SocialArena-Platform.
  • Closing the Loop on Unverifiable Reward Modeling: While SocialArena diagnoses why models fail at the perception-action interface, formalizing dense, verifiable reward functions that distill these 100 human rubrics into stable policy gradients remains an open challenge.
  • vs. Role-Playing Benchmarks (InCharacter, CharacterBench, CoSER): Role-play benchmarks focus almost exclusively on superficial stylistic consistency and persona retention; SocialArena treats persona stability as a baseline prerequisite and elevates the evaluation to strategic mental state reasoning and behavioral efficacy.
  • vs. Theory-of-Mind Benchmarks (OpenToM, SimpleToM, ToMATO): Prior ToM benchmarks rely on synthetic multi-choice QA or passive narrative probes; SocialArena tests whether an agent can actively deploy its ToM inferences to navigate open-ended, multi-agent strategic stakes.
  • vs. Social Intelligence Benchmarks (SOTOPIA, SAGE, SI-Bench): Earlier social benchmarks evaluate narrow scenarios (e.g., dating or single-room negotiations) and rely heavily on uncalibrated LLM judges; SocialArena scales to 5,702 structured scenarios and anchors its leaderboard on 17,676 rigorous human votes.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Groundbreaking formulation of the U-TRIMF scenario space and unified three-tier social intelligence architecture.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Impressive $132.4K human annotation effort yielding 17.6K pairwise comparisons across 15 premier models with zero length gaming.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous prose, seamless interdisciplinary grounding, and exceptionally insightful diagnostic analyses.
  • Value: ⭐⭐⭐⭐⭐ A milestone contribution that exposes critical blind spots in post-training and sets the standard for social intelligence benchmarking.