Skip to content

PIVOTS-Bench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://ciossayin.github.io/pivots-bench/
Area: Multimodal Vision Reasoning
Keywords: Multimodal Large Language Models, Fine-Grained Interpersonal Relationships, Bidirectional Reasoning, Causal Visual Cues, Social Intelligence Benchmark

TL;DR

PIVOTS-Bench introduces the first multimodal benchmark for evaluating fine-grained bidirectional interpersonal relationship reasoning across six psychological dimensions, revealing substantial deficiencies of current MLLMs in grounded visual causality and implicit social dynamics.

Background & Motivation

Humans possess an innate capability to decode fine-grained interpersonal relationships from everyday dynamic interactions, effortlessly synthesizing verbal dialogue with nonverbal signals such as facial expressions, gaze shifts, and posture to deduce social hierarchy, emotional intimacy, and collaborative intent. Mastering such interpersonal perception is foundational for artificial social intelligence, empowering systems to navigate complex multi-agent collaboration, negotiations, persuasion, and conflict resolution. However, traditional Social Relationship Recognition (SRR) typically reduces human relationships into coarse-grained, static categorical taxonomies (e.g., family, friends, colleagues, or strangers), which fundamentally fails to capture the drastically different social dynamics that can emerge within the very same nominal role pair across differing contexts.

Crucially, human social perception is intrinsically bidirectional and asymmetric. In any given dyadic interaction, the two participants often form divergent relational perceptions due to information asymmetry, differing social power, distinct communicative goals, and conflicting emotions. Furthermore, linguistic transcriptions alone frequently lack essential social context or conceal crucial subtext, making nonverbal visual cues indispensable for disambiguating underlying relationship states. Although recent multimodal large language models (MLLMs) have achieved remarkable performance on standard visual perception benchmarks, whether they can reliably integrate visual and verbal cues to reason about bidirectional interpersonal dynamics remains largely unexplored.

To address this critical gap, this work establishes a rigorous foundation grounded in classical interpersonal psychology and affective science, mapping complex relationships onto six continuous, bipolar relational axes. The core idea is to establish PIVOTS-Bench, the first multimodal benchmark evaluating fine-grained bidirectional interpersonal relationship reasoning through six-dimensional scoring, key-frame identification, and grounded visual cue causal analysis across explicit and implicit social interaction contexts.

Method

Overall Architecture

PIVOTS-Bench establishes an end-to-end evaluation paradigm comprising three primary phases: theoretical formulation of the six-dimensional bipolar system, curated multi-source video collection with human-calibrated verification, and a three-tiered hierarchical evaluation protocol. Given an input interaction video clip \(V\) (30 seconds), transcribed utterances \(U\), and a designated directional subject pair \(D^\alpha_{A \to B}\), evaluated models must predict continuous ordinal ratings across specified dimensions and ground their decisions in causal visual evidence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Video clip V + Utterances U + Directed Pair Aโ†’B"] --> B["Six-Dimensional PIVOTS Framework<br/>P/I/V/O/T/S 5-point bipolar Likert scale [-2, +2]"]
    B --> C["Curated Dual-Source Dataset<br/>Social-IQ 2.0 (Explicit) + YouTube (Implicit)"]
    C --> D["Task 1: Six-Dimensional Relational Scoring<br/>Separate / Joint / Pairwise Prediction"]
    C --> E["Task 2: Key Frame Identification<br/>Identify most representative visual moment"]
    C --> F["Task 3: Visual Cue Causal Analysis<br/>Select supporting cues from expressions/posture/attire"]

Key Designs

1. Six-Dimensional Bidirectional PIVOTS Relational Framework: Beyond Coarse Categorical Taxonomies

To overcome the expressive limitations of discrete, coarse relationship categories, the benchmark adapts and extends Wish et al.'s classical interpersonal perception theory and Russell's circumplex affect model into six continuous bipolar dimensions. Each axis is operationalized as a discrete five-point Likert scale ranging from -2 to +2: - P (Egalitarian vs. Hierarchical Power): Evaluates power balance and decision-making agency, spanning extreme self-dominance (+2), egalitarian symmetry (0), to other-dominance (-2); - I (Superficial vs. Intense Involvement): Measures psychological intimacy and emotional investment, from deep vulnerability and merged identity (+2) to purely transactional encounters (-2); - V (Positive vs. Negative Valence): Captures emotional satisfaction and pleasantness versus interpersonal hostility and distress, ranging from highly restorative (+2) to antagonistic (-2); - O (Socioemotional vs. Task-Oriented Objective): Contrasts relationships driven purely by personal connection (+2) against those governed strictly by formal functional tasks (-2); - T (Temporary vs. Enduring Permanence): Assesses expected relational longevity, from lifelong enduring commitment (+2) to one-off ephemeral interactions (-2); - S (Cooperative vs. Competitive Stance): Reflects goal alignment and resource sharing, from deep collaborative synergy (+2) to zero-sum adversarial competition (-2).

A core feature is its directional asymmetry: for any interacting dyad, subject A's evaluation toward subject B (\(D^\alpha_{A \to B}\)) is modeled independently from subject B's evaluation toward subject A (\(D^\alpha_{B \to A}\)), faithfully reflecting human perceptual divergence.

2. Dual-Source Interaction Video Curation and Model-Assisted Human Calibration

To avoid overfitting to overt, exaggerated theatrical behaviors, PIVOTS-Bench incorporates two complementary video sources: - Social-IQ 2.0 Subset: 121 curated video clips focusing on explicit, daily interactions and formal debates, filtering out artificial interviews and text overlays to deliver localized, salient behavioral cues. - YouTube In-Depth Context Subset: 70 carefully curated real-world interactions characterized by restrained, subtle, and psychologically nuanced dynamics (e.g., estranged partners, long-lost siblings, ideologically polarized discussions). In these clips, interpersonal tensions emerge from global discourse and relational histories rather than localized gestures.

The annotation follows a model-first, human-calibrated protocol: comprehensive prompts encoding dimensional rubrics and contextual transcripts are provided to GPT-5 to generate initial scores, which are subsequently scrutinized, verified, and adjusted by human researchers. Human annotators revised 25.3% of the initial scores. An independent validation study on 35 videos achieved a Cohen's \(\kappa\) of 0.8399 against pure human annotation, and the overall dataset attained a Krippendorff's \(\alpha\) of 0.8125, validating the high reliability of the ground-truth ratings.

3. Three-Tiered Hierarchical Benchmark Tasks: From Dimensional Rating to Grounded Visual Causality

To probe whether MLLMs genuinely perceive social dynamics or merely rely on superficial textual heuristics, the benchmark formulates three hierarchical evaluation tasks: - Task 1: Six-Dimensional Scoring: Given video \(V\), transcript \(U\), and directed dyad \(D^\alpha_{A \to B}\), the model predicts the discrete Likert rating \(r \in \{-2, -1, 0, 1, 2\}\): $\(R^\alpha_{A \to B} = \arg\max_{r \in \{-2, -1, 0, 1, 2\}} p(r \mid V, U, D^\alpha_{A \to B})\)$ The benchmark examines three operational settings: separate prediction per dimension, joint multi-dimensional prediction, and pairwise bidirectional modeling. - Task 2: Key Frame Identification: Formulated as a 5-option multiple-choice task, requiring the model to identify the specific video frame \(T^\alpha_{A \to B}\) that provides the strongest visual evidence for a given dimension score. - Task 3: Visual Cue Causal Analysis: Conditioned on the identified key frame, the model must select the two most decisive visual elements (from candidates covering facial expressions, hand gestures, body posture, physical distance, and attire/context), evaluated using partial (0.5) and full (1.0) credit scoring: $\(C^\alpha_{A \to B} = \arg\max_{c \in \{o_1, \dots, o_n\}} p(c \mid T^\alpha_{A \to B}, D^\alpha_{A \to B})\)$ This verifies whether the model grounds its relational inferences in authentic visual evidence.

Key Experimental Results

Main Results

Evaluations were conducted across proprietary frontier models (GPT-5, Gemini-2.5-pro) and open-source models (Qwen3-4B/8B/32B-VL) on both the Social-IQ 2.0 and YouTube subsets. Task 1 accuracy reflects exact-match 5-class classification (chance level 20%), Task 2 evaluates 5-way frame selection (chance 20%), and Task 3 assesses 5-choose-2 visual cue attribution (expected random baseline 40%).

Dataset Subset Model Avg Acc Power (P) Involve (I) Valence (V) Objective (O) Time (T) Stance (S) Task 2 Task 3
Social-IQ 2.0 Gemini-2.5-pro 47.71 61.30 50.77 33.75 51.63 46.98 41.86 46.63 48.00
Social-IQ 2.0 GPT-5 56.91 65.12 60.31 55.73 42.33 56.59 61.40 62.15 60.25
Social-IQ 2.0 Qwen3-4B-vl 24.33 51.05 20.77 23.99 13.73 25.93 10.47 28.59 30.17
Social-IQ 2.0 Qwen3-8B-vl 27.90 59.74 22.22 22.87 18.58 26.89 17.07 34.75 31.00
Social-IQ 2.0 Qwen3-32B-vl 34.54 62.62 23.82 33.82 23.77 30.28 31.50 35.29 33.50
YouTube Gemini-2.5-pro 54.66 69.49 55.08 31.78 61.02 54.24 56.36 - -
YouTube GPT-5 58.88 73.71 50.44 48.28 61.14 60.68 58.85 - -
YouTube Qwen3-4B-vl 30.03 71.98 16.81 33.62 6.90 29.31 21.55 - -
YouTube Qwen3-8B-vl 35.19 66.67 26.85 31.48 18.52 32.41 35.19 - -
YouTube Qwen3-32B-vl 40.81 67.92 32.38 34.91 29.47 35.85 43.14 - -

Ablation Study

Ablations on input modalities, explicit social role masking (replacing kinship/honorific terms like "father" or "boss" with [PERSON]/[TITLE]), acoustic inputs, and prediction paradigms were conducted using Qwen3-32B-VL and Qwen3-Omni-30B.

Configuration Social-IQ 2.0 Avg YouTube Avg Dimension-Specific Findings & Notes
Video Only 33.70 31.18 Decent baseline in overt daily scenes, but plummets in subtle dialogues
Original Conv. Only 31.44 37.81 Text-only suffices for global interview context, but lacks visual alignment
Redacted Conv. Only 28.97 33.64 Removing explicit role tokens degrades performance by 2.47% and 4.17%
Video + Original Conv. 34.54 40.81 Full multimodality yields optimal reasoning (+3.10% and +3.00% gains)
Video + Redacted Conv. 35.83 32.22 Visual signals compensate strongly (+6.86% gain) when explicit roles are absent
Qwen3-Omni-30B (w/ Audio) 40.93 40.10 Acoustic signals and pitch inflection significantly enhance social affect reasoning
Qwen3-Omni-30B (w/o Audio) 32.61 38.30 Removing audio drops Social-IQ performance by 8.32%, confirming prosodic importance
Joint Prediction 43.26 38.63 Multi-dimensional synergy benefits overt scenes, but causes cross-axis interference on YouTube
Pairwise Prediction 42.19 39.67 Bidirectional modeling excels on Power (86.96% on YouTube P-axis), but hurts fine-grained axes

Key Findings

  • Proprietary Frontier Models Substantially Outperform Open-Source MLLMs: GPT-5 achieves 56.91% and 58.88% overall accuracy across the two benchmarks, whereas the strongest open-source model (Qwen3-32B-VL) scores 34.54% and 40.81%. Crucially, on Task 3 (visual cue causal analysis), open-source models score below the 40% random chance baseline (30.17%โ€“33.50%), showing severe blindness to micro-level physical cues.
  • Valence (V) and Objective (O) Pose Persistent Bottlenecks: All models struggle significantly on emotional valence and functional objectives. Valence inference heavily depends on subtle vocal inflection and fleeting micro-expressions, while objective classifications falter when formal task-driven interactions are accompanied by intense emotional displays.
  • Paradigm Inversion Between Explicit and Implicit Social Contexts: In Social-IQ's overt interactions, Joint prediction dominates Separate prediction (43.26% vs. 34.54%), and heuristic multi-stage prompting boosts accuracy by +6.69%. In contrast, in YouTube's restrained, in-depth context, Separate prediction performs best (40.81%), while rigid multi-stage prompting causes a sharp decline (-9.33%), demonstrating that rigid reasoning heuristics can misguide models in subtle social settings.

Highlights & Insights

  • Theoretically Grounded Continuous Multimodal Formulation: Transforms interpersonal relationship modeling from simplistic categorical labeling into an asymmetric, six-dimensional continuous coordinate system grounded in established social psychology.
  • Grounded Causal Visual Cue Verification: By coupling outcome scoring with keyframe localization and visual cue attribution, the benchmark effectively diagnoses whether an MLLM truly understands multimodal social dynamics or merely exploits textual linguistic shortcuts.
  • Actionable Takeaways for Socially Aware AI: Highlights the necessity of building context-adaptive reasoning architecturesโ€”leveraging holistic joint reasoning for overt everyday encounters while preserving decoupled, dimension-specific focus in subtle, restrained interactions.

Limitations & Future Work

  • Author-Internal Annotation: Annotations were conducted by the research authors rather than third-party crowdsourced workers. While ensuring rigorous adherence to the psychological rubrics, this introduces potential shared prior assumptions; additionally, pre-annotation with GPT-5 may induce slight anchoring bias.
  • Cultural and Linguistic Scope: The current benchmark is restricted to English-language interactions within Western-centric cultural representations, omitting cross-cultural etiquette, varied nonverbal norms, and multilingual dynamics.
  • Directions for Future Research: Extending the evaluation to multi-party group dynamics, integrating 3D gaze tracking and micro-gestural stream encoders, and leveraging the benchmark data for preference alignment and reinforcement learning in socially aware AI agents.
  • vs Social Relationship Recognition (SRR): Prior methods (e.g., VISR, Dual-glance) predominantly focus on static images or coarse categorical roles (e.g., leader, friend). PIVOTS-Bench establishes a fine-grained, bidirectional, six-dimensional continuous scale for dynamic interactions.
  • vs SCRIPTS: SCRIPTS is a text-only benchmark evaluating dialogue scripts with undirected labels; PIVOTS-Bench models multimodal behavioral grounding and bidirectional cognitive asymmetry.
  • vs MMToM-QA / Social Genome: Prior Theory-of-Mind benchmarks rely heavily on synthetic environments or general social QA; PIVOTS-Bench provides a standardized quantitative evaluation of real-world interpersonal dynamics and causal visual grounding.

Rating

  • Novelty: โญโญโญโญโญ First multimodal benchmark formalizing fine-grained bidirectional interpersonal dynamics via six psychological axes.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluations across frontier closed/open MLLMs, multimodal ablations, token redactions, and reasoning paradigms.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear mathematical formulation, rigorous psychological foundation, and insightful experimental analysis.
  • Value: โญโญโญโญโญ Sets a crucial foundation for evaluating and advancing social intelligence, affective understanding, and grounded social reasoning in MLLMs.