Skip to content

PIVOTS-Bench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://ciossayin.github.io/pivots-bench/
Area: VLM Reasoning
Keywords: interpersonal relationship reasoning, multimodal social intelligence, directional asymmetry, visual causal analysis, PIVOTS framework

TL;DR

This paper introduces PIVOTS-Bench, the first multimodal benchmark evaluating fine-grained interpersonal relationship reasoning in MLLMs across six bidirectional psychological dimensions, revealing pronounced performance gaps and unexpected behavioral reversals between explicit dialogues and nuanced social interactions.

Background & Motivation

Humans naturally decipher nuanced interpersonal relationships during everyday social encounters. This social cognition ability underpins collaboration, negotiation, empathy, and conflict resolution. Yet authentic human relationships are highly fluid and multi-dimensional, eluding simplistic categorical labels such as friends, family, colleagues, or acquaintances. For instance, two distinct interactions between parents and children can manifest diametrically opposed relational dynamicsโ€”one characterized by warm, collaborative banter, and the other by bitter accusation and hostility. Traditional Social Relationship Recognition (SRR) approaches typically cast this challenge as coarse-grained discrete classification over static images or short clips, largely ignoring the fine-grained, continuous spectrum of interpersonal dynamics.

Recent investigations into the social intelligence of foundation models reveal two major limitations. First, existing computational linguistic benchmarks operate almost exclusively on textual dialogue corpora, whereas critical interpersonal signalsโ€”including power imbalances, interpersonal tension, and affective disengagementโ€”are fundamentally carried by non-verbal multimodal cues such as subtle facial micro-expressions, body postures, eye gaze avoidance, and vocal tone. Second, prior relationship modeling frameworks largely presume symmetric or unidirectional dynamics. In real-world interactions, information asymmetry, differing communicative intents, and social roles frequently cause dyadic participants to hold drastically asymmetric perceptions of each other. A senior mentor may view an encounter as benevolent guidance, whereas the junior mentee experiences it as intrusive micromanagement.

The core tension lies in the fact that conventional categorical labels and text-only modalities cannot represent the subtle, asymmetric, and multimodally grounded psychological states of human dyads. To resolve this limitation, this paper frames social reasoning through a formal psychological dimensional system grounded in video interactions. Core idea: Grounded in classical social psychology, construct a six-dimensional bidirectional evaluation benchmark (PIVOTS) spanning power, involvement, valence, objective, permanence, and stance on a 5-point bipolar Likert scale, coupled with keyframe identification and visual causal attribution to systematically probe MLLMs' fine-grained social reasoning across explicit and implicit interaction regimes.

Method

Overall Architecture

PIVOTS-Bench evaluates multimodal foundation models through a hierarchical three-task structure grounded in video interactions. Task 1 is the primary six-dimensional relational scoring task: given a video clip \(V\), transcribed utterance \(U\), and a specified directional dyadic query \(D_{AB}^\alpha\) (representing subject A's perception of subject B along dimension \(\alpha\)), the model predicts a discrete rating from \(-2\) to \(+2\). Task 2 evaluates keyframe identification, requiring the model to localize the specific video frame that offers the most decisive visual evidence for the assigned score. Task 3 performs visual cue causal analysis, prompting the model to identify the key visual elements (e.g., facial expressions, gestures, posture, or environment) that causally justify the dimensional rating.

The dataset construction pipeline deliberately balances explicit daily interactions from Social-IQ 2.0 with emotionally charged, implicit interactions curated from YouTube. Annotations follow a model-first, human-calibrated protocol where GPT-5 generates initial candidate ratings, followed by comprehensive verification and correction by expert human annotators.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Interaction Input<br/>Video clip V + Utterance transcript U + Directional dyad Aโ†’B"] --> B["PIVOTS Six-Dimensional Bidirectional Framework<br/>P / I / V / O / T / S 5-point bipolar Likert scale"]
    B --> C["Hierarchical Three-Task Evaluation Design<br/>Task 1: 6D Scoring | Task 2: Keyframe Localization | Task 3: Visual Causal Grounding"]
    C --> D["Dual-Source Complementary Data Pipeline<br/>Social-IQ 2.0 explicit casual scenes + YouTube implicit nuanced encounters"]
    D --> E["Model-First Human-Calibrated Annotation Protocol<br/>GPT-5 initial draft โ†’ Expert revision and de-biasing"]
    E --> F["Output: Fine-Grained Social Dynamics Benchmark<br/>765 dyadic score pairs + 483 auxiliary causal grounding questions"]

Key Designs

1. PIVOTS Six-Dimensional Bidirectional Framework: Replacing discrete labels with continuous psychological axes

To overcome the representational rigidity of categorical social roles and capture relational nuances, the benchmark operationalizes six orthogonal dimensions grounded in social psychology (Wish et al., 1976; Russell, 1980) using a 5-point bipolar Likert scale from \(-2\) to \(+2\): - P (Egalitarian vs. Hierarchical Power): Measures the distribution of agency, authority, and decision-making control (\(+2\) denotes total dominance over a compliant party, \(0\) indicates balanced peer equality, and \(-2\) reflects complete subjugation). - I (Superficial vs. Intense Involvement): Gauges emotional intimacy, psychological vulnerability, and mutual disclosure (\(+2\) signifies profound soulmate-level bonding, whereas \(-2\) corresponds to depersonalized transactional exchanges). - V (Positive vs. Negative Valence): Evaluates core affective quality (\(+2\) represents uplifting, psychologically safe interactions, whereas \(-2\) captures aggressive, hostile, or threatening confrontations). - O (Socioemotional vs. Task-Oriented Objective): Disentangles interactional goals (\(+2\) reflects interactions existing purely for relationship bonding, while \(-2\) marks strictly utilitarian, procedural, or rule-bound execution). - T (Temporary vs. Enduring Permanence): Quantifies the expected temporal durability of the bond (\(+2\) indicates lifelong family or marriage ties, while \(-2\) represents fleeting, one-time encounters). - S (Cooperative vs. Competitive Stance): Reflects goal alignment and resource sharing (\(+2\) denotes highly collaborative mutual dependence, whereas \(-2\) marks zero-sum, obstructive competition).

For any interacting dyad, predictions are formulated bidirectionally:

\[R_{AB}^\alpha = \arg\max_{r \in \{-2, -1, 0, 1, 2\}} p\left(r \mid V, U, D_{AB}^\alpha\right)\]

This directional formulation explicitly accommodates interpersonal asymmetries, providing a unified and standardized quantitative space for social dynamics.

2. Hierarchical Multimodal Evaluation: Probing macro judgments down to temporal frames and visual cues

To prevent models from relying on linguistic superficial shortcuts and verify true non-verbal comprehension, PIVOTS-Bench supplements dimension scoring with two grounded auxiliary tasks. In Task 2 (Key Frame Identification), models evaluate sampled frame candidates \(\{I_1, I_2, \dots, I_n\}\) to identify the single frame \(T_{AB}^\alpha = \arg\max_{t} p(t \mid V, D_{AB}^\alpha)\) containing decisive visual evidence for dimension \(\alpha\). In Task 3 (Visual Cue Causal Analysis), given the keyframe, models choose the two most salient visual factors \(C_{AB}^\alpha\) among five candidates (e.g., facial expressions, body posture, hand gestures, physical distance, attire, or environmental context). Partial credit (\(0.5\)) is awarded for one correct cue, and full credit (\(1.0\)) for both, directly evaluating the model's ability to chain low-level perceptual observations into high-level social inferences.

3. Dual-Source Data Pipeline and De-biased Annotation: Combining explicit and implicit social contexts

The benchmark curates videos from two complementary regimes: 121 30-second clips from Social-IQ 2.0 representing everyday casual conversations, debates, and group activities with explicit social signals (59.4% positive tone); and 70 YouTube clips targeting complex, in-depth interactions such as high-stakes interviews, reconciliations, and ideological disputes characterized by restrained affect and profound subtext (61.6% serious tone).

To ensure rigorous data quality across 765 directional score pairs, the benchmark implements a model-first, human-calibrated annotation workflow. Initial 6D scores generated by the GPT-5 API are thoroughly inspected and revised by researchers via a dedicated annotation tool. Annotators revised 25.3% of the initial scores, actively countering the model's tendency toward safe neutral answers (\(0\)). Inter-annotator agreement measured by ordinal Krippendorff's \(\alpha\) reached \(0.8125\) (\(83.87\%\) exact agreement), confirming high annotation consistency.

Loss & Training

As an evaluation benchmark, PIVOTS-Bench involves no parameter training. Inference evaluates models in zero-shot settings using standard OpenAI ChatCompletion prompts with 8 uniformly sampled frames. The authors investigate three prediction formulations and two human-inspired prompting paradigms: - Separate Prediction: Scores each character pair and dimension independently. - Joint Prediction: Prompts the model to output all six PIVOTS scores for a directional pair in a single vector. - Pairwise Prediction: Prompts the model to output both directional score vectors (\(A \to B\) and \(B \to A\), 12 scores total) simultaneously. - Multi-Stage Prompting: Simulates a four-stage psychological appraisal protocol: Phase 1 affective calibration (V and I), Phase 2 structural dimension analysis (P, O, T, S), Phase 3 blind independent scoring, and Phase 4 discrepancy reconciliation, constrained by an explicit instruction against neutral bias. - In-Context Prompting: Provides a micro-to-macro reasoning demonstration linking subtle behavioral cues to intermediate affective states and final scores.

Key Experimental Results

Main Results

Evaluations encompass leading proprietary models (GPT-5, Gemini-2.5-pro) and open-source models (Qwen3 series: 4B, 8B, 32B). Random chance baselines are 20.00% for Task 1 (5-class discrete scoring), 20.00% for Task 2 (5-choice frame selection), and 40.00% expected accuracy for Task 3 (choosing 2 out of 5 cues).

Performance on the Social-IQ 2.0 evaluation subset:

Model Category Model Task 1 Avg (%) Power P Involv. I Valence V Object. O Perman. T Stance S Task 2 Keyframe (%) Task 3 Causal (%)
Proprietary GPT-5 56.91 65.12 60.31 55.73 42.33 56.59 61.40 62.15 60.25
Proprietary Gemini-2.5-pro 47.71 61.30 50.77 33.75 51.63 46.98 41.86 46.63 48.00
Open-source Qwen3-32B-vl 34.54 62.62 23.82 33.82 23.77 30.28 31.50 35.29 33.50
Open-source Qwen3-8B-vl 27.90 59.74 22.22 22.87 18.58 26.89 17.07 34.75 31.00
Open-source Qwen3-4B-vl 24.33 51.05 20.77 23.99 13.73 25.93 10.47 28.59 30.17

Performance on the YouTube evaluation subset (Task 1 per-dimension accuracy):

| Model Category | Model | Task 1 Avg (%) | Power P | Involv. I | Valence V | Object. O | Perman. T | Stance S | |---|---|---|---|---|---|---|---| | Proprietary | GPT-5 | 58.88 | 73.71 | 50.44 | 48.28 | 61.14 | 60.68 | 58.85 | | Proprietary | Gemini-2.5-pro | 54.66 | 69.49 | 55.08 | 31.78 | 61.02 | 54.24 | 56.36 | | Open-source | Qwen3-32B-vl | 40.81 | 67.92 | 32.38 | 34.91 | 29.47 | 35.85 | 43.14 | | Open-source | Qwen3-8B-vl | 35.19 | 66.67 | 26.85 | 31.48 | 18.52 | 32.41 | 35.19 | | Open-source | Qwen3-4B-vl | 30.03 | 71.98 | 16.81 | 33.62 | 6.90 | 29.31 | 21.55 |

Ablation Study

Ablation studies utilizing Qwen3-32B-VL evaluate the influence of visual modalities, textual role identifiers (kinship/honorific terms redacted into generic tokens), and audio signals:

Input Configuration Social-IQ 2.0 Avg (%) YouTube Avg (%) Note
Video Only 33.70 31.18 Visual frames without transcripts
Original Conv. 31.44 37.81 Text-only with explicit role identifiers
Redacted Conv. 28.97 33.64 Text-only with explicit role tokens removed
Video + Original Conv. 34.54 40.81 Full multimodal baseline
Video + Redacted Conv. 35.83 32.22 Visual signals compensating for text redaction
Qwen3-Omni-30B (Audio included) 40.93 40.10 Audio-visual-text unified reasoning
Qwen3-Omni-30B (No audio) 32.61 (-8.32%) 38.30 (-1.80%) Removing acoustic speech signals

Comparison of prediction formats across the two interaction regimes:

Prediction Setting Social-IQ 2.0 Avg (%) YouTube Avg (%) Power P (YouTube) Note
Separate Prediction 34.54 40.81 67.92 Standard independent scoring
Joint Prediction 43.26 38.63 82.24 Predicting all 6 dimensions together
Pairwise Prediction 42.19 39.67 86.96 Predicting bidirectional dyadic scores

Key Findings

  • Substantial generation gap between proprietary and open-source models: GPT-5 achieves 56.91% and 58.88% across Social-IQ 2.0 and YouTube, outperforming Qwen3-32B-VL (34.54% and 40.81%) by nearly 20 percentage points. In Task 3 (visual cue causal analysis), Qwen3-32B-VL scores 33.50%, falling below random chance (40.00%), highlighting severe deficits in attributing social judgments to specific visual behaviors.
  • Valence (V) and Objective (O) represent pervasive failure modes: All evaluated models struggle disproportionately on Valence and Objective. Valence depends heavily on prosodic nuances and subtle micro-expressions, which remain poorly captured by current MLLMs. Objective judgments are frequently derailed when emotionally expressive arguments occur in formal work or task-oriented settings.
  • Context-dependent strategy reversal between explicit and implicit domains: On Social-IQ 2.0, joint prediction (+8.72%) and multi-stage prompting (+6.69%) deliver marked improvements by leveraging cross-dimensional shared representations. Conversely, on YouTube's implicit, in-depth interactions, both joint prediction (-2.18%) and multi-stage prompting (-9.33%) cause substantial performance drops, whereas independent dimension scoring proves superior. Rigid prompting heuristics appear to confuse models when analyzing nuanced social dynamics.
  • Disparate reliance on audio signals: Removing the audio modality drops Qwen3-Omni-30B's performance by 8.32% on Social-IQ 2.0, but only by 1.80% on YouTube, reflecting how everyday casual interactions rely heavily on conversational acoustics, whereas restrained YouTube dialogues depend on subtle global discourse.

Highlights & Insights

  • Operationalizing social psychology into an actionable MLLM benchmark: Rather than relying on superficial classification tags, the benchmark adapts the classic PIVOTS taxonomy to establish a standardized, continuous coordinate system for machine social cognition.
  • Dual constraints of bidirectional asymmetry and causal grounding: Requiring models to differentiate \(A \to B\) from \(B \to A\) perceptions while grounding decisions in temporal keyframes and specific visual cues effectively penalizes spurious text-matching shortcuts.
  • Empirical evidence for adaptive social reasoning: Highlights that a uniform reasoning pipeline cannot handle diverse social encounters; models require adaptive mechanisms that toggle between holistic joint modeling for casual interactions and decoupled independent analysis for high-stakes, implicit encounters.

Limitations & Future Work

  • Internal annotation by authors: All labels were established by the authors to ensure deep comprehension of the six-dimensional criteria; however, this creates potential shared inductive biases that lack validation from broader demographic cohorts.
  • Anchoring risks from model-assisted drafting: While 25.3% of GPT-5's pre-annotated ratings were modified, human raters may still be subject to subtle anchoring effects, potentially giving GPT-5 an advantage during evaluation.
  • Linguistic and cultural homogeneity: The benchmark is restricted to English-language interactions, and annotators share non-native English backgrounds, leaving cross-cultural non-verbal variations and localized social norms unexamined.
  • Absence of fine-tuning or alignment recipes: The work focuses purely on benchmarking without proposing targeted post-training or reinforcement learning methods to enhance multimodal social reasoning.
  • vs SCRIPTS / SocialEval / SocialMaze: Text-only benchmarks evaluate relationship reasoning without sensory grounding; PIVOTS-Bench establishes multimodal visual and acoustic evidence as indispensable causal anchors.
  • vs Social Genome / SIV-Bench: Prior video benchmarks focus on coarse action labels, atomic social interactions, or discrete social roles; PIVOTS-Bench pioneeringly targets fine-grained psychological dimensions and bidirectional perceptual asymmetries.

Rating

  • Novelty: โญโญโญโญโญ Grounding six-dimensional psychological axes and bidirectional causal attribution in multimodal foundation model evaluation is highly original.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive cross-model evaluation across proprietary and open-source models, paired with detailed modality and prompt ablations.
  • Writing Quality: โญโญโญโญโญ Rigorous conceptual framing, clear task formalization, and transparent discussion of empirical anomalies.
  • Value: โญโญโญโญโญ Provides a crucial stepping stone toward developing socially aware multimodal agents endowed with genuine Theory of Mind capabilities.