Skip to content

MMDS-Bench: Benchmarking Multimodal Large Language Models on Dynamic Stance in Social Media Interactions

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/whu-yzding/MMDS-Bench
Area: Multimodal VLM
Keywords: Multimodal Dynamic Stance Detection, Social Media Interactions, Diagnostic Benchmark, Multimodal Large Language Models, Relational Reasoning Bottleneck

TL;DR

MMDS-Bench constructs the first diagnostic benchmark for multimodal dynamic stance classification across 3,482 social media parentโ€“reply interactions with a 7-label taxonomy and an 800-instance structured reasoning subset, revealing across 12 prominent MLLMs that while models readily comprehend individual parent and reply messages, their fundamental bottleneck resides in inferring the local dynamic stance relation connecting them.

Background & Motivation

Stance detection plays a pivotal role in public opinion mining, rumor debunking, and argument analysis across social platforms. For years, the prevailing research formulation has treated stance detection as a static or target-oriented task, predicting whether an isolated post supports, opposes, or remains neutral toward a predefined global topic or claim such as climate change or vaccination mandates. However, real-world online discourse is intrinsically interactive and conversational: users directly address preceding comments by agreeing, disputing, contributing supplementary evidence, seeking clarification, or deliberately shifting the topic. This pragmatic reality has given rise to dynamic stance classification, which shifts the analytical focus away from detached global claims to how a reply message locally responds to its immediate parent message.

While dynamic stance detection has seen initial progress in text-only environments, everyday social interactions on platforms like X (Twitter) have become predominantly multimodal. Both original parent posts and direct responses routinely integrate screenshots, memes, reaction images, and embedded text graphics. Under these interactional settings, communicative intent is frequently encoded through cross-modal interplay, visual metaphors, or sarcasmโ€”for instance, a standalone reaction meme conveys no explicit stance on its own, but constitutes sharp mockery or dissent when juxtaposed against the specific premise of a parent post; similarly, a captured statistical chart can serve as visual evidence that elaborates on a preceding claim. Consequently, the core computational challenge transcends conventional visual question answering, OCR, or descriptive captioning, demanding that models jointly interpret textual and visual cues to decode how the reply dynamically engages with the parent.

Existing multimodal stance benchmarks nonetheless remain largely target-centric, examining posts in isolation against external topics rather than inspecting pairwise conversational interactions. Concurrently, while state-of-the-art multimodal large language models (MLLMs) demonstrate impressive perceptual and visual reasoning competencies, no systematic diagnostic framework has examined whether they can reliably infer fine-grained conversational stance dynamics from multimodal interactions. Core idea: construct MMDS-Bench, the first diagnostic benchmark comprising 3,482 multimodal parentโ€“reply pairs with a seven-label dynamic stance taxonomy, an 800-instance structured reasoning diagnostic subset spanning parent understanding, reply understanding, and stance-relation inference, five annotated interaction challenge factors, and a reference-grounded LLM-judge protocol to systematically pinpoint the relational reasoning bottlenecks of modern MLLMs.

Method

Overall Architecture

MMDS-Bench focuses on pairwise parentโ€“reply interactions on social media platforms, where both the parent message \(P\) and direct reply \(R\) incorporate visual content (encompassing text-and-image or image-only variations across four modality compositions). The benchmark evaluates how accurately MLLMs determine the dynamic stance relation of the reply with respect to the parent. The evaluation framework comprises two sequential tiers: Task 1 (Multimodal Dynamic Stance Classification) tests final 7-way stance label predictions across the full benchmark of 3,482 instances; Task 2 (Diagnostic Multimodal Dynamic Stance Reasoning) evaluates structured reasoning trajectories across an 800-instance difficulty-balanced subset, requiring models to explicitly generate parent understanding, reply understanding, and stance-relation reasoning prior to the final prediction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Parent-Reply Data Collection & Filtering<br/>X Platform Interaction Pairs (3,482 instances)"] --> B["Seven-Label Dynamic Stance Taxonomy<br/>Agree / Disagree / Elaborates / Query / Neutral / Unrelated / NA"]
    B --> C["Five Interaction Challenge Factors Annotation<br/>Multimodal Fusion / Parent Framing / Non-Literal / Reasoning / Ambiguity"]
    C --> D["Dual Evaluation Framework & Task Design<br/>Task 1 Full Benchmark (3,482) + Task 2 Diagnostic Subset (800)"]
    D --> E["Reference-Grounded LLM-Judge Diagnostic Protocol<br/>PUS / RUS / SRS Independent Scoring & Bottleneck Metric"]
    E --> F["Benchmark Output & Multi-Dimensional Diagnosis<br/>Evaluation of 12 MLLMs & Identification of Reasoning Bottleneck"]

Key Designs

1. Seven-Label Dynamic Stance Taxonomy: Shifting from global topics to local conversational pragmatics Standard stance schemes relying on generic positive/negative polarities fail to capture nuanced communicative intents within threaded social dialogue. MMDS-Bench adapts the dynamic stance taxonomy to multimodal contexts by formalizing seven mutually exclusive pragmatic classes: Agree (explicitly endorsing the parent's stance or assertion), Disagree (directly refuting the parent's claim, including sarcasm, mockery, and critical rhetorical questions), Elaborates (extending discussion with background, examples, or contextual perspective without explicit endorsement or rejection), Query (sincerely requesting evidence, elaboration, or clarification without taking a stance), Neutral (staying on topic while expressing no stance and adding no substantive argument), Unrelated (deviating entirely from the parent's subject), and NA (ambiguous instances admitting multiple equally plausible interpretations or suffering from unresolvable informational deficits). Annotations strictly anchor the parent as the contextual baseline, determining the reply's primary communicative intent when multiple rhetorical signals coexist.

2. Five Interaction Challenge Factors Annotation: Deconstructing multimodal and interactional complexity To move beyond aggregate classification accuracy and identify specific structural failure modes, MMDS-Bench annotates five independent binary challenge factors for each instance: - Multimodal Fusion (MF): The image itself is inherently intricate or the textโ€“image relationship is ambiguous, rendering overall meaning recovery challenging; - Parent Framing (PF): The parent post lacks an explicit proposition, argumentative target, or clear emotive tone, leaving the reply's intended angle of response ambiguous; - Non-Literal Reply (NLR): Stance is conveyed predominantly through irony, sarcasm, memes, hyperbole, or analogical metaphors rather than literal assertions; - Interaction Reasoning (IR): Mapping the reply to the parent requires multi-step deductive chaining or external contextual knowledge; - Label-Boundary Ambiguity (LBA): The communicative intent straddles the borderline between neighboring stance categories (e.g., Agree vs. Elaborates). By summing the active challenge factors per instance (ranging from 0 to 5), the benchmark provides a continuous indicator of task hardness to structure error analysis.

3. Dual Evaluation Framework & Task Design: Disentangling message comprehension from relational inference To diagnose whether model failure stems from misreading individual inputs or failing to map their conversational relationship, the benchmark establishes two complementary tasks. Task 1 covers all 3,482 instances under a constrained JSON schema returning solely pred_stance, testing overall classification robustness under natural class imbalance. Task 2 deploys an 800-instance diagnostic subset uniformly sampled across challenge-count strata, compelling models to output concise textual summaries (\(\le 50\) words each) for parent_understanding, reply_understanding, and stance_reasoning before emitting pred_stance. This structured trajectory exposes the intermediate cognitive steps of evaluated MLLMs.

4. Reference-Grounded LLM-Judge Diagnostic Protocol: Quantifying multi-stage cognitive capabilities and reasoning bottlenecks Evaluating the open-ended reasoning generated in Task 2 presents significant challenges. The authors introduce a multi-judge reference-grounded LLM-as-a-judge framework utilizing Gemini 2.5 Flash, Qwen3-VL-32B, and Gemma-4-31B. Given the original multimodal interaction pair, gold dynamic stance label, human-authored reference rationale, and model generation, each judge independently assigns three 1โ€“5 integer scores: Parent Understanding Score (PUS), Reply Understanding Score (RUS), and Stance Reasoning Score (SRS). In addition to the aggregate Process Score \(PS = \frac{PUS + RUS + SRS}{3}\), the protocol calculates the Bottleneck Score: $\(BS = \min(PUS, RUS, SRS)\)$ The scoring rubric enforces penalization for unsupported visual hallucinations, drift into global topic commentary, or failure to explicitly bridge the reply's communicative intent back to the parent message.

Experimental Results

Main Benchmark Results

Evaluation covers 12 state-of-the-art MLLMs spanning proprietary APIs, massive open-source reasoning/instruct architectures, and efficient lightweight models. Task 1 is conducted across the full 3,482 instances, and Task 2 reports averaged metrics across three LLM judges on the 800 diagnostic instances.

Model Task 1 Acc (%) Task 1 Macro-F1 (%) Task 2 Acc (%) Task 2 Macro-F1 (%) PUS (1-5) RUS (1-5) SRS (1-5) BS (1-5)
Gemini 2.5 Pro 79.93 42.22 72.25 48.38 4.92 4.89 4.33 4.32
Claude Sonnet 4.6 72.46 37.29 70.62 51.99 4.92 4.87 4.32 4.30
GPT-5.1 71.63 41.54 64.62 41.50 4.89 4.80 4.04 4.02
Kimi-K2.5 65.88 37.85 61.00 46.20 4.89 4.81 4.04 4.01
Qwen3-VL-235B-Thinking 62.95 33.46 53.62 33.51 4.75 4.57 3.62 3.58
Qwen3-VL-235B-Instruct 53.39 28.81 50.38 31.59 4.74 4.58 3.49 3.44
GLM-4.6V 52.30 25.93 45.50 24.46 4.64 4.53 3.30 3.24
Llama-4-Maverick-17B 46.41 25.36 48.38 27.39 4.69 4.50 3.50 3.44
Gemma-3-12B-IT 45.55 23.22 45.50 30.42 4.35 4.25 3.13 2.99
Qwen3-VL-8B-Thinking 55.23 27.19 45.75 20.55 4.66 4.43 3.31 3.27
Qwen3-VL-8B-Instruct 50.40 25.51 38.38 16.23 4.55 4.26 2.95 2.87
Ministral3-8B-2512 43.34 19.21 35.12 12.08 4.42 4.12 2.80 2.71

Diagnostic Reasoning and Challenge Stratification Analysis

The table below displays the breakdown of reasoning quality scores across prediction outcome types in Task 2 (aggregated across all model-instance predictions), alongside overall accuracy and stance reasoning score by active challenge factor count.

Evaluation Dimension / Slices Cases Count PUS (1-5) RUS (1-5) SRS (1-5) Performance Metric (Acc / Trend)
Correct Predictions (Correct) 5,049 4.87 4.83 4.81 High accuracy backed by sound reasoning
Incorrect Predictions (Incorrect) 4,551 4.51 4.24 2.20 High single-side scores, sharp collapse in SRS
Correct but weak SRS (\(\le 3\)) 192 3.91 3.33 2.49 Accounts for only 2.00% (rare guessing)
Incorrect but high SRS (\(\ge 4\)) 362 4.96 4.96 4.49 Reflects label-boundary ambiguity
Challenge Count = 0 144 - - 4.29 Task 1 Acc: 76.35% / Task 2 Acc: 77.31%
Challenge Count = 1 153 - - 4.01 Task 1 Acc: 65.40% / Task 2 Acc: 69.88%
Challenge Count = 2 148 - - 3.46 Task 1 Acc: 59.92% / Task 2 Acc: 49.38%
Challenge Count = 3 149 - - 3.36 Task 1 Acc: 51.31% / Task 2 Acc: 42.67%
Challenge Count = 4 148 - - 3.01 Task 1 Acc: 38.95% / Task 2 Acc: 33.78%
Challenge Count = 5 (All Active) 58 - - 2.89 Task 1 Acc: 26.44% / Task 2 Acc: 27.30%

Key Findings

  • Stance-Relation Reasoning is the Primary Bottleneck: While evaluated MLLMs demonstrate strong message comprehension (averaging 4.3โ€“4.9 on PUS and RUS), their stance reasoning ability (SRS) lags substantially behind. In the 4,551 incorrect prediction instances, PUS and RUS remain remarkably high at 4.51 and 4.24, whereas SRS collapses to 2.20. Models reliably recognize visual entities and transcribe text, but struggle to deduce how communicative intent bridges the two posts.
  • Reasoning-Oriented Models Consistently Outperform Instruct Variants: Test-time thinking variants show decisive gains over standard instruction-tuned checkpoints. Qwen3-VL-235B-Thinking outscores Qwen3-VL-235B-Instruct by 9.56% on Task 1 and 3.24% on Task 2 accuracy; similarly, Qwen3-VL-8B-Thinking achieves a 7.37% accuracy advantage over Qwen3-VL-8B-Instruct on Task 2, showing that explicit inference chains help resolve pragmatic ambiguities.
  • Compounding Challenge Factors Drive Severe Performance Degradation: Accuracy monotonically drops from 76.35% for instances with zero challenge factors down to 26.44% when all five factors co-occur. In error gap analysis, Interaction Reasoning (IR, +14.10%) and Label-Boundary Ambiguity (LBA, +13.65%) exhibit the highest error concentrations.
  • Systematic Bias Toward the Disagree Category: Confusion matrix analysis indicates that while Disagree is recognized with the highest within-class accuracy (79.3%), models exhibit a pronounced negative bias. For ground-truth Agree, Elaborates, and NA cases, 39.6%, 37.5%, and 58.7% of predictions are mistakenly misclassified as Disagree.
  • Explicit OCR Augmentation Yields Marginal Benefit: Providing EasyOCR transcriptions of images to efficient models (Gemma-3-12B-IT, Qwen3-VL-8B-Instruct) produces mixed resultsโ€”Gemma-3-12B-IT drops in accuracy from 45.55% to 44.51%, while Qwen3-VL-8B-Instruct slightly improves accuracy (50.40% to 52.01%) but loses Macro-F1. Social media memes depend on cultural and visual pragmatics rather than explicit text parsing; OCR noise can interfere with holistic visual processing.

Highlights & Insights

  • Standardized Multi-Stage Diagnostic Protocol: Structuring the diagnostic inference into parent_understanding \(\to\) reply_understanding \(\to\) stance_reasoning effectively opens the black-box generation process, pinpointing exactly where relational deduction breaks down.
  • Robust Reference-Grounded Judging Framework: LLM judges show high inter-judge agreement on stance reasoning (\(r_s = 0.92\)) and strong correlation with human experts (\(r_s = 0.93\)), demonstrating that grounding LLM evaluation in gold labels and reference rationales offers a scalable, reproducible assessment methodology.
  • Shift to Interaction-Centric Pragmatic Understanding: Moving beyond topic-oriented benchmarks toward threaded conversational stance detection aligns benchmarking with authentic multimodal social communication dynamics.

Limitations & Future Work

  • Platform and Modality Constraints: Data collection is restricted to pairwise interactions on X where both parent and reply contain images. The benchmark does not currently cover short video clips, audio snippets, multi-turn dialogue trees, or alternative online platforms (e.g., Reddit, TikTok).
  • Absence of Task-Specific Supervised Baselines: As a diagnostic benchmark, MMDS-Bench does not supply an extensive fine-tuning training corpus, leaving open whether supervised training on specialized interaction data can bridge the relational reasoning gap.
  • Potential Judge Biases on Subtle Pragmatics: While human validation demonstrates high aggregate alignment, automatic LLM judges may still exhibit blind spots when encountering niche subculture memes, ephemeral slang, or culturally localized satire.
  • vs. Target-Oriented Stance Detection (e.g., SemEval-2016, JointCL, T-MAD): Target-oriented methods evaluate standalone posts against global propositions, ignoring how conversational replies dynamically adapt meaning based on immediate context. MMDS-Bench formalizes stance as a relational edge between two communicative acts.
  • vs. Multimodal Conversational Stance Datasets (e.g., MmMtCSD, MT2-CSD): Although prior conversational datasets introduce multi-turn threads, stances remain anchored to global claim entities. MMDS-Bench classifies the local pragmatic intent between consecutive conversational turns.
  • vs. General Multimodal Benchmarks (e.g., MMBench, MMMU): Standard vision-language benchmarks emphasize perceptual grounding, factual knowledge retrieval, and scientific diagram reasoning. MMDS-Bench uniquely probes high-level social cognition, pragmatic inference, and cross-modal irony.

Rating

  • Novelty: โญโญโญโญโญ (Pioneering benchmark formalizing multimodal dynamic stance in social interactions with comprehensive factor annotations)
  • Experimental Thoroughness: โญโญโญโญโญ (Evaluates 12 diverse MLLMs, featuring thorough challenge factor stratifications, modality ablations, and extensive human validation)
  • Writing Quality: โญโญโญโญโญ (Exemplary clarity in problem formulation, rigorous empirical presentation, and insightful analytical conclusions)
  • Practical Value: โญโญโญโญโญ (Highlights critical blind spots of frontier MLLMs in social interaction reasoning, providing foundational methodologies for conversational AI development)