Skip to content

Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://www.uni-bamberg.de/en/nlproc/projects/item
Area: Social Computing
Keywords: Multimodal Emotion Analysis, Cognitive Appraisal Theory, Social Media, Emotion Expression and Perception, Mult2EMo Dataset

TL;DR

This paper introduces Mult2EMo, the first multimodal social media emotion dataset (7,200 posts) jointly capturing author experience, post expression, and reader perception, demonstrating that understanding the triggering event is decisive for reconstructing authors' emotions, while text remains the dominant modality over images.

Background & Motivation

Multimodal communication on social media has become ubiquitous, with users routinely weaving text and images together to convey nuanced emotional states. However, emotion analysis in natural language processing has long overlooked two fundamental dimensions: first, a pervasive modality imbalance, where research remains heavily text-centric and neglects how images modulate emotional expression; second, an isolated focus on surface post content that disconnects the expressed emotion from the real-world inciting event that originally elicited it. Human emotions are essentially private internal states, whereas social media posts represent authors' attempts to share these private states externally through text and visuals—a process frequently shaped by social desirability, selective omission, and contextual assumptions.

This separation introduces a central tension: readers and computational models attempting to reconstruct authors' emotions are doubly removed from the original experience. At the authoring stage, writers often omit granular details or presuppose shared context; at the reading stage, third-party observers must reconstruct complex internal states from sparse and ambiguous multimodal cues, leading to frequent divergence in interpretation. Prior work has been unable to disentangle whether perceptual inaccuracies stem purely from subjective reader variance or from an intrinsic disconnect between the inciting event and post representations. Furthermore, conventional taxonomies remain confined to coarse emotion categories or continuous dimensions, failing to account for cognitive appraisals—the subjective evaluations of events that psychological theory identifies as the driving mechanism of emotion elicitation.

To overcome these limitations, this paper investigates the complete emotion communication trajectory by decomposing it into three interconnected processes: experience, expression, and perception across text and vision. Core idea: construct a dual-perspective multimodal dataset, Mult2EMo, containing annotations from both original post authors and independent readers, and quantitatively analyze how inciting event alignment and text–image interplay dictate the feasibility and fidelity of emotion reconstruction.

Method

Overall Architecture

The study establishes Mult2EMo, comprising 7,200 multimodal posts spanning six basic emotions (anger, disgust, fear, joy, sadness, surprise), alongside a comprehensive reconstruction benchmark for human readers, fine-tuned multimodal models, and zero-shot multimodal LLMs. The end-to-end framework operates in three sequential phases: first, post authors recall a specific inciting event, annotating experienced emotions, 21 cognitive appraisal dimensions (following Scherer's Component Process Model), and post details; second, independent readers review the posts in a blind setup to infer and reconstruct the author's internal state and inciting event; finally, cross-encoder semantic similarity and multimodal classifiers evaluate reconstruction fidelity and quantify the relative contribution of textual versus visual signals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Inciting Real-World Event<br/>Original Contextual Stimulus"] --> B["Author Dual-Level Annotation<br/>Experience / 21 Appraisals / Multimodal Post"]
    B --> C["Reader Reverse Reconstruction<br/>Multimodal Reading / Author Intent Inference"]
    C --> D["Event Alignment & Modality Analysis<br/>Cross-Encoder Similarity / Text-Image Dynamics"]
    D --> E["Model Emotion & Appraisal Benchmark<br/>Fine-Tuned CLIP / Zero-Shot Qwen3-VL"]

Key Designs

1. Author Dual-Level Annotation: Bridging Private Experience and Public Expression To resolve the flaw of substituting author ground truth with reader annotations, the authors instituted a direct collection protocol with active social media users via Prolific. Authors were prompted to recall a specific event that elicited a target emotion and about which they authored a multimodal post. Annotation proceeded across six structured steps: (1) Event Details: Free-text description of the triggering event, experienced primary emotion, intensity (1–5 Likert scale), recall confidence, and duration; (2) Cognitive Appraisals: Evaluation of 21 appraisal dimensions (including pleasantness, goal support, own responsibility, external standards, suddenness) to model the dynamic cognitive checks shaping the emotional state; (3) Post Details: Primary and secondary emotions expressed in the post along with text-level emotion stimulus spans; (4) Image Details & Taxonomy: Categorization of image types (personal photo, professional photo, screenshot, meme, graphic) and functional motivations for image inclusion; (5) Text–Image Interplay: Mutual dependency ratings between text and image and their individual explicitness in conveying emotion; (6) Social Context: Purpose of posting, intended audience, and prerequisite cultural/situational context.

2. Reader Reverse Reconstruction: Disentangling Observer Bias from Information Loss To establish how accurately external observers interpret authors' internal states, a separate Reader Phase was conducted on a platform-balanced test subset of 1,440 posts (from Facebook, Instagram, Twitter/X). Each post was annotated by three independent readers (yielding 4,320 total reader annotations). Crucially, readers were instructed to act as third-party decoders: instead of reporting their own induced feelings, they reconstructed what they believed the author experienced during the event and intended to express in the post. Readers estimated the author's inciting event description, the original emotion experienced, the 21 appraisal ratings, and the post's expressed emotions. By evaluating individual reader accuracy (R), aggregated majority voting (V), and inter-reader agreement (RR), the framework uniquely quantifies the gap between consensus reader interpretation and actual authorial intent.

3. Event Alignment & Modality Analysis: Quantifying Context Loss and Visual Asymmetry Addressing the bottleneck of implicit context and detail omission during posting, this design systematically traces where information breaks down. First, an STS-trained cross-encoder measures semantic similarity between reader-inferred event descriptions and author ground-truth descriptions, partitioning the test set into quartiles (Q1 to Q4) to expose the causal relationship between event comprehension and emotion recovery; second, the BLIP model evaluates text–image alignment and image-to-event relevance via visual question answering; third, the protocol isolates posts exhibiting emotion mismatches—where the author's experienced primary emotion diverges from the post's expressed primary emotion. This design illuminates why negative emotions are disproportionately pruned during social media expression and pinpoints textual context as the indispensable anchor for disambiguating emotional signals.

Loss & Training

The computational benchmark evaluates fine-tuned CLIP models (clip-vit-base-patch32) and zero-shot Qwen3-VL-8B-Instruct. CLIP models were fine-tuned across three modality configurations: text-only (T), image-only (I), and feature concatenation (T+I). Emotion classification was trained using standard categorical cross-entropy loss: $\(\mathcal{L}_{\text{emo}} = - \sum_{c=1}^{C} y_c \log \hat{y}_c\)$ Appraisal prediction employed a multi-head regression architecture featuring 21 distinct heads on top of the multimodal representation, optimized via mean squared error (MSE): $\(\mathcal{L}_{\text{appr}} = \frac{1}{21} \sum_{k=1}^{21} (r_k - \hat{r}_k)^2\)$ Supervised models were trained using 4 NVIDIA L40 GPUs with learning rate \(1\times 10^{-5}\), batch size 8, 5 epochs, and early stopping. A control baseline with randomly shuffled images (CLIP Rand.) was evaluated under identical settings to verify whether visual features provide genuine informational utility.

Key Experimental Results

Main Results

The primary benchmark compares emotion classification Macro F1 scores across individual readers (R), majority voting over readers (V), inter-reader consistency (RR), fine-tuned CLIP, and zero-shot Qwen3-VL on the 1,440-post Mult2EMo test set.

Model / Evaluator Anger Disgust Fear Joy Sadness Surprise Macro Avg.
Reader (RR Inter-Reader) .52 .44 .52 .82 .67 .44 .57
Reader (R Individual) .44 .38 .44 .58 .60 .35 .47
Reader (V Majority Vote) .47 .40 .45 .58 .63 .34 .48
CLIP (Text) .43 .39 .53 .58 .60 .48 .50
CLIP (Image) .27 .23 .32 .41 .31 .25 .30
CLIP (Text + Image) .45 .36 .56 .58 .60 .50 .51
CLIP Rand. (T + I) .46 .38 .53 .58 .58 .49 .50
Qwen3-VL (Text) .54 .29 .58 .54 .57 .30 .47
Qwen3-VL (Image) .36 .20 .33 .42 .33 .14 .30
Qwen3-VL (T + I) .52 .35 .59 .56 .61 .32 .49

Ablation Study

The ablation investigates how reader reconstruction performance scales with event comprehension by stratifying the test set into quartiles (Q1 lowest, Q4 highest) based on cross-encoder similarity between author and reader event descriptions. Metrics report emotion classification Macro F1 and root mean square error (RMSE, lower is better) on key appraisal dimensions.

Event Similarity Quartile Mean Sim. Emotion Macro F1 Pleasantness RMSE Unpleasantness RMSE Own Resp. RMSE Goal Support RMSE Overall Appr. RMSE
Q1 (Lowest Alignment) .02 .34 1.72 1.82 1.70 1.68 1.81
Q2 (Low-Mid Alignment) .21 .44 1.38 1.53 1.53 1.51 1.70
Q3 (Mid-High Alignment) .39 .51 1.14 1.35 1.44 1.46 1.65
Q4 (Highest Alignment) .59 .61 0.99 1.18 1.29 1.35 1.60
Q4 vs Q1 Delta +0.57 +0.27 (+79.4%) -0.73 -0.64 -0.41 -0.33 -0.21

Key Findings

  • Event comprehension is the primary driver of emotion reconstruction: Moving from Q1 to Q4 event similarity boosts emotion classification F1 from .34 to .61 (a 79.4% relative gain), with disgust F1 more than doubling from .23 to .51. Similarly, pleasantness RMSE drops dramatically from 1.72 to 0.99, showing that contextual information loss during writing is the chief barrier to emotion recovery.
  • Limited visual utility and multimodal fusion collapse: Multimodal CLIP with true images achieves an F1 of .51, barely outperforming text-only (.50) and matching the random-image control (.50). With image-only models reaching only .30 F1, the results show that naive multimodal fusion fails to extract synergistic visual emotion features beyond what text provides.
  • Experienced vs. expressed emotion mismatches degrade performance: In 10% of posts where primary experienced and expressed emotions diverge, reader classification F1 drops by .25 and model F1 declines by .21. Joy exhibits the steepest decline (-.46 for readers, -.45 for models), underscoring the confounding effect of social desirability filtering.
  • Readers converge on shared misinterpretations: Inter-reader event description similarity (.41) substantially exceeds author–reader similarity (.30), matching the inter-reader agreement gap in emotion classification (RR .57 vs. R .47). Readers reliably agree with one another on misinterpretations induced by ambiguous post text.

Highlights & Insights

  • Full-lifecycle formulation of emotion communication: Transcends conventional surface-level emotion classification by articulating a triadic paradigm—experience (inciting event), expression (multimodal post), and perception (third-party decoding)—establishing an empirical benchmark for social computing.
  • Empirical grounding of cognitive appraisal theory: Successfully operationalizes 21 appraisal dimensions on multimodal social data, revealing that computational regression heads outperform individual human readers on objective appraisals (overall RMSE 1.38 vs. 1.70).
  • Rigorous multimodal sanity checks: Incorporating random-image controls and cross-encoder event stratification provides an exemplary diagnostic protocol that exposes modality collapse in multimodal affective computing.

Limitations & Future Work

  • Author-admitted limitations: The participant pool is restricted to native English speakers in the UK and Ireland, limiting cross-cultural generalization; recruitment challenges for negative emotions (notably disgust) introduced minor timeline differences; baseline architectures relied on basic concatenation rather than advanced dynamic fusion.
  • Observed limitations: Appraisals were gathered retrospectively from authors, which may introduce recall bias; third-party annotators were drawn from the general population rather than the authors' immediate social circles, potentially exaggerating contextual misunderstandings.
  • Concrete future directions: Developing asymmetrical multimodal fusion architectures with adaptive modality gating; integrating personalized user priors and social circle context to infer latent inciting events.
  • vs GoEmotions / SemEval Emotion Corpora: Traditional datasets (Demszky et al., 2020; Strapparava & Mihalcea, 2007) rely exclusively on third-party crowd annotations on text, conflating reader perception with authorial intent. Mult2EMo demonstrates that reader labels exhibit systematic divergence from author ground truth, requiring event grounding for rectification.
  • vs Multimodal Meme and Sentiment Benchmarks: Prior multimodal affective studies predominantly examine internet memes (Sharma et al., 2020) or coarse sentiment polarity. This paper explores the broad natural distribution of personal social media posts across 5 image categories, exposing severe modality imbalance in everyday communication.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering triadic formulation of emotion experience, expression, and perception with dual-perspective annotations.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation of 7,200 posts, bootstrap significance testing, random-image controls, event quartile ablations, and MLLM comparison.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative structure with tight alignment between theoretical grounding and experimental validation.
  • Value: ⭐⭐⭐⭐⭐ Sets a new standard for event-centric, cognitively grounded multimodal emotion analysis in social computing.