Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

Conference: ECCV 2026
Paper: ECCV Official
Code: https://information-fusion-lab-umass.github.io/video2reaction-bench.github.io/
Area: Time Series
Keywords: Induced Emotion Recognition, Label Distribution Learning, Video Understanding, Multimodal VLM, Audience Reaction Forecasting

TL;DR

Addressing the challenge of predicting diverse and non-stationary viewer emotional responses to video content in the wild, this paper presents Video2Reaction—the first large-scale benchmark comprising 10,348 movie clips (389 hours) paired with 21-class fine-grained audience reaction distributions derived from social media comments—supported by a scalable two-stage multi-agent LLM annotation pipeline and comprehensive evaluations showing that finetuned vision-language models substantially outperform classical distribution baselines.

Background & Motivation

In affective computing and multimedia understanding, a fundamental distinction exists between perceived emotion—the emotion depicted or conveyed by the video characters and content creators—and induced emotion—the actual affective response experienced by the viewing audience. The vast majority of existing video emotion datasets (such as IEMOCAP, MELD, and CMU-MOSEI) center almost exclusively on perceived emotions. The few benchmarks addressing induced emotion (such as LIRIS-ACCEDE and DEAP) are typically constrained to low-dimensional continuous valence-arousal spaces and rely on small cohorts of participants in controlled laboratory settings, failing to capture the rich ecological diversity of how general audiences react in the wild.

In real-world media consumption, audience responses to identical cinematic sequences are inherently subjective and multimodal. A suspenseful horror scene crafted with terrifying intent may indeed provoke genuine fear and distress in some viewers; however, others may find amusement in predictable genre tropes, while individuals with past personal trauma may experience deep distress and anxiety. Collapsing such heterogeneous, multifaceted reactions into a single majority-vote label or deterministic multi-label assignment completely discards the natural variance of audience engagement. Furthermore, collective audience sentiment is non-stationary: viewer interpretations shift dynamically across years due to evolving cultural contexts and contemporary references, rendering traditional static datasets built through laborious manual annotations obsolete and prohibitively costly to update.

To overcome these barriers, this work formalizes audience reaction forecasting as a label distribution learning (LDL) problem, leveraging naturally occurring viewer commentary from social media to map multimodal video inputs directly to continuous probability distributions over categorical reactions. Core idea: construct Video2Reaction, the first large-scale in-the-wild benchmark for video-to-reaction-distribution prediction (spanning 10,348 movie clips and 21 fine-grained emotions), powered by a two-stage multi-agent LLM annotation pipeline with blind human verification, and establish a dual-axis benchmark evaluating both full distribution alignment and dominant reaction forecasting.

Method

Overall Architecture

Video2Reaction formulates the predictive task as mapping multimodal input video content to an audience reaction probability distribution across 21 fine-grained emotion categories. The end-to-end framework consists of three principal phases: large-scale cinematic data curation from licensed content, scalable two-stage multi-agent comment annotation with majority voting, and predictive modeling via label distribution learning across classical algorithms and foundation vision-language models. The predictive pipeline takes video visual frames, audio tracks, and textual descriptions as input and produces a continuous, normalized probability distribution vector over the emotion taxonomy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data<br/>Movie Clips + Social Media Comments"] --> B["Two-Stage Multi-Agent Comment Annotation Pipeline<br/>Stage 1 Rephrase & Filter + Stage 2 Majority Voting"]
    B --> C["21-Class Fine-Grained Reaction Taxonomy & LDL Formulation<br/>Aggregation to Normalized Probability Distributions"]
    C --> D["Dual-Axis Full Distribution & Dominant Reaction Benchmark<br/>Distributional Divergence + Dominant Classification"]
    D --> E["Foundation Vision-Language Model Adaptation via LoRA<br/>Parameter-Efficient Tuning + Label Perturbation"]
    E --> F["Output Predictions<br/>Audience Reaction Distribution & Dominant Reactions"]

Key Designs

1. Two-Stage Multi-Agent Comment Annotation Pipeline: decoupling implicit context and filtering noise via heterogeneous LLM majority voting
Extracting genuine induced emotions from wild user comments is severely complicated by sarcasm, implicit sentiment, and off-topic discussion. A comment such as "I'm so disappointed this actor didn't win an Oscar" would be naively misclassified as disappointment by a standard classifier, even though it expresses deep admiration for the actor's performance in that scene. To address this, the pipeline decouples annotation into two sequential stages: Stage 1 conditions on the clip description to rephrase comments into explicit reaction statements while simultaneously filtering out completely off-topic remarks (achieving 0.83 accuracy and 1.00 sensitivity, guaranteeing no relevant comments are dropped); Stage 2 deploys an ensemble of three medium-sized open-source LLMs (LLaMA-3.1-8B-Instruct, Qwen2.5-14B-Instruct, and DeepSeek-R1-Distill-Qwen-7B) to extract categorical labels via majority voting (\(\ge 2\) agreeing agents). In dual-blind human verification over 1,000 randomly sampled comments, this pipeline achieves 86% correctness, with an inter-rater Spearman correlation of 0.402 that closely matches the natural agreement among human annotators (0.428), providing an extensible and highly cost-effective annotation engine.

2. 21-Class Fine-Grained Reaction Taxonomy & LDL Formulation: transforming discrete subjective feedback into continuous probability distributions
Unlike traditional classification tasks where soft targets represent label ambiguity around a single ground truth, the label distribution here reflects the empirical proportion of a viewing population experiencing each emotional response. Starting from the 28-category GoEmotions taxonomy, the authors drop 7 categories whose average probability mass is below 0.01%, establishing a compact 21-category taxonomy (7 positive, 10 negative, and 4 ambiguous emotions). Given an input video clip \(x\), audience reactions are formalized as a discrete distribution \(d_x = \{d_{xm}\}_{m=1}^M\) (\(M=21\)), where each component \(d_{xm} \in [0, 1]\) represents the fraction of retained comments assigned to category \(m\), satisfying the probability simplex constraint: $\(\sum_{m=1}^{M} d_{xm} = 1\)$ This distribution corresponds to the empirical conditional probability \(p(m|x)\). Statistical analysis reveals that the median Chebyshev distance between different clips from the same movie reaches 0.48, and dominant reaction probabilities vary widely with a median of 0.4, confirming that audience reactions vary dramatically across scenes and demand clip-level distribution modeling.

3. Dual-Axis Full Distribution & Dominant Reaction Benchmark: balancing global geometric fidelity with dominant response identification
Predicting audience response requires both recovering the complete shape of collective sentiment and precisely identifying the primary emotions that dictate user engagement, recommendation success, or content moderation. The benchmark establishes two complementary evaluation axes: the Full Distribution axis includes Chebyshev distance (measuring worst-case per-class error), Clark distance, Kullback-Leibler (KL) divergence, Cumulative Absolute Distance (CAD, which maps emotions to the valence-arousal space to penalize semantically distant classification errors), Cosine similarity, and Intersection; the Dominant Reaction axis measures Top-1 Probability Error (TPE), Mean Reciprocal Rank (MRR), and class-weighted Top-\(k\) F1 scores (\(k \in \{1, 2, 3\}\)). This dual-axis protocol prevents evaluation bias caused by severe label imbalance (imbalance factor \(\epsilon = 28.36\)).

4. Foundation Vision-Language Model Adaptation via LoRA: overcoming zero-shot failure through parameter-efficient tuning
Evaluating whether general-purpose multimodal pretraining allows models to infer audience reactions, the benchmark assesses Gemini 2.5 Flash, LLaVA-Next-Video-7B, and Qwen2.5-VL in zero-shot settings. Strikingly, all pretrained foundation VLMs completely fail zero-shot prediction (Top-1 F1 scores remain below 0.30 and Cosine similarity below 0.51). To bridge this gap, low-rank adaptation (LoRA, rank \(r=8\), \(\alpha=16\)) is applied to open-source VLMs for 3 epochs, incorporating label taxonomy random reordering and synonym replacement during training to guard against label-order memorization. Lightweight finetuning dramatically unlocks the forecasting capacity of VLMs, yielding state-of-the-art results across both distributional similarity and dominant emotion classification while exhibiting positive zero-shot cross-dataset transfer on the external VCE benchmark.

Loss & Training

For specialized label distribution learning algorithms such as SA-BFGS, optimization directly minimizes the Kullback-Leibler divergence between the empirical distribution \(d_x\) and the predicted distribution \(\hat{d}_x\) using the quasi-Newton BFGS method: $\(\mathcal{L}_{\text{KL}}(d_x, \hat{d}_x) = \sum_{m=1}^{M} d_{xm} \ln \left( \frac{d_{xm}}{\hat{d}_{xm}} \right)\)$ For foundation vision-language models, the task is framed as conditional generation over prompt-formatted inputs comprising visual keyframes, audio cues, and textual clip descriptions. Models are trained with cross-entropy loss under causal language modeling objectives. During post-processing, temperature scaling is applied on the validation split to optimize probability calibration across all 21 categories.

Key Experimental Results

Main Results

The benchmark compares four distinct model families on Video2Reaction: Problem Transformation (PT), Specialized Algorithms (SA), Algorithm Adaptation (AA), and Foundation VLMs. Evaluation metrics span both full distribution alignment and dominant reaction classification:

Model Family	Model Name	Chebyshev (Cheb) ↓	KL Divergence (KL) ↓	Intersection (Inter) ↑	Cosine (Cos) ↑	Dominant MRR ↑	Dominant Top-1 F1 ↑	Dominant Top-3 F1 ↑
Foundation VLM	Gemini 2.5 Flash (Zero-shot)	0.3425	4.8102	0.3787	0.5029	0.4378	0.2735	0.3794
	LLaVA-Next-Video-7B (Zero-shot)	0.4110	1.5547	0.2970	0.4185	0.1992	0.0143	0.1374
	LLaVA-Next-Video-7B (LoRA Finetuned)	0.1882	3.1765	0.6861	0.8663	0.7833	0.6521	0.7672
	Qwen2.5-VL (Zero-shot)	0.3985	1.5216	0.3140	0.4401	0.3088	0.1958	0.3037
	Qwen2.5-VL (LoRA Finetuned)	0.2047	3.4431	0.6656	0.8427	0.7548	0.6577	0.7725
Problem Transformation (PT)	PT-Bayes	0.9668	21.1994	0.0144	0.0272	0.1535	0.0001	0.0741
	LDSVR	0.2584	4.9794	0.6146	0.7872	0.7054	0.5034	0.5696
Specialized Algorithms (SA)	SA-BFGS	0.2306	0.5976	0.6254	0.8089	0.7163	0.5283	0.6265
	LDL-LRR	0.3293	2.2569	0.5242	0.7115	0.6700	0.4965	0.5803
	TLRLDL	0.3368	7.9606	0.4264	0.5968	0.5559	0.4516	0.4858
Algorithm Adaptation (AA)	CubeMLP	0.2738	0.6900	0.5624	0.7513	0.5996	0.2376	0.5587
	CTEN	0.2432	0.6033	0.6071	0.7977	0.6939	0.4826	0.5109
	MMIM	0.2442	0.6076	0.6019	0.7946	0.6749	0.4503	0.5775

Ablation Study

To evaluate the individual and combined impact of multimodal inputs (Visual, Audio, and Textual clip descriptions), ablation experiments were performed on representative models SA-BFGS and LLaVA-Next:

Model	Input Modality Configuration	Chebyshev (Cheb) ↓	KL Divergence (KL) ↓	Dominant MRR ↑	Dominant Top-1 F1 ↑	Note
SA-BFGS	Visual + Audio	0.314	0.907	0.552	0.306	Lacks high-level narrative context
	Visual + Text	0.234	0.588	0.709	0.515	Substantial boost from scene descriptions
	Visual + Audio + Text (Full)	0.231	0.598	0.716	0.528	Audio provides minor marginal gain
LLaVA-Next	Visual	0.218	4.148	0.731	0.586	Strong baseline from visual reasoning
	Text	0.202	3.693	0.750	0.605	Description alone slightly outperforms raw frames
	Visual + Text	0.188	3.177	0.783	0.652	Multimodal synergy achieves best performance

Key Findings

• Finetuning is essential to unlock VLM audience empathy: Zero-shot foundation VLMs (including proprietary Gemini 2.5 Flash) fail to accurately forecast viewer reactions (Top-1 F1 \(< 0.30\)). LoRA finetuning on Video2Reaction cuts Chebyshev distance by more than half, doubles Cosine similarity (reaching 0.8663), and boosts dominant Top-1 F1 by over 40-fold (reaching 0.6521 on LLaVA-Next).
• Specialized LDL algorithms and VLMs exhibit complementary strengths: SA-BFGS attains superior KL divergence scores (0.5976 vs. 3.1765 for finetuned LLaVA-Next) with minimal computational footprint, accurately capturing broad distribution tails. However, it severely underestimates peak probabilities in low-entropy, unimodal scenes, where finetuned VLMs dominate in ranking (MRR 0.7833) and Top-1 F1 (0.6521).
• Textual descriptions provide critical contextual grounding: Across both model families, textual scene descriptions contribute the most significant performance gains, indicating that high-level plot comprehension is indispensable for anticipating viewer sentiment. Acoustic features provide only minor marginal benefits once visual and textual inputs are integrated.
• Effective cross-dataset transfer indicates reusable affective representations: Pretraining Qwen2.5-VL on Video2Reaction combined with only 1% of the VCE dataset yields a Top-3 accuracy of 0.46 on VCE, outperforming a model trained on 10% in-domain VCE data (0.35), demonstrating that the learned reaction priors generalize beyond the source distribution.

Highlights & Insights

• From perceived to induced emotion modeling: By formalizing audience response prediction as label distribution learning over social commentary, the paper transitions affective computing from subjective, single-label authorial intent to population-level, ecological distribution forecasting.
• Scalable multi-agent self-updating curation: The two-stage multi-agent annotation framework (rephrasing followed by multi-model majority voting) eliminates human annotation bottlenecks while matching human-level consensus (86% blind verification accuracy), establishing an accessible blueprint for tracking non-stationary societal reaction shifts over time.
• Balanced dual-axis evaluation methodology: Introducing the CAD metric (grounded in valence-arousal space) alongside dominant Top-\(k\) classification provides a rigorous template for evaluating future multi-modal distribution learning tasks.

Limitations & Future Work

• Single-platform sampling and demographic opacity: All video clips and comments originate from YouTube, potentially introducing platform-specific commentary habits and recommendation biases. Due to privacy policies, demographic variables (age, gender, geographical background) cannot be disaggregated.
• Pronounced long-tail label imbalance: With an imbalance factor of 28.36, rare affective reactions such as embarrassment and grief exhibit low representation, leaving severe recall challenges on infrequent emotion categories.
• Static aggregation of longitudinal shifts: Although the dataset reveals that movie clips undergo an average of 3.46 dominant emotion transitions across years, current predictive models learn static aggregated target distributions rather than modeling time-conditioned trajectories.

Related Work & Insights

• vs VCE (Mazeika et al., NeurIPS 2022): While VCE encompasses 61k videos, its ratings were gathered from a closed pool of 400 lab/crowd workers and cannot scale or adapt over time; Video2Reaction leverages wild social media interactions, capturing genuine population variance with continuous update capability.
• vs CMSV (Xu et al., NeurIPS 2024): CMSV predicts induced sentiment conditioned on both the video and a specific user comment; Video2Reaction tackles the significantly harder forecasting challenge of predicting the entire audience reaction distribution directly from video content alone.
• vs Classical LDL (Geng, TKDE 2016): Traditional LDL focused on static images and facial age estimation; this work pioneers label distribution learning on complex multimodal cinematic sequences while benchmarking modern foundation VLMs.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the task of in-the-wild video-to-audience-reaction-distribution forecasting with a novel large-scale benchmark and multi-agent pipeline]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluates 12 baseline models across 4 families, 9 metrics over dual axes, multimodal ablations, and extensive blind human validation]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, rigorous conceptual distinction between perceived and induced emotion, and complete empirical analysis]
• Value: ⭐⭐⭐⭐⭐ [Provides an invaluable resource for video recommendation, content creation pre-testing, movie trailer optimization, and VLM affective alignment]