Benchmarking Dynamic Affective Reasoning: A Viewer-Centric Video Emotion Dataset¶

Conference: ECCV2026
Paper: ECCV official page
Full Text: Paper PDF
Project: https://github.com/Zhang-Zhiyan/DAR
Area: VLM Reasoning / Video Emotion Analysis
Keywords: dynamic affective reasoning, viewer perspective, event-level localization, differential captioning, GRPO

TL;DR¶

DAR redefines video emotion understanding through event boundaries, 27 viewer-emotion categories, and contextual causal explanations, while DAR-R1, obtained by applying SFT and GRPO to Qwen2.5-VL-3B, achieves 41.5% segment-count accuracy, 52.3% temporal mIoU, and 28.6% emotion accuracy on its test set.

Background & Motivation¶

Video emotion understanding commonly asks which emotion a clip conveys, but this can conflate two different targets: what a person in the scene expresses and what someone watching the scene might feel. A character's expression need not match the viewer's reaction; even without a change in that expression, unfolding events can move the viewer from curiosity to surprise. Resources such as DFEW primarily support facial-expression recognition. VCE moves toward the viewer perspective and fine-grained categories, but lacks dense event boundaries and explanations of emotional transitions.

A single label for an entire video also erases when a change occurs. Uniformly splitting a video into windows and classifying each window does not solve this: an event can cross a window boundary, and an emotional turning point may occupy only a small part of a window. Drawing on Affective Events Theory, or AET, the paper treats emotions as responses triggered by successive events and influenced by earlier stimuli. Evaluation must therefore ask when an emotion changes, what it becomes, and why it changes, rather than merely expand the classification vocabulary.

DAR primarily contributes a supervision format and benchmark; DAR-R1 tests whether a model can learn the resulting output structure. Core idea: use events as temporal units, connect current visual evidence to the preceding affective context, construct viewer-centric emotional transition chains, and train a model to generate those chains through structured supervision and task-specific rewards.

Method¶

Overall Architecture¶

The work separates offline annotation from model training. Starting with VCE videos, the annotation pipeline performs event-aligned segmentation, incremental differential captioning, and stream-of-affect reasoning and verification, producing timestamps, an emotion label, and an explanation for each segment. Two-stage training then teaches this supervision to a video MLLM. At inference, DAR-R1 receives a video and task instruction and directly generates the segment sequence; it does not need to invoke the entire annotation committee for each prediction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["VCE videos<br/>Duration and static filtering"] --> B["Event-aligned segmentation"]
    B --> C["Incremental differential<br/>captioning"]
    C --> D["Stream-of-affect reasoning<br/>and verification"]
    D -->|Low-score revision feedback| D
    D -->|Accepted annotations| E["Two-stage training"]
    E --> F["DAR-R1<br/>Intervals, emotions, explanations"]

The resulting DAR dataset contains 15,087 videos and 36,908 affective segments, with 13,646 training videos and 1,441 test videos. Mean video and segment durations are 14.3 seconds and 5.8 seconds, respectively, and explanations average 118.8 words. These are not moment-by-moment measurements collected from actual viewers. They are model-generated and model-verified annotations conditioned on existing viewer-emotion candidates, a provenance distinction that limits how the results should be interpreted.

Key Designs¶

1. Event-aligned segmentation: find semantic turning points before correcting timestamps

The original VCE collection contains 61,046 videos. Preprocessing removes clips shorter than 1 second and uses perceptual hashes and optical flow to remove static imagery. Segmentation itself is not based on fixed durations: Gemini-2.5-Pro first proposes boundaries based on narrative changes or affective turning points. PySceneDetect then detects hard cuts and fades, snapping semantic boundaries to the corresponding cut frame when the distance is no more than 0.5 seconds. The semantic model decides where a split is meaningful; the visual detector supplies precision that generated timestamps often lack.

A shot change is not necessarily an emotional transition, however, and many complete events contain no hard cut. InternVL3.5 therefore checks the integrity of segments without hard-cut support, recursively re-segmenting or merging incomplete events. The design does not equate every shot with an independent emotion. Instead, it uses cuts as local calibration signals while retaining semantic control over the final event units.

2. Incremental differential captioning: emphasize changes instead of repeating the setting

Independent captions for successive segments tend to reintroduce the same people and scenery, obscuring the new action that actually changes the viewer's response. Qwen3-VL describes the current segment while reading the preceding description and is explicitly asked to focus on changes in visual content, actions, and atmosphere. The initial historical description is empty. Background information can consequently persist through context while the current description foregrounds new stimuli, making the next stage's transition reasoning more focused.

Grounding DINO checks whether salient entities mentioned in a description can be localized in the associated video frames. This provides an object-level evidence filter, not a proof of the entire causal explanation: an entity's presence does not establish that an interpretation of its action, intention, or emotional effect is correct. The subsequent verification stage remains necessary for precisely this reason.

3. Stream-of-affect reasoning and verification: connect history, current stimuli, and emotion candidates

Qwen3-VL receives the previous segment with its description and emotion, the current segment with its differential description, and the Top-3 candidate emotions from VCE. It generates 4 candidate emotion-reason pairs. Reasoning follows visual evidence, contextual appraisal, and emotion trigger: it identifies what changed before explaining how that change affects the viewer, instead of guessing a label and decorating it with a rationale. When adjacent segments receive the same emotion, the pipeline merges them into a sustained affective phase and regenerates the explanation. The final number of affective segments therefore need not equal the initial number of event segments.

InternVL3.5 and Qwen3-omni then verify candidates along five dimensions: visual grounding, causal logic, viewer centricity, temporal consistency, and answer consistency. Low-scoring candidates return to generation with specific revision feedback; only annotations passing a quality threshold are retained. This checks both whether the rationale supports the label and whether a character's emotion has been substituted for the viewer's reaction. The cache does not specify the numerical acceptance threshold, exact score aggregation, or maximum feedback iterations, so this procedure should not be taken as evidence that subjectivity or labeling bias has been eliminated.

4. Two-stage training: learn valid outputs before optimizing localization and explanation proxies

DAR-R1 uses Qwen2.5-VL-3B as its backbone. SFT minimizes standard token-level negative log-likelihood over target segment sequences, each containing start time, end time, emotion, and causal explanation. Its main role is to overcome the mismatch between general video understanding and the required JSON schema. GRPO subsequently samples a group of outputs for each input, estimates advantages through within-group reward comparison, and constrains policy changes through clipped updates and a KL term without requiring a separately trained critic. This is an application of an existing optimizer: the emphasis is on task supervision and reward configuration, not new policy-optimization theory.

Five rewards address different failure modes. The structural component checks JSON and temporal validity, excessively short segments, and runaway output length. The count component discourages over- and under-segmentation. Temporal localization matches each predicted segment to the ground-truth segment with maximum temporal IoU and accounts for both interval overlap and boundary distance. Emotion reward requires the correct label with sufficient temporal overlap. Explanation reward uses target length and repetition between adjacent explanations as proxies. Temporal IoU is intersection duration divided by union duration; a correct emotion label assigned to the wrong interval is not a correct localized emotion prediction.

Explanation training reward must be distinguished from explanation evaluation: the former primarily checks length and repetition, rather than directly asking a judge to verify causal truth. The readable length-scoring expression in the cache is:

\[ \phi(l)=\exp\left(-\frac{|l-\mu_{\mathrm{len}}|}{\sigma_{\mathrm{len}}}\right),\qquad \mu_{\mathrm{len}}=120,\quad \sigma_{\mathrm{len}}=40. \]

Length here is the word count of each segment's explanation, with the score decaying away from 120 words. Repetition between adjacent explanations is measured with Jaccard similarity, using a stated threshold of 0.75. Appropriate length and different wording do not automatically establish correct reasoning, which is an important boundary when interpreting the results.

A Worked Example¶

Figure 4 shows an 8.2-second video. DAR-R1 labels 0.0–1.5 seconds as Interest, attributing the viewer's curiosity to a person positioned on a tree branch. A sudden swinging action during 1.5–2.5 seconds changes the label to Surprise. The loss of control, landing, and consequences during 2.5–8.2 seconds are labeled Empathic Pain, with an explanation tied to bodily movements and the outcome. This is a model output illustrated in the paper, not a separately collected viewer experiment.

The example shows how the three tasks constrain each other: boundaries should align with action changes, labels should describe the viewer rather than the character's internal state, and explanations should connect consecutive emotions. The backbone instead interprets later events as Relief or Aesthetic Appreciation of the forest. Perceiving scenery and actions is thus not equivalent to identifying the salient event that drives the viewer's response.

Loss & Training¶

Training uses 4 H100 GPUs, freezes the vision encoder in both stages, and updates the LLM and alignment module. SFT uses AdamW for 0.5 epoch with a learning rate of \(1\times10^{-5}\); GRPO runs for 1 epoch with a learning rate of \(2\times10^{-6}\). The structural, count, temporal, emotion, and explanation reward weights are 0.10, 0.25, 0.25, 0.25, and 0.15, respectively.

The structural subcomponent weights are 0.5, 0.2, and 0.3, with a minimum segment duration of 0.4 seconds. The soft output limit is 1200 tokens, the hard limit is 2400 tokens, and the decay scale is 400. Count-penalty temperature is 1.5. Temporal localization uses a 0.5 coefficient to balance IoU and boundary terms and a boundary-decay temperature of 0.5.

Cached equations (1)–(9) contain missing operators, parentheses, or thresholds, and the visible inequality in explanation deduplication conflicts with the prose about penalizing repetition. This note retains only clearly stated prose and the intact length subexpression; it does not reconstruct the full GRPO, structural-reward, or deduplication formulas. Exact reproduction requires checking the original PDF or author implementation. The project URL is stated in the paper, but code availability was not checked online for this note.

Key Experimental Results¶

Main Results¶

The following results are selected from Table 2 and evaluated on the DAR test set. SC-Acc measures whether the predicted segment count is correct; mIoU is mean temporal intersection over union; Emo-Acc requires both the correct emotion label and temporal IoU ≥ 0.5. GPT-Score averages GPT-4o ratings over the five explanation dimensions on a 0–5 scale. The first three columns are percentages.

Model	SC-Acc	mIoU	Emo-Acc	GPT-Score
AffectGPT	24.3	11.9	5.8	1.1
InternVL-3.5-8B	26.0	39.2	11.5	2.2
Qwen2.5-VL-3B	25.4	41.7	15.0	2.3
Qwen2.5-VL-7B	31.9	45.2	14.9	2.6
Qwen3-VL-4B	29.2	48.2	17.9	2.7
Qwen3-VL-8B	26.2	43.1	17.0	2.6
DAR-SFT	37.7	47.6	25.8	3.0
DAR-R1	41.5	52.3	28.6	3.3

DAR-R1 improves mIoU over its own backbone by 10.6 percentage points and Emo-Acc by 13.6 percentage points. It outperforms the listed general-purpose baselines but receives DAR-specific training, so this does not establish that fewer parameters yield universally stronger affective understanding. Qwen3-VL-4B also has higher mIoU than the SFT-only model.

Ablation Study¶

The cache contains no independent reward-removal table and no results for removing differential captioning, boundary snapping, or the committee. The following real training-stage comparison is organized from Table 2. It tests the change from adding the complete RL stage, not the isolated causal contribution of individual modules.

Training Configuration	SC-Acc	mIoU	Emo-Acc	GPT-Score
Backbone without DAR adaptation	25.4	41.7	15.0	2.3
DAR-SFT only	37.7	47.6	25.8	3.0
DAR-SFT + GRPO	41.5	52.3	28.6	3.3
GRPO increment over SFT	+3.8	+4.7	+2.8	+0.3

The first three increments are percentage points; the last is in rating points. SFT already improves SC-Acc and Emo-Acc over the backbone by 12.3 and 10.8 percentage points, respectively, while GRPO further improves localization. GPT-rated causal logic rises from 3.0 to 3.8, but visual grounding stays at 3.1. Thus, not every explanation dimension improves, and this comparison cannot identify which reward independently caused a gain.

Table 3 additionally reports ratings by 5 experts on 100 randomly sampled videos. All columns use a 0–5 scale; averages are retained as reported.

Model	Visual Grounding	Causal Logic	Viewer Centricity	Temporal Consistency	Answer Consistency	Average
AffectGPT	0.6	1.8	0.2	0.1	1.0	0.7
Qwen2.5-VL-3B	2.5	2.3	2.5	2.4	2.8	2.5
Qwen3-VL-4B	3.1	3.5	2.7	2.8	3.0	3.0
DAR-SFT	3.4	3.5	4.0	3.1	3.8	3.6
DAR-R1	4.2	4.5	4.2	3.8	4.4	4.2

Key Findings¶

Task adaptation is more directly effective here than simply increasing parameter count. The matched 3B training-stage comparison supports this, but does not isolate the respective effects of data volume, computation, and reward design.
Localized emotion recognition remains difficult: final Emo-Acc is only 28.6%. This includes both category errors and failures to meet the temporal-overlap threshold, and is not classification accuracy with ground-truth segmentation supplied.
Both human and automatic evaluation prefer DAR-R1, but their average scores are 4.2 and 3.3, respectively. Agreement in ranking does not establish score calibration or a measured statistical correlation.

Highlights & Insights¶

Explicitly separating viewer emotion from character emotion prevents facial recognition alone from satisfying the task. Changing the target of evaluation reveals errors that merely adding categories would miss.
Semantic proposals and visual cuts have separate responsibilities, reducing dependence on generated timestamps. Similar event-level annotation pipelines can reuse this division while retaining integrity checks for events without cuts.
Differential descriptions suppress repeated background information, while affective history constrains subsequent explanations. The transferable supervision is an account of why the current change matters, not simply the emotion vocabulary.

Limitations & Future Work¶

The paper has no dedicated limitations section; the following points are analysis based on its setup and reported evidence.

Model-generated labels and model committees may share biases, while Top-3 candidates and same-emotion merging influence the final state sequence. The annotations cannot be assumed to represent reactions across cultures, experiences, or individual viewers; multiple-viewer continuous annotation and disagreement modeling would help.
AET supplies an organizing principle, but invoking the theory does not validate psychological causality. Future work should distinguish narrative coherence, sufficient visual evidence, and actual viewer experience.
Explanation length and deduplication remain proxy rewards. Without reward-level ablations, the source of better causal-logic ratings is unresolved; direct visual-evidence rewards should be compared under matched training budgets.
The distribution is long-tailed, with the top 17 categories accounting for approximately 80%. The cache reports no per-class results, cross-dataset generalization, seed variance, or confidence intervals. Human evaluation covers only 100 videos and cannot establish broad clinical or real-time interaction capability.
Damaged formula extraction and missing reproduction details limit offline verification. This note preserves explicit numbers rather than presenting speculative symbol repairs as original equations.

Relative to VCE: DAR inherits 27-category viewer-emotion semantics and candidate information, adding event-aligned phases and explanations. It constructs denser supervision on existing data rather than independently collecting all-new videos.
Relative to ArtEmis / Affection / StimuVAR: DAR continues stimulus-based explanations of viewer feelings, emphasizing dense boundaries and historical affective continuity in video instead of an explanation for an entire image or clip.
Relative to AffectGPT / Videmo: DAR binds temporal localization, viewer perspective, and explanation into one structured task. Low scores from existing affective models indicate transfer difficulty, not ineffectiveness on their original tasks.
Relative to GRPO: DAR-R1 uses existing group-relative optimization with a task-specific reward combination. The reusable lesson is to check format, localization, category, and textual quality separately, not assume that one aggregate score proves every capability.

Rating¶

Novelty: 4/5. Dynamic viewer-emotion supervision and evaluation are valuable; the training algorithm largely uses existing methods.
Experimental Thoroughness: 3/5. Multiple models, training-stage comparisons, and human evaluation are included, but component ablations, distribution-shift tests, and uncertainty reports are missing.
Writing Quality: 3/5. The task and pipeline are clear, but reward semantics need more precision; damaged cached formulas should not all be attributed to the original manuscript.
Value: 4/5. An actionable benchmark for event-level emotion understanding, with substantial distance remaining to general and personalized affect inference.