Skip to content

Why Do Vision Language Models Struggle To Recognize Human Emotions?

Conference: ECCV 2026
Paper: ECCV 2026 Poster #5341 · Project Page
Area: Multimodal VLM
Keywords: vision-language models, dynamic facial expression recognition, long-tail bias, temporal modeling, context enrichment

TL;DR

Using video-based dynamic facial expression recognition (DFER) as a probe task, this paper traces the emotion-recognition failures of contemporary VLMs to two root causes — head-class bias inherited from long-tailed pretraining data and a breakdown of temporal modeling under a fixed token budget — via three evidence chains (a lexical-frequency proxy, frame shuffling, and a frame-rate density sweep), and validates the diagnosis with balanced-data decoupled fine-tuning and an inference-time Multi-Stage Context Enrichment (MSCE) pipeline, reporting consistent F1 gains on the balanced test splits of MAFW and DFEW.

Background & Motivation

Emotion recognition requires inferring a person's affective state from behavioural signals that unfold over time (facial expression, voice, gesture), which demands two capabilities at once: fine-grained spatial sensitivity to subtle facial muscle movements, and temporal fidelity for signals as fleeting as micro-expressions. Humans are near-instinctive at this — they read micro-expressions lasting only 0.25–0.5 seconds, modulate their interpretation with context, and cope naturally with the heavy-tailed distribution of affective categories, in which high-frequency states such as neutral dominate while rare but consequential states such as contempt, helplessness and disappointment appear in only a handful of samples. Contemporary VLMs (the Gemini and Qwen families) are already strong at static object recognition and long-description grounding, yet they perform surprisingly poorly on video emotion recognition: they routinely conflate semantically adjacent states (sadness versus disappointment) and hold no clear advantage over vision-only specialist classifiers on this task.

The authors hypothesise that this gap is not because emotion is intrinsically too hard, but because DFER happens to hit the two places where contemporary VLMs are uniquely fragile. The first is head-class bias induced by the long-tailed distribution: VLM pretraining corpora are web-scraped image-text pairs whose visual features are aligned with alt-text embeddings during contrastive pretraining, so the frequency with which an emotion concept appears in human text directly dictates its representation density in the multimodal latent space; rare emotions are therefore systematically collapsed into high-frequency categories. The second is temporal representation under a fixed token budget: video frames are cut into tokens and treated on a par with text tokens, while adjacent frames are highly redundant, so denser sampling tends to bring attentional dilution rather than information, and the model degenerates into order-agnostic "bag-of-frames" processing. The difficulty is that both remain hypotheses: pretraining corpora are proprietary, so per-class emotion frequencies cannot be measured directly, and the temporal failure lacks a controlled experiment on natural high-resolution video that perturbs only time while holding spatial content fixed.

This paper's angle is to treat DFER as a probe — it simultaneously tests spatial fine-grainedness and temporal fidelity, and it is naturally long-tailed, which makes it well suited to turning both hypotheses into falsifiable evidence. For the long tail, the authors avoid guessing at the data and instead use historical lexical frequencies from Google Books Ngrams as an observable proxy for pretraining distribution bias, quantifying the correlation between category scarcity and per-class accuracy, then pushing that correlation towards causation with vision-only models and balanced-data fine-tuning as controls. For temporal behaviour, two perturbation experiments — frame shuffling and a frame-rate sweep — decompose "temporal failure" into two concrete symptoms: invariance to frame order, and attention saturating as visual token density rises. Core idea: rather than chasing higher DFER numbers, this paper uses three evidence chains — a lexical-frequency proxy, order/density perturbations, and balanced-data fine-tuning — to attribute VLM emotion-recognition failure to two layers (the head-class prior induced by long-tailed data, and attention diluted by dense visual tokens), and validates the temporal bottleneck with MSCE, which translates inter-frame motion into natural language.

Method

Overall Architecture

The paper has two halves. The first (Section 3) is a systematic diagnosis; the second (Section 4) attaches one plug-and-play intervention to each of the two pathologies the diagnosis identifies. For the diagnosis, MAFW (11 emotion classes) and DFEW (7 emotion classes) are re-split into class-balanced test sets, the same pool of models performs multi-class classification, and three instruments ask "where exactly is it failing": the lexical-frequency proxy answers which classes fail, the frame-shuffling probe answers whether the model uses temporal order at all, and the frame-rate sweep answers whether more visual information helps or hurts. For the intervention, balanced-data decoupled fine-tuning tests the causal role of the long tail, while Multi-Stage Context Enrichment (MSCE) recovers the temporal gaps discarded by sparse sampling at inference time. Diagnosis and intervention form a closed loop: the interventions are designed purely from the diagnostic findings, and their effects in turn validate those findings.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Balanced MAFW / DFEW test splits<br/>multi-class emotion classification"] --> B["Lexical-frequency proxy correlation analysis"]
    A --> C["Frame-shuffling probe"]
    A --> D["Frame-rate density sweep"]
    B -->|supports the long-tail hypothesis| G["Balanced-data decoupled fine-tuning"]
    C -->|confirms order invariance| M
    D -->|confirms attentional dilution| M
    subgraph M["Multi-Stage Context Enrichment (MSCE)"]
        direction TB
        M1["Stage 1: Motion-to-Text Translation"] --> M2["Stage 2: Interleaved Context Classification"]
    end
    G --> Z["Reduced head-class bias<br/>and recovered inter-frame cues"]
    M --> Z

One detail matters for reading every number below: the diagnosis uses a single balanced evaluation protocol across all models, taking 495 videos from MAFW (45 per class) and 700 from DFEW (100 per class), with the remaining videos kept for training. Because the test set is flattened by construction, WAR and UAR become mathematically identical, so a single reported accuracy reflects both overall and per-class behaviour — this design is itself part of the diagnosis, since WAR would otherwise mask failures on the tail.

Key Designs

1. Lexical-frequency proxy correlation analysis: replacing a proprietary pretraining prior with an observable linguistic statistic

Per-class frequencies in the pretraining corpus cannot be measured directly, so the authors quantify something that can be: if a VLM aligns visual features with web text during pretraining, then the frequency of an emotion concept in human text should govern its representation density. They take the historical frequency of each emotion label in Google Books Ngrams (July 2024 corpora), in both English and Chinese, and correlate it with the zero-shot per-class F1 of Gemini2.5-Flash and Qwen2.5-VL on the balanced MAFW test split, reporting Pearson coefficients with two-tailed t-test p-values. English frequencies correlate with per-class F1 at 0.7927 / 0.8041 (p = 0.0036 / 0.0029) for the two models; Chinese frequencies at 0.6343 / 0.7547 (p = 0.0361 / 0.0073); and the two frequency signals themselves correlate at 0.8825. The authors are explicit about the proxy's weaknesses — VLMs are trained predominantly on English web data so English n-grams should be closer, while Chinese affective lemmas are more polysemous and inflate raw frequency — so what supports the long-tail interpretation is not the absolute frequencies but the fact that two independent language statistics point in the same direction, suggesting a deeper conceptual scarcity rather than the statistics of one particular corpus.

Correlation alone is not enough, so two controls harden it. The first rules out "this is specific to VLM pretraining or context windows": MAE-DFER and HiCMAE, two vision-only classifiers (MAE architecture, pretrained on Voxceleb2, which carries no emotion labels and therefore serves only as a general audio-visual feature extractor), fine-tuned on the imbalanced MAFW training split, reproduce the same long-tail shape on the same balanced test set — happiness reaches F1 0.69/0.70 while helplessness, contempt and disappointment sit at only 0.04–0.22. The coupling therefore comes from the long-tailed training distribution itself, not from the VLM architecture. The second control examines the direction of confusion: rare classes collapse systematically into common ones (Disappointment → Sadness) while the reverse almost never holds. Pure semantic confusion would be bidirectional, so this strong asymmetry indicates that models fall back on high-frequency priors rather than failing to distinguish meanings. The authors formalise this with a Directional Asymmetry Index in the supplementary material (⚠️ the definition does not appear in the main text; refer to the original paper).

2. Frame-shuffling probe: unchanged accuracy under shuffled frames is an admission that order is unused

If a model genuinely reads temporal information, randomly permuting the frame order should cost it dearly. The authors construct two inputs with identical visual content — the same frames, the same appearance and spatial information — differing only in whether the temporal order is preserved or randomly shuffled. The contrast is stark: dedicated temporal models such as MAE-DFER and HiCMAE lose 15–16% macro-F1 (HiCMAE on MAFW drops from 0.3993 to 0.3345; MAE-DFER on DFEW from 0.5645 to 0.4802), whereas every VLM barely moves. Qwen2.5-VL even improves slightly on MAFW (0.2449 → 0.2506) and is essentially flat on DFEW (0.4552 → 0.4534); Qwen3-VL rises slightly on DFEW (0.5511 → 0.5538); the largest VLM drop, Gemini2.5-Flash, is only 0.6008 → 0.5778. This invariance is strikingly consistent across model scales and visual integration strategies, and the only reasonable reading is that VLMs perform order-agnostic aggregation of per-frame appearance. The sharper implication is that the implicit temporal ordering supposedly imposed by causal attention masking is not working either. The authors contrast this probe with SpookyBench: that work uses noise sequences to show models rely on spatial cues, whereas this probe keeps spatial content fully intact on natural high-resolution video and removes only order, thereby exposing a distinct failure mode — redundancy-driven attentional dilution.

3. Frame-rate density sweep: performance traces a quasi-bell curve against visual token density

The previous probe asks whether order matters; this one asks whether more visual information helps or harms. The authors hold Qwen2.5-VL and EmotionQwen fixed (open-source models permit fine-grained control of frame sampling, whereas the closed-source Gemini offers no such API control and is therefore excluded) and raise the sampling rate from 1 FPS through 5, 10, 15, 20 and 25 FPS, instructing the model to infer the emotional state from the given frames alone. The result is quasi-bell-shaped: accuracy improves from 1 to 5 FPS, as one would expect from having more evidence, but then degrades as the rate rises further, eventually falling below the 1 FPS level. The proposed mechanism is that visual token embeddings have a substantially larger vector norm than text token embeddings while adjacent frames are themselves highly correlated, so filling the context amounts to drowning the task-relevant signal in redundancy — the video manifestation of the "lost-in-the-middle" problem. This is especially damaging for emotion, since a micro-expression lasts only 0.25–0.5 seconds: it needs dense enough sampling to be captured at all, yet is diluted by the redundant tokens that dense sampling introduces. Sparse and dense are both wrong, and that deadlock is precisely what MSCE addresses.

4. Balanced-data decoupled fine-tuning: upgrading the long tail from correlation to causation

Lexical-frequency correlation can only say that rare emotions perform worse, not that imbalance causes it. The authors validate the causal claim with decoupled training: keep the feature-extraction ability learned from large-scale pretraining and retrain only the classification part on an artificially flattened distribution. Concretely, they sample roughly 1500 videos from MAFW with an equal number per class — about a six-fold reduction relative to the original training split — and fine-tune MAE-DFER and HiCMAE on it, plus Qwen2.5-VL with LoRA. Despite the six-fold data reduction, the distribution of predicted labels becomes appreciably more uniform, tail-class F1 rises, and head-class bias weakens: confusion matrices move from a strong diagonal on high-frequency classes plus scattered tail errors to a more balanced form. The value of this step is establishing that poor recognition of rare emotions is not a missing capability but a data prior pushing outputs towards high-frequency classes; the intervention is also lightweight, requiring neither extra modelling nor generative augmentation, and remains usable when the pretraining distribution is opaque and retraining is infeasible. For the closed-source model the authors report bias rather than a mitigation result: Gemini2.5-Flash reaches per-class F1 of 0.60 for happiness, 0.09 for helplessness and 0.26 for disappointment, against 0.54, 0.0 and 0.07 for Qwen2.5-VL — numbers that downstream users can treat as a calibration of trust.

5. Multi-Stage Context Enrichment (MSCE): a temporal bridge built by translating inter-frame motion into text

MSCE targets the deadlock described above: sparse sampling (1 FPS) necessarily misses micro-expressions, dense sampling (>10 FPS) dilutes attention, and the diagnostically valuable cues frequently sit precisely in the gaps that sparse sampling discards. Unlike frame-selection methods such as TCoT, which pick among candidate frames and discard everything unselected, MSCE discards nothing and instead changes modality, in two stages.

Stage 1 is Motion-to-Text Translation. Given a video, \(n\) keyframes \(K=\{k_1,\dots,k_n\}\) are sampled at the 1 FPS baseline, creating \(n-1\) temporal gaps; each gap is then sampled with a small, denser set of "in-between" frames \(G_j=\{g_{j1},\dots,g_{jm}\}\). The set is deliberately kept small (\(m=4\) in the experiments) so as not to reintroduce attentional dilution. Each such set is passed to the VLM, which produces a natural-language summary \(t_j\) of the facial motion in that interval. The point is not to push more frames into the context but to make the VLM perform a modality translation over a short, highly redundant visual input, compressing bulky, low-density visual tokens into compact, high-density text tokens. Stage 1 ends with \(n-1\) summaries \(T=\{t_1,\dots,t_{n-1}\}\), each describing motion that the keyframes missed.

Stage 2 is Interleaved Context Classification. The keyframes and the text summaries are interleaved in chronological order into a single prompt, \(\{k_1,t_1,k_2,t_2,\dots,k_{n-1},t_{n-1},k_n\}\), with each motion description explicitly tied to the gap between its two neighbouring keyframes and the candidate emotion labels listed, and the VLM is asked to produce the final classification from this enriched context. The summaries act as a temporal bridge: they supply explicit high-level semantics spanning the visual hole between \(k_i\) and \(k_{i+1}\), letting the model reason about how an expression evolves — a micro-expression appearing and vanishing — instead of continuing to aggregate per-frame appearance in an order-agnostic way. The design holds up because it shifts the burden from the capability VLMs are worst at (long-horizon temporal retention) to the one they are best at (short-horizon description); and because the text exists alongside the original keyframes rather than replacing them, generation noise from the summaries does not become a single point of failure.

A Worked Example

Take the fleeting smile in Figure 5: a three-second video, sparsely sampled at 1 FPS into four keyframes \([email protected]\text{s}\), \([email protected]\text{s}\), \([email protected]\text{s}\), \([email protected]\text{s}\), forming three gaps. The actual micro-expression occurs only between 0.25 s and 0.75 s, falling entirely inside the gap between \(k_1\) and \(k_2\); the baseline VLM sees four keyframes, sees nothing, and predicts Neutral. In Stage 1, MSCE samples four frames uniformly within gap 1 and asks the VLM to describe the motion, obtaining a summary along the lines of "the character's lips move upwards into a brief smile", with similar summaries for gaps 2 and 3. Stage 2 interleaves \(\{k_1,t_1,k_2,t_2,k_3,t_3,k_4\}\) into the prompt, at which point the model finally "sees" the half-second smile and outputs Happiness. No training is added anywhere in this chain — it is one extra VLM call at inference time.

Loss & Training

The paper proposes no new training objective; training-related settings concentrate in two places. First, model preparation for the diagnosis: Gemini2.5-Flash, Qwen2.5-VL and others are queried zero-shot on MAFW / DFEW to measure per-class bias, while MAE-DFER and HiCMAE are pretrained on the large-scale audio-visual corpus Voxceleb2 (which carries no emotion labels) and then fine-tuned on the MAFW training split, in order to test whether the long-tail effect is independent of VLM architecture. Second, the decoupled fine-tuning of Section 4.1: the classification part is retrained on roughly 1500 class-balanced MAFW videos, with Qwen2.5-VL adapted through LoRA, so that pretrained representations are preserved while the head-class prior is corrected. MSCE is a pure inference-time pipeline requiring no training, with only two hyper-parameters: keyframe density (a 1 FPS baseline) and the number of in-between frames per gap, \(m=4\).

Key Experimental Results

Main Results

Evaluation uses MAFW (11 classes) and DFEW (7 classes) with class-balanced test splits (495 and 700 videos respectively), so the reported macro-F1 is equivalent to balanced accuracy. Table 1 gives the central frame-shuffling comparison, in which visual content is identical and only temporal order is destroyed.

Model Type MAFW original MAFW shuffled DFEW original DFEW shuffled
HiCMAE vision-only specialist 0.3993 0.3345 0.5725 0.4797
MAE-DFER vision-only specialist 0.3602 0.3041 0.5645 0.4802
Gemini2.5-Flash closed-source generalist VLM 0.3758 0.3626 0.6008 0.5778
Qwen3-VL open-source generalist VLM 0.2738 0.2615 0.5511 0.5538
Qwen2.5-Omni open-source generalist VLM 0.3060 0.2972 0.4296 0.4226
Qwen2.5-VL open-source generalist VLM 0.2449 0.2506 0.4552 0.4534
EmotionQwen task-specific VLM 0.2581 0.2517 0.5010 0.4895
InternVL-3.0 open-source generalist VLM 0.2445 0.2326 0.5044 0.4985
LLaVA-NeXT-Video open-source generalist VLM 0.1438 0.1382 0.2969 0.2712
Video-LLaVA open-source generalist VLM 0.0870 0.0814 0.1654 0.1531

Table 2 reports the effect of MSCE. The most direct alternative, TCoT (frame-selection long-video reasoning, matched to Qwen2.5-VL), scores 0.241 / 0.450 on MAFW / DFEW — below the sparse-sampling baseline of 0.245 / 0.455 — while MSCE exceeds both.

Model MAFW F1 baseline MAFW +MSCE DFEW F1 baseline DFEW +MSCE
Qwen2.5-VL 0.2449 0.2731 (+0.0282) 0.4552 0.4820 (+0.0268)
EmotionQwen 0.2581 0.2683 (+0.0102) 0.5010 0.5147 (+0.0137)
LLaVA-NeXT-Video 0.1438 0.1715 (+0.0277) 0.2969 0.3171 (+0.0202)
Qwen2.5-VL + TCoT (frame-selection control) 0.245 0.241 0.455 0.450

Ablation Study

The paper has no conventional module ablation; diagnostic controls take its place. Table 3 is the correlation matrix for the lexical-frequency proxy (Pearson coefficients, two-tailed t-test p-values in parentheses): every pairing of a language statistic with a model's per-class F1 is significantly positive.

Signal pair Pearson r p-value
English Ngram frequency — Gemini2.5-Flash per-class F1 0.7927 0.0036
English Ngram frequency — Qwen2.5-VL per-class F1 0.8041 0.0029
Chinese Ngram frequency — Gemini2.5-Flash per-class F1 0.6343 0.0361
Chinese Ngram frequency — Qwen2.5-VL per-class F1 0.7547 0.0073
English Ngram — Chinese Ngram frequency 0.8825 0.0003
Gemini2.5-Flash — Qwen2.5-VL per-class F1 0.8176 0.0021

Two further analytical controls are not tabulated. First, vision-only models fine-tuned on the imbalanced MAFW training split show the same long tail (MAE-DFER: happiness 0.70, contempt 0.18, disappointment 0.22; HiCMAE: happiness 0.69, helplessness 0.04, disappointment 0.16), separating "the long tail comes from the data" from "the long tail comes from the VLM architecture". Second, the frame-rate sweep (Qwen2.5-VL and EmotionQwen only) yields the quasi-bell curve: rising from 1 to 5 FPS, degrading steadily beyond 5 FPS, and ending below the 1 FPS level.

Key Findings

  • The shuffling experiment is the paper's most telling evidence: specialist models lose 15–16%, while VLM drops are of order 0.01–0.02 and in a few settings (Qwen2.5-VL on MAFW, Qwen3-VL on DFEW) accuracy even rises. The problem is not that VLMs model time poorly; it is that they never use frame order at all.
  • The frame-rate sweep shows longer context is not free: a modest increase in visual evidence helps, but beyond a point redundant tokens dilute attention and performance falls below the lowest sampling rate. The two frequency-related failures are two sides of one coin — sparse sampling misses micro-expressions, dense sampling drowns them.
  • The MSCE-versus-TCoT comparison is clean: frame selection discards the content of unselected gaps and ends up below the sparse baseline (0.241 < 0.245), whereas MSCE keeps the gap information and merely changes its modality, beating both (0.273 / 0.482). This directly supports the claim that critical affective cues live in the discarded temporal gaps.
  • Balanced fine-tuning shows the long tail is mitigable: even with roughly six times less data, the predicted-label distribution becomes markedly more uniform, indicating that tail failures stem from the data prior rather than a missing capability.
  • One boundary deserves care. MSCE's gains are moderate (F1 +0.01 to +0.03), and the authors themselves frame them as diagnostically conclusive rather than leaderboard-oriented. In Table 1, Gemini2.5-Flash is indeed below vision-only HiCMAE on MAFW (0.3758 vs 0.3993) but above it on DFEW (0.6008 vs 0.5725) — so "VLMs lose to specialised vision classifiers" holds on MAFW but is not a universal conclusion, and in the main text the claim is made primarily about the Qwen family under the frame-rate sweep.

Highlights & Insights

  • Trading an unobservable pretraining distribution for an observable linguistic statistic. Proprietary pretraining corpora are a dead end, so the authors measure something that the corpora necessarily shape and that is fully public (Google Books Ngram frequencies), then reinforce the conclusion with two independent language statistics pointing the same way. This "proxy variable plus cross-lingual consistency" pattern transfers to any setting where a closed model's data prior must be inferred.
  • The shuffling probe is a remarkably cheap general check-up. Two inputs with identical visual content differing only in temporal order are enough to determine whether a model uses time at all — any model claiming video temporal understanding can be audited this way, without sweeping sampling rates or prompts.
  • Turning a frame-budget problem into a modality problem. Faced with "sparse misses it, dense drowns it", the conventional move is to design a better frame-selection policy; MSCE instead changes modality, compressing redundant visual tokens into dense text tokens, effectively using the VLM's language side to cover for its visual side. This is relevant to any task combining a continuous signal with a limited context (long-video QA, procedural understanding).
  • Selling a closed model's bias estimate as a deliverable. For models that cannot be fine-tuned or retrained, the coupling between per-class F1 and lexical frequency is a usable trust-calibration table, telling downstream users which emotions the model cannot be trusted on. This framing is uncommon in VLM evaluation.

Limitations & Future Work

  • Three limitations acknowledged by the authors. MSCE is a strategic intervention rather than a final resolution, and text summarisation introduces generation noise (they argue the burden shifts from weak long-horizon temporal retention to strong short-horizon description, and that the summaries remain a supporting rather than sole signal). Current VLM attention degrades over very long visual contexts, so the real remedy should be hierarchical or sparse temporal attention with better long-sequence positional encodings. And the Ngram analysis is a correlative, non-causal proxy, which additional frequency proxies would strengthen.
  • Limitations I would add. MSCE's key hyper-parameter is reported at a single setting (\(m=4\)), with neither an \(m\) sensitivity study nor an ablation on keyframe density, so the claim that small \(m\) avoids dilution lacks direct evidence. The frame-rate sweep covers only open-source Qwen-family models, leaving closed-source behaviour to indirect inference. Both test splits are manually balanced, so the numbers are not comparable with DFEW SOTA in the literature. The per-class correlations rest on only 11 and 7 data points, so the cross-lingual consistency argument retains a speculative element. And since a specialised vision model is overtaken by Gemini on DFEW, "VLMs lose to specialised models" is not a universal conclusion; the paper does not discuss this counterexample.
  • Concrete improvements. Apply self-consistency or multi-summary voting to the gap summaries to reduce noise; turn MSCE's text summaries from an inference-time trick into auxiliary training supervision so the model learns to build temporal bridges itself; and extend the probes to audio and textual cues, since MAFW carries multimodal annotations, to test whether the same failure appears in the speech channel.
  • vs TCoT (Temporal Chain of Thought): Both address temporal sparsity in long video. TCoT searches over candidate frames and discards the unselected temporal gaps, which is exactly where micro-expressions live, so in a matched setting (Qwen2.5-VL) it scores below the sparse baseline (MAFW 0.241 vs 0.245). MSCE discards nothing and changes modality instead, which is why it beats both — and this comparison is the paper's single most persuasive argument.
  • vs long-tail visual recognition work (e.g. The Neglected Tails in VLMs, decoupled training): That line establishes the general inheritance of long-tail bias over common concepts in VLMs and proposes remedies such as decoupled training. This paper lands the same phenomenon on emotion categories, shows the bias couples to cross-lingual historical word frequency and also appears in vision-only models, and deliberately reuses decoupled training for mitigation rather than proposing a new method — the contribution is the diagnosis.
  • vs face-specific DFER methods (DK-CLIP, PE-CLIP, Fine-CLIPER, AU-DFER, FaVChat): These improve DFER by injecting action-unit priors, landmarks, segmentation masks or structured textual descriptions, and substantially outperform generalist VLMs on DFEW. The authors read this as supporting their diagnosis: each succeeds precisely because it supplies the structure or specialisation that generalist VLMs lack.
  • vs SpookyBench / Lost in the Middle: SpookyBench uses noise video to show models rely on spatial cues; this paper keeps spatial content intact on natural high-resolution video and removes only order, reaching the same direction of conclusion while additionally exposing an independent failure mode — redundancy-driven attentional dilution, in which accuracy saturates and then falls as visual token density rises, something noise sequences cannot reveal.

Rating

  • Novelty: ⭐⭐⭐⭐ The diagnostic angle is new — splitting "why VLMs cannot read emotions" into a data layer and an architecture layer, each with falsifiable probes — though the method components are mostly combinations and reuses of existing parts.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Two datasets, roughly a dozen models, three diagnostic probes plus two interventions, forming a complete loop; the weaknesses are MSCE's single hyper-parameter setting and moderate gains, and the frame-rate sweep covering only open-source models.
  • Writing Quality: ⭐⭐⭐⭐ The argument chain is clear and the diagnose-intervene-validate loop is persuasive, with the proxy variable's flaws openly discussed; some experimental detail (such as the Directional Asymmetry Index) is deferred to the supplementary material.
  • Value: ⭐⭐⭐⭐ Offers researchers in affective computing and multimodal temporal understanding a reusable diagnostic checklist plus two lightweight interventions, and gives closed-model users a way to think about trust calibration.