Emotion Understanding in Streaming Video with Trajectory-Aware Reliability¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/APTX574/Trace
Area: Video Understanding
Keywords: Streaming Video Understanding, Multimodal Emotion Recognition, Belief Trajectory, Reliability Estimation, Adaptive Reinterpretation
TL;DR¶
Addressing the vulnerability of isolated prefix predictions and deceptive single-prefix confidence in streaming video, this paper proposes TRACE, a trajectory-aware reliability framework that continuously tracks emotion belief trajectories from lightweight audio streams, evaluates decision risk via a two-head reliability model, and selectively triggers multimodal contextual reinterpretation, matching full-context performance while reducing reinterpretation calls to 55.58%.
Background & Motivation¶
Video emotion understanding serves as a foundational pillar for affective computing, dialog systems, and real-time human-computer interaction. Most existing multimodal emotion recognition methods approach this problem under an offline classification formulation, presuming that the complete video segment, acoustic waveform, dialogue transcript, and conversational context are fully observed before any prediction is executed. However, emerging real-time interactive domainsโsuch as embodied intelligence, online mental health counseling, and live conference analyticsโmandate that systems perceive and respond to user emotions dynamically as the interaction unfolds. While contemporary streaming video understanding models can detect actions or scene transitions in real time, they generally wait for an event to terminate before making a commitment and treat outputs at consecutive time steps as isolated static predictions, failing to accommodate the temporal accumulation and evolving nature of affective signals.
The fundamental tension hindering the transition from offline recognition to streaming deployment lies in the asymmetric arrival latency of multimodal signals versus the deceptive unreliability of isolated prefix outputs. In streaming environments, acoustic signals arrive continuously and carry dense, low-latency emotional cues such as intonation, pauses, speaking rate, and vocal intensity, rendering audio the natural vehicle for fast online tracking. In contrast, visual observations and conversational transcripts provide crucial contextual grounding and semantic disambiguation, but incur heavy computational costs and transmission delays. Naively committing decisions based on a single audio prefix invites severe errors: transient vocal fluctuations can induce sharp, spuriously high confidence at an isolated step even while the underlying class distribution oscillates wildly across neighboring frames. Conversely, routing every streaming frame through heavy cross-modal models introduces prohibitive inference latency and destroys real-time responsiveness.
This paper addresses this challenge by reframing streaming video emotion understanding as a continuous belief evolution and reliability calibration process: the system must not only form incremental emotion beliefs from low-latency audio prefixes, but also track the temporal trajectory of these beliefs to ascertain whether early evidence is sufficient or whether heavier contextual reasoning is genuinely required. Core idea: reframe streaming video emotion understanding into a three-stage dynamic process of "online audio prefix belief formation, trajectory-calibrated reliability estimation, and selective contextual belief reinterpretation," utilizing trajectory dynamics such as confidence transitions, entropy shifts, and label-switching patterns to adaptively route computation between low-latency commitment and deep multimodal reinterpretation.
Method¶
Overall Architecture¶
The TRACE architecture decouples streaming emotion understanding into three complementary stages: low-latency Online Prefix Belief Formation, Trajectory-Calibrated Reliability Estimation, and selective Contextual Belief Reinterpretation. In the online streaming pipeline, incoming video is segmented into utterance units using voice activity detection (VAD). An incremental audio backbone continuously outputs emotion probability distributions across progressive prefixes to construct an evolving belief trajectory. A trajectory-aware reliability module extracts temporal dynamics across this trajectory and feeds them into a two-head network that estimates prediction error risk and potential contextual gain, thereby guiding an online policy to commit immediately, wait for further audio, or trigger reinterpretation. For ambiguous or volatile samples, the system invokes a heavyweight multimodal model to re-evaluate the utterance by jointly integrating visual frames, transcripts, preceding dialogue memory, and the prefix belief trajectory itself.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Streaming Video Utterance Input<br/>VAD Utterance Segmentation"] --> B["Online Audio Prefix Belief Formation"]
B -->|Incremental audio prefixes and belief sequence| C["Trajectory-Calibrated Reliability Estimation"]
C -->|Stable and confident: commit| D["Immediate Online Commitment<br/>Output current audio belief label"]
C -->|Insufficient evidence: wait| B
C -->|Volatile or ambiguous: reinterpret| E["Contextual Belief Reinterpretation"]
E --> F["Revised Decision Output<br/>Output revised label + rationale + updated memory"]
Key Designs¶
1. Online Audio Prefix Belief Formation: Progressive Non-Autoregressive Belief Tracking Against Isolated Prefix Vulnerability
Addressing the severe computational latency of full multimodal input and the error-compounding hazard of autoregressive history encoding, this design establishes an audio-first fast pathway for low-latency incremental inference. Audio streams provide dense affective prosody (e.g., pitch shifts, tremor, pauses, speech rate) with minimal processing delay. For an utterance \(V_k\) observed up to prefix step \(t\) with audio \(a_{1:t}\), the module utilizes Qwen2.5-Omni-3B to process the concatenated task query and audio prefix \([q, a_{1:t}]\), producing the emotion belief probability distribution over label set \(\mathcal{Y}\):
The predicted label at step \(t\) is \(\hat{y}_{k,t}^{on} = \arg\max_{y \in \mathcal{Y}} p_{k,t}(y)\). Crucially, these step-wise predictions are recorded externally to construct the temporal belief trajectory \(B_{k,1:t} = \{p_{k,1}, p_{k,2}, \dots, p_{k,t}\}\) rather than concatenated autoregressively into the input prompt of subsequent time steps. This non-autoregressive tracking guarantees that all belief revisions are driven purely by freshly arrived acoustic evidence, prevents hallucinated early labels from poisoning later inferences, and leverages KV caching to maintain sub-frame latency. During training, supervision is applied across all prefix steps weighted by the observation ratio \(r_t = t / T_k\) via an observation-weighted loss:
This formulation encourages the backbone to distill meaningful emotional signals from early partial waveforms while placing higher gradient penalties on near-complete prefixes.
2. Trajectory-Calibrated Reliability Estimation: Dual-Head Risk-Gain Routing Against Deceptive Single-Prefix Confidence
Addressing the limitation that single-prefix Softmax confidence merely reflects momentary distribution sharpness rather than cumulative stability, this module extracts a trajectory state vector \(z_{k,t} = \phi(B_{k,1:t})\) capturing temporal dynamics across the entire prefix history. The state vector incorporates: instantaneous statistics (maximum confidence \(c_t\), entropy \(H_t\), top-two margin \(m_t\)), temporal derivatives between consecutive prefixes (mean, maximum, and latest deltas \(\Delta c_i, \Delta H_i, \Delta m_i\)), adjacent distribution Jensen-Shannon divergence \(JS(p_{k,i}, p_{k,i-1})\), label-switching count \(S_t = \sum_{i=2}^t \mathbb{I}(\hat{y}_{k,i}^{on} \ne \hat{y}_{k,i-1}^{on})\), top-1 persistence ratio \(P_t = \frac{1}{t}\sum_{i=1}^t \mathbb{I}(\hat{y}_{k,i}^{on} = \hat{y}_{k,t}^{on})\), area under the entropy curve \(\text{AUC}_H(t)\), and observation ratio \(r_t\).
The trajectory representation \(z_{k,t}\) is fed into a shared-MLP dual-head network: the risk head \(q_\psi(z_{k,t})\) estimates the probability that the online prediction is incorrect (\(e_{k,t} = \mathbb{I}(\hat{y}_{k,t}^{on} \ne y_k)\)), while the gain head \(g_\omega(z_{k,t})\) estimates whether contextual reinterpretation can successfully rectify the error (\(b_{k,t} = \mathbb{I}(\hat{y}_{k,t}^{on} \ne y_k \land \hat{y}_k^{rev} = y_k)\)). The training objective combines both objectives with dynamic weighting:
where \(w_{k,t}\) prioritizes samples that genuinely benefit from reinterpretation over those corrupted by multimodal ambiguity. Finally, an online policy head \(f_\theta\) maps \((z_{k,t}, q_\psi, g_\omega)\) to an action distribution \(s_{k,t} = \text{softmax}(f_\theta(\cdot))\), selecting among "commit", "wait", and "reinterpret" via \(\arg\max_{a} s_{k,t}(a)\).
3. Contextual Belief Reinterpretation: Multimodal Fusion and Explainable Attribution for Ambiguous Trajectories
Addressing challenging cases involving sarcasm, subtle micro-expressions, or conflicting vocal cues that leave the online trajectory in persistent instability, the system selectively activates Qwen2.5-Omni-7B. Unlike standard offline classification, this module does not discard early processing; it ingests the accumulated prefix belief trajectory \(B_{k,1:t}\) as an explicit reference alongside the target utterance audio \(a^{(k)}\), visual frames sampled at 2 fps \(o^{(k)}\), transcripts \(s^{(k)}\), preceding utterance context \(V_{k-1}\), and historical dialogue memory \(M_{k-1}\):
The output comprises the revised emotion label \(\hat{y}_k^{rev}\), an explicit natural language rationale \(R_k\) grounding the decision in observable audio-visual evidence, and a condensed contextual summary \(M_k\) updating conversational memory for future utterances. Conditioning the model on the prior belief trajectory enables it to inspect why the online pathway struggled (e.g., resolving ambiguity between anger and anxiety), while forced rationale generation prevents ungrounded hallucinations by anchoring decisions to visual gaze, facial tension, and conversational subtext.
A Worked Example¶
In a conversational scenario from StreamMER, the speaker Erica confronts Joey: "Who? Sabrina. I know about you two! I saw you today, kissing in the doctor's lounge." - Online Prefix Belief Formation: During early audio prefixes (1.5sโ3.5s), Erica's high-pitched, rapid speech initially skews the model toward Joy, with probability outputs of 0.46, 0.77, 0.71, and 0.45. As she delivers the accusatory statement around 4.5sโ6.5s, the prefix prediction abruptly flips to Surprise (probabilities 0.62, 0.55, 0.51). - Trajectory Reliability Estimation: The trajectory monitor registers a label switch (\(S_t = 1\)), a collapse in top-1 persistence to 0.57, and heightened Jensen-Shannon divergence across neighboring distributions. The risk head outputs a high failure risk \(q_\psi = 0.82\), while the gain head outputs \(g_\omega = 0.78\). The policy halts immediate commitment and triggers contextual reinterpretation. - Contextual Belief Reinterpretation: The reinterpretation module receives the fluctuating Joy-Surprise trajectory, the complete audio, the sampled visual frames (depicting Erica's direct confrontational stare and rigid posture), and the dialogue context. The model generates the rationale: "Accusatory tone, direct gaze, and confrontational statement," successfully overrides the online prediction to output Anger, and updates dialogue memory with her jealousy over Joey's fictional romance.
Loss & Training¶
TRACE utilizes an isolated multi-stage training pipeline to avoid data leakage:
1. Backbone Fine-Tuning: Both Qwen2.5-Omni-3B (online belief) and Qwen2.5-Omni-7B (reinterpretation) are fine-tuned using LoRA with rank \(r=16\), scaling factor \(\alpha=32\), and dropout 0.05 across attention and feed-forward projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj). Training runs for 10 epochs on 8 GPUs via DeepSpeed using AdamW with learning rate \(8 \times 10^{-6}\), weight decay 0.01, and cosine learning rate scheduling.
2. Reliability Model Training: To strictly prevent evaluation leakage, trajectory states and risk/gain labels are computed using \(K\)-Fold out-of-fold predictions on the training set. The reliability MLP has a hidden dimension of 128, dropout 0.3, and batch size 128, optimized via AdamW (learning rate \(1 \times 10^{-3}\), weight decay \(1 \times 10^{-3}\)) with early stopping after 30 epochs of validation stagnation.
Key Experimental Results¶
Main Results¶
Evaluations were conducted on the streaming video benchmark StreamMER (derived from narrative scenes of Friends with preserved conversational context) alongside public affective benchmarks MELD and MER2024 under strict streaming protocols.
| Method | StreamMER Acc | StreamMER W-F1 | StreamMER UAR | MELD Acc | MELD W-F1 | MELD UAR | MER2024 Acc | MER2024 W-F1 | MER2024 UAR |
|---|---|---|---|---|---|---|---|---|---|
| Gemini-3-Flash | 57.98 | 56.68 | 58.59 | 50.46 | 50.34 | 36.75 | 77.07 | 78.17 | 75.71 |
| Gemini-3-Pro | 63.08 | 62.84 | 66.91 | 61.34 | 62.27 | 45.62 | 86.82 | 86.95 | 84.68 |
| Emotion-LLaMA | 31.26 | 29.18 | 27.18 | 28.85 | 24.10 | 33.61 | 43.28 | 45.73 | 50.99 |
| AffectGPT | 53.66 | 49.86 | 48.51 | 55.65 | 51.33 | 56.67 | 78.80 | 82.46 | 75.74 |
| Emotion-Qwen | 59.51 | 58.39 | 48.45 | 51.76 | 52.04 | 38.51 | 82.12 | 82.50 | 80.77 |
| ViDEmo | 55.46 | 58.10 | 32.25 | 57.13 | 59.20 | 46.05 | 66.55 | 66.92 | 66.98 |
| Online Belief Formation Only | 59.87 | 54.56 | 52.58 | 58.94 | 56.08 | 38.67 | 81.18 | 80.09 | 75.29 |
| Full Contextual Reinterpretation | 70.18 | 70.01 | 66.47 | 70.30 | 68.90 | 51.51 | 87.42 | 87.09 | 85.71 |
| TRACE (Ours) | 69.47 | 68.10 | 63.80 | 68.28 | 70.14 | 41.42 | 85.46 | 85.11 | 83.04 |
Streaming Decision Strategy Comparison¶
A comparative evaluation of streaming decision rules on StreamMER demonstrates the trade-off among recognition accuracy, latency, and invocation cost.
| Strategy | Acc (%) โ | W-F1 (%) โ | Avg. Decision Time (s) โ | NDR (%) โ | Reinterpretation Rate RR (%) โ |
|---|---|---|---|---|---|
| Early Commit (at 1.5s prefix) | 57.98 | 57.38 | 1.49 | 50.77 | - |
| Full Commit (full utterance audio) | 58.98 | 58.41 | 3.98 | 100.00 | - |
| Adaptive Commit (TRACE steps without reinterpretation) | 59.65 | 58.99 | 3.10 | 87.66 | - |
| Always Reinterp. (invoke on all samples) | 70.18 | 70.03 | 4.05 | 100.00 | 100.00 |
| TRACE (Ours) | 69.47 | 68.10 | 3.10 | 87.66 | 55.58 |
| Oracle Reinterp. (theoretical upper bound) | 75.50 | 75.25 | - | 100.00 | 16.52 |
Ablation Studies¶
1. Reliability Features and Estimation Models (StreamMER)¶
Comparison between single-prefix confidence versus trajectory-aware features under linear thresholding versus the dual-head reliability model:
| Reliability Method | Acc (%) โ | Macro-F1 (%) โ | Reinterpretation Rate RR (%) โ |
|---|---|---|---|
| Confidence Linear | 66.87 | 64.30 | 69.33 |
| Confidence Reliability | 67.98 | 65.87 | 64.66 |
| Trajectory Linear | 68.83 | 65.79 | 60.12 |
| Trajectory Reliability (TRACE) | 68.83 | 65.40 | 55.71 |
2. Fast-Path Modality Selection¶
Evaluation of input modality choices for the online fast path on StreamMER:
| Fast-Path Modality | Online Acc | Online W-F1 | Online RTF | TRACE Acc | TRACE W-F1 | TRACE RR (%) | TRACE RTF |
|---|---|---|---|---|---|---|---|
| Audio | 59.87 | 54.56 | 0.055 | 69.47 | 68.10 | 55.58 | 0.091 |
| Visual | 42.45 | 36.26 | 0.081 | 66.26 | 64.95 | 64.17 | 0.113 |
| Audio + Visual | 60.74 | 54.31 | 0.083 | 69.57 | 67.96 | 56.07 | 0.119 |
3. Rationale Generation and Memory Summary Ablation¶
Impact of rationale prediction and contextual memory in Contextual Belief Reinterpretation:
| Reinterpretation Setting | Acc (%) | W-F1 (%) | UAR (%) | Note |
|---|---|---|---|---|
| Full Reinterpretation (TRACE) | 70.18 | 70.00 | 58.29 | Jointly predicts label, rationale, and memory |
| w/o Memory Summary | 66.01 | 65.69 | 61.81 | Removes historical dialogue summary, drops 4.17% Acc |
| w/o Rationale | 63.31 | 61.84 | 55.31 | Removes step-wise explanation, drops 6.87% Acc |
Key Findings¶
- Belief trajectories provide critical reliability signals beyond static confidence: When online prefix predictions exhibit zero label switches, online audio accuracy reaches 72.6%. When trajectories suffer 1 or 3 label switches, online accuracy drops precipitously to 38.3% and 24.1%. Contextual reinterpretation on these unstable subsets achieves 74 and 9 positive corrections (versus only 19 and 2 negative regressions), validating trajectory instability as a powerful trigger for contextual reasoning.
- Audio-first streaming achieves superior latency-utility trade-offs: Injecting visual frames into every prefix increases the online RTF from 0.055 to 0.083, yet yields only an imperceptible 0.10% gain in final TRACE accuracy (69.47% vs 69.57%), confirming that reserving vision for selective reinterpretation is vastly more efficient.
- Substantial cost reduction with minimal accuracy degradation: TRACE preserves 99.0% (StreamMER), 97.1% (MELD), and 97.8% (MER2024) of the accuracy attained by full contextual reinterpretation, while slashing reinterpretation invocations by nearly half (55.58%, 54.63%, and 58.21% reinterpretation rates) and reducing the RTF to 0.091.
Highlights & Insights¶
- Asymmetric Evidence Trajectory Architecture: Leveraging audio signals for rapid, low-latency belief formation while deploying heavyweight vision-language models selectively as on-demand contextual arbiters resolves the core bottleneck of multimodal streaming computation.
- Trajectory Dynamics over Point Estimates: Moving beyond static Softmax probabilities to track temporal divergence, entropy integrals, and switching frequencies provides an effective defense against deceptive early confidence peaks caused by acoustic noise.
- Dual-Head Risk-Gain Decoupling: Jointly estimating error risk and contextual correction gain prevents wasteful reinterpretation calls on inherently irresolvable or label-ambiguous segments where extra context offers no discriminative benefit.
Limitations & Future Work¶
- Sensitivity of Calibration Distributions to Domain Shifts: The reliability MLP is trained on in-distribution belief trajectories; performance may degrade under extreme acoustic shifts, differing conversational pacing, or cross-cultural linguistic variances.
- Inference Latency during Reinterpretation: Although average RTF remains 0.091, reinterpreting an utterance via the 7B multimodal backbone introduces ~0.89s of wall-clock latency, which may still exceed tolerances in hard real-time safety scenarios.
- Neglect of Immediate Visual Salience in the Fast Path: If an interlocutor displays an exaggerated facial emotion (e.g., intense silent crying or gasping) while pausing speech, the audio-only online path may remain uninformative or momentarily misdirected. Future work could integrate sparse, ultra-lightweight keyframe glimpses into the fast path.
Related Work & Insights¶
- vs Offline Multimodal Affective LLMs (AffectGPT / Emotion-Qwen / ViDEmo): Offline models require fully observed video segments and transcripts before inference. TRACE is complementary: it acts as a real-time gatekeeper, filtering out simple cases via audio beliefs and conditionally activating full multimodal models only when ambiguity arises.
- vs Streaming Video LLMs (StreamingVLM / Flash-VStream / ViSpeak): Existing streaming video architectures focus heavily on continuous action recognition or open-ended instruction following. TRACE pioneers trajectory-calibrated reliability tracking specifically tailored for psychological and affective understanding.
Rating¶
- Novelty: โญโญโญโญโญ Formulating streaming emotion recognition as dynamic belief tracking coupled with trajectory-calibrated reliability routing is conceptually elegant and original.
- Experimental Thoroughness: โญโญโญโญโญ Introduces the dedicated StreamMER benchmark with thorough ablations covering modalities, context scopes, noise robustness, and decision strategies.
- Writing Quality: โญโญโญโญโญ Clearly structured, mathematically rigorous, and supported by concrete qualitative case studies.
- Value: โญโญโญโญโญ Offers an actionable blueprint for real-world affective computing in conversational AI, virtual agents, and interactive systems.