Skip to content

๐Ÿ“น Video Understanding

๐Ÿ’ฌ EMNLP2026 ยท 17 paper notes

๐Ÿ“Œ Same area in other venues: ๐Ÿง  NeurIPS2026 (41) ยท ๐ŸŽž๏ธ ECCV2026 (111) ยท ๐Ÿ“ท CVPR2026 (190) ยท ๐Ÿ”ฌ ICLR2026 (48) ยท ๐Ÿงช ICML2026 (17) ยท ๐Ÿค– AAAI2026 (27)

๐Ÿ”ฅ Top topics: Compression ร—4 ยท Reasoning ร—3 ยท Question Answering ร—3

CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales

Addressing the critical limitations of existing video captioning benchmarksโ€”short durations, static scenes, and neglect of identity trackingโ€”CapRiCorn-1K introduces a 1,000-video benchmark spanning 15 seconds to 10 minutes with rich scene transitions and clothing variations, proposing a novel Rand-Index-inspired metric to evaluate subject referential consistency alongside fine-grained keypoint coverage across audiovisual and visual-only settings.

Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

Caption-once, Frames-on-Demand (CFD) introduces a budget-aware edge-cloud agentic framework for long-video understanding that constructs reusable dual-track narrative memories offline (event skeleton plus clip micro-logs) and deploys an online Visual-Need Router to fetch bounded keyframes only when perceptual questions require direct pixel evidence, reducing per-query frame consumption to 5.8 frames on average while achieving competitive accuracy on Video-MME (67.5) and InfiniBench.

CausalChapter: Improving Long-Form Video Chaptering with Interventional Dependency Modeling

To tackle boundary error propagation and cross-segment context fragmentation in scalable segment-then-caption long-form video chaptering, CausalChapter introduces intervention-inspired predictive dependency modeling via Local Dependency Shift (LCDS) to detect smooth boundary transitions and Cross-Segment Support Selection (CSSE) to rerank historical contexts by output KL divergence shift, substantially outperforming prior state-of-the-art systems on AVLecture and VidChapters-7M.

CoFIE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

To tackle high front-end vision encoding costs and the limited latency gains of downstream token pruning in streaming video understanding, CoFiE decouples evidence selection into a pre-encoder coarse filter using grayscale histogram novelty and an LLM prefill fine ranker using text-to-vision attention, achieving up to a 2.54ร— end-to-end speedup while outperforming the full-frame baseline.

Emotion Understanding in Streaming Video with Trajectory-Aware Reliability

Addressing the vulnerability of isolated prefix predictions and deceptive single-prefix confidence in streaming video, this paper proposes TRACE, a trajectory-aware reliability framework that continuously tracks emotion belief trajectories from lightweight audio streams, evaluates decision risk via a two-head reliability model, and selectively triggers multimodal contextual reinterpretation, matching full-context performance while reducing reinterpretation calls to 55.58%.

HIPPO: Accelerating Video Large Language Model Inference via Holistic-aware Parallel Speculative Decoding

Targeting the visual token redundancy and draft latency bottlenecks in video-LLM inference, HIPPO integrates a multi-dimensional semantic-aware token preservation strategy with an overlapping video parallel speculative decoding algorithm, achieving up to 3.51ร— lossless inference speedup across diverse video benchmarks.

Long-to-Short Video Evidence Reasoning for Grounded Question Answering

Addressing temporal evidence duration imbalance and the optimization conflict between strict boundary overlap and answer reasoning in grounded video QA (GQA), this paper introduces LOVER, an RL post-training framework combining a long-to-short evidence curriculum, an evidence-spotting IoP reward, and adaptive timestamp rendering to achieve new state-of-the-art results among open-source models on NExT-GQA and ReXTime.

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

To tackle the challenge of sparse, scattered spatial cues being diluted by redundant visual tokens in long videos, this paper proposes LongSpaceโ€”a unified framework combining 3D Spatial Structure Perception (SSP) and Hierarchical KV Memory (HKM)โ€”alongside LongSpace-Bench, a benchmark spanning seconds to hours of indoor room tours, setting strong state-of-the-art results in long-horizon spatial recall.

MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Agentic Hierarchical Graph Memory

MemDreamer decouples long-video understanding into streaming construction of a pure-text Hierarchical Graph Memory and tool-augmented agentic retrieval via an Observation-Reason-Action loop, establishing new SOTA across four benchmarks with a 40โ€“124ร— smaller reasoning context.

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

Addressing the challenge that key evidence in long videos is sparse, brief, and temporally scattered while isolated frames lack coherent semantics, MemoryCard segments videos into topic-coherent sessions via VLM self-reading, renders event gists, speech transcripts, and representative frames into self-contained "Memory Cards", and performs question-conditioned retrieval with adaptive resolution allocation, boosting QA accuracy by up to 21.8% under comparable visual token budgets.

Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing

EBM-RL decouples video role-playing generation into an explicit See-Think-Speak pipeline and optimizes each stage via four complementary rewards under GRPO, eliminating "atmosphere-blind" responses and achieving situational persona consistency.

Self-Supervised On-Policy Distillation for Omni-Modal Video Captioning

SODA introduces a self-supervised on-policy distillation framework for omni-modal video captioning that exploits intrinsic task and input-budget asymmetries, auditing low-budget student drafts via dual-stream QA and distilling high-budget teacher revisions back into the student through token-level KL divergence without external teachers or human annotations.

Streaming Atomic Action Segmentation in Egocentric Videos via Iterative Boundary Refinement

This paper proposes CoTT (Chains of Temporal Transitions), an annotation-free framework that extracts dense atomic action pseudo-labels from unlabeled egocentric videos via decoupled LVLM transition captioning, language-based grouping, and visual boundary refinement, and distills them into a lightweight causal online model (CoTT-Lite) whose predictions iteratively bootstrap subsequent training rounds to surpass fully supervised online action localization baselines.

TempCore: Are Video QA Benchmarks Temporally Grounded?

The paper introduces Frame Selection Sensitivity (FSS) based on the contrast between most and least relevant frames and the Language Independence Score (LIS), auditing six popular Video QA benchmarks to reveal that merely 5.5%โ€“31% of samples genuinely demand temporal frame selection, and constructs the compact, temporally grounded benchmark subset TEMPCORE without manual re-annotation.

ToM-Video: A Theory-of-Mind Video Benchmark with Spatio-Temporal Grounded Reasoning

This paper introduces ToM-Video, a large-scale spatio-temporal grounded Theory-of-Mind benchmark comprising 1,487 real-world narrative videos and 41,206 multiple-choice questions, and develops INSPIRE, a three-stage curriculum post-training framework that progressively aligns format compliance, spatio-temporal evidence grounding, and causal mental-state reasoning, substantially narrowing the gap between open-source models, proprietary frontier systems, and human experts.

VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration

Addressing the superficial shortcuts of multiple-choice QA and the sparsity of human caption matching, this paper introduces VidOmni-Bench, a sentence-wise event verification benchmark spanning 5 spatio-temporal complexity types and durations from 4 seconds to 90 minutes that uses human-verified Video-LLM hallucinations as hard negatives, systematically diagnosing temporal reasoning bottlenecks and self-preference bias.

VISA: Video Skeleton-Aware Efficient Frame Selection for Long Video Understanding

VISA utilizes sparse native I-frames from video Group-of-Pictures (GOP) structures to build a query-agnostic temporal skeleton, aggregates hierarchical query relevance, allocates frame budgets via dynamic programming, and refines candidates across neighboring GOPs, reducing end-to-end latency by 32.9%โ€“53.6% while outperforming prior selection baselines.