Skip to content

📹 Video Understanding

🧠 NeurIPS2026 · 40 paper notes

📌 Same area in other venues: 🎞️ ECCV2026 (111) · 📷 CVPR2026 (190) · 🔬 ICLR2026 (48) · 🧪 ICML2026 (17) · 🤖 AAAI2026 (27) · 🧠 NeurIPS2025 (41)

🔥 Top topics: Compression ×6 · Anomaly Detection ×2 · Video Generation ×2 · Diffusion Models ×2 · Multimodal/VLM ×2

3DSPA: A 3D Semantic Point Autoencoder for Evaluating Video Realism

Addressing the challenge of evaluating generative video realism and physical plausibility without reference videos, this paper introduces 3DSPA—an autoencoder integrating 3D point trajectories, occlusion masks, and DINOv2 visual semantic features to quantify video realism via query reconstruction error (Average Jaccard), significantly outperforming 2D trajectory models and frontier vision-language models in physical rule violation detection and human alignment.

A Principled Optimal Transport Framework for Frame Selection in Long Video Understanding

Addressing the mismatch between long video length and limited visual context budgets as well as the heuristic nature of prior frame selectors, this paper introduces OTFS, a training-free framework that formulates frame selection as semi-unbalanced optimal transport of heterogeneous evidence atoms coupled with coverage-aware exact dynamic programming, substantially outperforming prior strong baselines on Video-MME, LongVideoBench, and MLVU.

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

By quantifying layer-wise representation divergence under frame order reversal, this paper reveals that VideoLLMs acquire temporal information at intermediate layers but allow it to fade toward the output, and proposes Temporal Activation Injection (TAI)—a training-free inference-time intervention that extracts and reinjects temporal vectors along the measured decay schedule to consistently enhance temporal reasoning.

Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception

Introduces Behavioral Geometric Supervision (BGS), a hybrid objective combining local triplet margin constraints and global representational similarity analysis (RSA) via low-rank adaptation (<2% parameters), which aligns video foundation models with human social perception, dramatically outperforming caption-based text embeddings and inducing emergent social-affective dimensions.

Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG

This paper reexamines scene segmentation as the chunking foundation for movie retrieval-augmented generation (Movie RAG), discovers that existing segmentation methods lag behind naive uniform chunking due to over-relying on superficial visual changes, and proposes NarraScene—a narrative-centric benchmark grounded in Event Segmentation Theory that demonstrates substantial gains on downstream reasoning.

CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Querying

Addressing the failure of existing video memory systems to distill long-horizon high-level semantics and recover lost fine-grained details during flat retrieval, CAM introduces evidence-driven asynchronous subgraph abstraction combined with a planner-executor-verifier workflow that adaptively orchestrates topological graph traversal and raw video/audio re-watching, boosting accuracy by up to 23.1 percentage points over SOTA baselines.

CoANeRV: Coordinate-Aware Token-Space Neural Video Representation

CoANeRV introduces a coordinate-aware token-space neural video representation framework that encodes a complete video clip into compact latent tokens via a feed-forward convolutional tokenizer and token former, and reconstructs continuous spatio-temporal queries with a shared coordinate decoder featuring axis-adaptive positional encoding, temperature-modulated cross-attention, and block-wise querying, completely removing per-video optimization while drastically lowering peak attention memory.

Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classification

Addressing the core dilemma that standard video transformers entangle concept channels and obscure attribution while static concept bottlenecks discard temporal dynamics, MoTIF introduces per-concept diagonal temporal self-attention and VLM-based concept discovery to achieve interpretable and intervenable video classification while bridging the gap to black-box baselines.

Conditional Multi-Event Temporal Grounding in Long-Form Video

To tackle the challenge of localizing and counting all events satisfying complex spatio-temporal conditions in long-form videos, this paper introduces CoMET-Bench along with a cheat-proof Rejection-F1 metric, and presents CoMET-Agent, a training-free search-and-aggregate agentic framework that significantly boosts multi-event temporal grounding precision.

Controllable Multi-label Video Safety Detection via Adaptive Tversky Policy Optimization

Addressing the limitations that existing video safety systems are confined to binary classification and lack precision-recall steerability, this paper proposes Adaptive Tversky Policy Optimization (ATPO), introducing feedback-driven Adaptive Tversky Rewards into multi-label GRPO to achieve dramatic performance gains and fine-grained, category-level operating point control.

COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection

To resolve the generalization gap and training–inference mismatch caused by static shared parameters in VLM-based video anomaly detection, COPRA trains a lightweight generator via reinforcement learning (GRPO) to predict instance-conditioned LoRA weights dynamically on the fly, delivering marked gains in both in-domain benchmarks and cross-domain video reasoning tasks.

CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding

Reframes KV cache compression in streaming video understanding as a training-free, query-agnostic coreset selection problem over joint KV representations with orthogonal anti-redundancy regularization, significantly outperforming heuristic token-pruning baselines and matching full-cache accuracy under tight memory budgets.

DeltaVid: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences

DeltaVid establishes a verifiable proxy-task framework by converting cross-video "spot-the-difference" into programmatic training signals; using controllable perturbation synthesis on real videos and rule-reward reinforcement learning (GRPO), it substantially improves spatiotemporal evidence localization and successfully transfers to general video understanding.

Don’t Pause! Every prediction matters in a streaming video

Addressing the flaw where current streaming benchmarks rely on retrospective "pause-ask-play" setups and overlook idle dead-time, this paper introduces SPOT-Bench—the first benchmark for continuous proactive streaming prediction—along with a time-sensitive metric Timeliness-F1, and proposes AsynKV, an asynchronous training-free KV cache adaptation that suppresses response echoing by utilizing dead-time.

EgoStream: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

EgoStream introduces a diagnostic benchmark for streaming episodic memory in egocentric vision with an Answer Validity Window (AVW) to isolate genuine model forgetting from world-state changes, accompanied by a unified streaming MLLM framework evaluating key KV-cache memory policies across seven cognitive dimensions.

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

FramePack-U introduces a training-free, content-aware frame packing framework that compresses arbitrarily long videos into a fixed token budget with \(O(1)\) constant memory complexity, dynamically allocating multi-tier spatial resolutions based on query relevance and visual diversity.

EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment

Addressing the lack of structured event alignment in Omni Language Models that leads to audiovisual temporal mismatch and character misattribution, EVA-Cap constructs the 63K EVA-Data with event-hit quality control and introduces a two-stage progressive GRPO post-training framework optimizing event completeness, intra-event attribution, and inter-event synchronization.

FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

Addresses the visual token explosion in long video understanding by introducing FrameScout, a lightweight keyframe selector that builds temporally-contextualized frame embeddings via sliding-window KV caching and backward successor aggregation under constant memory, and aligns them using frame-query contrastive learning for decoupled multi-query retrieval.

FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

Drawing inspiration from the Weber-Fechner law of logarithmic perception and hippocampal Sharp-Wave Ripples (SWRs), FreshMem introduces a hybrid frequency-space memory network combining multi-scale recursive Fourier updates with adaptive episodic thumbnails, achieving superior streaming video reasoning without mandatory model retraining.

HAPACT: A Benchmark for Human-Centric Physical Impact Localization in Movies

Addressing the lack of fine-grained spatio-temporal representations for abrupt external physical events, this paper introduces the human-centric impact localization task along with HAPACT, the first movie-based benchmark with 24 FPS temporal and 3D SMPL-X spatial groundings, and proposes an MLLM region-selection baseline.

Harnessing Streaming Video in the Wild

This paper introduces an end-to-end stack—Streaming-Train-248K dataset with asymmetric control loss, the StreamingHarness inference framework supporting 12-hour context with sub-second latency, and the Streaming-Eval benchmark—establishing state-of-the-art performance across in-the-wild streaming video tasks.

iCATS: Fast Video Generation via Interaction-Aware Sparse Attention and Timestep-Adaptive Sparsity

iCATS is a training-free sparse attention acceleration framework for Diffusion Transformers (DiTs) that reformulates Query-Key interaction importance estimation into a low-cost quadratic form, applies an SNR-guided inverted-sigmoid dynamic sparsity schedule across timesteps, and adopts a hardware-level tail-merging strategy, achieving up to 2.03× end-to-end acceleration with state-of-the-art visual quality preservation.

LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute

LookWhen factorizes video recognition into a selector-extractor framework: a shallow selector scores space-time token uniqueness via nearest-neighbor distance (top1-distance) on a downscaled video, while a deep extractor processes only the top-K selected tokens via multi-teacher distillation with time-normalized features, outperforming dense teacher baselines in accuracy-throughput efficiency.

Masked Diffusion Vision-Language Models for Temporal Action Localization

MDVLM-TAL adapts masked diffusion vision-language models to temporal action localization, discarding left-to-right autoregressive constraints in favor of bidirectional iterative denoising while introducing a planned training objective and a step-level IoU reward to jointly refine action semantics and temporal boundaries.

MedHorizon: Towards Long-context Medical Video Understanding in the Wild

MedHorizon presents an in-the-wild benchmark comprising 759 hours of uncurated clinical procedures across 8 datasets to evaluate full-procedure medical video understanding; with an ultra-sparse evidence ratio of only 0.166%, extensive evaluation of 22 representative MLLMs reveals that the strongest system achieves merely 41.1% accuracy, highlighting non-monotonic frame scaling, dual bottlenecks in retrieval and clinical interpretation, and severe attention drift toward redundant anatomy.

MobileWan: Closing the Quality Gap for Mobile Video Diffusion

MobileWan deploys a server-scale 5-billion parameter video diffusion transformer (Wan2.2-5B) directly onto a commercial mobile NPU by combining noise-biased learnable head pruning, recurrent linear-softmax hybrid distillation, few-step sampling distillation, and a long-receptive-field VAE decoder, synthesizing 5-second 480×832 videos at 16 FPS in 20 seconds end-to-end latency.

Parameterized Stripe Attention for Efficient Video Generation

Addressing the substantial computational latency of full spatio-temporal attention in Diffusion Transformers (DiTs), this paper reveals the periodic diagonal stripe regularity of attention maps, formalizes a unified parameterized stripe attention (PSA) with an efficient single CUDA kernel, and introduces training-free offline search to achieve 1.37×–1.57× end-to-end speedups on Wan 2.1 and HunyuanVideo with negligible quality degradation.

Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

Addressing the blind spots of prior long-video VLMs that fixate on native resolution token filtering while ignoring spatial downscaling and front-end decoding latency, this paper presents LoHi, a training-free, single-pass framework that pairs a dense low-resolution video stream with a sparse high-resolution image stream, outperforming prior efficiency baselines by +5.2% on average under matched token budgets while cutting front-end decoding latency by up to 7× on hour-scale videos.

SLVMBench: Skill Learning from Video Memory

SLVMBench is the first benchmark designed to jointly evaluate whether video-LLMs can acquire procedural skills from tutorial videos embedded in up to 2-hour distractor streams and apply them to real-time predictive reasoning at precise sub-second decision points, revealing severe memory collapse across frontier models.

SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere

Addressing the conical collapse and cross-domain rotational offset of pre-trained multimodal LLM representations, SphereVAD recasts training-free video anomaly detection as von Mises–Fisher geodesic inference on the unit hypersphere, achieving massive zero-shot state-of-the-art gains with only minimal synthetic calibration samples.

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

StreamOV addresses the failure of offline omni-modal models in continuous streaming interactions by introducing dual-perspective decoupled multimodal evidence routing, a budget-constrained long-short term memory, and an intrinsic prefix hidden-state response trigger that eliminates auto-regressive silence tokens while setting new state-of-the-art results across streaming and offline benchmarks.

TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition

Addressing the objective inconsistency where fine-tuning is restricted to known training categories while evaluation targets unseen out-of-distribution classes, TACO preserves global representation geometry via Relative Structure Distillation with random hypersphere geometric anchors and decouples the representation space from optimization using a lightweight residual specialization projection, achieving state-of-the-art open-vocabulary generalization.

Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling

To overcome the lack of flexibility and noise susceptibility in hard keyframe selection for long video understanding, this paper proposes AdaQ, a training-free adaptive Quasi-Gaussian sampling approach that dynamically modulates the sampling temperature and \(3\text{-}\sigma\) interval via query-frame similarity variance, delivering substantial efficiency gains and outperforming existing keyframe selection methods and uniform sampling baselines across four MLLMs.

TrAction: Action Recognition with Sparse Trajectories

Addressing the heavy computational footprint and pronounced appearance/background shortcuts of dense RGB video models, TrAction proposes the first dedicated action recognition architecture operating purely on 2.5D sparse point trajectories via a lightweight Transformer, achieving strong temporal recognition accuracy without raw pixels and providing substantial complementary gains when fused with vision foundation models such as DINOv2 and V-JEPA 2.

Training-Free Temporal Abstraction for General Video Understanding

The paper introduces STITCH, a training-free temporal abstraction framework that encodes short video windows with a frozen InternVideo2 backbone, partitions the video timeline using kernel change-point detection, and produces reusable semantic chunks that directly serve event boundary detection, language moment retrieval, and budget-constrained frame selection for long-video VLMs.

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

TT-VidT reconstructs later frames using a first-frame anchor that retains full spatial information and compact learned motion tokens, substantially improving motion-sensitive tasks under a shared recipe of roughly 1.7 million videos and 8 epochs while reducing TT3D encoder compute to 456.1 GF, without establishing universal video-understanding superiority or strict appearance–motion disentanglement.

Video-Zero: Self-Evolution Video Understanding

Addressing the issue where full-video generation in video self-evolution exploits static cues and language priors, Video-Zero introduces an evidence-centered Questioner–Solver co-evolution framework that substantially boosts temporal grounding and video reasoning without any human annotations.

VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

To address the shallow reasoning bottleneck in long video evaluation where questions collapse into localized frame retrieval, this paper introduces VideoOdyssey—the first ultra-long-context and omni-modal benchmark quantifying cognitive load via continuous certificate length (average duration 109 minutes, average CCL 16 minutes)—revealing profound failures of frontier MLLMs in sustained reasoning, fine-grained perception, and non-verbal acoustics.

VideoSTF: Stress-Testing Output Repetition in Video Large Language Models

Introduces VideoSTF, the first comprehensive benchmark for stress-testing output repetition in VideoLLMs, revealing its pervasiveness across models, amplification under temporal perturbations, vulnerability to query-efficient black-box attacks, and underlying link to local temporal disruption.

ViMU: Benchmarking Video Metaphorical Understanding

Addressing the limitation that existing video models remain confined to literal perceptual reasoning without comprehending deeper subtext, this paper introduces ViMU, the first hint-free benchmark evaluating video metaphorical and social subtext understanding, where even frontier models achieve below 50% overall performance.