Towards Long-Form Spatio-Temporal Video Grounding¶

Conference: ECCV 2026
arXiv: 2602.23294
Code: https://github.com/HengLan/ART-STVG
Area: Video Understanding
Keywords: Spatio-Temporal Video Grounding, Long Video Understanding, Autoregressive Transformer, Memory Selection, Cascaded Decoding

TL;DR¶

This paper formalizes and explores the Long-Form Spatio-Temporal Video Grounding (LF-STVG) task for the first time. It proposes ART-STVG, a memory-augmented autoregressive Transformer architecture that processes videos frame-by-frame as streaming inputs. By employing spatial/temporal memory banks with selective retrieval strategies and a cascaded spatio-temporal decoding design, ART-STVG significantly outperforms existing methods on 1 to 5-minute long video grounding tasks without relying on additional long-form video training data.

Background & Motivation¶

Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube (i.e., bounding boxes of the target object across frames and the start/end times of the corresponding event) given an untrimmed video and a natural language query. Over the past few years, methods like TubeDETR, STCAT, CG-STVG, and TA-STVG have made significant progress. However, existing works almost exclusively focus on short videos of tens of seconds—with average durations in popular benchmarks like HCSTVG-v2 and VidSTG being only 20 and 35 seconds, respectively. Real-world applications, such as video retrieval, surveillance, and sports analysis, often involve videos spanning several minutes or even hours. Moving from 20-second clips to minute-level videos introduces three major challenges: first, loading all frames into GPU memory at once for joint localization causes severe memory bottlenecks; second, a longer temporal span drastically increases localization complexity; third, abundant irrelevant background information distracts the model from identifying the target event.

The authors point out a key conflict: existing STVG methods invariably adopt a non-autoregressive "look at all frames \(\rightarrow\) predict all at once" paradigm. Under this paradigm, models must compute the spatio-temporal relations across hundreds of frames simultaneously, making it hard to scale computationally or capture long-range spatio-temporal dependencies in long videos. Conversely, long videos are inherently streaming in nature—the model does not need to see the entire content at once. Instead, it can observe frame-by-frame and incrementally build memory of the target, much like a human observer. Core Idea: Treating the video as a streaming input, the paper proposes a memory-augmented autoregressive Transformer architecture (ART-STVG). It sequentially processes the video frame-by-frame and maintains cross-frame consistency through selective spatial and temporal memory. Concurrently, spatial localization results are cascaded into the temporal localizer, leveraging fine-grained target information to assist complex temporal localization.

Method¶

Overall Architecture¶

ART-STVG consists of a multimodal encoder and a set of cascaded autoregressive decoders. For the \(i\)-th frame of a video, the encoder extracts appearance, motion, and text features, merging them into multimodal features. The decoder pipeline first runs a spatial decoder to predict the bounding box of the current frame, and then feeds the RoI motion features of this box into the temporal decoder to predict the event start/end probabilities. Both spatial and temporal decoders maintain respective memory banks, accumulating historical information frame-by-frame and selectively retrieving only the most relevant memories for the current frame to enhance decoding. This sequential process iterates frame-by-frame as memory banks update, enabling infinite scaling for long video inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video Stream<br/>Frame-by-frame Input"] --> B["Multimodal Encoder<br/>ResNet-101 + VidSwin<br/>+ RoBERTa + Feature Fusion"]
    B --> C["Spatial Decoder<br/>with Selective Spatial Memory"]
    C --> D["Predict Bounding Box<br/>SpatialHead"]
    D --> E["RoI Pooling<br/>Extract Target Motion Features"]
    E --> F["Temporal Decoder<br/>with Selective Temporal Memory"]
    F --> G["Predict Event Start/End Probabilities"]
    G -->|Update Memory Banks| H["Spatial/Temporal Memory Banks<br/>Accumulate History Information"]
    H -->|Next Frame| B

Key Designs¶

1. Streaming Autoregressive Decoding: Frame-by-Frame Prediction over One-Time Global Prediction

Existing STVG methods process all frames of a video at once, which is memory-constrained on long videos (processing 284 frames requires ~80G GPU memory) and struggles to capture long-range spatio-temporal dependencies. ART-STVG treats videos as streaming inputs. After extracting multimodal features for each frame independently, it sequentially predicts the frame's bounding box and event probabilities using an autoregressive decoder, updating the memory bank with the current frame for subsequent ones. The core advantage is that inference memory overhead does not grow with video length (requiring only ~7.9G), theoretically scaling to videos of arbitrary length—an hour-long video requires only 22.5MB of memory storage. Ablation studies show that the autoregressive baseline (without memory) achieves much better m_tIoU on long videos (16.2% \(\rightarrow\) 9.2%) compared to traditional global prediction methods (TubeDETR achieves only 7.8% on 5-minute videos), proving the necessity of the streaming paradigm.

2. Selective Spatial Memory: Text-Guided Retrieval of Historical Target Positions

The spatial decoder maintains a spatial memory bank \(B^s\) with \(K\) partitions (one for each decoder block). While decoding the \(i\)-th frame, the current spatial query \(q_i^{k-1}\) is inserted into the corresponding memory partition. Then, the cosine similarity between each memory item and the text feature is calculated to select the top-\(N_s\) memories (\(N_s=32\) is optimal) to form the selective spatial memory \(\mathcal{M}^s_{i,k}\). The spatial query first performs cross-attention with \(\mathcal{M}^s_{i,k}\) to yield memory-enhanced query features, followed by cross-attention with the current frame's appearance and text features. The design motivation is straightforward: historical memories vary in importance for localizing the target in the current frame—excessively distant memories or those after dramatic scene changes can introduce noise. Filtering memories at the semantic level using text queries effectively screens out irrelevant history. Experiments show that using all spatial memories only improves m_tIoU by 0.8% (21.3% \(\rightarrow\) 22.1%), whereas incorporating the selection strategy yields an additional 0.9% gain (22.1% \(\rightarrow\) 23.0%). Attention maps further verify that selective memory guides the model to focus more on the target object.

3. Selective Temporal Memory: Event Boundary-Driven Filtering of Temporal Context

The temporal decoder similarly maintains a temporal memory bank \(B^t\), but utilizes a completely different memory selection strategy. Since temporal localization centers on locating the boundaries of the target event, the authors draw inspiration from TextTiling: they compute the cosine similarity between temporal memories of adjacent frames, and sudden drops in similarity (below a threshold \(\theta_t=0.8\)) are identified as event boundaries. The model then selects only the memories from the most recent event segment relative to the current frame to form the selective temporal memory \(\mathcal{M}^t_{i,k}\). The elegance of this design lies in its ability to automatically segment video events solely through changes in memory vector similarity, requiring zero event-level annotations. Long videos often present multiple unrelated events (e.g., "a person enters a room, sits down, and opens a laptop" contains three sub-events). Retrieving all temporal memories indiscriminately floods the model with distracting info from irrelevant events—ablation studies reveal that using entire temporal memories causes m_tIoU to drop from 16.7% to 9.6%, while applying the selection strategy pushes it back to 23.0%, a huge 13.4% absolute gain.

4. Cascaded Spatio-Temporal Decoding: Assisting Temporal Localization with Spatial Target Refinement

Unlike prior methods that execute spatial and temporal decoding in parallel, ART-STVG adopts a cascaded design. After the spatial decoder predicts the current frame's bounding box \(b_i\), RoI Pooling is applied to the corresponding motion feature map to extract fine-grained object-level motion features \(\bar{f}^m_i\). This refined feature (rather than the full-frame motion feature) is passed alongside text features to the temporal decoder. The intuition is that temporal localization in long videos is significantly harder than in short ones (due to the distraction of numerous irrelevant frames), and spatial localization offers a critical "where is the target" prior. Knowing the exact location of the target enables accurate analysis of when it appears or disappears. Ablation studies represent that the cascaded design outperforms the parallel alternative by 1.5% m_tIoU (21.5% \(\rightarrow\) 23.0%) and 2.8% [email protected] (17.3% \(\rightarrow\) 20.1%), demonstrating that spatial cues guide temporal localization effectively.

Loss & Training¶

The total loss consists of the spatial localization loss and the temporal localization loss:

\[\mathcal{L} = \lambda_k(\mathcal{L}_{KL}(\mathcal{H}_s^*, \mathcal{H}_s) + \mathcal{L}_{KL}(\mathcal{H}_e^*, \mathcal{H}_e)) + \lambda_l \mathcal{L}_1(\mathcal{B}^*, \mathcal{B}) + \lambda_u \mathcal{L}_{IoU}(\mathcal{B}^*, \mathcal{B})\]

The temporal branch employs KL-divergence to supervise the start/end probability distributions of the event, while the spatial branch uses Smooth L1 + IoU loss to supervise the bounding boxes. The loss weights are set to \(\lambda_k=10,\ \lambda_l=5,\ \lambda_u=3\).

Key Experimental Results¶

Main Results¶

Since existing long-video STVG benchmarks are absent, the authors extend the validation set of HCSTVG-v2 to evaluate three temporal scales (1, 3, and 5 minutes) on the LF-STVG benchmark. For a fair comparison, all models are trained solely on the original 20-second training set.

Benchmark	Method	m_tIoU	m_vIoU	[email protected]	[email protected]
LF-STVG-1min	TA-STVG (SOTA)	38.4	25.2	35.5	12.1
LF-STVG-1min	ART-STVG	39.1	26.1	36.8	17.6
LF-STVG-3min	TA-STVG (SOTA)	13.9	8.5	3.3	0.2
LF-STVG-3min	ART-STVG	23.0	15.3	20.1	9.5
LF-STVG-5min	TA-STVG (SOTA)	7.7	4.5	0.3	0.0
LF-STVG-5min	ART-STVG	15.0	10.0	11.4	4.7

On 3-minute and 5-minute videos, existing methods almost completely fail ([email protected] drops close to 0), whereas ART-STVG maintains reasonable localization accuracy, showing a more pronounced advantage as video length increases.

Ablation Study¶

Configuration	m_tIoU	m_vIoU	Description
Full (w/ All Designs)	23.0	15.3	Full Model
No Temporal Memory + No Selection	16.7	11.1	Baseline
W/ Temporal Memory + No Selection	9.6	6.2	Entire memory degrades performance
No Spatial Memory + No Selection	21.3	13.9	Spatial baseline
W/ Spatial Memory + No Selection	22.1	14.2	Minor improvement
Parallel Decoding (W/o Cascade)	21.5	13.9	Replace cascade with parallel

Key Findings¶

Temporal memory selection is the most critical contributor: Removing the selection strategy to use all temporal memories degrades performance significantly by 7.1% m_tIoU. Introducing the selection strategy brings a net increase of 13.4%, clearly demonstrating the pitfall of "excessive memory acting as noise" in long-form videos.
The marginal contribution of spatial memory is relatively small (only a 1.7% m_tIoU improvement), indicating that the spatial localization itself is already robust under the frame-by-frame autoregressive framework, where memory primarily serves a stabilizing role.
Cascaded design outperforms parallel design: Yielding gains of +1.5% in m_tIoU and +2.8% in [email protected], proving that leveraging spatial localization cues to assist temporal localization is particularly effective for long videos.
Exceptional GPU memory efficiency: ART-STVG requires only 7.9G of GPU memory during inference, less than one-third of TA-STVG's requirement (25.1G), and remains constant regardless of video length.

Highlights & Insights¶

First work to formally define and address the LF-STVG task, creating a complete closed loop from task definition and benchmark extension to methodological design. This provides highly valuable guidance to the community, as fine-grained spatio-temporal localization in long videos has been mostly unexplored.
The comparison of memory selection strategies is highly convincing: Using all memories without selection drops performance drastically (m_tIoU 16.7% \(\rightarrow\) 9.6%), whereas incorporating selection boosts it to 23.0%. This cleanly validates the intuition that "more memory is not always better in long videos."
A streaming inference paradigm with memory footprint independent of video length represents an ingenious system design, enabling the proposed method to process hour-long videos on consumer-level GPUs. This approach can be extended to other long video tasks (e.g., long video action detection, video paragraph captioning) as a plug-and-play inference framework.

Limitations & Future Work¶

Performance decay as video length increases: On 5-minute videos, m_tIoU drops to 15.0%. Although dramatically higher than existing methods (7.8%), there remains a substantial gap toward practical utility. The authors suggest that a more fine-grained, discriminative memory system is needed.
Inability to run in real-time: The inference speed stands at 1.09s per 64 frames (~18 FPS), which lies below real-time requirements. Model compression or quantization represents potential improvement avenues.
Failure under ambiguous event boundaries: When transition boundaries between adjacent events are faint or unclear, the similarity-based event segmentation strategy is prone to errors, causing incorrect temporal memory selection. Introducing external event-level signals or multimodal cues could address this.
Limitations in benchmark extension: The evaluation is only based on one dataset (HCSTVG-v2), and the long validation sequences are synthesized by random padding rather than capturing natural long-form videos, which might underestimate the complexity of real-world scenarios.

vs. TubeDETR / STCAT / CG-STVG: These methods employ a non-autoregressive, parallel spatio-temporal decoding paradigm that processes all frames simultaneously. While highly effective on short videos, they severely degrade in long videos due to extreme GPU memory usage and background noise. The autoregressive, streaming mechanics of ART-STVG bypasses this bottleneck structurally.
vs. General Long-Video Understanding Methods (e.g., MovieChat, VidLLaMA): These approaches typically resort to global video-level memory for video QA or captioning. In contrast, ART-STVG maintains text-guided spatial instance boundaries and temporal event clues, specifically designed for fine-grained localization.

Rating¶

Novelty: ⭐⭐⭐⭐ First systematic exploration of the long-form STVG task. The overall layout of autoregressive decoding, selective memory, and cascaded decoding is solid, although the individual modules are not purely pioneering concepts.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated against five other methods across three video length scales. The ablation studies cover all key designs, along with supplemental efficiency analysis and failure case analysis.
Writing Quality: ⭐⭐⭐⭐ Clear arguments, well-reasoned motivations, and thoroughly explained methods, though some notations and formulas could be simplified.
Value: ⭐⭐⭐⭐⭐ Identifies and defines a practically valuable new task. The proposed streaming paradigm exhibits clear efficiency benefits in long video localization, establishing a solid baseline and methodological groundwork for future directions.