ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/TenMinutes209/ScanFocus
Area: Video Understanding
Keywords: Spatio-Temporal Video Grounding, Coarse-to-Fine, Multimodal Fusion, Temporal Aggregator, Deformable Attention

TL;DR

ScanFocus proposes a coarse-to-fine spatio-temporal video grounding framework that decouples the task into a global spatio-temporal scan and a local boundary focus, combining lightweight deformable semantic-motion fusion with a Semantic-Guided Temporal Aggregator (SGTA) to capture boundary cues with halved computation.

Background & Motivation

Spatio-Temporal Video Grounding (STVG) aims to retrieve the visual trajectory of a specific target (bounding box sequence and temporal start/end timestamps) in an untrimmed video given a natural language query. Recent Transformer architectures, such as TubeDETR, STCAT, and CG-STVG, have significantly advanced the field through unified end-to-end encoder-decoder pipelines. However, due to the prohibitive computational cost of processing long video streams, existing approaches predominantly resort to low-rate uniform temporal subsampling. This coarse sampling inevitably suppresses high-frequency temporal boundary cues and often skips frames around the true boundaries, fundamentally restricting the model from accessing precise boundary information and resulting in temporal boundary ambiguity.

A deeper limitation lies in the lack of explicit temporal modeling and the optimization bottleneck caused by tri-modal feature overloading. Prevailing frameworks build upon the static MDETR paradigm, forcing a heavy Transformer encoder to concurrently couple appearance, linguistic, and motion features in a shared latent space. This fully-coupled interaction fails to decouple the disparate feature representations required for spatial versus temporal localization. Furthermore, existing methods rely primarily on motion features implicitly extracted by backbones within individual frames, lacking dedicated mechanisms to capture explicit inter-frame dependencies during rapid action transitions. An oracle experiment shows that replacing predicted intervals with ground-truth timestamps boosts [email protected] from 42.2% to 86.9%, proving that temporal ambiguity—not spatial localization—is the primary bottleneck in STVG.

To resolve this conflict, inspired by human visual perception that executes a global glance followed by local scrutiny, ScanFocus decouples STVG into a global spatio-temporal scan and a local boundary focus. Core idea: decouple spatio-temporal video grounding into a sparse global scan for coarse tubelet/interval proposal generation and a high-rate local boundary focus, employing a Semantic-Guided Temporal Aggregator (SGTA) to explicitly model inter-frame dependencies and high-frequency motion transitions around boundary windows.

Method

Overall Architecture

ScanFocus operates through two cascaded stages: the Global Spatio-Temporal Scan stage and the Local Boundary Focus stage. In the coarse stage, the input video is sparsely sampled at a low frame rate. A unified vision-language fusion encoder (BEiT-3) and a frozen video motion encoder (VideoMAE) extract aligned features, which are cross-modally fused through a lightweight Deformable Semantic-Motion Fusion module. Dual DETR-style decoders then predict coarse spatial bounding box sequences and temporal intervals. In the refinement stage, dense frame sequences are sampled within local windows centered at the coarse boundaries; coarse bounding boxes are linearly interpolated to provide spatial priors, and the Semantic-Guided Temporal Aggregator (SGTA) models explicit inter-frame interactions before task-specific refine decoders predict fine-grained timestamps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Untrimmed Video + Text Query"] --> B["Global Spatio-Temporal Scan<br/>Low-rate uniform sampling of Tc frames"]
    B --> C["Deformable Semantic-Motion Fusion<br/>BEiT-3 + VideoMAE sparse multi-scale alignment"]
    C --> D["Dual Coarse Decoders<br/>Output coarse boxes Bc and interval (tcs, tce)"]
    D --> E["Boundary Dense Sampling & Spatial Prior<br/>High-rate local windows + linear interpolation Bst"]
    E --> F["Semantic-Guided Temporal Aggregator<br/>RoI pooling + semantic modulation + local MHSA"]
    F --> G["Dual Refine Decoders<br/>Output fine-grained timestamps & action confidence"]

Key Designs

1. Deformable Semantic-Motion Fusion: resolving tri-modal coupling bottlenecks

To alleviate the optimization bottleneck of standard Transformer encoders that concurrently crush appearance, motion, and language representations, ScanFocus employs a decoupled fusion strategy. Visual and textual features \(F_a \in \mathbb{R}^{T_c \times N_a \times C_a}\), \(F_l \in \mathbb{R}^{T_c \times L \times C_a}\), and context token \(F_c \in \mathbb{R}^{T_c \times 1 \times C_a}\) are extracted via a unified vision-language foundation model (BEiT-3) pre-trained on web-scale data, eliminating cross-modal discrepancies at the source. Motion features \(F_m\) are extracted via frozen VideoMAE. Projecting these features into a unified dimension forms a multi-level feature pyramid \(\mathcal{X} = \{F_a, F_m, F_t, F_c\}\), processed by multi-scale deformable attention:

\[\text{Fusion}(z_q) = \sum_{l=1}^{4} \sum_{k=1}^{K} A_{lqk} \cdot \Phi(\mathcal{X}^l; p_q + \Delta p_{lqk})\]

where \(p_q\) is the reference point, \(\Delta p_{lqk}\) represents learnable offsets, and \(A_{lqk}\) denotes attention weights. This formulation enables linguistic tokens to attend sparsely to salient visual and motion regions with linear computational complexity, filtering background noise and reducing fusion compute by 50%.

2. Boundary Dense Sampling and Spatial Prior Alignment: recovering suppressed high-frequency boundary cues

While sparse global sampling keeps long-video computation feasible, it discards essential action boundary cues. To recover fine-grained details without incurring full-sequence overhead, ScanFocus establishes local observation windows centered at predicted coarse boundaries \(t_c^s\) and \(t_c^e\). Within each window, \(N_w\) frames are densely extracted using a high sampling rate \(f_r = k_r \cdot f_c\) (\(k_r > 1\)), generating dense sequences \(V_{rs}\) and \(V_{re}\).

To bypass the heavy computational cost of deploying external object detectors in refinement, the model directly reuses coarse bounding box sequences \(B_c\). By slicing the subset corresponding to the local window and applying temporal linear interpolation, a dense spatial prior \(B_{st} \in \mathbb{R}^{N_w \times 4}\) is produced with negligible compute, providing accurate object localization priors across the dense boundary frames.

3. Semantic-Guided Temporal Aggregator: explicit modeling of short-term action transitions

Naively processing dense frames introduces redundant background motion and distraction from non-target entities. The Semantic-Guided Temporal Aggregator (SGTA) first isolates target-specific appearance representations via RoI Pooling guided by the spatial prior \(B_{st}\): \(F_{ps}^a = \text{ROIPool}(F_{rs}^a, B_{st})\). The motion features \(F_{rs}^m\) are then modulated by target appearance through Hadamard product to filter background dynamics, followed by text feature injection:

\[F_{sg} = (\phi_a(F_{ps}^a) \odot F_{rs}^m) \oplus \phi_t(F_{rs}^t)\]

To capture high-frequency transitions and establish explicit inter-frame dependencies, \(F_{sg}\) is flattened into a unified sequence of length \(N_w \times N_m\) and passed through stacked temporal self-attention layers: \(F_{rs}' = \text{MHSA}_{\times L_t}(\text{Flatten}(F_{sg}))\). This flattened local structure allows all tokens within the local boundary window to interact comprehensively, enabling the refine decoder to detect rapid state transitions and regress precise boundary probabilities.

Loss & Training

To protect well-learned global semantic priors from disturbance by fine-grained boundary gradients, ScanFocus utilizes a decoupled two-stage training scheme:

1. Coarse Stage: Jointly trains the feature encoders, fusion module, and coarse decoders using \(\mathcal{L}_c = \lambda_{box} (\mathcal{L}_{L1}(B_c, B^*) + \mathcal{L}_{IoU}(B_c, B^*)) + \lambda_{tmp} (\mathcal{L}_{KL}(\hat{P}_s, t_s^*) + \mathcal{L}_{KL}(\hat{P}_e, t_e^*))\), combining L1 and GIoU loss for spatial boxes and Gaussian-smoothed KL divergence for coarse intervals.
2. Refinement Stage: Freezes all coarse stage parameters and trains SGTA and the refine decoders exclusively. Boundary timestamp classification and action confidence are supervised using Binary Cross-Entropy loss: \(\mathcal{L}_r = \lambda_{ref} \mathcal{L}_{BCE}(P_{st/ed}, y_{st/ed}^*) + \lambda_{act} \mathcal{L}_{BCE}(A_{st/ed}, y_{act}^*)\).

Key Experimental Results

Main Results

ScanFocus is evaluated against leading state-of-the-art approaches across three standard benchmarks: HC-STVGv1, HC-STVGv2, and VidSTG. It exhibits significant improvements across all metrics, especially under strict localization thresholds ([email protected]).

Dataset	Metric	Ours	Prev. SOTA (TA-STVG)	Gain
HC-STVGv1 (Test)	m_tIoU	55.5	53.0	+2.5%
HC-STVGv1 (Test)	m_vIoU	41.8	39.1	+2.7%
HC-STVGv1 (Test)	[email protected]	67.5	63.1	+4.4%
HC-STVGv1 (Test)	[email protected]	42.2	36.8	+5.4%
HC-STVGv2 (Val)	m_tIoU	62.4	60.4	+2.0%
HC-STVGv2 (Val)	m_vIoU	41.7	40.2	+1.5%
HC-STVGv2 (Val)	[email protected]	68.4	65.8	+2.6%
HC-STVGv2 (Val)	[email protected]	39.3	36.7	+2.6%
VidSTG (Declarative)	m_tIoU	53.3	51.7	+1.6%
VidSTG (Declarative)	[email protected]	36.0	33.5	+2.5%
VidSTG (Interrogative)	m_tIoU	51.4	50.2	+1.2%
VidSTG (Interrogative)	[email protected]	29.4	28.0	+1.4%

Ablation Study

Ablations on HC-STVGv1 (input resolution 224×224) evaluate component contributions and fusion mechanisms.

Config	tIoU	vIoU	[email protected]	[email protected]	Note
Coarse (Baseline)	50.9	38.2	60.7	38.0	global coarse scan baseline
+ DS (Dense Sampling)	52.4	39.1	62.6	37.9	dense sampling + refine decoders without SGTA
+ SGTA (full model)	53.7	40.0	64.0	39.5	complete model with SGTA
SGTA w/o TA	52.8	39.3	63.1	38.7	removing explicit temporal self-attention
SGTA w/o SG	53.1	39.5	63.4	39.0	removing RoI appearance and text guidance
Standard Self-Attn Fusion	53.3	39.8	-	36.5	standard self-attention fusion (516 GFLOPs)
Deformable SM Fusion	53.7	40.0	-	39.5	deformable semantic-motion fusion (260 GFLOPs)

Key Findings

• Crucial role of SGTA: Adding dense sampling alone (+DS) improves tIoU from 50.9% to 52.4% due to higher frame density, but slightly degrades [email protected] from 38.0% to 37.9%, revealing that unmodulated dense frames introduce noise. Adding SGTA raises [email protected] to 39.5%, confirming that semantic modulation and explicit inter-frame modeling are vital for boundary refinement.
• Optimal window size \(N_w\): Testing window sizes \(N_w \in \{4, 6, 8, 10, 12\}\) demonstrates peak performance at \(N_w = 8\) (tIoU 53.7%, vIoU 40.0%). Smaller windows lack context, whereas larger windows (\(N_w > 8\)) incorporate irrelevant background frames that dilute boundary-specific cues.
• Halved computational overhead: Deformable Semantic-Motion Fusion slashes fusion complexity from 516 GFLOPs to 260 GFLOPs (a 50% reduction) while outperforming standard dense attention by 3.0% on [email protected], successfully suppressing background noise.

Highlights & Insights

• Decoupled coarse-to-fine paradigm hits the core bottleneck: Oracle analysis reveals that temporal ambiguity is the predominant limiter in STVG. The coarse-to-fine structure recovers suppressed boundary cues while circumventing the quadratic memory explosion of uniform dense sampling across entire videos.
• Linear interpolation avoids heavy detector overhead: Slicing and interpolating coarse spatial boxes to generate local dense spatial priors eliminates the need for expensive secondary object detectors in the refinement stage.
• Generalizable temporal refinement module: The pipeline of "dense window sampling + semantic-guided motion modulation + local temporal attention" is readily transferable to other fine-grained temporal tasks like Video Temporal Grounding (VTG) and Highlight Detection.

Limitations & Future Work

• Dependency on coarse proposal recall: If the initial coarse stage produces an interval that completely misses the true boundaries outside the local window, the refinement stage cannot recover from the severe false-positive localization.
• Two-stage isolated optimization: Freezing the coarse stage ensures stability but prevents joint end-to-end gradient feedback from the boundary refiner to the global feature encoder.
• Future directions: Adaptive dynamic window sizing based on predicted action duration, as well as end-to-end reinforcement learning fine-tuning, represent promising next steps.

Related Work & Insights

• vs TubeDETR / STCAT / CG-STVG: These methods uniformly downsample the entire video in a single stage, permanently forfeiting high-frequency boundary cues; ScanFocus recovers these cues via local dense focus while cutting multimodal fusion complexity by 50%.
• vs TA-STVG: While TA-STVG enhances target-aware spatio-temporal reasoning, it remains constrained by low-rate global downsampling; ScanFocus introduces local dense sampling and SGTA, obtaining 2.0%~5.4% gains under strict [email protected] thresholds.

Rating

• Novelty: ⭐⭐⭐⭐☆ Decouples STVG into a global scan and local boundary focus, effectively resolving the temporal ambiguity bottleneck.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across three benchmarks, accompanied by Oracle experiments, detailed ablations, parameter sensitivity tests, and FLOPs comparisons.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, lucid argumentation with strong empirical and conceptual alignment.
• Value: ⭐⭐⭐⭐⭐ Provides an efficient, highly practical design paradigm for precise multimodal video grounding.

Related Papers

• [ECCV 2026] Towards Long-Form Spatio-Temporal Video Grounding
• [CVPR 2026] SARL-STG: A Spatially Aware Reinforcement Learning Framework for Refining MLLMs in Spatio-Temporal Video Grounding
• [ICLR 2026] OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding
• [CVPR 2026] OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios
• [ECCV 2026] TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding