SV-TAD: Native Sparse Convolutions for Efficient Temporal Action Detection

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/pcr-upm/eccv26_tad
Area: Video Understanding
Keywords: Temporal Action Detection, Native Sparse Convolutions, Token Selection, Parameter-Efficient Fine-Tuning, Auxiliary Task Supervision

TL;DR

SV-TAD introduces native sparse 2D convolutions and an adapter architecture designed directly for dynamically pruned Vision Transformer token sequences, eliminating dense grid reconstruction overhead and reducing VideoMAEv2-L computation by 64% with a 2.2x inference speedup while preserving state-of-the-art detection accuracy.

Background & Motivation

In large-scale long video understanding and temporal action detection (TAD), input video sequences typically span hundreds to thousands of frames. Directly fine-tuning billion-parameter video foundation models such as VideoMAEv2 or InternVideoNext incurs prohibitive training and memory footprints. Recent parameter-efficient fine-tuning (PEFT) approaches freeze the backbone and insert lightweight convolutional adapter modules (such as ST-Adapter, LoSA, and AdaTAD) to achieve parameter efficiency during training. However, existing adapters must still compute over complete, dense token grids during inference, leaving computational complexity scaling linearly or quadratically with video sequence length, which fundamentally hampers long-video scalability.

Dynamic token selection methods (such as EViT, DynamicViT, and ToMe) effectively alleviate the self-attention bottleneck by discarding uninformative background patches based on attention weights. Nevertheless, token pruning shatters the regular 2D spatial grid topology. Existing convolutional adapters, feature pyramid networks, and local spatial operators inherently assume fixed spatial neighborhood structures. To run convolutions over pruned token sets, existing pipelines are forced to perform expensive scatter-gather dense reconstructions—scattering sparse tokens back onto an empty dense grid, running standard convolutions with zero-padding, and then gathering the retained features. This reconstruction step not only inflates memory to full-grid levels, but its overhead also largely offsets any speedup gained from attention sparsification. Meanwhile, 3D sparse convolutional libraries (such as MinkowskiEngine and Submanifold Sparse Conv) rely on hash-table coordinate management designed for point clouds, which is fundamentally incompatible with the dynamic layer-varying sparsity and compact packed-tensor format of Vision Transformers.

The authors observe that a local 2D convolution requires only the adjacency between each output position and its \(K \times K\) spatial neighbors, rather than the materialization of the full dense feature map. Core idea: construct a native sparse 2D convolution primitive (SparseConv2D) backed by precomputed neighbor index tables and custom CUDA kernels, enabling lightweight bottleneck adapters to execute directly and losslessly on dynamically pruned token sequences without dense grid reconstruction while naturally supporting auxiliary task token supervision.

Method

Overall Architecture

SV-TAD builds upon a frozen pretrained video Vision Transformer backbone (e.g., VideoMAEv2 or InternVideoNext). The end-to-end framework comprises stage-wise token selection modules, native sparse bottleneck adapters (SparseConv2D Adapters), and an ActionFormer temporal action detection head. The input consists of sampled long video frames alongside learnable auxiliary task tokens. The initial shallow layers retain the complete dense grid to establish robust low-level spatial features, whereas intermediate and deep layers progressively prune tokens via attention-guided selection. The resulting irregular sparse token sets are fed directly into adapters containing sparse spatial convolutions and an asymmetric cross-attention mechanism, before the refined representations enter the detection head to output action boundaries and classification scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long video frames + Auxiliary token Xaux"] --> B["Shallow Transformer blocks<br/>First 3 blocks processed densely"]
    B --> C["Attention-guided token selection<br/>Retain Nkept visual patches with grid indices"]
    C --> D["Precompute neighbor index table<br/>Two-step O(Nkept) lookup map and neighbor table"]
    D --> E["Native sparse 2D convolution<br/>SparseConv2D directly gathers via neighbor table"]
    E --> F["Asymmetric cross-attention<br/>Xaux queries sparse visual features for temporal context"]
    F --> G["ActionFormer detection head<br/>Predicts temporal action segments and category scores"]

Key Designs

1. Static Neighbor Index Table: Decoupling Spatial Convolutions from Dense Grid Materialization in \(O(N_{kept})\)

To allow standard \(3 \times 3\) convolutions to bypass the full \(H \times W\) spatial grid, the method leverages the fact that ViT patch coordinates already reside on a known regular 2D lattice. A neighbor index table \(\mathbf{N} \in \mathbb{Z}^{N_{kept} \times 9}\) is constructed to map each retained token to the indices of its spatial neighbors within the sparse token sequence. The construction executes in two lightweight steps: first, a dense 2D lookup array \(\mathbf{M} \in \mathbb{Z}^{H \times W}\) is initialized to \(-1\), and for each retained token \(k\) at spatial coordinate \((y_k, x_k)\), the mapping \(\mathbf{M}[y_k, x_k] = k\) is established; second, for each retained token \(k\), its 8 spatial neighbors are retrieved via directional offsets \((\delta_y, \delta_x)\):

\[ \mathbf{N}[k, j] = \begin{cases} \mathbf{M}[y_k + \delta_{y,j}, x_k + \delta_{x,j}], & \text{if neighbor is retained} \\ -1, & \text{if neighbor was pruned or is out-of-bounds} \end{cases} \]

Both steps run on the CPU with \(O(N_{kept})\) complexity in under 3ms, and the lookup array \(\mathbf{M}\) is reused across multiple adapter layers within the same pruning stage until the next selection point.

2. Native Sparse 2D Convolution Kernels: Dual-Strategy CUDA Execution Eliminating Memory and Dispatch Redundancy

Conventional scatter-gather dense convolution requires \(O(THW \cdot C)\) memory, which degrades severely on long videos. The authors design dedicated forward and backward CUDA kernels that gather features and compute convolution directly over the retained token set:

\[ \mathbf{y}_k = \sum_{j=1}^{9} \mathbf{W}_j \mathbf{x}[\mathbf{N}[k, j]] \]

where \(\mathbf{N}[k, j] = -1\) automatically evaluates to zero-padding. The kernels incorporate an automatic dual-dispatch strategy: for wide channel widths (\(C_{out} \ge 256\)), an implicit GEMM path via cuBLAS utilizes Tensor Core throughput; for narrower channels, a custom tiled kernel fuses feature gathering and GEMM into a single execution launch. A chunked execution mechanism completely decouples memory from total video sequence length, keeping peak memory nearly flat at approximately 800MB even under 6,144 frames—an 87% memory reduction compared to dense processing.

3. Asymmetric Cross-Attention for Temporal Recombination: Recovering Inter-Frame Dynamics with Negligible Overhead

While native sparse 2D convolutions efficiently model spatial neighborhoods within each frame, temporal action detection demands strong cross-frame modeling. Directly executing sparse 3D convolutions across frames increases memory stride distances by approximately \(8\times\), breaking GPU warp coalescing and L2 cache locality. Instead, the authors incorporate a lightweight asymmetric cross-attention module inside the bottleneck adapter: auxiliary tokens (such as the CLS token) act as Queries, while down-projected sparse visual features serve as Keys and Values:

\[ \mathbf{X}'_{aux} = \text{CrossAttn}(\mathbf{X}'_{vis}) \]

Only the low-dimensional auxiliary tokens are updated, adding a negligible compute cost (+0.11 TFLOPs). Subsequently, the frozen ViT self-attention layers naturally broadcast this enriched global temporal context back to all visual tokens across all frames, compensating for the lack of inter-frame convolutions without incurring hardware memory bottlenecks.

4. Auxiliary Task Token Supervision: Exploiting Explicit Spatial Indexing for Fine-Grained Assembly Detection

Because SV-TAD explicitly maintains original grid coordinates and survival indices for all retained tokens, the framework seamlessly accommodates auxiliary task supervision. During token selection, auxiliary tokens are integrated into the importance scoring function with learnable weights, biasing selection toward task-relevant visual regions (e.g., hands and manipulated objects). In fine-grained assembly detection benchmarks such as ATTACH, auxiliary pose landmarks and heatmap supervision are injected directly through the adapter's cross-attention interface without requiring separate dense reconstruction pipelines.

Loss & Training

SV-TAD is optimized end-to-end with the backbone frozen while training the adapters and ActionFormer detection head. The overall loss combines classification Focal Loss \(\mathcal{L}_{cls}\) and boundary regression DIoU Loss \(\mathcal{L}_{reg}\):

\[ \mathcal{L} = \mathcal{L}_{cls} + \lambda_{reg} \mathcal{L}_{reg} + \lambda_{aux} \mathcal{L}_{aux} \]

When auxiliary landmark supervision is enabled, the auxiliary landmark loss \(\mathcal{L}_{aux}\) is added. The early layers use standard 2D convolutions, while token selection prunes sequences at layers 4, 8, 12, and 16 (for ViT-L with keep rate \(k_r=0.6\)), with subsequent layers executing SparseConv2D.

Key Experimental Results

Main Results

On both THUMOS-14 and ActivityNet-1.3, SV-TAD matches or surpasses dense adapter baselines such as AdaTAD and LoSA at a fraction of their computational cost:

Dataset	Backbone	Method	TFLOPs↓	[email protected] (%)	Avg. mAP (%)
THUMOS-14	VideoMAEv2-B	AdaTAD (CVPR'24)	17.90	74.31	70.67
THUMOS-14	VideoMAEv2-B	SV-TAD (Ours)	10.29 (-43%)	75.28	72.44 (+1.77)
THUMOS-14	VideoMAEv2-L	AdaTAD (CVPR'24)	59.40	76.84	73.50
THUMOS-14	VideoMAEv2-L	SV-TAD (Ours)	21.25 (-64%)	77.22	73.47
THUMOS-14	InternVideoNext-L	SV-TAD (Ours)	87.05	78.22	74.13
THUMOS-14	VideoMAEv2-G	AdaTAD (CVPR'24)	176.87	77.61	73.87
ActivityNet-1.3	VideoMAEv2-B	AdaTAD (CVPR'24)	8.05	56.66	38.31
ActivityNet-1.3	VideoMAEv2-B	SV-TAD (Ours)	4.75 (-41%)	57.09	38.80 (+0.49)
ActivityNet-1.3	InternVideoNext-L	SV-TAD (Ours)	34.00 (-59%)	58.79	40.06 (+0.86)

Note: On THUMOS-14, SV-TAD with InternVideoNext-L achieves 74.13% Avg. mAP using 87.05 TFLOPs, surpassing AdaTAD's billion-parameter VMAEv2-G (73.87% at 176.87 TFLOPs) at roughly half the compute. Under matched InternVideoNext-L backbones on ActivityNet-1.3, SV-TAD (34.0 TFLOPs / 40.06%) substantially outperforms AdaTAD (95.3 TFLOPs / 39.72%).

Ablation Study

The following table details the impact of convolution primitives, temporal cross-attention, keep rate sweeps, and auxiliary landmark supervision:

Experiment Type	Configuration	TFLOPs↓	Avg. mAP (%)	Note
Convolution Type (THUMOS-14, VMAE-B)	DenseConv1D (w/ recon)	17.90	69.60	Temporal 1D convolution with scatter-gather
Convolution Type (THUMOS-14, VMAE-B)	DenseConv2D (w/ recon)	17.90	68.83	Spatial 2D convolution with zero-fill padding
Convolution Type (THUMOS-14, VMAE-B)	SparseConv2D (Ours)	10.29	69.35	Native sparse convolution bypassing reconstruction
Module Ablation (THUMOS-14, VMAE-B)	SV-TAD (w/o CrossAttn)	10.18	71.80	Disables auxiliary temporal aggregation
Module Ablation (THUMOS-14, VMAE-B)	SV-TAD (full model)	10.29	72.44	Recovers temporal context with +0.11 TFLOPs (+0.64%)
Keep Rate \(k_r\) (InternVidNext-L)	\(k_r = 0.7\)	87.05	74.13	Throughput 0.42 vid/s, highest accuracy
Keep Rate \(k_r\) (InternVidNext-L)	\(k_r = 0.6\)	74.09	73.64	Throughput 0.50 vid/s, balanced performance
Keep Rate \(k_r\) (InternVidNext-L)	\(k_r = 0.5\)	60.13	73.00	Throughput 0.60 vid/s, compute drops 31% with -1.13% mAP
Auxiliary Task (ATTACH Dataset)	SV-TAD Baseline	10.29	16.28	Visual token selection without landmark tokens
Auxiliary Task (ATTACH Dataset)	SV-TAD + Kp (Keypoints)	11.17	18.69	Landmark-guided token selection (+2.41% mAP)

Key Findings

• Compounding Efficiency in Deep Models: In a 24-layer ViT-L, progressive four-stage token pruning reduces the effective token keep rate to 0.13 in deeper blocks. This places the majority of layers well below the 55% crossover point where SparseConv2D outperforms dense convolutions, translating into a 2.2x real-world throughput gain (1.57 vs. 0.73 vid/s), a 31% training peak memory drop (9.23GB vs. 13.30GB), and a 1.7x training speedup.
• Asymmetric Cross-Attention as a Hardware-Friendly Temporal Bridge: Spatial sparse convolutions avoid inter-frame stride thrashing in GPU memory, while cross-attention with a single CLS token restores global temporal context at a cost of only 0.11 TFLOPs, yielding a +0.64% mAP gain over the variant without cross-attention.
• Graceful Performance Degradation: Sweeping \(k_r\) from 0.7 down to 0.5 increases throughput from 0.42 to 0.60 vid/s (+43%) while average mAP drops smoothly by only 1.13%, enabling flexible adaptation to edge deployment budgets.

Highlights & Insights

• Bypassing the False Dilemma of Sparse Hash Tables vs. Dense Reconstruction: Rather than borrowing heavy 3D point cloud hash managers or resorting to wasteful dense zero-padding, the authors exploit the fixed 2D grid geometry of ViT patches to build a lightweight, reusable \(O(N_{kept})\) lookup table, providing an elegant algorithmic solution.
• Hardware-Aware Spatio-Temporal Decoupling: Recognizing that sparse 3D temporal convolutions shatter memory locality and warp coalescing, the design isolates sparse convolutions to spatial slices and delegates temporal communication to attention mechanisms, reconciling modeling capacity with GPU hardware efficiency.
• Broad Versatility Beyond TAD: SparseConv2D serves as a general computational primitive applicable to any Vision Transformer architecture where local convolutional operators (such as FPNs or prediction heads) follow dynamic token selection.

Limitations & Future Work

• Hardware and Backend Constraints: The custom CUDA kernels are heavily optimized for NVIDIA architectures and Tensor Cores; deployment on AMD ROCm, Apple Metal, or edge NPUs requires developing corresponding low-level kernels.
• Heuristic Pruning Schedules: The keep rates and layer selection points are statically preconfigured across fixed intervals; exploring dynamic, input-adaptive pruning schedules could further enhance efficiency on simpler video segments.
• Unified 3D Spatio-Temporal Primitives: Currently, spatial convolution and temporal aggregation are decoupled; investigating hardware-efficient sparse graph or 3D spatio-temporal operators with high cache locality remains an open challenge.

Related Work & Insights

• vs. AdaTAD (CVPR 2024): AdaTAD pioneered parameter-efficient 1D temporal adapters but computes over the full dense grid at inference; SV-TAD introduces dynamic pruning and native sparse convolutions, achieving a 64% reduction in TFLOPs and 2.2x faster inference on VideoMAEv2-L.
• vs. LoSA (WACV 2025): LoSA scales adapters to VMAEv2-G but suffers from performance saturation and full-grid computational overhead; SV-TAD with InternVideoNext-L outperforms LoSA while requiring less than half the compute.
• vs. MinkowskiEngine / Submanifold Sparse Convolutions: Traditional 3D sparse engines incur high hash table probe overhead and COO tensor conversions that mismatch dense ViT activations; SV-TAD's \(O(N_{kept})\) index table integrates natively with standard PyTorch tensor layouts.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First native sparse 2D convolution primitive designed specifically for dynamically pruned Vision Transformers, resolving the long-standing incompatibility between token selection and convolutional adapters.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across THUMOS-14, ActivityNet-1.3, and ATTACH, with rigorous FLOP-matched comparisons, comprehensive ablations, and kernel-level micro-benchmarks.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear exposition of the problem setting, rigorous mathematical definitions, and comprehensive hardware efficiency analyses.
• Value: ⭐⭐⭐⭐⭐ Provides an impactful computational primitive and open-source library that significantly advances efficient long video foundation model adaptation.

Related Papers

• [ICLR 2026] VideoNSA: Native Sparse Attention Scales Video Understanding
• [ECCV 2026] MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction
• [ECCV 2026] STEP: Score-Based Temporal Energy for Human Pose Video Anomaly Detection
• [ECCV 2026] ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistency
• [ICCV 2025] Sparse-Dense Side-Tuner for Efficient Video Temporal Grounding