S2Gest: Split-Scan State Space Models for Dynamic Hand Gesture Recognition¶

Conference: ECCV 2026
Paper: CVF / Official Poster
Code: https://github.com/Chen-Ke-Fan/S2Gest
Area: Human Understanding
Keywords: Dynamic Hand Gesture Recognition, State Space Models, Bidirectional Temporal Modeling, Lightweight Architecture

TL;DR¶

Designed for continuous always-on local execution on consumer-grade hardware, S2Gest integrates a spatial-first multi-scale stem with adaptive feature modulation and introduces a parameter-neutral Split-Scan S2-Block that partitions channels to capture bidirectional temporal dynamics in linear time, matching heavy 3D CNNs and Transformers with under 3M parameters at over 100 clips/s.

Background & Motivation¶

Dynamic Hand Gesture Recognition (DHGR) is a crucial human-computer interaction modality for modern smart devices, smart cockpits, and wearable AR/VR systems. In real-world consumer deployments—such as dimly lit vehicle cabins or privacy-sensitive home environments—continuous, always-on recognition is a strict local requirement rather than merely a latency preference. This operational regime imposes rigorous constraints on consumer-grade and edge processors regarding computational FLOPs, thermal power limits, and runtime memory bandwidth. However, existing architectural paradigms remain caught in an efficiency-accuracy dilemma: heavyweight 3D CNNs and Video Transformers provide strong temporal modeling but suffer from prohibitive Memory Access Costs (MAC) and dynamic KV-cache memory expansion, while compact 2D CNNs lack the receptive field and temporal capacity required to model complex kinematic trajectories.

The root cause of this breakdown lies in the fundamental disconnect between discrete, uniformly sampled video frames and the fluid, continuous physical evolution of human hand kinematics. State Space Models (SSMs), particularly Mamba, provide a compelling solution via continuous ODE formulations and constant-memory recurrence. Nonetheless, adapting standard SSMs directly to DHGR encounters two critical structural obstacles: first, noisy RGB and sparse depth streams require robust yet cheap spatial feature extraction prior to temporal scanning; second, standard SSMs are fundamentally causal and unidirectional, unable to absorb non-causal retrospective context. Naive bidirectional scanning strategies (such as full-channel dual-branch passes) immediately double both parameter count and activation memory, violating strict edge constraints.

To break this efficiency-accuracy bottleneck, this paper re-evaluates dynamic gesture modeling through continuous state-space evolution. Core idea: by combining a spatial-first multi-scale visual stem and adaptive contextual modulation with a channel-partitioned Split-Scan S2-Block, the model decouples forward gesture onset from backward gesture verification across independent channel subspaces, enabling global bidirectional temporal modeling with zero parameter overhead in under 3M parameters.

Method¶

Overall Architecture¶

S2Gest processes dynamic hand gesture sequences within a sliding temporal window of \(T\) continuous frames. As depicted in the pipeline flow, the architecture operates across three sequential phases: first, raw input video tensors pass through a shared Visual Stem where the Multi-Scale Spatial Block (MS-Block) compresses spatial redundancy early while extracting rich multi-scale structural semantics; next, the representation branches into a Context-Aware Dual-Stream setup, where a lightweight Context Stream generates holistic spatiotemporal priors that dynamically recalibrate the high-resolution Focal Stream via Adaptive Feature Modulation (AFM), followed by Content-Adaptive Positional Encoding (CPE) for arbitrary sequence-length extrapolation; finally, the modulated features are processed by cascaded S2-Blocks, where a Patch-Interleaved Tokenizer (PIT) structures the spatial grid into localized temporal tubes, and a parameter-neutral Split-Scan engine models bidirectional kinematics before a linear classification head yields gesture class probabilities.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Input Video Tensor<br/>C × T × H × W"] --> B["1. Visual Stem & Spatial-First Embedding<br/>MS-Block Multi-Scale Redundancy Reduction with Conv-BN Fusion"]
    B --> C["2. Context-Aware Dual-Stream & Modulation<br/>Lightweight Context Prior Stream + AFM Affine Recalibration"]
    C --> D["Dynamic 3D Positional Encoding CPE<br/>3×3×3 Depthwise Conv for Arbitrary Sequence Extrapolation"]
    D --> E["Patch-Interleaved Tokenizer PIT<br/>Eulerian Tube-Major Scanning Preserves Spatial Aperture Continuity"]
    E --> F["3. Split-Scan S2-Block Engine<br/>Channel-Partitioned Forward & Backward Bidirectional SSM"]
    F --> G["3D Depthwise Conv Smoothing & Residual Merge<br/>Mitigates Tube Boundary Discontinuity"]
    G --> H["Classification Head Prediction<br/>Gesture Class Probabilities"]

Key Designs¶

1. Spatial-First Embedding via MS-Block: Early Redundancy Compression with Inference Conv-BN Fusion

Early video representations exhibit massive spatial redundancy, making dense 3D cubic convolutions (\(k \times k \times k\)) computationally wasteful and severely memory-access bound (MAC). S2Gest adopts a "Spatial-First" design philosophy within its Visual Stem, utilizing an Inception-inspired Multi-Scale Spatial Block (MS-Block). The module routes features through four parallel heterogeneous branches: a point-wise \(1 \times 1 \times 1\) convolution, two parallel \(1 \times 3 \times 3\) spatial convolution branches providing diverse receptive field scales, and a parameter-free \(3 \times 3 \times 3\) max-pooling branch to extract early motion cues without dense 3D MAC:

\[\operatorname{MS-Block}(\mathbf{X}) = \mathcal{H}_{\mathrm{proj}}\left( \operatorname{Concat}_{o \in \mathcal{O}} \mathcal{B}_o(\mathbf{X}) \right)\]

where \(\mathcal{O} = \{\text{Conv}_{1\times 1\times 1}, \text{Conv}^{(1)}_{1\times 3\times 3}, \text{Conv}^{(2)}_{1\times 3\times 3}, \text{MaxPool}_{3\times 3\times 3}\}\). During deployment inference, all spatial convolutions and batch normalizations are fused (Conv-BN fusion), dramatically accelerating operational throughput and preserving computational headroom for downstream deep temporal reasoning.

2. Context-Aware Dual-Stream Architecture and Adaptive Feature Modulation (AFM): Global Prior Conditioning

Complex in-cabin lighting variations and background clutter easily obscure subtle hand articulations. Performing full global self-attention across high-resolution tokens is prohibitive on consumer hardware. Instead, S2Gest bifurcates features after the stem into an asymmetric dual-stream design: an ultra-lightweight Context Stream applies aggressive spatiotemporal downsampling and spatial global average pooling to extract a compact prior sequence \(\mathbf{Z}_{\mathrm{ctx}} \in \mathbb{R}^{D_{\mathrm{ctx}} \times T}\) with negligible overhead.

The Adaptive Feature Modulation (AFM) module then serves as an affine semantic filter between the two streams. By applying temporal average pooling to \(\mathbf{Z}_{\mathrm{ctx}}\), AFM generates channel-wise scale parameters \(\boldsymbol{\gamma} \in \mathbb{R}^C\) and shift parameters \(\boldsymbol{\beta} \in \mathbb{R}^C\) via learnable projections:

\[\mathbf{F}_{\mathrm{mod}} = \mathrm{AFM}(\mathbf{F}_{\mathrm{focal}}, \mathbf{Z}_{\mathrm{ctx}}) = \boldsymbol{\gamma} \odot \mathbf{F}_{\mathrm{focal}} + \boldsymbol{\beta}\]

Unlike naive additive or concatenative feature fusion that injects unconstrained background noise into local activations, AFM selectively amplifies motion-sensitive channels and suppresses static ambient clutter before deep temporal sequence modeling begins.

3. Patch-Interleaved Tokenizer (PIT) and Split-Scan S2-Block: Parameter-Neutral Bidirectional SSM

Standard serialization methods (e.g., Frame-Major scanning) destroy spatial locality by sweeping entire spatial frames sequentially, forcing 1D SSMs to bridge large spatial gaps and triggering acute hidden state decay. PIT reorders tokens into a "Tube-Major" structure, grouping tokens by spatial patches of size \(P \times P\) across all \(T\) frames into tubes \(\mathbf{X}_{\mathrm{tube}} \in \mathbb{R}^{N \times T \times D_{\mathrm{emb}}}\). Under an Eulerian perspective, the SSM continuously observes motion within a static spatial window, allowing Mamba's input-dependent step size \(\Delta(x_t)\) to maintain context within each tube and reset across tube boundaries.

To eliminate the causal unidirectionality of standard SSMs without duplicating parameter count, S2Gest introduces the "Split-Transform-Merge" Split-Scan mechanism. The token channels are split into disjoint forward and reverse subgroups \(\mathbf{U}_{\mathrm{fwd}}, \mathbf{U}_{\mathrm{rev}} \in \mathbb{R}^{T \times (D_{\mathrm{emb}}/2)}\), each processed simultaneously:

\[\mathbf{H}_{\mathrm{fwd}} = \operatorname{SSM}(\mathbf{U}_{\mathrm{fwd}}), \quad \mathbf{H}_{\mathrm{rev}} = \operatorname{Flip}\left(\operatorname{SSM}\left(\operatorname{Flip}(\mathbf{U}_{\mathrm{rev}})\right)\right)\]

\[\mathbf{H} = \mathcal{F}_{\mathrm{mix}}\left( \operatorname{Norm}\left( \operatorname{Concat}[\mathbf{H}_{\mathrm{fwd}}, \mathbf{H}_{\mathrm{rev}}] \right) \right)\]

Because linear projection parameter count scales quadratically with channel width, halving the channel dimension reduces the internal transformation cost per branch to \(0.25 D_{\mathrm{emb}}^2\), making the combined dual-branch cost exactly half that of a full-channel baseline (\(2 \times 0.25 D^2 = 0.5 D^2\)). Subsequent linear mixing \(\mathcal{F}_{\mathrm{mix}}\) restores cross-channel interactions. Activation analysis demonstrates that the forward branch acts as a causal trigger peaking at gesture onset, while the reverse branch acts as an anti-causal validator peaking at gesture completion, providing structural regularization against feature collapse. A terminal 3D depthwise convolution smoothes out spatial boundary artifacts before residual addition.

Loss & Training¶

The network is trained end-to-end using standard cross-entropy loss:

\[\mathcal{L} = -\sum_{k=1}^K y_k \log \hat{y}_k\]

Input clips are uniformly sampled to \(T = 40\) frames resized to \(192 \times 256\). RGB models are initialized with Jester pre-trained weights, whereas Depth models are trained entirely from scratch. Training utilizes AdamW with an initial learning rate of \(5 \times 10^{-5}\), weight decay of 0.05, and a multi-step decay schedule on NVIDIA A40 GPUs. A small batch size of 4 is deliberately adopted as an implicit regularizer, preventing overfitting on high-dimensional video manifolds under constrained parameter regimes.

Key Experimental Results¶

Main Results¶

S2Gest was evaluated against state-of-the-art methods across NVGesture, EgoGesture, and Briareo benchmarks for both RGB and Depth modalities (Table 1 from original paper):

Method	Params (M)	FLOPs (G)	NVGesture (RGB / Depth)	EgoGesture (RGB / Depth)	Briareo (RGB / Depth)
NAS	127.10	120.20	83.61% / 86.10%	93.31% / 94.13%	— / —
ResNeXt-101	47.53	14.93	78.63% / 83.82%	93.75% / 94.03%	— / —
MotionRGBD	38.14	73.85	89.58% / 90.62%	— / —	— / —
GestFormer	24.08	60.40	75.41% / 80.21%	— / —	94.44% / 96.18%
MDRA	23.90	45.30	83.87% / 86.60%	93.96% / 94.20%	97.84% / 96.92%
MVTN	19.55	60.22	77.50% / 85.21%	— / —	97.69% / 97.92%
ConvMixFormer	13.57	59.98	76.04% / 80.83%	— / —	98.26% / 97.22%
DSTCNet	9.60	42.56	82.50% / 88.33%	93.70% / 94.26%	97.81% / 98.10%
MobileNetV2 1.0	2.38	4.76	74.07% / 75.73%	93.49% / 94.18%	93.52% / 91.20%
S2Gest-Small (Ours)	2.59	14.90	86.10% / 90.66%	96.47% / 97.35%	97.69% / 98.15%
S2Gest-Tiny (Ours)	1.17	4.53	85.06% / 87.76%	96.27% / 97.27%	97.69% / 98.61%
S2Gest-Nano (Ours)	0.68	4.02	85.48% / 82.99%	95.94% / 97.07%	98.15% / 97.69%

Ablation Study¶

Ablation experiments conducted on NVGesture (Depth modality) validate the individual contributions of each architectural design (Table 2 from original paper):

Row #	Spatial Stem	Temporal Module	Context Guidance	Scan Strategy	Params (M)	Top-1 Acc (%)	Note
Row 0 (Full Model)	MS-Block	S2-Block	AFM	Split-Scan	2.59	90.66%	Full proposed model baseline
Row 1 (Stem Ablation)	3D ResNet-18	Identical	Identical	Identical	2.66	83.82%	-6.84% drop, proving MS-Block multi-scale capacity
Row 2 (SSM vs Attn)	Identical	Transformer	Identical	N/A	2.79	86.51%	-4.15% drop, confirming SSM continuous kinematic advantage
Row 3 (No Guidance)	Identical	Identical	None	Identical	2.24	88.80%	Removing global context degrades performance
Row 4 (Additive Fusion)	Identical	Identical	Add	Identical	2.53	87.76%	Unconditional addition injects noise, worse than None
Row 5 (Concat Fusion)	Identical	Identical	Concat	Identical	2.52	89.21%	Naive concatenation underperforms AFM affine modulation
Row 6 (Unidirectional)	Identical	Identical	Identical	Uni-Direct.	2.80	89.42%	Lacks retrospective context, dropping 1.24%
Row 7 (Full Bidirectional)	Identical	Identical	Identical	Full Bi-Direct.	3.26	90.04%	+26% params with channel redundancy, inferior to Split-Scan

Key Findings¶

Structural Regularization of Split-Scan: Full-channel bidirectional scanning (Row 7, 3.26M params) introduces channel redundancy and drops 0.62% behind Split-Scan (Row 0, 2.59M params, 90.66%). Explicitly separating channels into disjoint forward and reverse subspaces prevents representational collapse while curbing quadratic linear scaling.
Extreme Downscaling as Noise Regularizer: In heavily cluttered datasets like EgoGesture and Briareo, S2Gest-Nano (0.68M) achieves 97.07% Depth accuracy and outscores Tiny and Small on Briareo RGB (98.15% vs. 97.69%). The restricted parameter capacity acts as an implicit regularizer, preventing the network from overfitting to background pixels and focusing purely on salient kinematics.
Hardware-Sustained Real-Time Throughput: On an NVIDIA RTX 3060 desktop GPU in interactive latency mode (BS=1), Nano delivers 114.37 clips/s (8.74 ms latency) and Small achieves 65.45 clips/s. In high-throughput batched mode (BS=32) on a laptop RTX 4060, Nano and Tiny sustain 175.53 and 170.76 clips/s, completely bypassing the memory-wall constraints common to Transformer KV-caches.

Highlights & Insights¶

Parameter-Neutral True Bidirectional SSM: By leveraging the quadratic complexity property of projection layers with respect to channel dimensions, splitting channels into forward and backward sub-paths halves internal transformation costs, turning bidirectional non-causal scanning into a parameter-neutral operation.
Eulerian Tube-Major Scanning: Structuring the 3D tensor into continuous spatial tubes aligns SSM sequential processing with fixed-aperture physical hand motion, drastically mitigating the temporal forgetting caused by frame-major token flattening.
Dynamic CPE for Arbitrary Extrapolation: Employing dynamic 3D depthwise convolutions for positional encoding decouples the model from rigid sequence lengths, enabling plug-and-play extrapolation across varied temporal frame rates without architectural re-tuning.

Limitations & Future Work¶

Temporal Aliasing in Rapid Erratic Gestures: ZOH discretization relies on the assumption of relatively stable execution speeds. Highly rapid or erratic cyclical hand gestures may disrupt the selective gating mechanism \(\Delta(x_t)\), occasionally triggering premature trajectory truncation.
Spatial Downsampling Blur on Articulated Fingers: To satisfy tight edge compute budgets, the Visual Stem downsamples spatial resolution early, occasionally confusing fine-grained finger topologies under severe motion blur (e.g., misclassifying two-finger vs. three-finger swipes).
Future Directions: Exploring dynamic, learnable ODE solvers in place of zero-order holds and incorporating adaptive salient-token preservation within the spatial stem.

vs VideoMamba / Bi-Mamba: These architectures run duplicate backward SSM branches over full feature channels, incurring a \(2\times\) parameter and activation memory overhead. S2Gest achieves bidirectional modeling at zero parameter addition through structural channel partitioning.
vs GestFormer / ConvMixFormer: Transformer-based gesture models suffer quadratic complexity and expanding KV-caches during streaming deployment. S2Gest utilizes selective SSMs with constant memory to match or exceed their accuracy with an order-of-magnitude smaller parameter footprint.
vs Lightweight CNNs (MobileNetV2): Standard 2D separable CNNs suffer from restricted temporal receptive fields; S2Gest-Nano achieves a 7.26% gain over MobileNetV2 on NVGesture Depth while using less than one-third of its parameters (0.68M vs. 2.38M).

Rating¶

Novelty: ⭐⭐⭐⭐ [The channel-split bidirectional SSM design and tube-wise spatiotemporal serialization provide an elegant, mathematically sound solution for edge video modeling.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated extensively on three major benchmarks across RGB and Depth modalities, with thorough component ablations and real-world consumer GPU throughput/latency benchmarks.]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and insightful activation visualizations detailing the causal vs. anti-causal branch behaviors.]
Value: ⭐⭐⭐⭐⭐ [Directly resolves the edge memory-wall and compute conflict for always-on human-computer interaction, setting a new sub-3M SOTA standard.]