Skip to content

Recognizing Co-Speech Gestures in-the-Wild

Conference: ECCV 2026
Paper: ECCV Official Poster
Code: https://www.robots.ox.ac.uk/~vgg/research/grw
Area: Human Understanding
Keywords: gesture recognition, co-speech gestures, temporal localization, multimodal video understanding, cross-modal temporal alignment

TL;DR

Oxford VGG introduces the first large-scale in-the-wild co-speech gesture dataset GRW (~140k video clips, 155-word vocabulary) along with temporal Transformer models that leverage hand priors and broad motion context to solve semantic gesture classification, word-level recognition, and dense frame-level temporal localization.

Background & Motivation

Humans naturally gesture with their hands and upper body during spoken discourse. However, only a sparse subset of these movements are truly meaning-bearing and semantically grounded in specific spoken words, such as iconic, deictic, or metaphoric gestures (e.g., tracing a circle while saying "circle", or spreading hands apart when saying "massive"). The vast majority of physical movements are non-semantic beat gestures providing rhythmic cadence or incidental fidgeting. Understanding and recognizing these subtle semantic gestures in unconstrained, real-world videos presents severe challenges: they occur sparsely in time, exhibit extreme kinematic diversity across different speakers expressing the same concept, and are loosely misaligned with the spoken audio signal.

Existing gesture benchmarks predominantly focus on isolated command-style actions (e.g., swipe gestures in Jester or EgoGesture) devoid of spoken context, or rely on laboratory-recorded motion capture setups with few speakers and homogeneous speech registers. While large-scale in-the-wild monologue datasets exist for speech-driven gesture generation, they lack precise word-level groundings and temporal boundaries, leading models to struggle with producing or recognizing semantically faithful movements. Moreover, modern general-purpose vision-language foundation models (such as InternVL, Qwen-VL, and Gemini) exhibit substantial brittleness when confronted with fine-grained, non-rigid co-speech gestures in natural video streams.

To bridge this data and methodological deficit, the authors introduce the Gesture Recognition in the Wild (GRW) benchmark and uncover fundamental kinematic insights into speech-gesture temporal envelopes. Core idea: leverage sign-language pretrained hand feature representations and model extended background motion context via cross-attention to contrast against a speaker's default rhythmic beat baseline, thereby accurately isolating and localizing sparse semantic gestures.

Method

Overall Architecture

The framework processes an input video sequence through a frozen hand-centric visual backbone to extract multi-layer representations, which are dynamically fused via layer-wise adaptive aggregation. Downstream predictions are handled by two specialized Transformer architectures: (1) a semantic gesture classification model that uses a Transformer encoder-decoder cross-attention mechanism to contrast a target query clip against an extended 10-second motion context, predicting whether a semantic gesture is present; and (2) a joint word recognition and localization model that performs dense frame-level binary classification and pools frame features into a global representation via localization-probability weighting to classify the gestured word.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Video Stream<br/>V in R^(T×H×W×3)"] --> B["Hand Feature Backbone<br/>SHuBERT extracts multi-layer states"]
    B --> C["Weighted Layerwise Aggregation<br/>MLP aggregates states into feature f"]
    C --> D{"Task Branch"}
    D -->|Long-range Context| E["Extended Motion Context & Cross-Attention<br/>RoPE encoder + query cross-decoder"]
    E --> F["Semantic Binary Classification<br/>p_sem predicts semantic gesture presence"]
    D -->|Target Segment| G["Dense Frame-level Localization<br/>Transformer encoder outputs p_loc"]
    G --> H["Probability-Weighted Word Recognition<br/>p_loc weighted pooling + MLP classifier"]

Key Designs

1. Hand Feature Backbone with Weighted Layerwise Aggregation: Establishing Fine-Grained Hand Priors Standard video backbones typically suffer from spatial downsampling and fail to preserve rapid hand articulations and delicate finger shapes in unconstrained videos. The authors employ SHuBERT, a Transformer-based visual backbone pretrained on large-scale sign language corpora, to provide robust representations of fine-grained hand kinematics. For an input clip, hidden states from \(L\) layers form a multi-layer feature tensor \(f_{\text{multi}} \in \mathbb{R}^{L \times T \times d_{\text{in}}}\) with \(d_{\text{in}} = 768\). Rather than hand-picking a single heuristic layer, a lightweight MLP computes scalar scores normalized across layers via Softmax into mixing weights \(\alpha \in \mathbb{R}^L\), producing an adaptive linear combination: $\(f = \sum_{l=1}^L \alpha_l f_{\text{multi}}^{(l)}\)$ The collapsed sequence \(f \in \mathbb{R}^{T \times d_{\text{in}}}\) serves as the unified input representation for subsequent temporal Transformer modules.

2. Extended Motion Context & Cross-Attention: Contrasting Against Rhythmic Beat Baselines In natural speech, repetitive beat gestures establish an individual's idiosyncratic baseline rhythm. Sparse semantic gestures deviate markedly from this baseline. To enable the network to perceive this distinction, the classification model consumes an extended temporal context of \(T=250\) frames (10 seconds), encoding it with Rotary Position Embeddings (RoPE) inside a Transformer encoder to produce a context memory \(M \in \mathbb{R}^{T \times d_{\text{model}}}\). The target query interval features (\(Q=100\) frames, 4 seconds) then act as queries \(q \in \mathbb{R}^{Q \times d_{\text{model}}}\) in a Transformer decoder cross-attending to \(M\). By grounding the query segment against the speaker's broader kinematic cadence, the network isolates genuine semantic deviations, outputting a binary semantic probability \(p_{\text{sem}}\) after temporal average pooling and sigmoid activation.

3. Dense Frame-Level Localization and Probability-Weighted Word Recognition: Mutual Alignment Rather than relying on region proposals or sliding windows, the model formulates temporal boundary detection as dense per-frame binary classification. Query features are processed by a Transformer encoder to yield temporally contextualized representations \(f_{\text{enc}} \in \mathbb{R}^{Q \times d_{\text{model}}}\). A per-frame MLP and sigmoid head directly predict frame-wise gesture presence probabilities \(p_{\text{loc}, q} \in [0, 1]\). To classify the conceptual word without introducing non-gesture frame noise, the model computes a soft-masked global video embedding \(\bar{f}\) weighted by the predicted frame probabilities: $\(\bar{f} = \frac{\sum_{q=1}^Q p_{\text{loc}, q} f_{\text{enc}, q}}{\sum_{q=1}^Q p_{\text{loc}, q} + \epsilon}\)$ A two-layer classification MLP maps \(\bar{f}\) to a Softmax probability distribution \(p_{\text{rec}}\) across \(C\) vocabulary classes, creating an end-to-end synergy between temporal localization and semantic recognition.

Loss & Training

The recognition and localization model follows a two-stage training paradigm: (1) weakly-supervised pre-training on 67,214 pseudo-labeled positive clips mined automatically by the high-confidence classifier (\(p_{\text{sem}} \ge 0.9\)), utilizing word-specific average boundary offsets as surrogate targets; and (2) fine-tuning on 15,340 clean, manually verified clips with frame-accurate human boundaries. The multi-task training loss jointly optimizes categorical cross-entropy for word prediction and binary cross-entropy for frame localization: $\(\mathcal{L} = -\sum_{c=1}^C y_c \log(p_{\text{rec}, c}) - \frac{\lambda_{\text{loc}}}{Q} \sum_{q=1}^Q \left[ a_q \log(p_{\text{loc}, q}) + (1 - a_q) \log(1 - p_{\text{loc}, q}) \right]\)$ where \(y_c\) is the one-hot ground-truth word label, \(a_q \in \{0, 1\}\) is the frame-level boundary label, and \(\lambda_{\text{loc}}\) balances the two components. For the semantic classification task, class-balanced sampling and positive sample loss weighting are utilized to tackle the 1:8 positive-to-negative class imbalance.

Key Experimental Results

Main Results

The proposed framework was evaluated on the unseen test splits of GRW against frozen feature representations, fine-tuned dense gesture models, and state-of-the-art vision-language models.

Table 1: Semantic gesture classification performance (original paper Table 5) | Method | Accuracy (%) | Precision (%) | Recall (%) | Semantic class accuracy for high-conf samples (≥0.9) (%) | |---|---|---|---|---| | Random Baseline | 50.00 | 50.00 | 50.00 | n/a | | Clip4Clip | 59.13 | 55.64 | 52.34 | 62.95 | | Sapiens | 60.10 | 60.92 | 57.80 | 68.01 | | SemMomDinov3 | 61.77 | 60.91 | 58.85 | 67.31 | | GestSync (Fine-tuned) | 61.58 | 59.90 | 62.17 | 64.91 | | JEGAL (Fine-tuned) | 63.94 | 61.02 | 59.94 | 66.93 | | Intern-VL | 61.58 | 56.67 | 58.35 | 66.68 | | Qwen3-VL | 55.05 | 53.24 | 51.28 | 58.61 | | Gemini-3.5 | 67.30 | 65.78 | 71.10 | 68.21 | | Ours | 75.83 | 79.91 | 69.00 | 93.20 |

Table 2: Word-level gesture recognition & localization performance (original paper Table 6, 100-word test vocabulary) | Method | Acc @ 1 (%) | Acc @ 5 (%) | Acc @ 10 (%) | Localization mIoU | |---|---|---|---|---| | Random | 1.00 | 5.00 | 10.00 | 0.1975 | | Clip4Clip | 8.70 | 19.25 | 27.60 | n/a | | Sapiens | 9.45 | 18.80 | 28.60 | n/a | | SemMomDinov3 | 10.05 | 21.15 | 29.65 | n/a | | GestSync | 9.12 | 20.37 | 30.18 | 0.5172 | | JEGAL | 10.43 | 22.71 | 31.88 | 0.5368 | | Intern-VL | 4.35 | 12.50 | 15.71 | 0.3932 | | Qwen3-VL | 5.92 | 13.19 | 19.27 | 0.2071 | | Gemini-3.5 | 13.95 | 30.15 | 41.10 | 0.4658 | | Ours | 18.35 | 37.30 | 51.90 | 0.6726 |

Ablation Study

The authors conducted ablations on temporal context window sizes for classification and the utility of weakly-supervised pre-training.

Table 3: Impact of temporal context length on high-confidence classification accuracy (original paper Table 7) | Context Length | Equivalent Duration (s) | High-conf Accuracy (%) | Note | |---|---|---|---| | 100 frames | 4.0s (no extra context) | 87.27 | Lacks baseline motion reference | | 150 frames | 6.0s | 89.82 | Moderate context gain | | 200 frames | 8.0s | 91.56 | Continuous improvement | | 250 frames | 10.0s | 93.20 | Optimal performance peak (+5.93%) | | 300 frames | 12.0s | 92.95 | Slight degradation from temporal noise |

Table 4: Impact of pre-training before fine-tuning on clean annotations (original paper Table 8) | Training Setup | Acc @ 1 (%) | Acc @ 5 (%) | Acc @ 10 (%) | Localization mIoU | |---|---|---|---|---| | Train on 67k Pre-train only | 11.50 | 28.15 | 38.70 | 0.5737 | | Train on 15k Clean only | 16.35 | 35.60 | 49.40 | 0.6473 | | Pre-train + FT on clean (Ours) | 18.35 | 37.30 | 51.90 | 0.6726 |

Key Findings

  • Extended motion context is crucial: Providing a 10-second temporal window (250 frames) to evaluate a 4-second target segment improves high-confidence accuracy by +5.93% (from 87.27% to 93.20%), validating that capturing the speaker's background gesturing rhythm is vital to detect sparse semantic gestures.
  • Asymmetric gesture-speech temporal envelopes: Quantitative analysis reveals that 97.5% of semantic gestures initiate before the spoken word begins, and 85.7% conclude after the spoken word ends. Gestures span substantially longer durations than speech (e.g., "spiral" speech averages 0.46s whereas gesture averages 2.17s).
  • Generalist VLMs struggle with fine-grained temporal localization: While Gemini-3.5 achieves a competitive 13.95% top-1 recognition accuracy due to strong semantic priors, its temporal localization mIoU remains at 0.4658, substantially behind the specialized architecture (0.6726 mIoU).

Highlights & Insights

  • Pioneer in-the-wild benchmark: GRW establishes the first large-scale unconstrained dataset for semantic co-speech gestures across 37k identities and 140k annotated video segments.
  • Baseline-contrast modeling: Formulating semantic gesture detection not as static pattern matching, but as finding salient deviations against an encoded personal rhythmic baseline via cross-attention.
  • Self-sustaining pseudo-labeling loop: Achieving 93.20% accuracy on high-confidence samples enables the classifier to harvest 67k weakly-labeled training clips automatically, proving the feasibility of bootstrapping gesture annotations.

Limitations & Future Work

  • Register bias of source videos: The dataset is sourced from public lectures, talks, and interviews (MultiVSR/YouTube), where speakers naturally present more projected and deliberate gestures than in casual interpersonal conversations.
  • Kinematic ambiguity across related words: Certain word pairs (e.g., hello vs. bye, small vs. little) share virtually identical visual gestures, limiting top-1 accuracy without deeper multimodal audio-text integration.
  • vs Jester / EgoGesture / ChaLearn: Command gesture datasets isolate explicit, artificial actions without speech alignment; GRW captures continuous, naturalistic co-speech gestures embedded in spoken language.
  • vs BEAT / Trinity / TalkingWithHands: MoCap datasets provide clean 3D markers but suffer from sterile lab environments and minimal speaker diversity; GRW scales to 37k speakers in varied natural video conditions.
  • vs JEGAL: JEGAL introduced cross-modal retrieval and spotting but lacked frame-level boundary localization and binary semantic classification baselines. On the AVS-Spot transfer benchmark, the proposed model achieves 77.3% spotting accuracy compared to JEGAL's 63.6%.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Introduces the first large-scale in-the-wild benchmark with dual temporal boundaries (speech & gesture) and uncovers empirical kinematic offset envelopes.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates three distinct tasks, compares against diverse frozen, fine-tuned, and frontier VLM baselines, and validates transferability on external datasets.
  • Writing Quality: ⭐⭐⭐⭐⭐ Extremely lucid narrative structure, compelling linguistic motivation, and comprehensive statistical analysis.
  • Value: ⭐⭐⭐⭐⭐ Foundational resource for gesture understanding, conversational AI, and embodied avatar synthesis.