Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Event-AHU/OpenESL/tree/main/EvSLT
Area: Human Understanding
Keywords: Event Camera, Sign Language Translation, Mamba Networks, Hypergraph Neural Network, Spatiotemporal Fusion

TL;DR

To overcome motion blur, lighting variations, and privacy concerns in RGB sign language translation, this paper establishes Event-CSL—the first large-scale, 1280×720 HD real-world event stream benchmark (14,827 videos)—and presents EvSLT, an end-to-end gloss-free framework coupling Mamba-based clip spatial memory aggregation with graph-guided spatiotemporal fusion (GSTF).

Background & Motivation

Sign Language Translation (SLT) is an indispensable pillar in AI-assisted accessibility, aiming to bridge the communication gap for deaf communities by converting continuous sign language videos into natural language sentences. Despite remarkable progress, conventional SLT pipelines remain overwhelmingly tethered to visible-light (RGB) cameras. In realistic scenarios involving rapid hand gestures and frequent finger articulations, RGB cameras suffer heavily from shutter motion blur and low frame rates. Furthermore, under extreme lighting conditions—such as harsh outdoor direct sunlight or dim indoor rooms—image quality deteriorates dramatically, crippling gesture recognition accuracy. Compounding these physical bottlenecks are severe ethical and legal hurdles: RGB sensors indiscriminately record unanonymized facial biometrics and surrounding private environments, raising widespread privacy concerns that hinder real-world deployment.

Bio-inspired event cameras capture pixel-level asynchronous brightness changes at microsecond temporal resolution, possessing natural advantages such as high dynamic range (HDR), low latency, low power consumption, and absence of motion blur. Crucially, the sparse binary pulse stream offers an innate layer of biological privacy protection. Nevertheless, research into event stream-based sign language translation has stalled due to two critical impediments. First, benchmark data is extraordinarily scarce. Early studies relied on synthetic pseudo-event streams rendered from existing RGB datasets, which fail to exhibit real sensor physics and noise profiles. Existing real event datasets (e.g., SL-Animal-DVS, EV-ASL, EvCSLR) suffer from diminutive resolutions (128×128 or 346×260) and cater solely to isolated sign recognition. The only continuous SLT dataset, EvSign, deliberately masked signers' head regions for privacy, which severely corrupts event signal continuity and strips away critical linguistic cues embedded in facial expressions and head nods. Second, on the architectural side, to avoid the daunting computation required by fine-grained spatial modeling, current approaches overwhelmingly focus on 1D temporal convolution, neglecting the vital role of detailed spatial geometry in disambiguating fine hand configurations.

To resolve these empirical and methodological bottlenecks, this paper collects and releases Event-CSL using a Prophesee EVK4-HD event camera, establishing the first large-scale, full-definition benchmark for event-based continuous SLT. Concurrently, the authors develop EvSLT, a tailored gloss-free architecture. Core idea: exploit the linear-complexity sequence modeling of Mamba to compress and progressively aggregate clip-level spatial memories without computational explosion, while constructing a clip hypergraph to bridge lightweight dual-branch attention between fine-grained spatial details and temporal motion dynamics for end-to-end text generation.

Method

Overall Architecture

EvSLT employs an end-to-end, gloss-free translation architecture. Asynchronous event streams are first aggregated across non-overlapping fixed time windows into stacked event frame sequences and fed into a ResNet-18 backbone to extract frame-level visual feature representations. The architecture then diverges into two parallel branches: a temporal branch that performs global spatial average pooling followed by a two-stage 1D temporal convolution block to capture continuous motion dynamics; and a spatial branch that partitions video features into fixed-length clips, applying a bidirectional Vision Mamba module to compress fine-grained spatial configurations and transfer memory representations across successive clips. Subsequently, these representations are integrated in a Graph-guided Spatiotemporal Fusion (GSTF) module, where a hypergraph built over compressed clip nodes serves as a relational bridge across self-attention and cross-attention mechanisms. Finally, the fused spatiotemporal representations are projected through a sign embedding layer and decoded into spoken Chinese text by a pre-trained mBART sequence-to-sequence model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Event Pulse Stream<br/>Stacked into Event Frames via Temporal Windows"] --> B["ResNet-18 Visual Backbone<br/>Multi-scale Downsampling & Feature Extraction"]
    B --> C1["Temporal Feature Branch<br/>Global Spatial Pooling + 2-Stage 1D Conv"]
    B --> C2["Mamba Memory Aggregation<br/>Clip Spatial Compression & State-Space Memory"]
    C2 --> D1["Hypergraph Relational Modeling<br/>K-NN Construction + HGNN Higher-Order Propagation"]
    C1 --> E["Hypergraph-Bridged Dual Attention Fusion<br/>SA Temporal Dynamics / CA Spatial Detail Retrieval"]
    D1 --> E
    C2 --> E
    E --> F["Sign Embedding & mBART Decoder<br/>Pre-trained Multilingual LM Autoregressive Generation"]
    F --> G["Output Target Spoken Sentences"]

Key Designs

1. Mamba-based Clip Memory Aggregation: Capturing Fine-Grained Spatial Details at Linear Complexity Standard self-attention applied directly across dense high-definition event tokens induces prohibitive quadratic complexity, whereas naive global pooling discards subtle hand shapes and finger positions. To resolve this dilemma, continuous video features \(F \in \mathbb{R}^{B \times T \times C' \times H' \times W'}\) are split along the time axis into \(n\) fixed-length clips (each consisting of 8 frames). For the \(i\)-th clip, global spatial pooling produces a compact summary token \(F_i^g \in \mathbb{R}^{B \times 1 \times C'}\), which is concatenated with the clip's fine-grained spatial tokens and processed by a bidirectional Vision Mamba (Vim) block. Governed by the selective state-space equation discretized via Zero-Order Hold (ZOH): $\(h' = \overline{A}h + \overline{B}x', \quad y = Ch'\)$ forward and backward SSM scans capture omnidirectional spatial context. The updated summary feature \(F_g^{(i)\prime}\) is recursively passed into the \((i+1)\)-th clip as an accumulated memory vector: $\(F_g^{(i+1)\prime}, F_c^{(i+1)\prime} = \text{Mamba}([F_g^{(i)\prime}, F_{i+1}^g, F_{i+1}^c])\)$ This mechanism progressively integrates historical spatial context across the entire sequence while maintaining linear complexity, yielding memory-enriched spatial features \(F_s\) and clip memory tokens \(F_g\).

2. Hypergraph-Guided Higher-Order Clip Relationship Modeling: Transcending Euclidean Proximity for Gesture Semantics Sign language exhibits complex non-local and co-occurring semantic relationships where actions in temporally distant clips share linguistic dependencies. Treating each compressed clip feature vector in \(F_g\) as a node, the method builds a K-nearest neighbor hypergraph (\(K=10\)) based on Euclidean feature distances. Unlike standard graphs where edges connect pairs of nodes, each hyperedge encompasses multiple related clip vertices. Using the node-hyperedge incidence matrix \(H\) alongside vertex and hyperedge degree matrices \(D_v, D_e\), the normalized hypergraph propagation matrix is formulated as: $\(G = D_v^{-\frac{1}{2}} H D_e^{-1} H^\top D_v^{-\frac{1}{2}}\)$ A Hypergraph Neural Network (HGNN) performs message passing over \(G\) via \(F'_g = G(F_g W + b)\). This models multi-clip semantic clusters, producing high-order relational representations \(F'_g\) that act as structural bridges.

3. Dual-Branch Attention Spatiotemporal Fusion via Graph Intermediaries: Mitigating Dense Cross-Modal Attention Overhead Directly computing full cross-attention between dense spatial tokens (which number in the hundreds or thousands) and temporal features leads to severe memory strain. The proposed GSTF module leverages compact graph features \(F'_g\) as an efficient intermediary across two parallel attention streams: $\(F_{SA}^t, F_{SA}^g = \text{SA}([F'_t, F'_g]), \quad F_{CA}^g = \text{CA}(Q=F'_g, K=F_s, V=F_s)\)$ In the self-attention (SA) branch, temporal features \(F'_t\) and graph tokens \(F'_g\) are concatenated to reinforce global motion coherence. In the cross-attention (CA) branch, the low-dimensional graph nodes serve as Queries to retrieve fine spatial details from Keys and Values formed by \(F_s\), keeping computation minimal. The resulting representations are fused via element-wise multiplication (\(F_{SA}^g \odot F_{CA}^g\)) and combined with temporal representations through a feedforward network, realizing synergistic spatiotemporal alignment.

4. Gloss-Free End-to-End Sequence Generation: Bypassing Intermediate Gloss Annotation Bottlenecks Conventional gloss-based SLT pipelines rely on frame-level or segment-level gloss supervision, suffering from high annotation expense and compounding cascading errors during inference. EvSLT adopts an end-to-end direct mapping strategy: the fused spatiotemporal representations are mapped through a lightweight sign embedding layer (Linear + ReLU) and fed directly into the cross-attention modules of a pre-trained mBART-large architecture (configured with 3 encoder and 3 decoder layers), directly outputting the spoken sentence under standard autoregressive text modeling.

A Worked Example

Consider translating a sentence with subtle count distinctions: "Five boys are standing on a stage in front of a tent": 1. Pulse Stacking & Feature Extraction: Raw asynchronous event pulses are binned into \(T=120\) frames over temporal windows and passed to ResNet-18, outputting downsampled feature maps of dimension \(120 \times C' \times H' \times W'\); 2. Dual Temporal-Spatial Encoding: The temporal branch performs spatial average pooling and 1D temporal convolution to encode continuous global arm kinematics; concurrently, the spatial branch partitions the sequence into \(n=15\) clips (8 frames each), where bidirectional Mamba scans sequentially update clip memories, preserving subtle hand shapes such as the five spread fingers indicating "five"; 3. Hypergraph Propagation & Attention: A 15-node hypergraph links semantically co-occurring clips ("boys", "standing", "stage", "tent"). In GSTF, graph features query the dense spatial memory for finger and object cues while self-attention aligns with arm trajectories; 4. Direct Generation: The aligned representation drives mBART decoding token-by-token to generate the precise target text, correctly capturing the numeral modifier "five" and spatial prepositions where models lacking spatial detail mistakenly output generic phrases like "A boy stands outside".

Loss & Training

The network is optimized end-to-end using standard cross-entropy loss over target sentence tokens \(S = \{\langle\text{bos}\rangle, w_1, w_2, \dots, w_U, \langle\text{eos}\rangle\}\): $\(\mathcal{L} = -\sum_{u=1}^{U} \log p(w_u \mid w_{<u}, \mathcal{E})\)$ Training utilizes the SGD optimizer with an initial learning rate of 0.01 and a cosine annealing scheduler across 200 epochs. The mini-batch size is set to 6 on a server equipped with NVIDIA A800 GPUs. Input frames are resized to 224×224 pixels.

Key Experimental Results

Main Results

EvSLT is evaluated on the newly proposed Event-CSL benchmark alongside the public EvSign benchmark against representative gloss-based and gloss-free baselines. On Event-CSL, EvSLT achieves state-of-the-art performance among all gloss-free approaches, outperforming ViT and LLM-based competitors. On EvSign, it outperforms all existing methods across every metric.

Table 1: Translation performance comparison on the Event-CSL benchmark

Algorithm	Venue	Backbone	Paradigm	ROUGE-L	BLEU-1	BLEU-2	BLEU-3	BLEU-4
Joint-SLT	CVPR 2020	ViT	Gloss-based	60.89	61.24	55.38	47.41	42.30
Chen et al.	CVPR 2022	S3D	Gloss-based	75.60	77.26	68.47	62.31	55.68
Sign-XmDA	EMNLP 2023	ViT	Gloss-based	65.81	66.96	60.18	52.54	48.15
TSPNet	NeurIPS 2020	I3D	Gloss-free	26.63	30.91	17.84	10.89	7.25
GASLT	CVPR 2023	ViT	Gloss-free	31.69	34.27	23.41	16.73	12.64
GFSLT	ICCV 2023	CNN	Gloss-free	67.23	69.00	60.62	53.77	48.20
SignCL	NeurIPS 2024	CNN	Gloss-free	67.76	69.12	60.88	54.18	48.52
Sign2GPT	ICLR 2024	ViT	Gloss-free	68.74	70.35	61.98	55.13	49.23
SpaMo	NAACL 2025	ViT	Gloss-free	67.78	68.59	61.27	55.12	49.08
MMSLT	ICCV 2025	CNN, BERT	Gloss-free	68.95	70.70	62.18	55.42	49.95
EvSLT (Ours)	ECCV 2026	CNN, Mamba	Gloss-free	69.80	71.36	63.01	56.28	50.76

Table 2: Translation performance comparison on the EvSign benchmark

Algorithm	Venue	Modality	Backbone	ROUGE-L	BLEU-1	BLEU-2	BLEU-3	BLEU-4
Joint-SLT	CVPR 2020	RGB	ViT	40.05	39.84	23.54	15.60	10.63
VAC+TH	ICCV 2021	RGB	CNN, BiLSTM	39.08	38.74	23.90	15.88	10.19
CorrNet+TH	CVPR 2023	RGB	CNN, BiLSTM	39.41	39.45	23.74	15.68	10.57
Joint-SLT	CVPR 2020	Event	ViT	41.54	40.13	24.36	16.04	10.87
GFSLT	ICCV 2023	Event	CNN	51.98	54.26	35.91	24.97	17.88
Zhang et al.	ECCV 2024	Event	CNN, BiLSTM, ViT	42.43	41.44	25.61	17.55	12.37
MMSLT	ICCV 2025	Event	CNN, BERT	52.14	54.29	37.68	26.09	19.12
EvSLT (Ours)	ECCV 2026	Event	CNN, Mamba	53.43	55.45	38.45	27.41	20.20

Ablation Study

Table 3: Component-wise ablation on the Event-CSL dataset

Config	Temporal Branch	Spatial Branch	Mamba Memory	GSTF Fusion	ROUGE-L	BLEU-4	Params (M)
(a) Temporal Only	✓	✗	✗	✗	67.23	48.20	115.5
(b) Spatial Only	✗	✓	✓	✗	56.31	35.12	119.8
(c) MLP w/o Mamba	✓	✓	✗	✓	69.14	50.08	134.4
(d) w/o GSTF Module	✓	✓	✓	✗	69.02	49.94	130.7
(e) Full Model (EvSLT)	✓	✓	✓	✓	69.80	50.76	138.2

Table 4: Analysis of fusion strategies and sequence backbones (Event-CSL)

Dimension	Variant	ROUGE-L	BLEU-1	BLEU-4	Note
Fusion Type	Concatenation	65.79	67.32	47.00	Simple concat lacks inter-modal conditioning
	Cross-Attention	69.02	70.47	49.94	Heavy computational overhead, lacks higher-order graph
	Ours w/o Graph	69.08	70.81	50.17	Drops hypergraph, reducing relational context
	Ours (GSTF)	69.80	71.36	50.76	Graph-bridged dual attention yields top accuracy
Backbone	BiLSTM	68.07	69.90	48.68	Memory decay across long sign video sequences
	Transformer	69.16	70.51	49.74	Quadratic scaling restricts dense spatial processing
	Mamba (Ours)	69.80	71.36	50.76	Linear selective state-space yields best representation

Key Findings

• Dominance and Complementarity of Temporal vs. Spatial Streams: The temporal-only baseline reaches 48.20 BLEU-4, far surpassing the spatial-only baseline (35.12 BLEU-4), verifying that sign semantics are rooted primarily in motion flow. However, incorporating fine spatial details lifts BLEU-4 by +2.56 points (from 48.20 to 50.76). Qualitative inspections confirm that spatial features are critical for differentiating fine-grained descriptive modifiers (e.g., number of objects, hand orientation).
• Mamba State-Space Superiority: Replacing Mamba with standard MLPs degrades BLEU-4 by 0.68, and Mamba outstrips the Transformer alternative by +1.02 BLEU-4, demonstrating that selective state-space recursions offer an ideal inductive bias for video clip memory propagation.
• Structural Value of Hypergraph Modeling: Omitting hypergraph modeling causes a 0.59 BLEU-4 drop. Hyperedges effectively group distant clips sharing higher-order linguistic associations, overcoming the localized receptive field of temporal convolutions.
• Superior Computational Efficiency: In execution profiling (Table 6 in the paper), EvSLT achieves an inference latency of 86 ms per video, which is approximately \(16\times\) faster than MMSLT (1382 ms) and nearly \(3\times\) faster than Sign2GPT (247 ms), while maintaining a compact footprint of 138.2M parameters and 252.0G FLOPs.

Highlights & Insights

• First High-Definition Real Event Dataset: Event-CSL establishes a benchmark with 14,827 full-body, unmasked videos at 1280×720 resolution recorded across indoor and outdoor settings, breaking free from synthetic toys and privacy-motivated head amputations that degraded prior resources.
• Graph-Bridged Asymmetric Attention: Instead of computing exhaustive attention across hundreds of spatial tokens, EvSLT compresses clips into hypergraph nodes to act as Queries against spatial Keys/Values. This asymmetric structure achieves fine-grained spatial indexing with negligible computational bloat.
• Cross-Domain Reusability: The paradigm of chunking dense event streams into short clips, maintaining temporal memory with Mamba, and capturing clip relations via hypergraph message passing is directly transferable to event-based micro-expression recognition, robotic hand manipulation tracking, and dynamic surgical instrument localization.

Limitations & Future Work

• Temporal Quantization into Frame Bins: Converting continuous microsecond event streams into discrete frame bins via temporal windows discards asynchronous temporal precision and sparsity advantages.
• Spatial Resolution Downsampling: Although the dataset is recorded at 1280×720 HD, practical training scales images down to 224×224 due to GPU memory constraints, discarding micro-edge details of distant finger joints.
• Future Directions: Exploring end-to-end continuous event backbones (e.g., spiking neural networks or point-trajectory models) and scaling self-supervised masked autoencoding pre-training on unannotated event sign streams.

Related Work & Insights

• vs. EvSign (ECCV 2024): EvSign introduced the first real event SLT dataset (6,773 videos) but deliberately masked facial and head regions, destroying crucial grammatical markers. Event-CSL preserves complete unmasked facial expressions at double the resolution (1280×720) and more than twice the sample volume.
• vs. GFSLT (ICCV 2023) & MMSLT (ICCV 2025): GFSLT relies predominantly on temporal convolutions, missing fine spatial details. MMSLT incorporates heavy language backbones (641.9M parameters, 1382 ms latency). EvSLT achieves higher translation fidelity (50.76 BLEU-4) with a lightweight 138.2M parameter model running at 86 ms per sample.

Rating

• Novelty: ⭐⭐⭐⭐☆ (First high-definition real-world event SLT benchmark; elegant Mamba memory and hypergraph-guided spatiotemporal fusion)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluates 10 diverse baselines across two benchmarks; thorough ablations across backbones, fusion strategies, frame rates, and latency)
• Writing Quality: ⭐⭐⭐⭐⭐ (Cohesive motivation, clean mathematical formulation, intuitive qualitative and quantitative proofs)
• Value: ⭐⭐⭐⭐⭐ (Establishes pivotal benchmark infrastructure and a performant baseline for event camera-based accessibility technology)