E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes¶

Conference: ECCV 2026
arXiv: 2604.04834
Code: Pending release
Area: robotics
Keywords: Event camera, VLA model, Low-light manipulation, Motion blur robustness, Event-count window, Multimodal fusion

TL;DR¶

E-VLA integrates event camera streams into the SmolVLA vision encoding pipeline via two lightweight methods—a zero-parameter overlay or a 13M hierarchical adapter—by converting event streams into frame-like accumulated representations through a novel event-count window. This elevates the robot Pick-and-Place success rate from 0% to 90% under extremely low light (20 lux), recovers it from 0% to 25% under severe motion blur (1000 ms), and maintains 45% success under OOD generalization at 20 lux using only 200 lux training data.

Background & Motivation¶

Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in open-world manipulation tasks by combining frame-based RGB vision encoders with language-conditioned policy learning. Representative systems like RT-2, OpenVLA, and SmolVLA share a common pathway: pre-training on large-scale multimodal data, followed by fine-tuning on robot interaction data. However, the perception of these models entirely relies on the same physical process—frame-based cameras integrating photons over an exposure window. While this is not an issue in bright laboratory environments, real-world deployment often involves sudden illumination drops, which severely deteriorate the signal-to-noise ratio. Increasing the exposure time to compensate for brightness inevitably introduces motion blur and extra latency; in extreme cases, the camera output becomes entirely black-clipped, leading to total information loss. More fundamentally, image enhancement and data augmentation can only operate on already degraded signals; if the information is irreversibly lost at the time of capture, post-processing cannot recover it from nothing.

Event cameras offer a fundamentally different imaging principle: asynchronously detecting pixel-level brightness changes instead of global exposure, featuring microsecond-level temporal resolution and a dynamic range exceeding 120 dB. This naturally renders event cameras robust against motion blur and enables them to stably capture edges and motion structures in scenes under extremely low light. However, integrating event perception into modern VLAs is not a simple sensor replacement—event streams are sparse, non-uniform, and severely mismatched with the dense and regular RGB pre-training distribution on which VLA models rely. Furthermore, wrist-mounted event cameras move with the robotic arm, causing event trigger rates to fluctuate wildly across different stages (e.g., approaching, grasping, and placing), resulting in highly unstable temporal distributions. Therefore, the core challenge lies in designing a representation conversion and fusion mechanism that is compatible with the pre-trained distribution while remaining stable under manipulation dynamics. The core idea of this work is to define windows based on event counts rather than fixed durations, accumulating the event stream into frame-like representations, and then incorporating event structural cues into the VLA vision encoding pipeline via either zero-parameter pixel-level overlay or light-weight adapters with hierarchical fusion modules without altering the pre-trained token distribution.

Method¶

Overall Architecture¶

E-VLA uses SmolVLA as its baseline skeleton. A DAVIS346 event camera outputs an asynchronous event stream ($(x,y,t,p)$ with microsecond-level timestamps) alongside RGB frames (30 FPS), which are naturally aligned spatially. The event stream first selects the latest $N=2000$ events via a "recency event-count window", projects them into a polarity-independent grayscale map based on pixel locations, and then normalizes and demosaics it to obtain a frame-like three-channel representation $E$. This event frame can then enter one of two fusion pathways: (1) Overlay—directly overlaying events onto RGB pixels according to spatial positions (pixels with events are replaced by polarity-mapped colors), introducing zero parameters, and feeding it into the SigLIP encoder; (2) Hierarchical Event Adapter—the event frame passes through a lightweight 4-layer ViT (hidden dimension 384 + weight-sharing patch embedding), progressively fusing with image features layer-by-layer at layers 3, 6, 9, and 12 of the SigLIP encoder via $\text{MLP}(\text{Concat}(\cdot))$. The fused visual tokens are concatenated with the language instruction tokens and robot state tokens, before being fed into a frozen SmolVLM LLM backbone. Finally, a 40-step action chunk is decoded by an action expert via cross-attention (asynchronously executed at 30 Hz).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    EV["Event Stream<br/>(x,y,t,p)"] --> WIN["Recency Count Window<br/>2000 Event Accumulation"]
    WIN --> REP["Accumulated Grayscale<br/>→ Norm → Demosaic"]
    REP --> SPLIT{"Fusion Strategy"}

    RGB["RGB Frame"] --> SPLIT

    SPLIT -->|Overlay| OVL["Zero-parameter Overlay<br/>Pixel-level Replacement"]
    SPLIT -->|Adapter| ADP["Hierarchical Event Adapter<br/>4-layer ViT + Progressive Fusion"]

    OVL --> SIG["Frozen SigLIP<br/>Vision Encoder"]
    ADP --> FUSE["MLP-Concat Fusion<br/>at layers 3/6/9/12 of SigLIP"] --> SIG

    SIG --> TOK["Fused Vision Tokens"]
    LANG["Language Instruction"] --> LLM["Frozen SmolVLM<br/>LLM Backbone"]
    STATE["Robot State Tokens"] --> LLM
    TOK --> LLM
    LLM --> AE["Action Expert<br/>Cross-Attention"]
    AE --> ACT["40-step Action Chunk<br/>@30Hz Async Execution"]

Key Designs¶

1. Recency Event-Count Window: Tailoring Event Representation Conversion for VLA Manipulation Dynamics

The critical premise of E-VLA is converting sparse, asynchronous event streams into a frame-like structure compatible with ViT encoders. The simplest strategy is a fixed-duration window (e.g., collecting events every 20 ms), but this commits a fundamental error in robot manipulation scenarios: the wrist-mounted camera moves with the robotic arm, meaning movement is slow and events are scarce during approach, while movement is fast and events surge during post-grasp motion. A fixed duration results in sparse events (perception failure) at low speeds and event piling (motion-blur-like artifacts) at high speeds. The core insight of E-VLA is to define the window via event count rather than time—each time slicing the latest $N$ chronologically sorted events. At low speeds, the window naturally stretches to collect enough events; at high speeds, it automatically shrinks to avoid overshooting. Given the current frame's exposure end time $t_e$, the window is defined as: $$\mathcal{W}_I = \{e_k\}_{k=|\mathcal{E}_{t_e}|-N+1}^{|\mathcal{E}_{t_e}|}$$ where $\mathcal{E}_{t_e} = \{e_i \mid t_i \leq t_e\}$. Experiments show that a 2000-event window maintains a stable success rate of 90-100% across the full illumination range (20-75 lux), far exceeding the best fixed-duration window (20 ms drops to 45% at the low-light end).

2. Zero-Parameter Overlay Fusion: The Simplest is Often the Most Effective

Because the events and RGB frames corresponding to the DAVIS camera are naturally aligned spatially, an extremely simple fusion scheme emerges: directly overlaying spatial and polarity information of the events onto RGB pixels—pixels with events are replaced by color-mapped polarity, while pixels without events retain their original RGB values. This introduces zero extra parameters and virtually zero FLOPs, rendering it purely plug-and-play. Surprisingly, this brute-force approach outperforms sophisticated image enhancement methods (RetinexNet, Retinexformer, EvLight, E2VID) in almost all low-light conditions: jumping from 0% (image baseline) to 60% under 20 lux, and to 20-25% under 1000 ms motion blur. The underlying reason is that VLA models struggle for spatial structural cues indicating rough object locations rather than aesthetically pleasing pseudo-images; the high-frequency edges provided by events are sufficient when RGB is completely degraded. Especially in black-clipped scenes, the event signal becomes the sole channel providing spatial information. However, the overlay scheme only reaches 60% under 20 lux, indicating that the upper bound of zero parameters is constrained by the underutilization of event information.

3. Hierarchical Event Adapter: Progressive Cross-Modal Feature Interaction

The Overlay method involves only pixel-level one-way replacement, preventing bi-directional interaction between events and RGB in deep feature spaces. To address this, E-VLA designs a Hierarchical Event Adapter: a compact 4-layer ViT (hidden dimension 384, with only 13.3M parameters, < 3% of the total model) that independently encodes event frames, and then progressively injects event features into image features at layers 3, 6, 9, and 12 of the SigLIP encoder using MLP(Concat). Formally, letting $\mathcal{G}_l$ be the $l$-th layer of the adapter and $\mathcal{F}_l$ be the $l$-th group of layers in SigLIP, the fusion process is formulated as: $$E^{(l+1)} = \mathcal{G}_{l+1}(E^{(l)})$$ $$F^{(l+1)} = \mathcal{F}_{l+1}(\text{Fuse}(F^{(l)}, E^{(l)}))$$ where $\text{Fuse}$ is MLP(Concat). This design exhibits three layers of elegance: first, the lightweight adapter combined with the frozen massive encoder avoids the need for large-scale event pre-training from scratch; second, progressive fusion allows event features to interact with image features at different granularities instead of destroying the pre-trained distribution through a one-off merge; third, a weight-sharing patch embedding ensures mapping consistency between event frames and RGB frames at the low level, mitigating convergence difficulties during cross-modal fusion. Consequently, the event adapter achieves 90% at 20 lux (vs. 60% with Overlay) and still retains 35% under extreme 2 lux conditions—where the RGB frame has long been completely black-clipped. In OOD generalization tests (trained only at 200 lux), the adapter maintains 45% at 20 lux, significantly leading the image baseline's 0%.

Loss & Training¶

Training consists of three phases. Action stage (20k iter, lr=2e-4): Freeze the VLM backbone and only train the action expert and projection layers to adapt the model to the deployment environment. Event stage (10k iter, lr=5e-4): Train the event adapter and fusion modules from scratch, while freezing all other components. Joint stage (10k iter, lr=1e-4): Jointly fine-tune the adapter, fusion modules, action expert, and projection layers. A key trick is applying a 50% random dropout to the image branch—forcing the model to actively rely on event signals during training rather than taking a shortcut when image signals are available. Ablation shows that with 0% dropout, performance under low-to-medium light drops by about 14 percentage points, whereas an 80% dropout harms performance under normal illumination.

Key Experimental Results¶

Main Results¶

Low-light Manipulation (Pick-Place, Success Rate %)

Method	75 lux	40 lux	35 lux	30 lux	25 lux	20 lux	Average
Image Baseline	100	80	70	35	0	0	47.5
+RetinexNet	100	100	85	80	25	10	66.7
+Retinexformer	100	80	80	75	20	10	60.8
+EvLight	100	95	95	75	45	10	70.0
+E2VID (Reconstruction)	80	60	55	10	5	5	35.8
Overlay	100	100	85	75	65	60	80.8
Event Adapter	100	100	95	90	90	90	94.2

Motion Blur (1000 ms Exposure, Success Rate %)

Method	Pick-Place	Sorting
Image Baseline	0	5
Overlay	20	32.5
Event Adapter	25	32.5

OOD Generalization (Trained only at 200 lux, Pick-Place Average %)

Method	Average
Image Baseline	39.0
+Retinexformer	46.0
+EvLight	40.5
Overlay	54.0
Event Adapter	75.0

Ablation Study¶

Configuration	Pick-Place Average %	Note
Full Model (2000 events + Adapter)	94.2	Best
Overlay Only	80.8	Zero-parameter, already outperforms all enhancement baselines
500-Event Window	67.5	Too few events, insufficient perceptual information
4000-Event Window	85.8	Too many events, introduces stale informational noise
5 ms Fixed-duration Window	60.8	Sparse events at low speeds cause perception failure
20 ms Fixed-duration Window	82.5	Best fixed-duration but still inferior to event-count window
No Image Dropout (~estimated)	≈80	Model takes RGB shortcut, drops under low light
Action→Joint (Without Event Stage)	75.0	Skipping event training leads to under-learned features
Action→Event→Joint (Without Shared Weights)	80.0	Shared patch embedding contributes an additional ~10%

Key Findings¶

Ranking of Module Contributions: Event Adapter (+46.7 percentage points over baseline) > Overlay (+33.3) > EvLight (+22.5) > RetinexNet (+19.2) > E2VID (-11.7, worse than baseline). The 13.4 percentage point advantage of the Adapter over Overlay primarily stems from the extremely low light regime (20 lux: +30)
Event-Count Window is a Fundamental Design Choice: Replacing it with a 20 ms fixed duration drops the average success rate from 94.2% to 82.5%, indicating that the windowing strategy is more fundamental than the fusion scheme—signal quality matters more than fusion granularity
Event Insensitivity to Color is a Physical Ceiling: Under 20 lux, the Event Adapter only achieves 70% in the Sorting task (vs. 90% in Pick-Place), as the accumulated event representation discards polarity direction information and cannot reliably distinguish absolute colors
Cross-Backbone Generality: On the pi0.5 backbone, the Event Adapter improves performance from 55.8% to 95.0%, showing that the method is not coupled to SmolVLA
Data Augmentation is Insufficient to Bridge the Gap: Aggressive augmentation (brightness reduction + directional blur) only yields a marginal improvement from 47.9% to 51.7%, validating that perceptual level degradation cannot be salvaged by late-stage augmentations
Extreme Low-light Boundaries: The Event Adapter still maintains 80% at 4 lux and only drops to 35% at 2 lux, where event sensor noise begins to dominate

Highlights & Insights¶

Design Philosophy of Lightweight Event Fusion: No image reconstruction, no increase in the number of visual tokens, and no alteration of the pre-trained distribution. Overlay has zero parameters, and the adapter requires only 13M—this minimally invasive design allows E-VLA to be plugged-and-played into any ViT-based VLA backbone, lowering the entry barrier for event-based VLAs.
"Resurrection" Capability Under Black Clipping: Under 25 lux and below, the image baseline completely collapses (0%), while the event adapter still holds 90% success. This contrast visually demonstrates that the blind spots of frame-based cameras are precisely the sweet spots of event cameras, forming an ideal sensing complement.
The Balancing Art of Image Dropout: Randomly dropping half of the images during training forces the model to learn to rely solely on event cues in the absence of RGB information. The 50% balance point itself represents a valuable training strategy discovery.
Edge Structural Cues > Reconstructed Images: E2VID, which reconstructs events into RGB frames before ingestion, performs even worse than the image baseline (35.8% average vs. 47.5%). In contrast, directly utilizing the edge structural cues of events brings a huge boost—verifying that under degraded conditions, VLA needs spatial structure rather than aesthetically enhanced pseudo-images.

Limitations & Future Work¶

Event Insensitivity to Color: The accumulated event representation discards polarity direction information, which prevents reliable distinction of object colors. The Sorting task success rate is about 20 percentage points lower than Pick-Place. Potential improvements include specialized color demosaicing methods (such as Color4e) or combining RGB residual information for explicit color recovery.
Wrist Camera Occlusion: Operated objects naturally block the field of view of the wrist-mounted camera, limiting Stacking success to only 40% under 20 lux. This perceptual bottleneck exists in both event and RGB modes and is a generic constraint, which can be mitigated by improving sensor placement or introducing a third-person view camera.
Passive Event Utilization: Currently, E-VLA primarily responds to brightness changes triggered by the robot arm's movement. Static but crucial objects in the scene (e.g., target cups) struggle to generate signals. Introducing active illumination modulation or active motion jitter to "light up" static regions could be a breakthrough.
Data Scarcity: With only 724 episodes, 3 tasks, and a single platform (SO100 + DAVIS346), the scene diversity is limited. Larger-scale, multi-scene datasets are required to validate generalization capabilities.
OOD Generalization Gap: The Event Adapter's success rate drops from 90% (ID) to 45% (OOD) under 20 lux. Although significantly better than the baseline (0%), it is still not practical enough. Training on a broader range of illuminations and domain adaptation strategies could further narrow the gap.

vs SmolVLA: E-VLA directly superimposes the event fusion module onto SmolVLA without changing the backbone. This demonstrates that event perception can seamlessly interface with existing VLA frameworks as a lightweight plug-in, providing a baseline design for subsequent event-based VLA research.
vs EvLight / RetinexNet / Retinexformer: Image enhancement cannot restore information once the signal is severely degraded (achieving at most 10% under 20 lux). E-VLA introduces complementary sensors at the acquisition end, fundamentally bypassing the "restoration ceiling".
vs E2VID: Event-to-image reconstruction performs the worst in manipulation scenarios (even lower than the image baseline). This is because event rates mutate heavily with mechanical action in VLAs, making reconstruction quality highly unstable. Abandoning reconstruction and directly utilizing events as structural cues is a key design decision of E-VLA.
vs Traditional Event Robotic Systems: Most event robotics works (grasping, obstacle avoidance, slip detection) employ modular pipelines. E-VLA is the first to integrate event perception into an end-to-end, general-purpose VLA learning framework, maintaining policy generality and language-conditioning capabilities.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First to systematically integrate event cameras into the VLA framework, with a precise entry point (perception bottleneck vs. common algorithm-side improvements). The open-sourced teleoperation platform and dataset will continuously drive the domain.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 3 tasks × 6 illumination levels × motion blur × OOD generalization × window strategy ablation × training strategy ablation × event representation ablation × cross-backbone validation × extreme low-light boundary experiments—the depth of control-variable experiments is uncommon in VLA papers.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational logic (physical limitations of frame-based imaging $\rightarrow$ event complementary advantages $\rightarrow$ three major fusion challenges), with the methodology step-by-step progressing from the simple Overlay to the Adapter, accompanied by in-depth experiments and honest error analysis.
Value: ⭐⭐⭐⭐⭐ Paves a brand-new, physically-grounded technical route for the perceptual robustness of VLAs; design insights such as the windowing and training strategies have universal guiding significance for the entire event VLA field.