E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes¶

Conference: ECCV 2026
arXiv: 2604.04834
Code: To be released
Area: robotics
Keywords: Event camera, VLA model, Low-light manipulation, Motion blur robustness, Event count window, Multimodal fusion

TL;DR¶

E-VLA transforms event camera streams into frame-like accumulated representations through a recent event count window. It then integrates these representations into the SmolVLA visual encoding pipeline via two lightweight approaches: zero-parameter overlay or a 13M hierarchical adapter. This enables the robot's Pick-and-Place success rate to jump from 0% to 90% under extremely low illumination of 20 lux, recovery from 0% to 25% under severe motion blur of 1000 ms, and maintaining 45% success under 20 lux in OOD generalization tests training only on 200 lux data.

Background & Motivation¶

Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in open-ended manipulation tasks by combining frame-based RGB visual encoders with language-conditioned policy learning. The common path of representative systems like RT-2, OpenVLA, and SmolVLA is pre-training on large-scale multimodal data and then fine-tuning on robot interaction data. However, the perception of these models entirely relies on the same physical process—frame-based cameras integrating the total number of photons within an exposure window. While this is not an issue in bright laboratory environments, real-world deployment with sharp drops in illumination severely deteriorates the signal-to-noise ratio. Increasing exposure time to compensate for brightness inevitably introduces motion blur and additional latency; in extreme cases, the camera output saturates completely to black (black clipping), resulting in a total loss of information. Fundamentally, image enhancement and data augmentation can only operate on already degraded signals—if information is irreversibly lost at the moment of acquisition, post-processing cannot recover it from thin air.

Event cameras offer a drastically different imaging principle: asynchronously detecting pixel-wise luminance changes instead of global exposure, possessing microsecond-level temporal resolution and a dynamic range of over 120 dB. This means event cameras are naturally robust to motion blur and can still stably capture edges and motion structures in scenes under extremely low light. However, inserting event perception into contemporary VLAs is not as simple as replacing sensors—event streams are sparse, non-uniformly distributed, and severely mismatch the dense and regular RGB pre-training distribution that VLA models rely on. Furthermore, wrist-mounted event cameras move with the robotic arm, causing event trigger rates to fluctuate wildly across different stages such as approaching, grasping, and placing, which leads to highly unstable temporal distributions. Therefore, the core challenge is designing a representation conversion and fusion mechanism that is compatible with the pre-trained distribution while remaining stable under manipulation dynamics. The core idea of this paper is to define windows by event count rather than fixed duration, accumulating the event stream into frame-like representations, and then using zero-parameter pixel-level overlay or a lightweight adapter with hierarchical fusion modules to integrate event structural cues into the VLA visual encoding pipeline without altering the pre-trained token distribution.

Method¶

Overall Architecture¶

E-VLA uses SmolVLA as its baseline backbone. A DAVIS346 event camera simultaneously outputs an asynchronous event stream (\((x,y,t,p)\) with microsecond-precision timestamps) and RGB frames (at 30 FPS), which are naturally aligned spatially. First, the event stream selects the most recent \(N=2000\) events using a "recent count window", which are accumulated based on pixel positions into a polarity-independent grayscale map, and then normalized and demosaiced to obtain a frame-like three-channel representation \(E\). This event frame can then be integrated into one of two fusion paths: (1) Overlay—directly overlaying events onto RGB pixels based on spatial location (replacing pixels with events using a polarity-mapped color), which requires zero additional parameters and is sent to SigLIP for encoding; (2) Hierarchical Event Adapter—the event frame is fed into a lightweight 4-layer ViT (hidden dimension of 384 + weight-shared patch embedding), and progressively fused with image features layer-by-layer at the 3rd, 6th, 9th, and 12th layers of the SigLIP encoder via \(\text{MLP}(\text{Concat}(\cdot))\). After fusion, the visual tokens are concatenated with language instruction tokens and robot state tokens, and sent to the frozen SmolVLM LLM backbone. Finally, an action expert decodes a 40-step action chunk via cross-attention (executed with 30 Hz asynchronous queue streaming).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    EV["Event Stream<br/>(x,y,t,p)"] --> WIN["Recent Count Window<br/>2000 Event Accumulation"]
    WIN --> REP["Accumulated Grayscale<br/>→Norm→Demosaic"]
    REP --> SPLIT{"Fusion Strategy"}

    RGB["RGB Frame"] --> SPLIT

    SPLIT -->|Overlay| OVL["Zero-param Overlay<br/>Pixel-level Replacement"]
    SPLIT -->|Adapter| ADP["Hierarchical Event Adapter<br/>4-layer ViT + Gradual Fusion"]

    OVL --> SIG["Frozen SigLIP<br/>Vision Encoder"]
    ADP --> FUSE["MLP-Concat Fusion<br/>at SigLIP Layers 3/6/9/12"] --> SIG

    SIG --> TOK["Fused Visual Tokens"]
    LANG["Language Instruction"] --> LLM["Frozen SmolVLM<br/>LLM Backbone"]
    STATE["Robot State Tokens"] --> LLM
    TOK --> LLM
    LLM --> AE["Action Expert<br/>Cross-attention"]
    AE --> ACT["40-step Action Chunk<br/>@30Hz Async Queue Stream"]

Key Designs¶

1. Recent Count Window: Event Representation Conversion Tailored for VLA Manipulation Dynamics

The key premise of E-VLA is that the sparse, asynchronous event stream must be transformed into a frame-like structure to be consumed by the ViT encoder. The simplest strategy would be a fixed-duration window (e.g., extracting a batch of events every 20 ms), but this makes a fundamental error in robot manipulation scenarios: the wrist-mounted camera moves with the robotic arm; when approaching an object, the motion is slow and events are scarce, whereas when moving after grasping, the motion is fast and events burst. A fixed-duration window results in sparse events (perception failure) at low speeds, and event accumulation (causing motion-blur-like artifacts) at high speeds. The core insight of E-VLA is to use event count rather than time to define the window—extracting the latest \(N\) events sorted chronologically at each step. At low speeds, the window naturally stretches to collect enough events; at high speeds, it automatically shrinks to avoid overshooting. Given the current frame exposure end time \(t_e\), the window is defined as \(\mathcal{W}_I = \{e_k\}_{k=|\mathcal{E}_{t_e}|-N+1}^{|\mathcal{E}_{t_e}|}\), where \(\mathcal{E}_{t_e} = \{e_i \mid t_i \leq t_e\}\). Experiments show that a 2000-event window maintains a stable success rate of 90-100% across the entire illumination range (20-75 lux), vastly outperforming the best fixed-duration window (the 20 ms window drops to 45% at the low-light end).

2. Zero-parameter Overlay Fusion: Simple is Often Most Effective

Since the events and RGB frames of the DAVIS camera are naturally aligned spatially, an extremely simple fusion scheme emerges: directly overlaying the location and polarity information of events onto RGB pixels—pixels with events are replaced with polarity-mapped colors, and pixels without events retain their original RGB values. This introduces zero additional parameters and almost zero FLOPs, representing a purely plug-and-play approach. Surprisingly, this brute-force approach outperforms elaborately designed image enhancement methods (RetinexNet, Retinexformer, EvLight, E2VID) under almost all low-light conditions: pulling the image baseline from 0% to 60% under 20 lux, and from 0% to 20-25% under 1000 ms motion blur. The underlying reason is that VLA models truly need spatial structural cues showing where the object roughly is, rather than aesthetically pleasing pseudo-images; the high-frequency edges provided by events are just sufficient when RGB is completely degraded. Especially in black-clipping scenarios, the event signal is the only channel still providing spatial information. However, the overlay scheme only achieves 60% at 20 lux, indicating that the upper limit of the zero-parameter approach is constrained by insufficient utilization of event information.

3. Hierarchical Event Adapter: Progressive Cross-modal Feature Interaction

Overlay only performs pixel-level unidirectional replacement, failing to enable bidirectional interaction between events and RGB in the deep feature space. E-VLA further designs a hierarchical event adapter: a compact 4-layer ViT (hidden dimension of 384, featuring only 13.3M parameters, which is less than 3% of the total model) independently encodes the event frame. Then, at layers 3, 6, 9, and 12 of the SigLIP visual encoder, event features are progressively injected into the image features via MLP(Concat). Mathematically, let \(\mathcal{G}_l\) be the \(l\)-th layer of the adapter and \(\mathcal{F}_l\) be the \(l\)-th group layer of SigLIP. The fusion process is defined as: \(E^{(l+1)} = \mathcal{G}_{l+1}(E^{(l)})\) and \(F^{(l+1)} = \mathcal{F}_{l+1}(\text{Fuse}(F^{(l)}, E^{(l)}))\), where \(\text{Fuse}\) is MLP(Concat). There are three exquisite aspects to this design: first, a lightweight adapter combined with a frozen large encoder avoids the need for large-scale event pre-training from scratch; second, progressive fusion allows event features to interact with image features step-by-step at different granularities instead of a one-time merge that disrupts distributions; third, the weight-shared patch embedding ensures consistent low-level projection of event and RGB frames, reducing convergence difficulty in cross-modal fusion. Consequently, the event adapter achieves 90% success under 20 lux (vs. 60% with Overlay), and maintains 35% even under the extreme condition of 2 lux—where RGB has long been completely black-clipped. In OOD generalization tests (trained only at 200 lux), the adapter still maintains 45% at 20 lux, significantly leading the image baseline of 0%.

Loss & Training¶

Training consists of three phases. Action stage (20k iter, lr=2e-4): The VLM backbone is frozen, and only the action expert and projection layers are trained to adapt the model to the deployment environment. Event stage (10k iter, lr=5e-4): The event adapter and fusion modules are trained from scratch while freezing other components. Joint stage (10k iter, lr=1e-4): Jointly fine-tune the adapter, fusion modules, action expert, and projection layers. A key technique is applying 50% random dropout to the image branch—forcing the model to actively rely on event signals during training instead of taking shortcuts when images are available. Ablation studies show that with 0% dropout, performance drops by about 14 percentage points under medium-to-low light, whereas 80% dropout impairs performance under normal light.

Key Experimental Results¶

Main Results¶

Low-Light Manipulation (Pick-Place, Success Rate %)

Method	75 lux	40 lux	35 lux	30 lux	25 lux	20 lux	Average
Image Baseline	100	80	70	35	0	0	47.5
+RetinexNet	100	100	85	80	25	10	66.7
+Retinexformer	100	80	80	75	20	10	60.8
+EvLight	100	95	95	75	45	10	70.0
+E2VID (Reconstruction)	80	60	55	10	5	5	35.8
Overlay	100	100	85	75	65	60	80.8
Event Adapter	100	100	95	90	90	90	94.2

Motion Blur (1000 ms Exposure, Success Rate %)

Method	Pick-Place	Sorting
Image Baseline	0	5
Overlay	20	32.5
Event Adapter	25	32.5

OOD Generalization (Trained only at 200 lux, Pick-Place Average %)

Method	Average
Image Baseline	39.0
+Retinexformer	46.0
+EvLight	40.5
Overlay	54.0
Event Adapter	75.0

Ablation Study¶

Configuration	Pick-Place Average %	Description
Full Model (2000 events + Adapter)	94.2	Optimal
Overlay-only Fusion	80.8	Zero parameters, already outperforms all enhancement baselines
500-event Window	67.5	Too few events, insufficient perceptual information
4000-event Window	85.8	Too many events, introducing stale information noise
5 ms Fixed-duration Window	60.8	Sparse events at low speed lead to perception failure
20 ms Fixed-duration Window	82.5	Optimal fixed-duration but still lagging behind the count window
No Image Dropout (~estimate)	≈80	Model takes RGB shortcuts, degrading in low light
Action→Joint (Without Event Stage)	75.0	Skipping event training leads to under-learned features
Action→Event→Joint (No Shared Weights)	80.0	Shared patch embedding contributes an extra ~10%

Key Findings¶

Ranking of Module Contributions: Event Adapter (+46.7 pp vs. baseline) > Overlay (+33.3) > EvLight (+22.5) > RetinexNet (+19.2) > E2VID (-11.7, worse than baseline). The 13.4 percentage point (pp) advantage of the Adapter relative to Overlay primarily stems from the extremely low-light range (20 lux: +30 pp).
Count Window is a Fundamental Design Choice: Replacing it with a 20 ms fixed-duration window drops the average success rate from 94.2% to 82.5%, indicating that the window strategy is more fundamental than the fusion strategy—signal quality is more critical than fusion sophistication.
Event Insensitivity to Color is a Physical Ceiling: Under 20 lux, the Event Adapter only achieves 70% in the Sorting task (vs. 90% in Pick-Place) because the accumulated event representation discards polarity direction information and cannot reliably distinguish absolute colors.
Cross-Backbone Generalizability: On the pi0.5 backbone, the Event Adapter boosts performance from 55.8% to 95.0%, indicating that the method is not bound to SmolVLA.
Data Augmentation is Insufficient to Bridge the Gap: Aggressive augmentation (brightness reduction + directional blur) only yields a marginal rise from 47.9% to 51.7%, validating that perception-level degradation cannot be compensated for by post-hoc augmentation.
Extremely Low-light Boundary: The Event Adapter still maintains 80% at 4 lux and only drops to 35% at 2 lux, where event sensor noise begins to dominate.

Highlights & Insights¶

Design Philosophy of Lightweight Event Fusion: No image reconstruction, no increase in the number of visual tokens, and no alteration of the pre-trained distribution. Overlay requires zero parameters, and the adapter has only 13M—this minimally invasive design allows E-VLA to be plug-and-play with any ViT-based VLA backbone, lowering the entry barrier for event-based VLAs.
Ability to "Revive" Under Black Clipping: Below 25 lux, the image baseline collapses completely (0%), while the Event Adapter still maintains 90%. This contrast most intuitively demonstrates that the blind spot of frame-based cameras is precisely the comfort zone of event cameras, forming an ideal complement in the perception dimension.
The Art of Balancing Image Dropout: Randomly dropping half of the images during training forces the model to learn to operate solely using event cues when RGB information is unavailable—this 50% balance point is itself a valuable training strategy discovery.
Edge Structural Cues > Reconstructed Images: Reconstruction of events to RGB via E2VID before inputting actually yields poorer performance than the image baseline (average 35.8% vs. 47.5%). Conversely, directly utilizing the edge structural cues of events brings a massive boost—indicating that what VLAs need in degraded conditions is spatial structural information, rather than aesthetically pleasing pseudo-images.

Limitations & Future Work¶

Event Insensitivity to Color: The accumulated event representation discards polarity direction information, making it unable to reliably distinguish object colors. The Sorting task is about 20 percentage points lower than the Pick-Place task. Potential improvement directions include dedicated color demosaicing methods (such as Color4e) or combining residual RGB information for explicit color recovery.
Wrist-mounted Camera Occlusion: The manipulated object naturally occludes the field of view of the wrist-mounted camera, limiting the Stacking success rate to only 40% under 20 lux. This perceptual bottleneck exists in both event and RGB modes, representing a general limitation. This can be mitigated by improving the mounting position or introducing a third-person perspective camera.
Passive Event Utilization: Currently, E-VLA primarily responds to illumination changes triggered by the movement of the robotic arm. Stationary but critical objects in the scene (such as a target cup) struggle to generate signals within the event stream. Introducing active illumination modulation or active motion jittering to "illuminate" static areas could be a potential breakthrough.
Data Scarcity: With only 724 episodes, 3 tasks, and a single platform (SO100 + DAVIS346), the scene diversity is limited. Larger-scale datasets across diverse scenarios are required to validate generalization performance.
OOD Generalization Gap: The Event Adapter's success rate drops from 90% (ID) to 45% (OOD) under 20 lux. Although far superior to the baseline (0%), this is still not practical enough. Training across a wider range of illuminations and employing domain adaptation strategies could further narrow the gap.

vs. SmolVLA: E-VLA directly superimposes the event fusion module onto SmolVLA without altering the backbone. This demonstrates that event perception can seamlessly plug into existing VLA frameworks as a lightweight add-on, providing a baseline design for future event-based VLA research.
vs. EvLight / RetinexNet / Retinexformer: Image enhancement methods fail to recover information after the signal is already severely degraded (capped at 10% under 20 lux). E-VLA introduces complementary sensors from the acquisition side, fundamentally bypassing the "restoration ceiling".
vs. E2VID: Event-to-image reconstruction performs the worst in manipulation scenarios (even lower than the image baseline). This is because event rates mutate heavily with actions in VLA, leading to highly unstable reconstruction quality. E-VLA's key design decision is abandoning reconstruction and directly utilizing events as structural cues.
vs. Traditional Event-based Robotic Systems: Most prior event-based robotics works (grasping, obstacle avoidance, slip detection) adopt modular pipelines. E-VLA is the first to integrate event perception into an end-to-end general VLA learning framework, maintaining policy generalization and language conditioning capabilities.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Systematically integrates event cameras into the VLA framework for the first time. The entry angle is highly precise (targeting perceptual bottlenecks rather than common algorithmic improvements), and the open-sourced teleoperation platform and dataset will continuously drive the field forward.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 3 tasks × 6 illumination levels × motion blur × OOD generalization × window strategy ablation × training strategy ablation × event representation ablation × cross-backbone validation × extremely low-light boundary experiments—the depth of the controlled variable experiments is exceptionally rare in VLA literature.
Writing Quality: ⭐⭐⭐⭐⭐ Extremely clear motivational logic (physical limitations of frame-based imaging \(\rightarrow\) complementary advantages of events \(\rightarrow\) three key challenges of fusion). The method progressively advances from the simple Overlay to the Adapter, backed by layer-by-layer experimental validation and transparent error analysis.
Value: ⭐⭐⭐⭐⭐ Paves a brand-new, physically grounded technical path for securing VLA perceptual robustness; design insights such as the window strategy and training strategy have universal guiding significance for the entire event-based VLA domain.