Intra-Class Consistency Guided Class-Agnostic Event Segmentation¶
Conference: ECCV 2026
Paper: ECCV Paper
Code: To be released
Area: Semantic Segmentation
Keywords: Event Cameras, Class-Agnostic Segmentation, Intra-Class Consistency, Knowledge Distillation, SAM Adaptation
TL;DR¶
To tackle weak intra-class consistency caused by the spatial sparsity of event data, IC2-ESeg explicitly recovers noise, motion, and brightness representations in a learning-free, physics-grounded manner, injecting them into an image encoder via staged adapters and transferring SAM's open-world segmentation capability via cross-modal distillation.
Background & Motivation¶
Class-agnostic segmentation aims to partition visual scenes into generic foreground entities and background stuff based on universal "objectness" rather than predefined closed-world category labels. The rise of vision foundation models, notably the Segment Anything Model (SAM), has established unprecedented zero-shot generalization in open-world segmentation by leveraging web-scale pre-training. However, standard frame-based cameras suffer severe degradation under extreme conditions such as high-speed motion blur and over/under-exposure. Neuromorphic event cameras, which operate with microsecond-level temporal resolution, high dynamic range, and low latency, provide a compelling alternative for robust perception in these demanding visual regimes.
Extending class-agnostic segmentation to the event domain, however, encounters a severe modality gap. Existing approaches like EventSAM seek to transfer SAM's capabilities to events but restrict adaptation to high-level token alignments, neglecting fundamental discrepancies at the input level. While RGB images possess dense pixel intensity and texture continuity that naturally yield strong intra-class consistency, event streams asynchronously record only sparse changes in logarithmic illumination. Consequently, event data primarily appear as fragmented boundary edges, leaving object interiors devoid of signal. This absence of intra-class consistency severely undermines dense mask prediction, inducing foreground-background confusion, object adhesion, and broken details.
Addressing the weak intra-class consistency of event data requires mining the underlying physical mechanisms that generate events. Isolated shot noise events quantitatively encode scene brightness; spatio-temporal event distributions govern the local motion field; and event triggering density reflects the inner product between brightness gradients and velocity. Core Idea: Construct noise, motion, and brightness representations in a learning-free manner derived from the physical event generation mechanism, integrate them into the image encoder via staged residual and visual prompt adapters, and perform cross-modal distillation from SAM to achieve robust class-agnostic event segmentation.
Method¶
Overall Architecture¶
The IC2-ESeg framework comprises three coordinated modules: the learning-free Intra-Class Consistency Mining (ICM) module, the Lightweight Information Injection (LIJ) module, and the teacher-student Cross-Modal Distillation (CMD) module. The system takes discretized 3-bin event voxel grids as baseline input. The ICM module derives complementary noise, motion, and brightness representations from the raw event stream. The LIJ module integrates these heterogeneous physical representations into the student image encoder via staged residual addition and visual prompt adapters. Finally, the CMD module supervises the student network using feature-level distillation from a frozen SAM ViT-H teacher and offline pseudo-label mask loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Asynchronous Event Stream<br/>Raw Events"] --> ICM["Intra-Class Consistency Mining<br/>Noise, Motion & Brightness Repr."]
In --> Voxel["Event Voxelization<br/>Voxel Grid (3 bins)"]
ICM --> LIJ["Lightweight Information Injection<br/>Residual Addition + Visual Prompts"]
Voxel --> LIJ
LIJ --> Student["Student Image Encoder<br/>Tuned FFNs + Prompt Adapters"]
Teacher["Frozen Teacher Model<br/>SAM ViT-H (RGB input)"] -->|Feature Alignment Lsim| Student
Teacher -->|Offline Pseudo-Labels Lseg| Decoder["Mask Decoder<br/>Two-stage Finetuning"]
Student --> Decoder
Decoder --> Out["Class-Agnostic Masks<br/>Dense Instance Segmentation"]
Key Designs¶
1. Intra-Class Consistency Mining: Physics-Grounded Learning-Free Scene Reconstruction
To eliminate reliance on recurrent neural reconstruction backbones (e.g., E2VID) that introduce heavy latency and initialization artifacts, ICM recovers three complementary scene representations using sensor physics without any neural network training: - Noise Representation: Photon shot noise generates isolated events in the spatio-temporal domain. A KD-Tree indexes local neighborhoods \(\mathcal{N}(e_k)\) to filter out noise events whose count falls below \(N_{\min}\). Based on a Gaussian photon arrival model, higher background brightness requires larger intensity fluctuations to cross the temporal contrast threshold, making noise triggering probability inversely correlated with scene intensity. Evaluating positive and negative event probabilities via the error function and averaging yields an absolute scene intensity map. - Motion Representation: Based on the Surface of Active Events (SAE) formulation, motion velocity \((v_x, v_y)\) is quantitatively equal to the inverse gradient of the SAE. An adaptive multi-scale local plane fitting strategy across three spatial/temporal neighborhood scales constructs a robust, sparse motion velocity map. - Brightness Representation: Under the first-order brightness constancy assumption, event density \(E\) corresponds to the inner product between brightness gradient and motion velocity: \(E = -\nabla L \cdot \mathbf{v}\). Using the computed motion field \(\mathbf{v}\) and event counts \(E\), edge contrast gradients \(\nabla L\) are resolved across both polarities. Dense scene brightness is subsequently synthesized via Poisson surface reconstruction from these gradient maps using only a single event slice.
2. Lightweight Information Injection: Decoupled Staged Fusion of Physical Attributes
Direct channel-wise concatenation of noise, motion, brightness, and voxel grids harms representation learning due to disparate statistical distributions. LIJ implements a staged, decoupled injection strategy: - For the spatial brightness representation \(x_b\), which closely resembles natural images, a lightweight residual convolutional block (\(3 \to 16 \to 3\) channels) injects spatial details directly into the base event voxel \(x_v\): $\(y = x_v + \mathcal{F}(x_b; \Theta)\)$ - For the motion representation \(x_m\) and noise representation \(x_n\), downsampling convolutions map them to patch embeddings of dimension \(C/s\) matching \(F_{\text{pe}}\). The sum \(F_i = F_{\text{pe}} + F_m + F_n\) is processed through an adapter consisting of two MLPs and a GELU activation: $\(P_i = \text{MLP}_{\text{up}}(\text{GELU}(\text{MLP}_{\text{tune}}^i(F_i)))\)$ The resulting visual prompts \(P_i\) are injected across all Transformer blocks in the image encoder. This module adds only 76.7K tunable parameters (+0.08%) and 13.2% computational overhead.
3. Cross-Modal Distillation: Teacher-Student Knowledge Transfer with Selective Tuning
To inherit SAM's universal structural and geometric priors learned from 1.1 billion masks, CMD employs an asymmetric distillation setup. The frozen SAM ViT-H teacher extracts reference features and offline pseudo-labels from paired RGB images, while the student adopts the lightweight ViT-B architecture. To prevent catastrophic forgetting and maintain zero-shot generalization, the student's attention layers remain frozen, while only the initial patch embedding layer and the Feed-Forward Networks (FFNs) within each Transformer block are updated. The prompt encoder is frozen, and the mask decoder is updated only after two initial epochs with a small learning rate of \(2 \times 10^{-7}\).
Loss & Training¶
The overall training objective combines feature distillation and mask supervision: $\(\mathcal{L} = \mathcal{L}_{\text{sim}} + \alpha \mathcal{L}_{\text{seg}}\)$ where \(\alpha = 0.1\). The feature similarity loss \(\mathcal{L}_{\text{sim}}\) aligns pivotal token embeddings using correlation-aware weighting. The segmentation loss \(\mathcal{L}_{\text{seg}}\) linearly combines Focal Loss and Dice Loss in a 20:1 ratio. During training, 8 prompt points are randomly sampled per mask instance. Training is conducted on two NVIDIA RTX 3090 GPUs for 10 epochs using Adam, with initial learning rates of \(2 \times 10^{-4}\) for the encoder and \(1 \times 10^{-4}\) for the LIJ module.
Key Experimental Results¶
Main Results¶
On the RGBE-SEG benchmark (65,957 pairs derived from VisEvent and COESOT), IC2-ESeg establishes a new state of the art across both automatic and interactive prompt modes, substantially outperforming prior reconstruction and distillation baselines on the critical mIoU and aIoU metrics.
| Method | Modality / Mode | Easy (mIoU/aIoU) | Medium (mIoU/aIoU) | Hard (mIoU/aIoU) | All (mIoU) | All (aIoU) | All (mP) | All (mR) |
|---|---|---|---|---|---|---|---|---|
| SAM (RGB baseline) | RGB Auto | 0.39 / 0.58 | 0.25 / 0.41 | 0.15 / 0.29 | 0.26 | 0.43 | 0.39 | 0.73 |
| E2VID + SAM | Event Recon. Gray | 0.38 / 0.54 | 0.32 / 0.44 | 0.26 / 0.36 | 0.32 | 0.45 | 0.54 | 0.61 |
| ETNet + SAM | Event Video Recon. | 0.40 / 0.51 | 0.35 / 0.42 | 0.31 / 0.35 | 0.35 | 0.42 | 0.63 | 0.53 |
| DTL (Dual Transfer) | Event Feature Trans. | 0.40 / 0.59 | 0.28 / 0.46 | 0.20 / 0.35 | 0.29 | 0.46 | 0.44 | 0.71 |
| ESS (UDA) | Event Domain Adapt. | 0.40 / 0.57 | 0.27 / 0.45 | 0.18 / 0.32 | 0.28 | 0.45 | 0.42 | 0.74 |
| EventSAM (CVPR 2024) | Event Auto | 0.49 / 0.65 | 0.40 / 0.54 | 0.34 / 0.47 | 0.41 | 0.55 | 0.59 | 0.71 |
| IC2-ESeg (Ours) | Event Auto | 0.51 / 0.67 | 0.43 / 0.57 | 0.36 / 0.49 | 0.43 | 0.57 | 0.56 | 0.77 |
| EventSAM* (CVPR 2024) | Event Prompt | 0.56 / 0.73 | 0.49 / 0.60 | 0.43 / 0.53 | 0.49 | 0.61 | 0.57 | 0.83 |
| IC2-ESeg* (Ours) | Event Prompt | 0.67 / 0.74 | 0.65 / 0.71 | 0.63 / 0.69 | 0.65 | 0.71 | 0.83 | 0.77 |
Zero-shot scene transfer evaluated on MVSEC (indoor/outdoor) and the newly recorded high-resolution ComScene (life, office, sports, traffic) demonstrates strong domain generalizability: - On MVSEC, under point prompts, IC2-ESeg improves average mIoU by 28.0% (0.45/0.55 to 0.64/0.64) and aIoU by 13.6% over EventSAM. - On ComScene, point prompt evaluation yields an average mIoU gain of 48.0% (from 0.38 to 0.56) and an aIoU gain of 27.2%.
Ablation Study¶
Ablation experiments conducted on RGBE-SEG under prompt mode dissect the contribution of each algorithmic component:
| Config | Modification | mIoU | aIoU | Note |
|---|---|---|---|---|
| Full Model | Complete IC2-ESeg | 0.65 | 0.70 | Baseline setup |
| w/o Brightness Repr. | Remove Poisson brightness map | 0.55 | 0.63 | -0.10 mIoU drop (primary driver) |
| w/o Motion Repr. | Remove multi-scale SAE motion map | 0.61 | 0.65 | -0.04 mIoU drop |
| w/o Noise Repr. | Remove shot noise intensity map | 0.64 | 0.68 | -0.01 mIoU drop (complementary cue) |
| w/o Residual Conn. | Concatenate brightness directly | 0.62 | 0.68 | -0.03 mIoU drop (architecture decoupling) |
| w/o Visual Prompts | Remove adapter prompt path | 0.61 | 0.66 | -0.04 mIoU drop |
| w/o \(\mathcal{L}_{\text{sim}}\) | Train without feature distillation | 0.57 | 0.60 | -0.08 mIoU drop |
| w/o \(\mathcal{L}_{\text{seg}}\) | Train without mask loss | 0.48 | 0.59 | -0.17 mIoU drop |
Quantification of intra-class consistency via mean Cosine Similarity (mCS) and mean Feature Variance (mFV): - Ground-truth RGB features achieve 0.85 mCS and 0.12 mFV; - Baseline event features yield 0.77 mCS and 0.18 mFV; - Introducing ICM boosts event feature mCS to 0.80 (+3.9%) and suppresses mFV to 0.16 (-11.1%), validating tighter feature compactness.
Key Findings¶
- Brightness representation forms the structural backbone: Removing Poisson brightness causes mIoU to collapse by 10 points (0.65 to 0.55), confirming that restoring interior intensity gradients is vital for dense mask prediction.
- Noise representation benefits slow motion and strong illumination: Scene stratification reveals that the noise representation yields its largest gains under slow motion (0.612 to 0.636 mIoU) and intense illumination (0.621 to 0.639 mIoU), conditions where edge-generating motion events are sparse but shot noise reliably signals background intensity.
- Backbone agnosticism: Substituting the SAM encoder with DINOv2 or MAE achieves 0.64 and 0.67 mIoU respectively, proving that ICM and LIJ generalize cleanly to other visual foundation model backbones.
Highlights & Insights¶
- Turning sensor noise into illumination cues: Rather than filtering out spatio-temporally isolated events as corrupt shot noise, the method leverages the Poisson-Gaussian statistical distribution to derive an analytical, learning-free estimator of scene background intensity.
- Single-slice physics inversion bypassing recurrent networks: By combining SAE plane fitting with Poisson reconstruction, IC2-ESeg computes dense brightness in 24.3 ms without recurrent states, sequence initialization lag, or synthetic ghosting artifacts.
- Minimal parameter adaptation overhead: By tuning only FFN layers and adding 76.7K adapter parameters (+0.08%), the model transfers SAM's dense instance reasoning without compromising its generalized zero-shot capability.
Limitations & Future Work¶
- Preprocessing computational latency: Although ICM is learning-free, KD-Tree radius queries and Poisson equation solvers require approximately 24.3 ms, leaving room for further CUDA optimization in microsecond-scale event pipelines.
- Feedforward slice prediction lacking temporal recurrency: The architecture currently evaluates discrete time slices feedforwardly without explicitly modeling temporal continuity across consecutive video frames; exploring state-space models (e.g., Mamba) represents an attractive future avenue.
Related Work & Insights¶
- vs EventSAM (CVPR 2024): EventSAM pioneers class-agnostic event segmentation via pivotal token matching, but ignores the severe spatial sparsity of the raw event stream. IC2-ESeg addresses the input-level modality gap through explicit physical consistency mining, outperforming EventSAM by 4.9% mIoU in auto mode and 32.7% mIoU in prompt mode.
- vs E2VID / ETNet Frame Reconstruction: Video reconstruction networks require recurrent warm-up and introduce hallucinated artifacts. IC2-ESeg's single-slice Poisson formulation delivers sharper boundaries and higher fidelity for downstream segmentation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant derivation of scene intensity from shot noise physics combined with learning-free Poisson brightness estimation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation covering auto and prompt modes, ablation on all components, consistency metrics, and a new high-resolution dataset (ComScene).
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulations, transparent physical intuitions, and modular architectural exposition.
- Value: ⭐⭐⭐⭐⭐ Sets a compelling standard for neuromorphic perception by bridging physical event generation mechanisms with foundation models.