EventVGGT: Exploring Cross-Modal Distillation for Consistent Event-based Depth Estimation¶
Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/yinruiRen/EventVGGT
Area: 3D Vision
Keywords: event camera, monocular depth estimation, cross-modal knowledge distillation, multi-view geometry, temporal consistency
TL;DR¶
EventVGGT transfers VGGT's multi-view geometric knowledge into event sequences through mixed-feature auxiliary predictions, spatio-temporal feature alignment, and depth-change distillation, reducing EventScape's mean absolute depth error at the 30 m cutoff from EventDAM's 2.30 m to 1.06 m while using only events at inference time.
Background & Motivation¶
Event cameras record when pixel brightness changes rather than capturing complete images. They retain visual information under rapid motion and extreme lighting, but do not directly provide dense scene structure. Event-based monocular depth estimation therefore faces both scarce dense annotations and the need to infer geometry in regions that generate few or no events. Methods such as EventDAM use paired RGB images and vision foundation models to supply supervision without ground-truth depth, yet frame-wise predictions can still exhibit blurred boundaries, unstable scale, and depth flickering.
The distinction is not merely whether the student has temporal modules, but whether its teacher supplies cross-frame geometry. DepthAnyEvent already incorporates temporal modules, although its teacher remains a single-image model; describing it as completely ignoring time would be inaccurate. VGGT uses alternating frame-wise and global attention to process multiple views jointly, making its outputs a source of cross-view constraints for continuous event streams. However, directly forcing event features to replicate RGB representations encounters a substantial gap between sparse motion responses and dense appearance-based structure.
The paper organizes events into sequences synchronized with RGB and constrains auxiliary depth, internal representations, and consecutive depth changes together. Core Idea: transfer not only each frame's depth answer, but also how a multi-view teacher organizes spatial structure and cross-frame changes, so that the event student produces more consistent depth sequences without RGB at inference time.
Method¶
Overall Architecture¶
Training takes synchronized RGB sequences and corresponding event streams and learns to output dense depth for the event sequence. The RGB teacher supplies features and pseudo-depth; the student uses a VGGT-style sequence backbone and learns through Cross-Modal Feature Mixture (CMFM), Spatio-Temporal Feature Distillation (STFD), and Temporal Consistency Distillation (TCD). Here, annotation-free means not relying on ground-truth depth supervision, not dispensing with paired RGB training data.
Solid arrows below represent feature or prediction flow, whereas dotted arrows indicate training-only distillation supervision. CMFM is an auxiliary training branch, not an RGB fusion path required at inference time; STFD is likewise not a new network layer appended to the depth decoder.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
E["Event stream"] --> S["Sequence-Based<br/>Geometry Backbone"]
R["Synchronized RGB<br/>Training only"] --> V["VGGT teacher<br/>Features and pseudo-depth"]
S --> M["Cross-Modal<br/>Feature Mixture<br/>CMFM"]
V -->|RGB features| M
S --> H["Event depth decoding"]
M --> A["Mixed-sequence<br/>auxiliary depth"]
S -.-> F["Spatio-Temporal<br/>Feature Distillation<br/>STFD"]
V -.-> F
A -.-> T["Temporal Consistency<br/>Distillation<br/>TCD"]
V -.->|Teacher depth changes| T
V -.->|CMFM output supervision| A
H --> O["Event-only depth sequence"]
Inference retains event encoding, alternating attention, and depth prediction, without synchronized RGB or the teacher's auxiliary branches. The paper also illustrates extensions to camera pose and point map estimation, but its central quantitative task is depth estimation. These demonstrations should not be interpreted as a fully validated general-purpose event-based 3D reconstruction system.
Key Designs¶
1. Sequence-Based Geometry Backbone: expose both networks to cross-frame structure
An event specifies position, time, and polarity: \(e=(x,y,t,p)\), where \(p\in\{+1,-1\}\) indicates an increase or decrease in logarithmic brightness. Events are accumulated in 50 ms windows with 5 temporal bins per window and synchronized with RGB frames. The text describes a three-channel final event representation, but does not clearly explain how the temporal bins and positive/negative polarities map to those channels. Reproduction should neither assume that 5 bins mean 5 channels nor invent an unreported compression rule.
The RGB and event sequences are encoded into spatially corresponding patch features. Sequence modeling uses VGGT's alternating frame-wise and global attention with a depth prediction head. Frame-wise attention handles within-frame structure, while global attention shares geometry across temporal views, matching the fact that successive observations depict the same evolving scene. Training uses 24-frame sequences, but sequence processing does not imply strictly causal online inference: the paper does not explain how future context is handled during frame-by-frame deployment.
2. Cross-Modal Feature Mixture: support event features with RGB context
Dense RGB features contain much more spatial information than sparse event features, making direct output imitation difficult. CMFM does not average the modalities element by element. It randomly replaces selected frame positions in the RGB feature sequence with temporally aligned event features. The default replacement rate is 25%, with RGB features retained elsewhere. Shared alternating attention and a depth decoder process this mixed sequence to produce auxiliary depth supervised by the teacher's RGB depth.
The auxiliary task requires event features to contribute geometric predictions in a sequence that still has reliable spatial context, instead of relying entirely on sparse observations from the outset. The described output objective includes confidence-weighted errors and a spatial-gradient penalty to handle uncertain locations and preserve local structure. Equation (1) is corrupted in the cached text, and the prose and equation disagree in their notation for the confidence source, so no reconstructed exact loss is supplied here. The confirmed mechanism is teacher-depth supervision of mixed-sequence predictions, not the addition of ground-truth depth labels.
3. Spatio-Temporal Feature Distillation: align structure and its evolution
Matching output depth alone does not ensure that the student inherits the teacher's internal cross-view representation. STFD compares event and RGB features within corresponding frames as well as their differences between consecutive frames. The first comparison anchors event representations to spatial geometry; the second focuses on how the representation changes from one observation to the next, rather than treating all time steps as unrelated static samples.
The prose explicitly describes channel-wise cosine similarity for feature alignment, but Equation (2) is incompletely extracted, preventing reliable recovery of all sums, weights, and difference terms. The authors also describe matching the magnitude and direction of changes. Cosine similarity alone does not constrain vector norms, so the available formula does not establish which term enforces magnitude matching. The supported conclusion is that both within-frame representations and between-frame feature changes are distilled, without inventing missing implementation details.
4. Temporal Consistency Distillation: learn plausible changes instead of flattening time
TCD constrains how predicted depth evolves. Camera and object motion genuinely change depth, so simply penalizing differences between adjacent predictions could suppress meaningful dynamics. Instead, the teacher's consecutive depth changes supervise the auxiliary prediction's changes. This discourages unsupported jumps while allowing motion consistent with the teacher, rather than forcing a static depth sequence.
The prose specifies per-pixel depth changes and an L1 discrepancy, but the operators and nesting of absolute values in Equation (3) are corrupted. Its visible notation uses mixed-branch depth, whereas the surrounding text refers more generally to student outputs. This note therefore follows the identifiable mixed auxiliary branch without rewriting the objective as event-only output supervision or guessing whether the exact nesting compares signed differences or change magnitudes. The section also does not explicitly introduce optical-flow or reprojection alignment, so this should not be described as a reprojection-consistency loss with explicit geometric correspondences.
A Worked Example¶
Consider a synchronized driving sequence passing a streetlight. This is an illustration of the reported settings, not an additional experiment. After encoding 24 frames, a 25% replacement rate corresponds to 6 event-feature positions and 18 RGB-feature positions in the mixed branch. Even when the streetlight produces only sparse event edges at one position, RGB context elsewhere in the sequence can support the auxiliary depth prediction.
CMFM supervises the mixed sequence using teacher depth, while STFD compares corresponding event/RGB features and their consecutive changes. TCD then checks whether the streetlight's depth evolves in line with the teacher instead of jumping with fluctuations in event density. At test time, only the event sequence is supplied; the model relies on transferred geometry and no longer reads those 18 RGB-feature positions.
Loss & Training¶
The total objective combines CMFM, STFD, and TCD. The prose specifies a feature-distillation weight of \(\lambda_{\mathrm{STFD}}=0.1\) and a temporal-consistency weight of \(\lambda_{\mathrm{TCD}}=0.2\). Because Equations (1) through (4) all contain extraction damage, this note retains the verifiable components and hyperparameters without presenting an invented exact author formula.
EventScape RGB images and event representations are center-cropped to \(252\times504\), with invalid sky-depth regions masked. Optimization uses AdamW with a learning rate of \(1\times10^{-4}\). LoRA uses \(r=16\) and \(\alpha=32\), adding approximately 1.6 million trainable parameters. Training takes roughly 11 hours on 4 NVIDIA A800 GPUs with dynamic batch sizes. The 1.6 million figure counts additional trainable parameters, not the entire model, and LoRA does not itself compress the inference backbone.
Two evaluation protocols must be distinguished. In-domain experiments use the training splits of EventScape and MVSEC respectively; MVSEC training uses Day2, with evaluation on Day1 and Night1–3. Zero-shot experiments train only on EventScape and evaluate unseen DENSE and MVSEC data. Although the method states that it does not require ground-truth depth, Section 4.2 uses the phrase "supervised fine-tuning." This is retained as an ambiguity in the source rather than interpreted as evidence of additional ground-truth supervision.
Key Experimental Results¶
Main Results¶
Values below are mean absolute depth errors at different distance cutoffs, in meters; lower is better. E indicates event-only inference, and E+I indicates joint event/image input. DENSE is an unseen synthetic domain, not a real-world dataset; MVSEC provides the real-world evaluation.
| Dataset and protocol | Method | Inference input | 10 m | 20 m | 30 m |
|---|---|---|---|---|---|
| EventScape, in-domain, Table 1 | E2Depth | E | 1.79 | 5.35 | 8.31 |
| EventScape, in-domain, Table 1 | SRFNet | E+I | 1.27 | 1.68 | 2.76 |
| EventScape, in-domain, Table 1 | EventDAM | E | 0.56 | 1.52 | 2.30 |
| EventScape, in-domain, Table 1 | EventVGGT | E | 0.54 | 0.79 | 1.06 |
| DENSE, zero-shot, Table 3 | SRFNet | E+I | 1.50 | 3.57 | 6.12 |
| DENSE, zero-shot, Table 3 | EventDAM | E | 1.20 | 2.60 | 5.18 |
| DENSE, zero-shot, Table 3 | EventVGGT | E | 0.54 | 0.89 | 1.33 |
At the 30 m cutoff, EventScape error decreases by 1.24 m, approximately 53.9% relative to EventDAM; the corresponding DENSE reduction is approximately 74.3%. These results support improved long-range accuracy and transfer between synthetic domains, not superiority over every baseline at every distance in all real-world scenes.
In MVSEC Table 2, EventVGGT's 30 m errors on Night1, Night2, Night3, and Day1 are 2.61, 2.48, 2.64, and 1.63 m, respectively. Its Night1 error at 10 m is 1.67 m, however, worse than EventDAM's 1.39 m, so a claim of winning every metric is unsupported. Table 4 describes its Night1 comparison as zero-shot, yet EventVGGT's 1.67 / 2.02 / 2.61 values repeat the in-domain results in Table 2. The relationship between these training protocols requires further verification; the note does not merge them into one interpretation.
Ablation Study¶
The following values are taken directly from Table 5 on EventScape. The configuration without the three components is the paper's base model, not necessarily an entirely unadapted VGGT.
| Config | 10 m | 20 m | 30 m | Note |
|---|---|---|---|---|
| No CMFM / STFD / TCD | 0.69 | 0.94 | 1.26 | Base model |
| CMFM only | 0.57 | 0.86 | 1.13 | Mixed auxiliary prediction supervision |
| CMFM + STFD | 0.53 | 0.81 | 1.10 | Additional spatio-temporal feature alignment |
| CMFM + TCD | 0.56 | 0.84 | 1.12 | Additional depth-change supervision |
| CMFM + STFD + TCD | 0.51 | 0.79 | 1.06 | Full distillation combination |
Table 5 reports 0.51 at 10 m for the full configuration, whereas Table 1 reports 0.54. The source does not clearly explain this discrepancy. Table 8 also reports differing results for Town 1 and Towns 1–3. A favorable value from one table should not replace a value in another; they are recorded separately rather than forced into an artificially uniform experimental setting.
Key Findings¶
- CMFM supplies the largest initial gain in Table 5: 30 m error decreases from 1.26 to 1.13. The full combination reaches 1.06, but the absence of STFD-only and TCD-only configurations prevents a complete ranking of all components' independent contributions.
- Table 6 increases sequence length from 1 to 24, reducing 30 m error from 1.50 to 1.26. At 10 m, however, error is 0.70 with 8 frames and 0.71 with 24, so longer sequences do not improve every metric monotonically. This table should not be treated as the full model's main result.
- In Table 7, event-feature replacement rates of 100%, 25%, and 10% yield 30 m errors of 1.26, 1.13, and 1.16. Excessive replacement removes RGB geometry context, while insufficient replacement weakens the training signal reaching the event branch.
- In Table 8, extending training from Town 1 to Towns 1–3 reduces DENSE's 30 m error from 1.50 to 1.33. Transfer gains also depend on training-scene diversity and cannot all be attributed to an individual distillation component.
Highlights & Insights¶
- Teacher selection changes what information can be transferred. A single-image teacher mainly supplies per-frame spatial answers, whereas a multi-view teacher also supplies cross-view relationships, addressing missing temporal supervision more directly than adding temporal modules only to the student.
- CMFM frames cross-modal alignment as whether one modality's features can substitute for another's within the same decoding task. This auxiliary objective is closer to the final geometry task than requiring raw observations from different sensors to look alike.
- TCD follows teacher-supported changes rather than indiscriminately smoothing consecutive predictions. Distilling a target's evolution is transferable, but other applications still need to consider occlusion, correspondence, and the teacher's own temporal errors.
Limitations & Future Work¶
- The authors acknowledge that the teacher's far-field depth compression transfers to the student and can underestimate distant backgrounds. They propose future calibration with dense ground-truth depth; the current method does not provide unbiased geometric recovery.
- Temporal consistency and extensions to pose/point maps are mainly demonstrated through qualitative reconstructions, without a dedicated quantitative flicker or cross-frame consistency metric. Lower depth error supports greater accuracy but does not independently establish every claim about temporal stability.
- The paper reports 24 ms to process a \(252\times504\) event representation on one A800, without clearly specifying online updates of the 24-frame context, end-to-end waiting time, or memory use. This does not establish embedded real-time performance or 24 ms causal latency.
- Training still requires synchronized RGB/event sequences and adequate sensor alignment. Corrupted formulas, unclear three-channel construction, and discrepancies in some tables and protocol descriptions also limit exact reproduction from the present text cache alone.
Related Work & Insights¶
- vs EventDAM: Both use vision foundation models to address scarce event-depth annotations. EventVGGT advances from frame-wise dense-to-sparse distillation toward multi-view sequence knowledge, with mixed auxiliary predictions and cross-frame constraints.
- vs DepthAnyEvent: The distinction is not that DepthAnyEvent has no temporal modules. More precisely, its single-image teacher supplies different supervision from VGGT's multi-view geometry teacher.
- vs VGGT: This is not simply feeding event representations into an unchanged RGB model. It uses modality-specific adaptation and three-level distillation. In Table 4, raw VGGT with event input has a Night1 30 m error of 3.44 m, versus EventVGGT's 2.61 m, although that table's protocol ambiguity must remain explicit.
- vs fusion methods such as SRFNet: RGB use during training does not require RGB use during deployment. EventVGGT shifts RGB's role to knowledge transfer, simplifying inference inputs without removing the paired-data requirement during training.
Rating¶
- Novelty: 4/5. Combines a sequence-aware geometry foundation model with event-domain distillation through three clearly differentiated objectives, although the underlying alignment ideas are not entirely new.
- Experimental Thoroughness: 3/5. Includes in-domain, zero-shot, and multiple ablation evaluations, but lacks dedicated temporal metrics, error bars, and fully consistent descriptions of some protocols.
- Writing Quality: 3/5. The motivation is accessible, but some table values and training terminology need clarification; corrupted cached formulas separately limit readability.
- Value: 4/5. Provides a reusable route for combining event depth estimation with multi-view foundation models, particularly for long-range improvements under event-only inference.