SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/nesl/SWAN
Area: Autonomous Driving
Keywords: adaptive multimodal networks, runtime variations, 3D object detection, neural sort, token pruning
TL;DR¶
Addressing concurrent runtime variations in sensor quality of information, platform compute constraints, and sample complexity in autonomous driving, SWAN unifies a differentiable NeuralSort-based QoI allocation controller, an input-conditioned SkipGate module, and DETR token pruning, reducing compute by up to 49% with minimal accuracy degradation.
Background & Motivation¶
Deep multimodal perception systems, integrating cameras and LiDAR into architectures such as BEVFusion or CMT, have become the standard in autonomous vehicles (AVs) by providing crucial robustness against sensory noise through cross-modal redundancy. However, real-world deployment exposes these networks to severe, compounding runtime variations across three orthogonal dimensions: fluctuating input Quality of Information (QoI) caused by adverse weather, sensor soiling, or hardware failure; volatile platform compute budgets resulting from thermal throttling, battery preservation, or high-speed latency constraints; and varying sample complexity where sparse rural roads demand far less network representation capacity than dense urban intersections.
Current neural architectures isolate and address these dynamics individually, rendering them ill-equipped for realistic driving conditions. Reconfigurable networks such as Once-For-All or LayerDrop support variable depth at inference time but fail to allocate resources asymmetrically across modalities based on relative sensor health. Existing controller-based approaches like ADMN rely on Straight-Through gradient estimators that yield unstable, suboptimal layer allocations under high-dimensional AV settings, while ignoring computation reduction within a designated budget. Meanwhile, input-adaptive networks such as AdaViT optimize only average-case compute and cannot satisfy the hard worst-case latency boundaries critical to vehicular safety. When sensor degradation coincides with platform thermal throttling, prior methods either waste scarce FLOPs on corrupted inputs or drop essential sensory streams indiscriminately.
To reconcile these competing demands, this paper decouples macro-level inter-modal resource provisioning from micro-level sample-dependent compute skipping, grounding both in explicit quality signals and hard budget constraints. Core idea: develop SWAN (Sample and World-Aware Multimodal Network), which coordinates a differentiable NeuralSort-guided QoI controller to distribute layer budgets across modalities, an input-conditioned SkipGate module to conditionally bypass unneeded layers within that budget, and a spatial token pruning module to eliminate background features before the detection head.
Method¶
Overall Architecture¶
SWAN builds upon the CMT multimodal 3D detection framework, employing an 8-layer FlatFormer backbone for LiDAR point clouds and a 12-layer Swin-Tiny Transformer backbone for multi-view camera images (20 total backbone layers). The end-to-end pipeline operates in three coordinated stages. First, camera images and LiDAR inputs are processed by lightweight convolutional networks to extract a multimodal quality embedding; this embedding, concatenated with a sinusoidal positional encoding of the user-specified layer budget \(b\), is fed into the QoI-Aware Controller to predict layer importance logits across both backbones. At inference time, the top \(b\) layers with the highest logits are selected. Second, at the entrance of each selected layer, a modality-shared SkipGate module evaluates the intermediate representation alongside cross-modal budget contexts to decide whether that layer should be executed or skipped via an identity shortcut. Third, before the resulting features reach the Transformer detection head, a spatial token pruning module evaluates the semantic saliency of each token, filtering out uninformative background regions (such as empty sky or distant ground) to suppress the quadratic attention overhead of the DETR decoder.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Inputs (Camera Images + LiDAR Point Clouds)<br/>and User-Specified Layer Budget b"] --> B["QoI-Aware Controller: Inter-Modal Budget Allocation<br/>Lightweight Convs Extract QoI Vector + Budget Encoding"]
B --> C["NeuralSort Differentiable Sorting Relaxation<br/>Generates Soft Top-b Selection Mask Across Modalities"]
C --> D["SkipGate: Sample-Level Conditional Layer Dropping<br/>Evaluates Contextual Embeddings to Bypass Redundant Layers"]
D --> E["DETR Head Token Pruning: Saliency-Guided Sparsification<br/>Discards Background Tokens Prior to Transformer Decoder"]
E --> F["CMT Transformer Decoder & 3D Detection Head<br/>Outputs Final 3D Bounding Boxes and Category Probabilities"]
Key Designs¶
1. Differentiable QoI-Aware Resource Allocation Controller: Continuous Sorting via NeuralSort
Under a strict total layer budget \(b \in B\), the network must prioritize layers belonging to modalities that offer the highest marginal utility and lowest noise corruption. Prior gating methods using Straight-Through Estimators (STE) suffer from high gradient variance, causing the controller to settle on erratic allocations where corrupted modalities inappropriately consume compute. SWAN resolves this by training lightweight convolutional encoders \(f_{\omega_m}\) on each modality \(x_m\) to produce a concatenated corruption representation \(z = \text{Concat}_{m \in M}(f_{\omega_m}(x_m))\), explicitly supervised by an environmental classification loss \(\mathcal{L}_{env}\) against corruption ground truth. The vector \(z\) is concatenated with sinusoidal budget embedding \(e_b\) to form \(\tilde{z}\), which an MLP maps to layer allocation logits \(\omega = [\omega_1 \Vert \omega_2 \Vert \dots \Vert \omega_M] \in \mathbb{R}^L\) for all \(L = \sum_m L_m\) backbone layers. To render top-\(b\) selection fully differentiable during end-to-end training, SWAN incorporates NeuralSort. Given perturbed logits \(\tilde{\omega} = \omega + \text{Gumbel}(0, 1)\), NeuralSort calculates a relaxed permutation matrix \(\hat{P}_{\text{sort}}(\tilde{\omega})\) where each row \(i\) models the continuous probability of an item ranking \(i\)-th:
$$
\hat{P}{\text{sort}}(\tilde{\omega})[i, :] = \text{softmax}\left( \frac{(L + 1 - 2i)\tilde{\omega} - A \right)
$$
Here }}\mathbf{1}}{\tau\(A_{\tilde{\omega}}\) denotes the pairwise absolute difference matrix \(| \tilde{\omega}_j - \tilde{\omega}_k |\), and \(\tau\) is a temperature parameter annealed toward 0.1. Summing the first \(b\) rows produces a continuous top-\(b\) selection mask \(g = \sum_{i=1}^b \hat{P}_{\text{sort}}(\tilde{\omega})[i, :]\), partitioned into modality gates \([g_1 \Vert \dots \Vert g_M]\) that weight layer forward passes: \(h_{m,l} = g_{m,l} f_{m,l}(h_{m,l-1}) + (1 - g_{m,l}) h_{m,l-1}\). Supervised directly by 3D detection loss, the controller accurately masters the relative utility hierarchy across modalities.
2. Context-Conditioned SkipGate Module: Elastic Compute Below the Maximum Budget
The controller-assigned budget \(b\) defines the upper execution ceiling needed to avoid platform timeout; however, straightforward inputs can often be adequately processed using fewer layers. The SkipGate module operates inside each modality backbone to conditionally skip controller-selected layers. Because whether a layer is redundant depends heavily on both environmental quality and how much capacity was granted to the other modality, SkipGate conditions its decision on a comprehensive context vector: the spatial average of the preceding feature map \(\bar{h}_{l-1,m}\), the current layer index embedding \(e_{m,l}\), the controller's corruption vector \(z\), the number of remaining selected layers in the current modality \(e_{m,l_r}\), and the allocated layer count of the peer modality \(e_{m,l_o}\). These embeddings are concatenated into \(q\) and projected by an MLP to output an execution logit \(d_{m,l}\):
$$
q = \left[ \bar{h}{l-1,m} \,\Vert\, e} \,\Vert\, \text{MLPz(z) \,\Vert\, e} \,\Vert\, e_{m,l_o} \right], \quad d_{m,l} = \text{MLP{skip}(q)
$$
During training, differentiable decisions \(a_{m,l}\) are obtained using a two-variable Gumbel-Sigmoid operator with temperature annealing. To penalize gratuitous computation, a bounded hinge penalty loss is applied:
$$
\mathcal{L} + \vartheta) \cdot \vartheta_m
$$
The cut-off parameter } = \sum_{m \in M} \sum_{l=1}^{L_m} \text{ReLU}(d_{m,l\(\vartheta\) prevents the network from cheating the loss by pushing negative logits toward negative infinity, ensuring clean gradient cut-offs once skipping is determined.
3. DETR Head Spatial Token Pruning: Eliminating Background Attention Overhead
In dense multi-view 3D detection, the output feature maps contain tens of thousands of spatial tokens, the vast majority of which correspond to uninformative background such as road pavement and sky. Feeding these uncurated tokens into cross-modal Transformer decoders incurs massive latency penalties. SWAN inserts a two-layer convolutional subnet \(f_{\sigma_m}\) between the backbones and the detection head to predict token retention weights \(w_m = \sigma(f_{\sigma_m}(h_m)) \in \mathbb{R}^{H \times W}\). The weights are thresholded into discrete binary masks \(\tilde{w}_m\) and optimized via Straight-Through Estimation against a token utilization loss \(\mathcal{L}_{tp} = \sum_{m} \sum_{i,j} \tilde{w}_{m,i,j}\). This design discards roughly 74.5% of camera tokens and 51.2% of LiDAR tokens without clipping foreground object structures, providing substantial latency reductions where layer-level skipping alone is hampered by execution overhead.
A Worked Example¶
Consider an autonomous vehicle driving through dense fog at dusk while experiencing thermal throttling that restricts total execution to a tight 4-layer budget (\(b=4\)): 1. QoI Extraction & Budget Injection: Multi-view images and point clouds are scanned by the controller's lightweight convolutional heads, outputting environmental embedding \(z\) that identifies severe camera degradation alongside moderate LiDAR retention. Budget \(b=4\) is encoded into \(e_b\). 2. Global Layer Ranking: The controller generates 20 layer logits \(\omega\). Driven by NeuralSort, the controller identifies that Swin camera layers offer near-zero utility in heavy fog, allocating all 4 layers to FlatFormer LiDAR (4 LiDAR layers, 0 camera layers). 3. SkipGate Verification: As point cloud features traverse the LiDAR backbone on a clear stretch of road, SkipGate recognizes that 2 layers are sufficient to discern surrounding obstacles; the remaining 2 layers produce negative logits and are skipped via identity shortcuts. 4. Token Pruning & 3D Detection: The resulting 2-layer features enter the token pruner, which filters out unreflective ground and distant points, feeding only salient foreground tokens into the CMT decoder. The system produces accurate 3D bounding boxes in ~72 ms, preventing safety-critical timeouts under harsh constraints.
Loss & Training¶
SWAN employs a staged training regimen to guarantee stable convergence: 1. Stage 1 (Adaptive Backbone Pretraining): Train the base CMT network with LayerDrop (stochastic layer drop rate of 0.2) in Swin and FlatFormer backbones using the standard 3D detection loss \(\mathcal{L}_{detection}\), establishing depth elasticity. 2. Stage 2 (QoI Controller Training): Freeze the backbone weights and train the controller on MultiCorrupt nuScenes data using \(\mathcal{L} = \mathcal{L}_{detection} + \varpi_1 \mathcal{L}_{env}\), annealing the NeuralSort temperature \(\tau\). 3. Stage 3 (SkipGate Training): Freeze the backbone and controller; train the shared SkipGate modules using \(\mathcal{L} = \mathcal{L}_{detection} + \varpi_2 \mathcal{L}_{skip}\) under Gumbel-Sigmoid relaxation. 4. Stage 4 (Token Pruning Training): Freeze preceding modules and train the token pruning convolutional layers with \(\mathcal{L} = \mathcal{L}_{detection} + \varpi_3 \mathcal{L}_{tp}\), where \(\varpi_2\) and \(\varpi_3\) allow customized trade-offs between precision and inference efficiency.
Key Experimental Results¶
Main Results¶
Evaluated on nuScenes with 5 synthetic corruption types from the MultiCorrupt suite (LiDAR Beamsreduce, Camera Fog, Darkness, Camera Motionblur, LiDAR Motionblur). Metrics are NDS, mAP, GFLOPs, and median latency measured on an Nvidia RTX 4090:
| Model Variant | Budget | GFLOPs | Latency (ms) | LiDAR Beamsreduce (NDS/mAP) | Camera Fog (NDS/mAP) | Dark (NDS/mAP) | Camera Motionblur (NDS/mAP) | LiDAR Motionblur (NDS/mAP) |
|---|---|---|---|---|---|---|---|---|
| Base Network (CMT Full 20L) | 20 Layers | 651.00 | 124.84 | 41.80 / 28.97 | 67.57 / 61.30 | 67.65 / 61.31 | 66.85 / 59.45 | 63.93 / 58.32 |
| BEVFusion | Fixed | 630.73 | 118.76 | 32.08 / 10.77 | 68.38 / 62.89 | 69.49 / 64.88 | 68.28 / 62.56 | 58.18 / 53.60 |
| PETR (Camera-only) | Fixed | 1100.15 | 44.31 | 35.21 / 32.63 | 22.17 / 14.72 | 20.32 / 12.84 | 23.24 / 14.92 | 35.21 / 32.63 |
| SST (LiDAR-only) | Fixed | 372.85 | 92.79 | 22.06 / 1.38 | 66.42 / 58.65 | 66.43 / 58.65 | 66.43 / 58.65 | 54.82 / 42.73 |
| Naive Allocation 16 | 16 Layers | 556.41 | 120.93 | 39.48 / 25.40 | 67.48 / 61.17 | 67.21 / 60.85 | 66.70 / 59.37 | 63.02 / 56.79 |
| ADMN 16 | 16 Layers | 584.01 | 123.21 | 40.12 / 26.78 | 67.54 / 61.16 | 67.28 / 61.15 | 66.89 / 60.15 | 62.84 / 56.65 |
| SWAN-PSC 16 (Full Model) | 16 Layers | 428.37 | 83.42 | 36.42 / 25.43 | 65.08 / 57.43 | 64.95 / 57.20 | 64.72 / 57.34 | 61.40 / 55.18 |
| Naive Allocation 8 | 8 Layers | 426.56 | 106.09 | 32.92 / 16.16 | 64.88 / 56.74 | 64.39 / 56.34 | 64.74 / 56.47 | 59.33 / 50.97 |
| ADMN 8 | 8 Layers | 438.09 | 110.21 | 36.35 / 21.04 | 65.86 / 57.90 | 65.69 / 57.79 | 65.68 / 57.74 | 55.74 / 46.52 |
| SWAN-PSC 8 (Full Model) | 8 Layers | 373.42 | 77.15 | 34.42 / 23.23 | 63.99 / 55.95 | 64.20 / 56.24 | 64.94 / 57.14 | 56.82 / 47.60 |
| Naive Allocation 4 | 4 Layers | 362.00 | 98.81 | 24.71 / 3.88 | 57.83 / 44.70 | 57.71 / 45.02 | 56.73 / 43.62 | 45.82 / 29.45 |
| ADMN 4 | 4 Layers | 369.65 | 102.02 | 26.33 / 7.48 | 55.69 / 42.04 | 60.55 / 49.66 | 58.08 / 46.47 | 47.82 / 32.20 |
| SWAN-C 4 (Controller only) | 4 Layers | 358.90 | 103.73 | 29.17 / 16.12 | 64.47 / 55.80 | 64.77 / 56.49 | 64.46 / 56.13 | 54.38 / 41.97 |
| SWAN-PSC 4 (Full Model) | 4 Layers | 331.41 | 72.21 | 27.92 / 14.18 | 63.95 / 55.90 | 64.95 / 55.91 | 63.74 / 55.69 | 54.86 / 43.99 |
Ablation Study¶
The modular ablation of SWAN was evaluated on the MM-Fi multi-frame (15-frame) human gesture recognition benchmark with RGB and depth camera modalities under synthetic Gaussian noise, measured on an RTX 4090:
| Method | 6 Layers Acc / Latency | 10 Layers Acc / Latency | 16 Layers Acc / Latency | 20 Layers Acc / Latency |
|---|---|---|---|---|
| Naive Allocation | 23.2% / 12.7 ms | 33.0% / 18.2 ms | 45.7% / 26.3 ms | 44.4% / 31.9 ms |
| SWAN-C (Controller only) | 30.9% / 13.7 ms (+7.4%) | 39.2% / 19.3 ms (+6.1%) | 43.2% / 27.5 ms (+4.7%) | 46.0% / 33.1 ms (+3.6%) |
| SWAN-SC (Controller + SkipGate) | 29.6% / 12.3 ms (-3.3%) | 38.0% / 18.7 ms (+2.9%) | 42.9% / 24.0 ms (-8.5%) | 41.7% / 27.1 ms (-15.1%) |
Edge hardware benchmarks conducted on an Nvidia Jetson Orin under camera fog corruption demonstrate that SWAN-SC reduces latency over Naive Allocation by 7.4% (MAXN mode), 9.9% (50W mode), and 16.5% (30W mode). Enabling full token pruning (SWAN-PSC) cuts overall inference latency by more than 50% across all power tiers.
Key Findings¶
- Quality-aware allocation is indispensable under extreme compute starvation: When constrained to 4 total layers under LiDAR Beamsreduce corruption, Naive static allocation plunges to 3.88 mAP, whereas SWAN-C achieves 16.12 mAP (+12.24 mAP gain) and outperforms ADMN (7.48 mAP) by 8.64 mAP, validating NeuralSort's accurate cross-modal rank estimation.
- SkipGate extracts substantial compute savings under abundant budgets: Under a 16-layer budget, SkipGate dynamically sheds between 5 and 14 layers depending on scene complexity, reducing GFLOPs by over 18% compared to Naive Allocation with less than 1.5 NDS/mAP degradation.
- Token pruning breaks the latency barrier: On desktop GPUs, SkipGate FLOP reductions yield minimal latency gains due to PyTorch CPU-GPU orchestration overhead (~1.5 ms controller, ~0.5 ms SkipGates). Token pruning discards 74.46% of image tokens and 51.16% of LiDAR tokens, reducing cross-attention dimensions and slashing inference latency by up to 30.8% on RTX 4090 and over 50% on Jetson Orin.
Highlights & Insights¶
- Continuous sorting relaxation for discrete allocation: Formulating the combinatorial top-\(b\) layer selection as a NeuralSort continuous permutation matrix resolves the gradient instability of straight-through estimators without resorting to sample-inefficient reinforcement learning.
- Data-driven allocation overcomes human heuristics: In LiDAR motion blur scenarios, common intuition might favor reallocating all layers to clear cameras; however, SWAN learns to retain significant LiDAR allocation, reflecting the empirical fact that motion-blurred 3D geometry remains vastly more informative than monocular views.
- Multi-tiered adaptation architecture: Combining macro-level QoI budgeting, micro-level layer skipping, and spatial token pruning provides both strict worst-case runtime bounds and aggressive opportunistic efficiency.
Limitations & Future Work¶
- Layer count abstraction vs. asymmetric resource costs: The system treats layers as uniform units of budget, whereas Swin image layers incur ~2.4x more FLOPs but only ~1/3 the latency of FlatFormer LiDAR layers. Incorporating non-uniform, hardware-specific layer cost coefficients directly into the controller's objective is a necessary next step.
- Kernel launch and PyTorch host orchestration overhead: On desktop GPUs, dynamic control flow introduces host-device synchronization latency that partially offsets SkipGate FLOP gains. Leveraging graph compilers such as TensorRT with native GPU-side conditionals is needed to realize full latency benefits.
- Synthetic-to-real domain gap: Models trained on MultiCorrupt synthetic corruptions exhibit overly aggressive layer dropping on real-world nuScenes rainy data, necessitating downstream fine-tuning on natural corruption distributions.
Related Work & Insights¶
- vs ADMN (Wu et al., 2025): ADMN introduced controller-based layer adaptation but evaluated on toy networks, neglected within-budget sample adaptation, and suffered from noisy Straight-Through gradient propagation; SWAN scales to full 3D perception with NeuralSort and introduces complementary SkipGate and token pruning modules.
- vs AdaViT (Meng et al., 2022): AdaViT performs input-adaptive token and layer skipping for image classification but provides no guarantees on worst-case execution budget; SWAN explicitly enforces user-specified budget ceilings while retaining elastic compute within those bounds.
- vs BEVFusion (Liu et al., 2023) / CMT (Yan et al., 2023): Conventional multimodal architectures assume clean inputs and unconstrained platform resources; SWAN establishes a dynamic runtime adaptation framework for realistic autonomous vehicle operating conditions.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant integration of NeuralSort relaxation into a multi-tiered adaptive multimodal framework addressing sensor quality, compute bounds, and sample difficulty.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluations spanning nuScenes corruptions, varied layer budgets, sim-to-real transfer, cross-domain gesture benchmarks, and edge Jetson Orin deployment.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative with thorough analysis of structural dynamics, empirical failure modes, and hardware runtime nuances.
- Value: ⭐⭐⭐⭐⭐ Highly impactful for real-world autonomous driving and edge robotics, offering principled resilience against environmental degradations and platform resource fluctuations.