Intrinsically Stable Spiking Neural Networks: Overcoming the Performance Barrier in the Absence of Batch Normalization

Conference: ECCV 2026
arXiv: 2606.31695
Code: https://github.com/Ruichen0424/IS-SNN
Area: Neuromorphic Computing / Model Compression
Keywords: Spiking Neural Networks, Batch Normalization-Free Training, Weight Standardization, Firing-Rate Stability, Offline Reparameterization

TL;DR

IS-SNN stabilizes the firing rate of deep spiking neural networks without any activation normalization layers (Batch Normalization) through topology-aware weight standardization and modified residual connections. After training, the standardization operations are folded into static weights, achieving zero normalization overhead during inference. It achieves 68.05% accuracy on ImageNet while reducing FPGA LUT resource consumption by 96.4%.

Background & Motivation

Spiking Neural Networks (SNNs) hold promise for extremely low-power inference on neuromorphic hardware due to their event-driven, sparse computing nature. However, current high-performance deep SNNs heavily rely on Batch Normalization (BN) to stabilize training and enhance accuracy. Although standard static BN can be folded into synaptic weights or thresholds during inference, leading models commonly utilize time/batch-dependent dynamic BN variants (e.g., TEBN, TAB, tdBN). The normalization statistics of these variants must be recomputed step-by-step during runtime, making them unfoldable into static parameters. This reintroduces multiplier operations and statistical tracking, undermining the advantage of SNNs' accumulator-centric data paths.

A key dilemma thus arises: directly removing activation normalization is not a viable option. Forward signal analysis demonstrates that deep SNNs lacking statistical control suffer from catastrophic firing-rate decay—neuron firing rates drop rapidly or saturate as depth increases, leading to disrupted signal propagation and training collapse. Prior BN-free attempts (such as OTTT) only apply to shallow networks and small-scale datasets, lacking a general solution for modern deep architectures. Translating the perspective from "optimizing instability" to "signal survival," this work introduces a Core Idea: maintaining firing-rate homeostasis during training via topology-aware standardization in the weight space and residual variance control, completely eliminating normalization overhead during inference.

Method

Overall Architecture

IS-SNN replaces the minimum repeating unit (MRU) of Conv-BN-SN in standard SNNs with WS-Conv-SN. At the beginning of each training step, weight standardization is performed online on the convolutional weights (subtracting the mean, dividing by the standard deviation, and multiplying by a topology-aware scaling factor \(\gamma_\ell\)). These standardized weights are then used for forward propagation. After training, the weights and their statistics are frozen, and the standardization operation is mathematically folded into static weights. The network architecture during inference is identical to a vanilla BN-free network—with no runtime tracking of normalization statistics and no normalization-related multiplications, retaining only membrane potential accumulation and threshold comparison.

For plain networks (VGG), the MRU is a simple stack of WS-Conv \(\to\) SN. For residual networks, the MRU consists of a transition block (to reset variance statistics) and several residual blocks. Each residual block introduces a scaling factor \(\alpha\) on the skip connection branch to control variance growth before connecting to the identity mapping and passing to the next layer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Spike Sequence"] --> B["WS-Conv<br/>Weight Standardization Convolution<br/>(Online during training, folded during inference)"]
    B --> C["LIF Spiking Neuron<br/>Integrate-Fire-Reset"]
    C --> D{"Network Topology"}
    D -->|"Plain (VGG)"| E["Direct Propagation<br/>σ² remains constant across layers"]
    D -->|"ResNet"| F["α-scaled Residual Branch<br/>α=0.5 controls variance growth"]
    F --> G["Addition with Skip Connection<br/>σ²_{ℓ+1}=α²σ²_g+σ²_ℓ"]
    E --> H["Next Layer / Classification Output"]
    G --> H

Key Designs

1. Weight Standardization and Offline Reparameterization: Replacing Activation Space Normalization with Weight Space Constraints

The root cause of training collapse in deep SNNs without BN lies in the uncontrolled distribution of pre-activation—where firing rates decay exponentially or saturate with depth. The idea of IS-SNN is to directly impose constraints in the weight space so that the pre-activation \(x_{pa}\) approximates the standard normal distribution \(\mathcal{N}(0,1)\). Given the pre-activation mean and variance formulas \(\mu_{pa}=N\mu_{in}\mu_{W_i}\) and \(\sigma_{pa}^2=N\sigma_{in}^2(\sigma_{W_i}^2+\mu_{W_i}^2)\), the objective can be achieved simply by ensuring \(\mu_{W_i}=0\) and \(\sigma_{W_i}^2=1/(N\sigma_{in}^2)\). IS-SNN explicitly satisfies these two conditions through Weight Standardization:

\[\hat{W}_{i,j} = \gamma_\ell \cdot \frac{W_{i,j} - \mu_i}{\sqrt{N\sigma_i^2 + \epsilon}}\]

Here, \(\mu_i\) and \(\sigma_i^2\) are the statistical mean and variance of the learnable weights \(W_{i,j}\) in the current batch, \(\gamma_\ell\) is the layer-wise scaling factor derived from topology-aware variance propagation, and \(\epsilon=10^{-4}\). Dynamic standardization is executed at each training step before forward propagation. Once training completes, all parameters are frozen, and \(\hat{W}_{i,j}\) is pre-computed as static weights to be deployed on hardware. The key to this step is decoupling training and inference—enjoying the stabilizing effect of WS during training while incurring zero normalization overhead during inference, making the structure identical to a vanilla BN-free network. In contrast, standard static BN folding still introduces a constant bias current (\(b_{fused}\)) into the neural membrane potential, disrupting the event-driven sparsity, whereas the folded WS of IS-SNN only alters synaptic weight values without introducing any constant current injection.

2. Topology-Aware Variance Propagation: Deriving Layer-wise Scaling Factors \(\gamma_\ell\) for Various Network Topologies

The weight standardization formula itself is merely a tool; its effectiveness depends entirely on whether \(\gamma_\ell\) matches the network topology. To this end, IS-SNN establishes a topology-aware variance propagation framework, defining three core parameters: \(\sigma_\ell\) (the theoretical input standard deviation of layer \(\ell\), solely determined by the topology and remaining constant across time steps), \(\gamma_\ell \equiv 1/\sigma_\ell\) (the layer-wise scaling factor), and \(\sigma_g\) (the empirical output standard deviation of a specific spiking neuron under a standard normal input, which is an intrinsic property of the neuron).

For plain networks (VGG), the output of layer \(\ell\) directly serves as the input of layer \(\ell+1\), and the variance remains constant across layers: \(\sigma_{in,\ell+1}^2 = \sigma_{out,\ell}^2 = \sigma_g^2\). Thus, the scaling factors of all layers are uniformly set to \(\gamma_\ell \equiv 1/\sigma_g\).

For residual networks, the addition operation of skip connections accumulates variance block-by-block, which leads to signal explosion in deeper layers if uncontrolled. IS-SNN introduces modified residual connections:

\[x_{\ell+1} = \alpha \cdot \text{SN}(f(x_\ell)) + x_\ell\]

where \(\alpha\) controls the contribution of the residual branch (uniformly set to \(0.5\) across all architectures, which can be implemented as a bit shift rather than a multiplication). Tracking variance propagation yields \(\sigma_{\ell+1}^2 = \alpha^2 \cdot \sigma_g^2 + \sigma_\ell^2\). At the start of each stage, a transition block resets the statistics to \(\sigma_{reset}^2 = (1+\alpha^2) \cdot \sigma_g^2\). By analytically tracking \(\sigma_\ell^2\) layer-by-layer, \(\gamma_\ell = 1/\sigma_\ell\) can be computed for every layer in the entire ResNet topology. Ablation studies show that fixing \(\gamma_\ell\) to 1 (degenerating to naive WS) leads to a complete failure to train VGG (only 1.00% on CIFAR-100) and a severe drop in ResNet performance (from 76.02% to 70.69% on CIFAR-100), validating the necessity of topology-aware derivation.

3. Empirical Estimation of Spiking Neuron Output Variance \(\sigma_g^2\)

The aforementioned variance propagation framework depends on a key parameter \(\sigma_g^2\)—the output variance of a spiking neuron under standard normal inputs. For linear activation functions like ReLU, the input-output variance relation has an analytical closed-form solution. However, the spiking process of a spiking neuron involves discrete spike generation, membrane potential integration and leakage, and time-step dependent reset mechanisms, which are highly non-linear and difficult to analyze analytically. IS-SNN adopts an empirical simulation approach: stimulating LIF neurons of different configurations with random pre-activations sampled from \(\mathcal{N}(0,1)\) to measure the time-averaged output variance \(\overline{\sigma_g^2}\). For standard LIF (\(\tau=2\), no input decay), \(\overline{\sigma_g^2}=0.1234\) at \(T=4\); it drops to \(0.0290\) when input decay is introduced. These values vary minimally across \(T=4/8/16\), indicating temporal robustness of the estimation. This method generalizes to other neuron models (IF, PLIF, etc.). Once \(\sigma_g^2\) is estimated, it is fixed as an architectural constant that does not change with training or inference. Injecting up to 10% random perturbation into \(\sigma_g\) leads to only a 0.23%/0.31% drop in accuracy on CIFAR-10/100, showing that the method is insensitive to estimation errors of \(\sigma_g\).

Loss & Training

IS-SNN does not modify the loss function and employs the standard cross-entropy loss. Training utilizes the surrogate gradient method, approximating the Heaviside step function for forward spike generation with an arctan-type smooth function \(g(x)=\frac{1}{\pi}\arctan(\frac{\pi}{2}\alpha x)+\frac{1}{2}\), and using \(g'(x)=\frac{2\alpha}{4+(\pi\alpha x)^2}\) (\(\alpha=2\)) to replace the true gradient during backpropagation. The optimizer is SGD (momentum 0.9) with a cosine annealing learning rate scheduler. On CIFAR, the initial learning rate is 0.02, weight decay is 5e-4, batch size is 128, and training is run for 256 epochs. On ImageNet, the initial learning rate is 0.1, weight decay is 2e-5, batch size is 256, and training is run for 128/400 epochs. Online weight standardization is performed before each training step, and all standardization operations are folded into static weights once training is concluded.

Key Experimental Results

Main Results

IS-SNN is comprehensively compared with state-of-the-art (SOTA) methods utilizing dynamic BN across three architectures (VGG / ResNet / Spikformer) on static datasets (CIFAR-10/100, ImageNet) and neuromorphic datasets (DVS-Gesture, CIFAR10-DVS). Representative results are summarized below.

Dataset	Model	Timestep	w/o BN Acc	IS-SNN Acc	Dynamic BN SOTA	SOTA Method
CIFAR-10	VGG-9	4	10.00	92.91	94.50 (T=6)	TET
CIFAR-10	ResNet-19	4	90.46	94.32	94.44	TET
CIFAR-10	ResNet-19*	4	89.94	96.02	95.94	TAB
CIFAR-100	ResNet-19	4	67.99	76.02	76.13	TEBN
CIFAR-100	ResNet-19*	4	70.60	79.97	78.71	TEBN
CIFAR-100	ResNet-152	4	17.10	76.98	—	—
ImageNet	ResNet-34	4	32.58	68.05 (E400)	68.28	TEBN
DVS-Gesture	7B-Net	16	77.08	96.88	97.92	w/ BN
CIFAR10-DVS	VGGSNN	8	10.0	77.7	78.1	w/ BN

* indicates the use of strong data augmentation (Mixup/Cutmix). Key Findings: (1) The w/o BN baseline completely collapses on deep models (only 17.10% for ResNet-152, and 32.58% for ImageNet ResNet-34), while IS-SNN stably recovers performance to competitive levels; (2) IS-SNN outperforms TAB and TEBN under strong augmentation settings (yielding 79.97% on CIFAR-100 ResNet-19*, surpassing TEBN's 78.71%); (3) IS-SNN can be applied in a plug-and-play manner to Spikformer (a Transformer architecture), achieving 95.43%/78.92% on CIFAR-10/CIFAR-100, outperforming the original Spikformer's 95.19%/77.86%.

Ablation Study

Ablation Item	Dataset/Model	Metric	Key Conclusion
Full IS-SNN	CIFAR-100 ResNet-19*	79.97%	Full model
γ_ℓ ≡ 1 (degenerated to naive WS)	CIFAR-100 VGG-11*	1.00%	VGG completely collapses; topology-aware γ_ℓ is necessary
γ_ℓ ≡ 1 (degenerated to naive WS)	CIFAR-100 ResNet-19	70.69%	ResNet accuracy drops by 5.33%, residual structures have some fault tolerance but still need γ_ℓ
w/o BN	CIFAR-100 ResNet-19B (5 transition blocks)	1.00%	More transition blocks exacerbate firing-rate decay
IS-SNN	CIFAR-100 ResNet-19B (5 transition blocks)	78.71%	IS-SNN is robust to topological changes, only 1.26% lower than standard ResNet-19
IS-SNN + Dropout	CIFAR-100 ResNet-19	+0.21%	Removing BN allows recovery of randomness benefits via pure training-phase regularization
σ_g perturbation ±10%	CIFAR-10/100 ResNet-19	-0.23%/-0.31%	Highly robust to σ_g estimation error

Key Findings

• Topology-aware \(\gamma_\ell\) is the core pillar of the method: Fixing \(\gamma_\ell\) to 1 (naive WS) renders VGG untrainable and causes ResNet to suffer severe accuracy drops. This indicates that merely constraining the weight norm is insufficient to combat firing-rate decay, and the scaling factor must be calculated layer-by-layer by incorporating the network topology.
• The residual scaling factor \(\alpha\) exhibits a broad optimal plateau: \(\alpha=0.5\) consistently yields optimal performance with the minimum variance across 48 independent training runs. The accuracy gap between \(\alpha=0.5\) and \(\alpha=1.0\) is \(<0.2\%\), showing that the method is insensitive to \(\alpha\), allowing a unified reuse of 0.5 without architecture-specific tuning.
• IS-SNN possesses significant scaling capabilities for deep architectures: On CIFAR-100, ResNet-152 recovers from 17.10% (w/o BN) to 76.98%. On ImageNet, ResNet-34 improves from 32.58% to 68.05%, achieving an absolute gain of 35.47%.
• Significant hardware benefits: Training throughput raises from 401 to 461 imgs/sec/GPU (+15%), with a 17% reduction in GPU memory. On FPGA, the LUT resource consumption of the LIF neuron core is reduced by 96.4% (the hardware cost difference between multipliers and pure accumulators is \(\mathcal{O}(N^2)\) vs \(\mathcal{O}(N)\), aligning with the theoretical estimation of 93.75%).

Highlights & Insights

• Shifting normalization from "runtime correction in activation space" to "compile-time folding in weight space": This perspective shift is the most elegant design in the paper. Traditional BN modifies the distribution step-by-step in the activation space, whereas IS-SNN applies constraints once and for all in the weight space before folding—effectively transforming normalization from an "online service" into "offline preprocessing" with zero inference cost. This paradigm of "online standardization during training + static folding during inference" can be extended to other efficient network designs requiring runtime normalization.
• Using firing-rate decay as a core diagnostic signal: Instead of generalized "training instability" analyses, the paper uses forward signal analysis to precisely quantify the firing-rate ratio \(c=\mu_{L+1}/\mu_L\) and entropy \(H(S_L)\), converting the problem into measurable physical metrics that guide the solution. This "diagnosis before prescription" methodology is highly reusable.
• Empirical estimation of \(\sigma_g\) instead of analytical derivation: Facing the practical difficulty of analytically solving non-linear SNN dynamics, the authors directly utilize Monte Carlo simulation to calibrate the neuron output variance. Coupled with robustness verification (with ±10% perturbations barely affecting accuracy), this represents a pragmatic engineering choice—not pursuing theoretical perfection but grounding the design via empirical validation.
• Unified setting of \(\alpha=0.5\) and bit-shift implementation: Taking \(\alpha\) as an integer power of 2 (\(0.5 = 2^{-1}\)) allows the scaling operation to be implemented as highly efficient bit shifts instead of multiplications, obtaining cross-architecture universality while maintaining hardware friendliness.

Limitations & Future Work

• Only applicable to inference scenarios, online learning is unsupported: The offline reparameterization of IS-SNN assumes frozen weights post-training, which is suitable for edge inference deployment. For online learning (such as on-chip STDP or continuous adaptation), hardware support for runtime statistical tracking remains necessary.
• Hardware analysis focused on local data paths: FPGA experiments only quantified the LUT savings at the neuron core level (Component B, affine transformation), excluding the statistic aggregation overhead of dynamic BN variants (Component A) and the complete storage/routing/system scheduling overhead. End-to-end energy evaluation on neuromorphic chips remains future work.
• Non-monotonic interaction between max-pooling and spike firing rate: The paper notes that the input-output variance relation of max-pooling behaves non-monotonically with the firing rate (\(\sigma_{in}^2\) peaks at \(0.25\) when \(\mu_{in}=0.5\) and then drops), suggesting a preference for strided convolutions for downsampling. This constraint may limit architectural flexibility in practical deployments.
• Limited gains on Spikformer: On CIFAR-10 Spikformer, IS-SNN only outperforms the original Spikformer by 0.24% (95.43% vs 95.19%), implying that the attention mechanism of Transformer-like architectures already tolerates firing-rate decay to some extent, making the core benefit of IS-SNN primarily evident in convolutional architectures.
• Explorable directions: Leveraging the stable signal propagation characteristics of IS-SNN to systematically study the trade-off among firing rate, sparsity, and accuracy; adapting to hardware-friendly online learning paradigms (such as eligibility trace); extending to sequential tasks (speech, NLP) and self-supervised learning to verify the generalizability of BN-free SNNs.

Related Work & Insights

• vs Dynamic BN variants (TEBN / TAB / tdBN): These methods stabilize training by introducing dynamic statistics along temporal or batch dimensions, at the expense of requiring step-by-step multiplications and statistical tracking during inference, making them unfoldable into static parameters. IS-SNN replaces them with weight space standardization, achieving zero normalization overhead during inference. IS-SNN performs comparably or slightly better than TEBN in accuracy (CIFAR-100 ResNet-19*: 79.97% vs 78.71%) while significantly outperforming it in efficiency.
• vs OTTT (Online Training Through Time): OTTT also employs WS, but its goal is to reduce online training complexity, treating WS as a general weight reparameterization method. IS-SNN differs in that: (1) it is guided by a different problem (stabilizing deep BN-free SNN inference vs online training efficiency); (2) the topology-aware derivation of \(\gamma_\ell\) is unique to IS-SNN, which OTTT lacks; (3) IS-SNN systematically validates its scalability on deep architectures (ResNet-152, ImageNet), whereas OTTT is only evaluated on small-scale shallow networks.
• vs NFNet (Normalizer-Free ResNet): NFNet, designed for ANNs, uses scaled WS and residual branch scaling based on signal propagation analysis, serving as an important precursor to IS-SNN on the ANN side. However, a direct transfer to SNNs is unfeasible because spike activations are discrete, temporally accumulated, and governed by neuron firing dynamics. The core adaptation of IS-SNN lies in: (1) replacing the analytical variance relationship of ReLU in ANNs with empirically estimated \(\sigma_g^2\); (2) re-deriving the variance propagation formulas under spiking dynamics; (3) focusing signal stabilization on firing-rate decay rather than vanishing gradients.

Rating

• Novelty: ⭐⭐⭐⭐ Combines the concept of "normalization folding" with SNN firing-rate stability. The topology-aware \(\gamma_\ell\) derivation and empirical \(\sigma_g\) estimation constitute unique contributions distinct from ANN BN-free methods, though the core mechanism (WS + residual scaling) has precedents in ANNs.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 5 datasets, 3 architectures (VGG/ResNet/Transformer), deep scaling validation (ResNet-152), hardware benchmarks (FPGA LUT + GPU throughput), and multi-dimensional ablations (\(\gamma_\ell\) / \(\alpha\) / number of transition blocks / Dropout / \(\sigma_g\) perturbations). The experimental design is comprehensive and self-consistent.
• Writing Quality: ⭐⭐⭐⭐ Possesses clear progression from diagnosing firing-rate decay to describing WS mechanisms, topological derivations, and empirical estimations. The hardware analysis section properly matches theoretical expectations with empirical measurements, although some paragraphs are dense and may require reference back to the formulas upon first reading.
• Value: ⭐⭐⭐⭐⭐ Resolves a practical and urgent engineering conflict in the deep SNN field (high performance vs hardware efficiency). It provides a reproducible, complete solution with open-source code, holding clear practical significance for transitioning SNNs from academic benchmarks to actual neuromorphic deployments.

Related Papers

• [ICLR 2026] Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition
• [AAAI 2026] I2E: Real-Time Image-to-Event Conversion for High-Performance Spiking Neural Networks
• [ICLR 2026] Breaking Gradient Temporal Collinearity for Robust Spiking Neural Networks
• [ICML 2026] Bullet Trains: Parallelizing Training of Temporally Precise Spiking Neural Networks
• [ICLR 2026] Beyond Linear Processing: Dendritic Bilinear Integration in Spiking Neural Networks