Denoised Variance-Based Pruning with Optimal Brain Bias Compensation¶

Conference: ECCV2026
Paper: ECCV Official Page
PDF: Full Paper
Authors: Geon Tack Lee, Jaegul Choo, Kang Eun Jeon
Code: https://github.com/geontackee/DVBP_OB2C
Area: Model Compression
Keywords: structured pruning, activation variance, covariance denoising, random matrix theory, bias compensation

TL;DR¶

DVBP + OB2C selects MLP neurons using a mixture of denoised and raw activation variance, then solves for weight and bias compensation using centered covariance; with 50% of MLP hidden neurons pruned and no fine-tuning, it raises Swin-S ImageNet-1K Top-1 accuracy from VBP's 69.91% to 77.24%.

Background & Motivation¶

Structured pruning removes entire neurons or channels, directly reducing matrix dimensions without requiring sparse kernels to reduce computation. Whether a neuron can be removed, however, cannot be determined from weight magnitude alone. Even a neuron with substantial weights contributes approximately a constant if its activation barely changes across inputs, allowing its contribution to be absorbed into the next layer's bias. Variance-Based Pruning (VBP) exploits this observation by ranking activation variance and restoring the mean contribution after removal.

VBP's limitations follow from the same reasoning. Variance estimated from a finite calibration set contains statistical noise. Ranking errors become more consequential at high pruning ratios, when many moderately important neurons must also be considered for removal. Meanwhile, a bias can restore only a constant, not the input-dependent part of a removed neuron's contribution. Surviving neurons may contain correlated information, but unchanged outgoing weights cannot actively reassign that information to replace the removed pathway.

Rather than introducing a trainable pruning controller, this paper revises VBP's statistical selection and post-pruning recovery. Core Idea: improve neuron selection through spectral denoising, then incorporate bias compensation into layer-output reconstruction so that recovery becomes a closed-form weight update based on the same centered covariance statistics.

Method¶

Overall Architecture¶

The inputs are a pretrained model and calibration images; the output is a model with smaller hidden dimensions. The method targets MLP hidden neurons in DeiT, Swin, and ConvNeXt, without pruning attention heads or changing a block's external input and output dimensions. Online Centered Statistics provides means and covariances; Denoised Hybrid Scoring determines the network-wide removal set; OB2C Joint Compensation updates the second linear layer's weights and bias using the raw centered covariance before the corresponding dimensions are physically removed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pretrained model<br/>Calibration images"] --> B["Online Centered Statistics"]
    B --> C["Denoised Hybrid Scoring"]
    B -->|Mean and raw covariance| D["OB2C Joint Compensation"]
    C -->|Neuron removal set| D
    D --> E["Remove matching dimensions<br/>Compressed model"]

Key Designs¶

1. Online Centered Statistics: a shared statistical foundation for ranking and compensation

The statistics describe the MLP hidden activations received by its second linear layer. Whereas VBP needs only each neuron's mean and variance, weight compensation also requires information about how neurons vary together, so this method retains the full covariance. Diagonal entries describe individual activation variability; off-diagonal entries describe whether changes in a removed neuron can be explained by changes in surviving neurons. The latter relationship is essential to recovering outputs through weight compensation.

Saving every calibration activation would make storage grow with the sample count. Instead, the method maintains counts, means, and covariances batch by batch. Chan/Pébay-style merging combines the incoming and historical centered statistics, including an outer-product correction for the difference between their means. This correction matters: even if both batches have little internal variation, different batch means introduce variation that simply averaging their covariances would miss. Online accumulation avoids storing the full activation history, but each layer still requires a dense covariance matrix; its cost is therefore not independent of hidden width.

2. Denoised Hybrid Scoring: suppress spectral noise for ranking and adapt its influence to the pruning ratio

Beyond low-rank signal, the sample covariance spectrum contains a noise bulk induced by finite sampling. The method estimates its upper boundary using the Marchenko–Pastur (MP) distribution. It first estimates the noise scale from the ratio between the covariance eigenvalue median and the theoretical MP median, then retains only eigenvalues above the resulting upper boundary and their eigenvectors. Reconstructing these components in neuron coordinates and taking the diagonal produces denoised variance scores. Thus, the method still ranks original neurons rather than rotating the network into an eigenvector basis before pruning.

The aspect ratio for MP fitting is hidden width divided by the number of calibration images, not the total number of image patches, because patches from one image are highly correlated. This avoids counting correlated patches as additional independent observations, but it does not rigorously establish that real network activations satisfy the MP law's independent, identically distributed noise assumptions. Denoising may also discard weak but useful signals. Rather than replacing raw variance entirely, the method uses the target pruning proportion to mix the two scores.

\[ C_{\mathrm{denoised}}=\sum_{\lambda_i>\lambda_+}\lambda_i v_i v_i^\top, \qquad s_i=(1-p)C_{ii}+p(C_{\mathrm{denoised}})_{ii}. \]

Here, \(C\) is the raw centered covariance, \(\lambda_i,v_i\) are its eigenvalues and eigenvectors, \(\lambda_+\) is the estimated MP noise boundary, and \(p\in[0,1]\) is the target pruning proportion. Lower pruning ratios favor raw variance; higher ratios favor denoised variance. Hidden neurons across all targeted MLPs are globally ranked by this mixture, and the lowest-scoring fraction is removed without imposing a uniform layerwise ratio. Consequently, 50% refers to the overall fraction of targeted neurons removed, not a halving of every layer, all model parameters, or MACs.

3. OB2C Joint Compensation: eliminate the optimal bias before recovering weights with centered curvature

VBP's bias update addresses only the mean contribution of removed neurons. OB2C instead jointly treats compressed weights and an additive bias correction as optimization variables, minimizing the discrepancy between original and compressed linear-layer outputs on calibration data. Solving for the optimal bias first converts the problem into reconstruction of centered activations. The bias handles mean shifts, leaving weights to recover input-dependent variation. This explains why covariance is appropriate instead of directly applying OBC's uncentered second moment.

\[ \min_{\widehat W,b}\left\|WX-(\widehat W X+b\mathbf 1^\top)\right\|_F^2, \qquad b^*=(W-\widehat W)\mu, \qquad H=2\widetilde X\widetilde X^\top=2NC. \]

The columns of \(X\) are calibration activations, \(\mu\) is their mean, \(\widetilde X=X-\mu\mathbf 1^\top\), and \(C=\widetilde X\widetilde X^\top/N\). The variable \(b\) denotes compensation added to the existing bias, not a replacement that discards it. Here, \(H\) is the proxy Hessian of layerwise squared reconstruction error, not the Hessian of the full classification loss. It is proportional to covariance, rather than literally equal to covariance without normalization.

Once the removal index set \(P\) is fixed, OB2C uses the inverse raw centered covariance to compute a closed-form multi-weight update constrained to zero the selected columns:

\[ \delta W=-W_{:,P}\bigl([C^{-1}]_{P,P}\bigr)^{-1}[C^{-1}]_{P,:}, \qquad \widehat W=W+\delta W. \]

The update redistributes the removed columns' contributions through activation correlations, after which the optimal bias correction is computed from the difference between the original and updated weights. Adding only the original removed weights' mean contribution would be insufficient because surviving weights also change. Finally, the matching output dimensions and hidden bias entries of the first linear layer, and input dimensions of the second, are removed together while preserving the block's external interface.

These equations are organized from the variable relationships in Sections 3.1 and 3.4. Some equations in the local text have extraction damage; unspecified damping or inversion implementation details are not reconstructed. The two covariance uses must be distinguished: the spectrally truncated covariance participates only in scoring, whereas the OB2C derivation uses the raw centered covariance rather than directly inverting a matrix already truncated to low rank. Optimality is limited to the given removal set, calibration activations, and layerwise reconstruction objective; it does not establish a globally optimal pruning structure.

Loss & Training¶

The main results require neither gradient-based training nor post-pruning fine-tuning. Calibration uses 8,192 images randomly sampled with a fixed seed from the ImageNet-1K training split, without augmentation. After collecting statistics, scoring and closed-form updates perform pruning, followed directly by validation Top-1 evaluation. Here, zero-shot means no post-pruning fine-tuning, not the absence of calibration data or zero-shot recognition of unseen classes.

A separate recovery analysis fine-tunes DeiT-B and Swin-S after 70% MLP pruning, using VBP's official script, the unpruned model as teacher, AdamW, a learning rate of \(1.5\times10^{-5}\), cosine annealing, and batch size 32 over 10 epochs. This additional experiment should not be conflated with the main tables' fine-tuning-free results.

Key Experimental Results¶

Main Results¶

The metric is ImageNet-1K Top-1 accuracy, higher is better. Every row below uses 50% MLP hidden-neuron pruning without post-pruning fine-tuning. The authors locally evaluate full models and baselines under the same setup; these baseline accuracies should not automatically be treated as standard values for other pretrained checkpoints.

Model	Full-model Top-1 (%) ↑	VBP (%) ↑	DVBP + OB2C (%) ↑	Gain over VBP (percentage points)
DeiT-S	79.86	66.03	70.25	4.22
DeiT-B	81.98	68.92	75.85	6.93
Swin-S	83.32	69.91	77.24	7.33
Swin-B	85.27	70.86	76.16	5.30
ConvNeXt-T	84.17	15.44	44.90	29.46
ConvNeXt-B	85.81	56.12	70.76	14.64

Values come from Tables 1 and 3. ConvNeXt-T's 29.46 is a percentage-point difference, not a 29.46% relative improvement; its post-pruning accuracy remains only 44.90%, so this is not near-lossless compression. The abstract's generalization that Small/Base variants retain over 90% of original accuracy also does not hold for every row: DeiT-S achieves 70.25% against a full-model 79.86%.

For resources, Table 2 reports DeiT-B shrinking from 86.57M to 58.24M parameters and from 17.59G to 12.01G MACs, again showing that 50% MLP pruning does not halve the entire model. VBP and the proposed method also have slightly different ConvNeXt footprints: ConvNeXt-T uses 12.61M / 12.96M parameters and 2.95G / 2.94G MACs, respectively. These comparisons are not exactly parameter-matched.

Ablation Study¶

The following selection from Table 5 compares successive stages at the same 50% pruning setting. “+D” adds denoised scoring to VBP; “+D+O” additionally applies OB2C. All accuracy values are percentages, higher is better.

Model	VBP	+D	+D+O	OB2C gain over +D (percentage points)
DeiT-T	42.90	44.32	56.40	12.08
DeiT-S	66.03	65.37	70.25	4.88
Swin-S	69.91	71.21	77.24	6.03
Swin-B	70.86	70.11	76.16	6.05

Table 5 gives DeiT-T VBP as 42.90%, whereas Table 1 gives 42.89%; the respective source values are preserved. Adjacent ablation columns show a 12.08-percentage-point DeiT-T gain from adding OB2C, rather than the prose's characterization of nearly 14%.

Key Findings¶

OB2C is the larger recovery contributor in Table 5. Denoising alone is not universally beneficial: DeiT-S falls from 66.03% to 65.37%, and Swin-B from 70.86% to 70.11%. Better selection and better recovery should therefore be evaluated separately.
Denoising matters more at high pruning ratios. For Swin-S at 70% pruning, Table 6 reports 37.26% without denoising and 57.13% with MP upper-bound selection. At 50% pruning, however, retaining the top 50% of spectral components gives 77.76%, exceeding the MP method's 77.24%; the prose's claim of consistently highest accuracy is not supported by every row.
On an RTX 3090, Table 4 reports Swin-B pruning time increasing from VBP's 34.89 seconds to 66.92 seconds, and peak VRAM from 4.58G to 8.11G. These are offline compression costs, not deployed inference latency; the main paper does not use them to establish a measured edge-device speedup.

Highlights & Insights¶

Incorporating bias optimization into reconstruction is more explanatory than adding a mean correction after pruning. Centering leaves weights responsible only for residual variation, aligning the statistics with the compensation task.
Shared calibration statistics need not undergo identical processing for both purposes. Ranking can use denoised diagonal entries while recovery preserves correlations in the raw covariance, separating importance scores from reconstruction geometry.
Structural selection need not carry the entire recovery burden. Joint weight and bias updates can substantially improve accuracy even when denoised scoring alone slightly regresses, so evaluating ranking rules in isolation may mischaracterize the full pruning system.

Limitations & Future Work¶

The authors explicitly acknowledge that linearly mixing raw and denoised variance according to the pruning ratio is heuristic, without a theoretical optimality basis. Weak but useful spectral signals may be mistaken for noise.
The evaluation primarily covers ImageNet-1K classification and MLP-neuron pruning. It does not establish results for attention structures, other tasks, or every kind of linear layer. ConvNeXt-T retains a particularly substantial accuracy loss.
Real activation distributions may deviate from the MP noise model, and covariance inversion requires invertibility and numerical stability. The readable main paper does not specify singular-matrix handling; full algorithms and proofs are deferred to supplementary material absent from the current cache.
The ablations and tables support more conservative conclusions than some prose summaries, and no multi-seed uncertainty ranges are provided. Small denoising gains should not be treated as deterministic evidence of a mechanism's superiority solely because their decimal values differ.

vs VBP: VBP selects neurons using raw activation variance and absorbs their mean contribution. This method adds spectral denoising and correlation-based weight recovery, with additional covariance storage and computation but larger gains under aggressive pruning.
vs OBC / OBS: These methods provide second-order reconstruction-based compensation frameworks. By explicitly optimizing bias, this paper changes the compensation statistics to centered covariance. Selection uses a relatively direct hybrid variance score rather than full OBS sensitivity scoring.
vs OBS-Diff: The paper describes OBS-Diff as using Hessian/OBS calculations to select structures in diffusion models. Here, the emphasis is on separating selection from recovery and validating MLP pruning in classification backbones. Tasks and protocols differ, and no direct matched-setting performance comparison is provided.

Rating¶

Novelty: 4/5. The connection between bias elimination and covariance-based weight recovery is clear; spectral denoising and hybrid scoring mainly combine established tools.
Experimental Thoroughness: 3/5. Multiple backbones and scales, resource measurements, and ablations are covered, but the task scope is narrow, repeated-trial information is missing, and some comparisons have different footprints.
Writing Quality: 3/5. The central derivation is understandable, but the abstract and some experimental summaries make stronger claims than the tables support.
Value: 4/5. A practical recovery method for fine-tuning-free structured pruning, particularly useful for understanding bias compensation and centered statistics.