FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration

Conference: ECCV 2026
arXiv: 2606.28654
Code: https://github.com/nadarasarbahavan/FEDLAS
Area: Model Calibration / Neural Networks
Keywords: Label Smoothing, Model Calibration, Feature Norm, Bidirectional Calibration Gating, Adaptive Regularization

TL;DR

Proposes FeDLaS (Feature-modulated Bidirectional Label Smoothing), which utilizes the L1 norm of hidden layer features as a confidence proxy to bypass the validation limits of softmax saturation. Combined with bidirectional calibration gating, it dynamically determines the overconfidence/underconfidence state of each sample in real-time, adaptively modulating the label smoothing strength per sample to significantly reduce ECE and AECE while maintaining Top-1 accuracy.

Background & Motivation

Background: The softmax output confidence of deep neural network classifiers systematically deviates from the true correct probability. The one-hot labels in standard cross-entropy training force the model to assign a probability close to 1 to target classes, leading to overconfidence. Label Smoothing (LS) introduces entropy regularization by uniformly distributing probability mass across all classes, and Margin-based LS (MbLS) further models the regularization as a margin constraint on logit gaps. Park et al. theoretically demonstrated that many advanced calibration methods are essentially implicit, sample-adaptive variants of label smoothing.

Key Challenge: Existing LS and MbLS rely on preset uniform smoothing rules and only perform unidirectional correction on overconfidence. However, a large number of samples during training are underconfident—the model lacks confidence even in correct predictions. Crucially, the confidence state of samples changes dynamically during training (model non-stationarity), and existing methods cannot identify the current confidence state on a sample-by-sample, step-by-step basis to apply the corresponding level of smoothing. Park et al. summarized four unresolved limitations: Lim1 fails to cover the full confidence range; Lim2 only handles overconfidence unidirectionally; Lim3 does not adapt to non-stationary model training; Lim4 using softmax output itself is prone to overconfidence. Some methods (e.g., AdaFocal) attempt to handle underconfidence but require an extra validation set.

Key Insight: Feature norms have been theoretically and experimentally proven to serve as class-agnostic confidence proxies—the L1 norm of hidden layers is equivalent to the maximum logit of an implicit binary classifier, reflecting the overall discriminative strength of the model, which is free from softmax saturation limits and possesses a wider dynamic range. This paper designs a plug-and-play adaptive smoothing module based on this concept.

Core Idea: An Adaptive Smoothing Module (ASM) is constructed by combining a Norm-based Confidence Indicator (NCI) and a Bidirectional Calibration Gating (BCG) module. It dynamically evaluates the overconfidence/underconfidence state of each training sample in real-time and bidirectionally regulates the smoothing strength based on this state, thereby covering the full confidence range.

Method

Overall Architecture

FeDLaS decomposes a standard classification network into F_backbone → F_head → softmax. During training, feature vectors and classification headers are extracted from the backbone. The ASM module computes the adaptive smoothing coefficient \(\alpha^{(i)}\) for each sample, which is then injected into the convex combination of LS or MbLS loss. ASM consists of two sub-modules: NCI uses the feature L1 norm (normalized via EMA) to yield a continuous confidence signal, and BCG utilizes a small MLP on logits to output a binary \(\pm 1\) gating signal. Both are modulated by a sigmoid function to obtain \(\alpha^{(i)}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input x(i)"] --> B["Backbone<br/>F_backbone"]
    B --> C["Feature z(i)"]
    C --> D["Classification Head F_head"]
    D --> E["Logit s(i)"]
    C --> F["NCI<br/>L1 Norm +<br/>EMA Normalization"]
    E --> G["BCG<br/>MLP + STE"]
    F --> H["ASM<br/>Sigmoid Modulation"]
    G --> H
    H --> I["α(i) = α·f(·)"]
    E --> J["Loss Calculation<br/>LS / MbLS Convex Combination"]
    I --> J
    J --> K["Backpropagation"]

Key Designs

1. NCI (Norm-based Confidence Indicator): Using EMA-Normalized Feature Norm as a Confidence Proxy

Traditional methods rely on the maximum softmax probability as a confidence measure. However, softmax behaves with a saturation tendency—where even low-quality samples exhibit a maximum probability close to 1, losing fine-grained differences (experiments in Paper Fig. 1(c) verified that the dynamic range of the L1 norm is much wider than that of softmax probability). Park et al. proved that in the hidden layers of ReLU/GELU networks, the L1 norm of the feature vector converges to the maximum logit of an implicit binary classifier, serving as a class-agnostic metric of the model's discriminative strength.

NCI is computed as \(\gamma(z^{(i)}) = (\|\text{sg}[z^{(i)}]\|_1 - \mu_z^k) / \sigma_z^k\), where \(\text{sg}[\cdot]\) represents the stop-gradient operation. The key design is utilizing an Exponential Moving Average (EMA) to track the global mean and variance \(\mu_z^k, \sigma_z^k\) rather than relying on batch statistics. This addresses two issues: (1) it prevents fluctuations in feature norm variance from driving the sigmoid into saturation zones ("dead" adaptation); and (2) it bases the smoothing correction on the sample's relative position across the entire training dataset rather than its relative position within a local batch. The EMA updates as \(\mu_z^k = \theta \hat{\mu}_z^k + (1-\theta)\mu_z^{k-1}\), where momentum \(\theta\) controls the update pace. \(\gamma\) can also be substituted with the softmax probability or the maximum logit (the paper compares these variants in the ablation study).

2. BCG (Bidirectional Calibration Gating): Discrete Gating to Differentiate Overconfidence and Underconfidence

BCG outputs \(\psi(\mathbf{s}^{(i)}) \in \{-1, +1\}\), where \(+1\) represents the positive mode (suppressing overconfidence) and \(-1\) represents the negative mode (boosting underconfidence). It is implemented as a lightweight MLP \(\mathcal{F}_{gate}\) that takes the stop-gradient logit as input, and maps the softmax-processed output to a binary sign using fixed weights \([1, -1]^\top\): \(\psi(\mathbf{s}^{(i)}) = \text{sgn}\left(\mathbf{w}_{\text{gate}}^\top \cdot \sigma(\mathcal{F}_{gate}(\text{sg}[\mathbf{s}^{(i)}]))\right)\). Since \(\text{sgn}\) is non-differentiable, the Straight-Through Estimator (STE) is utilized to use discrete values during forward propagation and pass through continuous gradients during backward propagation, maintaining end-to-end trainability.

BCG stability experiments (Paper Fig. 3) demonstrate that: the proportion of overconfident/underconfident samples remains essentially stable during training, shifting only slightly during learning rate decay; the sample state flip rate continuously decreases, indicating gating convergence. Unlike AdaFocal, which requires an additional validation set, BCG autonomously learns confidence state classification entirely on the training data.

3. ASM (Adaptive Smoothing Module): Generating Dynamic Smoothing Coefficients by Integrating NCI and BCG

ASM combines NCI and BCG into a unified modulation function: \(\alpha^{(i)} = \alpha \cdot 2\phi\left(\beta \cdot \psi(\text{sg}[\mathbf{s}^{(i)}]) \cdot \gamma(\text{sg}[\mathbf{z}^{(i)}])\right)\) where \(\phi\) is the sigmoid function. Three properties of the sigmoid precisely match the design requirements: (1) Boundedness—outputs in \((0, 2\alpha)\), degrading to base methods when \(\beta=0\); (2) Monotonicity—under a fixed mode, \(\alpha^{(i)}\) is monotonically positive/negative with NCI; (3) Symmetry—rotational symmetry exists between the positive and negative modes. The scalar \(\beta\) controls sensitivity (selected via validation set; \(\beta=0.5\) for CIFAR-10, \(\beta=4.0\) for Tiny-ImageNet).

In the positive mode (\(\psi=+1\), overconfident), \(\alpha^{(i)}\) monotonically increases with NCI, applying stronger smoothing to high-confidence samples. In the negative mode (\(\psi=-1\), underconfident), \(\alpha^{(i)}\) monotonically decreases with NCI, applying weaker smoothing (or even "negative smoothing") to samples that should be confident but display low confidence, thereby boosting their prediction confidence.

Loss & Training

FeDLaS embeds two baseline losses in a convex combination form: - FeDLaS-LS: \(\mathcal{L}_{FeDLaS-LS}^{i} = (1-\alpha^{(i)})\mathcal{L}_{\text{CE}}^{i} + \alpha^{(i)}E(\hat{\mathbf{p}}^{(i)})\), where \(\alpha=0.05\) for baseline LS (0.1 for CIFAR-100). - FeDLaS-MbLS: \(\mathcal{L}_{FeDLaS-MbLS}^{i} = (1-\alpha^{(i)})\mathcal{L}_{\text{CE}}^{i} + \alpha^{(i)}R(\mathbf{s}^{(i)})\), with the original margin and \(\lambda\) configuration maintained for MbLS (\(m=6,\lambda=0.1\) / \(m=10,\lambda=0.05\)). Unlike MbLS, which isolatedly sums the regularizer, the convex combination form of FeDLaS-MbLS allows \(\alpha^{(i)}\) to control both the classification loss and the regularization weight simultaneously.

Training entirely replicates the standardized protocol of ACLS (SGD optimizer, cosine learning rate scheduler, standard data augmentations). ResNet extracts the feature vector prior to the classification head, and ViT extracts the [CLS] token. \(\beta\) and \(\theta\) are selected via the validation set; fine-grained tasks inherit the optimal configuration of Tiny-ImageNet.

Key Experimental Results

Main Results

Table 1 shows the calibration results (15 bins) across 6 architecture combinations on CIFAR-10/100 and Tiny-ImageNet. Rankings are based on the average across all scenarios:

Method	Avg. ECE Rank	Avg. AECE Rank	Avg. Top-1 Accuracy
CE	14.00	14.00	78.74
LS	7.00	9.00	79.69
MbLS	4.33	6.00	79.66
ACLS	5.50	4.83	79.78
DFL	7.50	6.33	78.89
AdaFocal	10.00	9.83	78.77
FeDLaS-LS	5.50	4.17	79.62
FeDLaS-MbLS	1.83	3.83	79.58

FeDLaS-MbLS achieves an average ECE ranking of 1.83, leading all methods comprehensively and significantly outperforming the baseline MbLS (4.33). This indicates that bidirectional adaptive modulation effectively bridges the calibration gap of margin-based losses. To avoid misinterpretation: FeDLaS-LS accuracy is 79.62 vs 79.69 for LS—with standard configurations, adaptive modulation does not compromise classification accuracy.

Ablation Study

Table 2 shows the decomposed ECE (CIFAR-10, ResNet-50) demonstrating the bidirectional correction effect of BCG:

Configuration	O-ECE↓	U-ECE↓	Description
CE	5.85	0.00	Severe overconfidence, almost no underconfidence
LS	1.16	2.45	Over-correction: O-ECE decreased but U-ECE surged
FeDLaS-LS	0.89	0.40	Bidirectional balance, LS U-ECE reduced from 2.45 to 0.40
MbLS	1.09	0.12	Moderate O-ECE, low U-ECE
FeDLaS-MbLS	1.11	0.02	O-ECE on par with MbLS, U-ECE approaches zero

Key Finding: CE exhibits unidirectional overconfidence (\(U\text{-ECE} \approx 0\)); LS and MbLS show reverse over-correction (drastically depressing \(O\text{-ECE}\) while \(U\text{-ECE}\) rises). FeDLaS variants maintain low rates in both error directions—FeDLaS-LS reduces the \(U\text{-ECE}\) of LS from 2.45 to 0.40 (an 83.7% reduction) while keeping \(O\text{-ECE}\) even lower.

Key Findings

• BCG is the Largest Contributor: As inferred from the ablation, removing BCG (which degrades the model to unidirectional smoothing) leads to a lack of underconfidence correction. FeDLaS-LS reduces \(U\text{-ECE}\) compared to LS from 2.45 to 0.40, a reduction entirely derived from the underconfidence detection capability of BCG.
• More Pronounced Advantages on Fine-Grained Tasks: On CUB-200 and FGVC-Aircraft, FeDLaS-LS achieves an average ECE rank of 1.5, far outperforming LS (3.0) and MbLS (4.5). Fine-grained tasks exhibit smaller inter-class differences and wider variations in sample difficulty, which further highlights the advantages of sample-adaptive smoothing.
• Restoring Out-Of-Distribution (OOD) Detection Performance: LS and MbLS significantly degrade OOD detection AUROC compared to CE, but FeDLaS variants substantially bridge this gap. For instance, in the CIFAR-100 \(\rightarrow\) SVHN scenario, the AUROC of LS is 72.92, whereas FeDLaS-LS improves it to 84.33 (+11.41%).
• Excellent BCG Stability: The proportions of overconfident and underconfident samples remain basically constant throughout training, with only minor fluctuations occurring when learning rate schedulers are triggered. The sample state flip rate continuously drops toward zero, illustrating gating convergence.

Highlights & Insights

• Systematic Application Expansion of Feature Norms: Previously, feature norms were primarily utilized for OOD detection and hard example mining. This study is the first to leverage them as a sample-adaptive confidence proxy during training for label smoothing. EMA normalization is a clean and effective engineering technique—enabling NCI to reflect the relative confidence of a sample across the entire training history rather than its position within a local batch, while avoiding sigmoid saturation.
• STE Maintains the Differentiability of Discrete Gating: BCG outputs discrete binary signs (\(\pm 1\)). Using the Straight-Through Estimator (STE) delivers discrete values in forward propagation and passes through continuous gradients in backward propagation. This maintains gating explicitness while keeping the overall framework end-to-end trainable.
• Bidirectionally Symmetric Mathematical Design: The relationship between NCI and \(\alpha^{(i)}\) is symmetric under positive/negative modes (monotonically increasing vs. decreasing). The boundedness of the sigmoid prevents out-of-distribution or anomalous samples from generating runaway smoothing coefficients. This "bounded + monotonic + symmetric" modulation framework is transferable to other scenarios necessitating bidirectional regularization adaptation.
• Plug-and-Play with Zero Inference Overhead: ASM relies solely on the gradient-detached computation of backbone features and logits. It does not alter the network structure or add inference overhead—only a miniature MLP and EMA tracking are introduced during training, while remaining completely silent during inference.

Limitations & Future Work

• Theoretical Assumption Limitations of NCI: Modeling feature norms as a confidence proxy relies on theoretical bounds defined for one-sided rectified activation functions such as ReLU/GELU (the inequality derivation in Eq. 2). For activation functions like Swish and ELU, or for networks lacking a clear backbone-head split, the theoretical foundation may not hold. The authors explicitly acknowledge this limitation.
• Tuning Requirements for Hyperparameters \(\beta\) and \(\theta\): \(\beta\) dictates sigmoid sensitivity, while \(\theta\) governs the EMA momentum, both of which require validation set tuning. Although fine-grained tasks can directly reuse configurations from Tiny-ImageNet, this still introduces extra parameter tuning costs in real-world deployment.
• BCG's Sole Reliance on Logits: Gating decisions determine confidence states based solely on logits, without utilizing representation layer features. Integrating feature space representations into the gating module could potentially improve the classification accuracy of underconfidence and overconfidence states.
• Exploration of Semi-Supervised/Self-Supervised Contexts Unaddressed: The paper only mentions the potential of the class-agnostic nature of NCI in unsupervised, self-supervised, semi-supervised, and noisy-label calibration settings within the conclusion, without actual experimental validation. This remains the most direct future direction, as model calibration under luxury annotation bottlenecks presents substantial challenges.

Related Work & Insights

• vs. Label Smoothing (LS): LS applies a fixed \(\alpha\) across all samples; FeDLaS-LS dynamically generates \(\alpha^{(i)}\) via ASM. On CIFAR-100, FeDLaS-LS reduces ECE from ~5.42 (for LS) to ~4.29, and \(U\text{-ECE}\) from ~2.69 to ~0.59.
• vs. Margin-based LS (MbLS): MbLS relaxes the rigid entropy constraints of LS, yet the regularization strength remains globally fixed. FeDLaS-MbLS improves the average ECE ranking to 1.83 from 4.33 for MbLS.
• vs. ACLS: ACLS introduces sample-adaptive margins based on MbLS, but is still restricted to unidirectional overconfidence correction. FeDLaS-MbLS adds an underconfidence gating system, reducing ECE on Tiny-ImageNet for ResNet-50 from ~1.37 (for ACLS) to ~1.22.
• vs. AdaFocal / DFL: AdaFocal requires an additional validation set to select the temperature parameter, and DFL has a limited regularization spectrum. FeDLaS does not depend on a validation set, and the cooperation between BCG and NCI spans the entire confidence landscape.

Rating

• Novelty: ⭐⭐⭐⭐ Introducing the feature norm from OOD detection to serve as a sample-adaptive label smoothing modulator during training is a novel perspective; the combination of bidirectional gating and EMA-based normalization is clean and effective. However, the overall method remains an adaptive coefficient extension under the LS/MbLS framework, qualifying its architectural innovation as incremental.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ It compares against over 10 baselines on 3 standard and 2 fine-grained classification datasets, additionally evaluating OOD detection, decomposed ECE analysis, and BCG stability visualizations. The training protocol is perfectly aligned with ACLS to ensure fair benchmarking.
• Writing Quality: ⭐⭐⭐⭐ The motivation is articulately presented, and the progression from theoretical proof to design motivation and then to engineering details is logical. Formulas in the Method section are slightly dense, and some derivations could be relegated to the appendix to enhance readability.
• Value: ⭐⭐⭐⭐ Model calibration is a critical challenge in real-world deployment; its plug-and-play and zero additional inference cost characteristics render high practical value. The general design ideology of bidirectional calibration can be extended to other regularization schemes.

Related Papers

• [ECCV 2026] Match-Any-Events: Zero-Shot Motion-Robust Feature Matching Across Wide Baselines for Event Cameras
• [ECCV 2026] Spectral Gating via Damped Oscillations for Adaptive Implicit Neural Representations
• [ECCV 2026] Intrinsically Stable Spiking Neural Networks: Overcoming the Performance Barrier in the Absence of Batch Normalization
• [ECCV 2026] Lost in the Tail: Addressing Geographic Imbalance in Urban Visual Place Recognition
• [ECCV 2026] Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning