FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration¶

Conference: ECCV 2026
arXiv: 2606.28654
Code: https://github.com/nadarasarbahavan/FEDLAS
Area: Model Calibration / Neural Networks
Keywords: Label Smoothing, Model Calibration, Feature Norm, Bidirectional Calibration Gating, Adaptive Regularization

TL;DR¶

Proposed FeDLaS (Feature-modulated Bidirectional Label Smoothing), which utilizes the L1 norm of hidden layer features as a confidence proxy to bypass the softmax saturation defect. Combined with a bidirectional calibration gating mechanism, it dynamically determines the over-confidence/under-confidence state of each sample in real-time and adaptively modulates label smoothing intensity per sample, significantly reducing ECE and AECE while maintaining Top-1 accuracy.

Background & Motivation¶

Background: Systematic biases exist between the softmax output confidence of deep neural network classifiers and the true prediction probability. Standard cross-entropy training with one-hot labels forces the model to assign probabilities close to 1 to target classes, leading to over-confidence. Label smoothing (LS) introduces entropy regularization by uniformly distributing probability mass across all classes, and Margin-based LS (MbLS) further models this regularization as a margin constraint on logit gaps. Park et al. theoretically proved that many advanced calibration methods are essentially implicit sample-adaptive label smoothing variants.

Key Challenge: Existing LS and MbLS methods rely on pre-set uniform smoothing rules and only perform unidirectional correction for over-confidence. However, a large number of samples remain in an under-confident state during training—where the model lacks confidence even in correct predictions. More importantly, the confidence states of samples dynamically change during training (model non-stationarity), whereas existing methods cannot identify the current confidence state and apply the corresponding degree of smoothing on a per-sample and per-step basis. Park et al. summarized four unresolved limitations: Lim1 cannot cover the entire confidence range; Lim2 only handles over-confidence unidirectionally; Lim3 does not adapt to non-stationary model training; Lim4 using the softmax output itself is prone to over-confidence. Some methods (e.g., AdaFocal) attempt to handle under-confidence but require an additional validation set.

Key Insight: Feature norms have been theoretically and experimentally proven to serve as class-agnostic confidence proxies. The L1 norm of the hidden layer is equivalent to the maximum logit of an implicit binary classifier, reflecting the overall discriminative strength of the model, and is free from softmax saturation limitations with a broader dynamic range. Based on this, a plug-and-play adaptive smoothing module is designed.

Core Idea: Combine a "Norm-based Confidence Indicator (NCI)" and a "Bidirectional Calibration Gating (BCG)" into an Adaptive Smoothing Module (ASM) to detect the over-confident/under-confident state of each training sample in real-time and bidirectionally adjust the smoothing strength accordingly, thereby covering the full confidence range.

Method¶

Overall Architecture¶

FeDLaS decomposes a standard classification network into F_backbone → F_head → softmax. During training, feature vectors and classification head logits are extracted from the backbone. The adaptive smoothing coefficient $\alpha^{(i)}$ for each sample is computed via the ASM module and subsequently injected into the convex combination of LS or MbLS losses. The ASM consists of two sub-modules: NCI, which utilizes the feature L1 norm (normalized by EMA) to provide a continuous confidence signal; and BCG, which utilizes logits passed through a small MLP to output a binary gating signal of $\pm 1$. The two signals are modulated via a sigmoid function to obtain $\alpha^{(i)}$.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input x(i)"] --> B["Backbone<br/>F_backbone"]
    B --> C["Feature z(i)"]
    C --> D["Classification Head F_head"]
    D --> E["Logit s(i)"]
    C --> F["NCI<br/>L1 Norm +<br/>EMA Normalization"]
    E --> G["BCG<br/>MLP + STE"]
    F --> H["ASM<br/>sigmoid modulation"]
    G --> H
    H --> I["α(i) = α·f(·)"]
    E --> J["Loss Calculation<br/>LS / MbLS convex combination"]
    I --> J
    J --> K["Backpropagation"]

Key Designs¶

1. NCI (Norm-based Confidence Indicator): Using EMA-normalized Feature Norm as a Confidence Proxy

Traditional methods rely on the maximum softmax probability as the confidence measure, but softmax tends to saturate—even low-quality samples can have a maximum probability close to 1, losing fine-grained differences (the experiments in Fig. 1(c) of the paper verify that the dynamic range of the L1 norm is much wider than that of softmax probabilities). Park et al. proved that in the hidden layers of ReLU/GELU networks, the L1 norm of the feature vector converges to the maximum logit of an implicit binary classifier, serving as a class-agnostic metric of model discriminative strength.

The NCI is computed as $\gamma(z^{(i)}) = (\|\text{sg}[z^{(i)}]\|_1 - \mu_z^k) / \sigma_z^k$, where $\text{sg}[\cdot]$ represents the stop-gradient operation. The key design is the utilization of EMA to track the global mean and variance $\mu_z^k, \sigma_z^k$, rather than relying on batch statistics. This addresses two issues: ① it prevents the fluctuations of feature norm variance from driving the sigmoid function into saturation zones ("dead" adaptation); ② it ensures that the smoothing correction is based on the relative position of the sample within the entire training dataset, rather than its relative position within the batch. The EMA updates as $\mu_z^k = \theta \hat{\mu}_z^k + (1-\theta)\mu_z^{k-1}$, where momentum $\theta$ controls the update speed. $\gamma$ can also be replaced by softmax probabilities or maximum logits (the paper compares these variants in the ablation study).

2. BCG (Bidirectional Calibration Gating): Discrete Gating to Differentiate Over-confidence and Under-confidence

BCG outputs $\psi(\mathbf{s}^{(i)}) \in \{-1, +1\}$, where $+1$ represents the positive mode (suppressing over-confidence) and $-1$ represents the negative mode (boosting under-confidence). It is implemented as a lightweight MLP $\mathcal{F}_{gate}$ that takes the stop-gradient logit as input, and maps the softmax-processed output to a binary sign using fixed weights $[1, -1]^\top$: $\psi(\mathbf{s}^{(i)}) = \text{sgn}\left(\mathbf{w}_{\text{gate}}^\top \cdot \sigma(\mathcal{F}_{gate}(\text{sg}[\mathbf{s}^{(i)}]))\right)$. Since $\text{sgn}$ is non-differentiable, the Straight-Through Estimator (STE) is used to pass through discrete values in the forward pass and continuous gradients in the backward pass, maintaining end-to-end trainability.

Gating stability experiments of BCG (Fig. 3 in the paper) show that the proportion of over-confident/under-confident samples remains largely constant during training, shifting only slightly when the learning rate decays. The sample state flip rate continuously decreases, indicating gating convergence. Unlike AdaFocal, which requires an extra validation set, BCG autonomously learns the classification of confidence states entirely on the training data.

3. ASM (Adaptive Smoothing Module): Dynamic Smoothing Coefficient Generation via Integrating NCI and BCG

ASM combines NCI and BCG into a unified modulation function: $$\alpha^{(i)} = \alpha \cdot 2\phi\left(\beta \cdot \psi(\text{sg}[\mathbf{s}^{(i)}]) \cdot \gamma(\text{sg}[\mathbf{z}^{(i)}])\right)$$ where $\phi$ is the sigmoid function. Three properties of the sigmoid function precisely match the design requirements: ① Boundedness—outputting in $(0, 2\alpha)$, falling back to the baseline method when $\beta=0$; ② Monotonicity—$\alpha^{(i)}$ is monotonically increasing/decreasing with respect to NCI under a fixed mode; ③ Symmetry—rotational symmetry between positive and negative modes. The scalar $\beta$ controls sensitivity (selected via validation; $\beta=0.5$ for CIFAR-10 and $\beta=4.0$ for Tiny-ImageNet).

Under the positive mode ($\psi=+1$, over-confidence), $\alpha^{(i)}$ increases monotonically with NCI, applying stronger smoothing to highly confident samples. Under the negative mode ($\psi=-1$, under-confidence), $\alpha^{(i)}$ decreases monotonically with NCI, applying weaker smoothing (or even "negative smoothing") to samples that should be confident but exhibit low confidence, thereby boosting prediction confidence.

Loss & Training¶

FeDLaS embeds two types of baseline losses in the form of a convex combination: - FeDLaS-LS: $\mathcal{L}_{FeDLaS-LS}^{i} = (1-\alpha^{(i)})\mathcal{L}_{\text{CE}}^{i} + \alpha^{(i)}E(\hat{\mathbf{p}}^{(i)})$, where baseline LS utilizes $\alpha=0.05$ (0.1 for CIFAR-100). - FeDLaS-MbLS: $\mathcal{L}_{FeDLaS-MbLS}^{i} = (1-\alpha^{(i)})\mathcal{L}_{\text{CE}}^{i} + \alpha^{(i)}R(\mathbf{s}^{(i)})$, where the margin $m$ and $\lambda$ of MbLS maintain their original configurations ($m=6,\lambda=0.1$ / $m=10,\lambda=0.05$). Unlike MbLS, which isolated and summed the regularization terms, the convex combination form of FeDLaS-MbLS allows $\alpha^{(i)}$ to simultaneously control both the classification loss and the regularization weights.

Training fully reproduces the standardized protocol of ACLS (SGD optimizer, cosine learning rate scheduler, standard data augmentation). ResNet takes the feature vector prior to the classification head, and ViT takes the [CLS] token. $\beta$ and $\theta$ are selected via the validation set, and fine-grained tasks inherit the optimal configuration of Tiny-ImageNet.

Key Experimental Results¶

Main Results¶

Table 1 shows the calibration results (15 bins) of 6 architecture combinations on CIFAR-10/100 and Tiny-ImageNet. Rankings are based on the average across all scenarios:

Method	Average ECE Rank	Average AECE Rank	Average Top-1 Accuracy
CE	14.00	14.00	78.74
LS	7.00	9.00	79.69
MbLS	4.33	6.00	79.66
ACLS	5.50	4.83	79.78
DFL	7.50	6.33	78.89
AdaFocal	10.00	9.83	78.77
FeDLaS-LS	5.50	4.17	79.62
FeDLaS-MbLS	1.83	3.83	79.58

FeDLaS-MbLS achieves an average ECE rank of 1.83, leading all methods comprehensively and significantly outperforming the baseline MbLS (4.33). This indicates that bidirectional adaptive modulation effectively bridges the calibration gap of margin-based losses. While the accuracy of FeDLaS-LS (79.62) vs LS (79.69) might be easily misconstrued as a downgrade, this falls within standard variance, proving that adaptive modulation does not compromise classification accuracy.

Ablation Study¶

Table 2 presents the decomposed ECE (CIFAR-10, ResNet-50), demonstrating the bidirectional calibration effect of BCG:

Configuration	O-ECE↓	U-ECE↓	Description
CE	5.85	0.00	Severe over-confidence, almost no under-confidence
LS	1.16	2.45	Over-correction: O-ECE decreased but U-ECE surged
FeDLaS-LS	0.89	0.40	Bidirectional balance, LS's U-ECE drops from 2.45 to 0.40
MbLS	1.09	0.12	Moderate O-ECE, low U-ECE
FeDLaS-MbLS	1.11	0.02	Comparable O-ECE with MbLS, U-ECE approaches zero

Key Finding: CE exhibits strictly unidirectional over-confidence ($U-ECE \approx 0$); LS and MbLS show reverse over-correction (drastically suppressing O-ECE but increasing U-ECE). FeDLaS variants maintain low errors in both directions—FeDLaS-LS reduces the U-ECE of LS from 2.45 to 0.40 (an 83.7% reduction) while keeping O-ECE even lower.

Key Findings¶

BCG makes the most significant contribution: As inferred from the ablation study, removing BCG (degenerating to unidirectional smoothing) leads to a deficiency in under-confidence correction. FeDLaS-LS improves the U-ECE from 2.45 (for LS) to 0.40, a reduction entirely attributed to BCG's under-confidence detection capability.
More pronounced advantages in fine-grained tasks: On CUB-200 and FGVC-Aircraft, FeDLaS-LS achieves an average ECE rank of 1.5, far outperforming LS (3.0) and MbLS (4.5). Fine-grained tasks exhibit smaller inter-class differences and larger absolute variation in sample difficulties, making the advantages of per-sample adaptive smoothing particularly prominent.
Recovery of OOD detection performance: While LS and MbLS significantly decrease the OOD detection AUROC (compared to CE), FeDLaS variants drastically bridge this gap. For instance, in the CIFAR-100$\rightarrow$SVHN benchmark, where LS yields an AUROC of 72.92, FeDLaS-LS improves it to 84.33 (+11.41%).
Good stability of BCG: The proportion of over-confident/under-confident samples remains basically constant during training, fluctuating only slightly when the learning rate scheduler is triggered; the sample state flip rate continuously decreases to near-zero, demonstrating gating convergence.

Highlights & Insights¶

Systematic expansion of feature norm utility: Previously, feature norms were primarily utilized for OOD detection and hard example mining. This work is the first to introduce them as a per-sample confidence proxy for label smoothing during training. EMA normalization represents a simple yet effective engineering heuristic—enabling the NCI to reflect the relative confidence of samples across the entire training trajectory rather than merely within a local batch, while avoiding sigmoid saturation.
STE maintains differentiability of discrete gating: BCG outputs discrete binary signs of $\pm 1$. By leveraging the Straight-Through Estimator to use discrete values in the forward pass and pass continuous gradients in the backward pass, the framework keeps the gating explicit while keeping the full pipeline end-to-end trainable.
Mathematically symmetric bidirectional design: The relationship between NCI and $\alpha^{(i)}$ is symmetric under positive and negative modes (monotonically increasing vs. decreasing), and the boundedness of the sigmoid function prevents outlier samples from producing runaway smoothing coefficients. This "bounded + monotonic + symmetric" modulation framework is readily transferrable to other scenarios demanding bidirectional regularization adaptation.
Plug-and-play with zero inference overhead: ASM relies solely on the backbone features and logits with gradient decoupling. It does not alter the network architecture or introduce inference overhead—adding only a tiny MLP and EMA maintenance during training, while remaining completely silent during inference.

Limitations & Future Work¶

Theoretical assumptions limiting NCI: Using feature norms as a confidence proxy relies on the theoretical bounds of single-sided rectified activation functions such as ReLU/GELU (inequalities in Eq. 2). For activation functions like Swish/ELU or networks without clear backbone-head decomposition, the theoretical foundation may not hold. The authors explicitly acknowledge this limitation.
Hyperparameters $\beta$ and $\theta$ requiring fine-tuning: $\beta$ controls sigmoid sensitivity and $\theta$ controls EMA momentum, both of which require validation set selection. Although fine-grained tasks can directly reuse configurations from Tiny-ImageNet, they still incur additional tuning costs in real-world applications.
BCG relying solely on logits: Gating decisions determine confidence states only using logits without leverage of feature-level details. Introducing feature information into the gating module could potentially improve the classification accuracy of under- and over-confidence.
Unexplored semi-supervised/self-supervised schemes: The paper only briefly mentions the potential of the class-agnostic nature of NCI in unsupervised, self-supervised, semi-supervised, and noisy-label calibration contexts in the conclusion, lacking empirical validation. This constitutes the most direct future direction, as the real-world demand for model calibration under annotation-expensive scenarios is significant.

vs Label Smoothing (LS): LS applies a fixed $\alpha$ to all samples; FeDLaS-LS dynamically generates $\alpha^{(i)}$ via the ASM. On CIFAR-100, FeDLaS-LS reduces ECE from ~5.42 (LS) to ~4.29, and U-ECE from ~2.69 to ~0.59.
vs Margin-based LS (MbLS): MbLS relaxes the strict entropy constraint of LS, but the regularization strength remains globally fixed. The average ECE rank of FeDLaS-MbLS rises to 1.83 from 4.33 for MbLS.
vs ACLS: ACLS introduces sample-adaptive margins based on MbLS, but still acts as a unidirectional correction for over-confidence. FeDLaS-MbLS incorporates under-confidence gating, reducing the ECE from ~1.37 (ACLS) to ~1.22 on Tiny-ImageNet for ResNet-50.
vs AdaFocal / DFL: AdaFocal requires an additional validation set to select the temperature parameter, and DFL has a limited regularization spectrum. FeDLaS does not rely on a validation set, with BCG and NCI cooperating to cover the complete confidence span.

Rating¶

Novelty: ⭐⭐⭐⭐ Introducing feature norms from OOD detection into training-time per-sample label smoothing modulation is a novel perspective; the combined design of bidirectional gating + EMA normalization is clean and effective. Overall, it still falls under the adaptive coefficient extension within the LS/MbLS framework, so the architectural innovation remains an incremental improvement.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across 3 standard and 2 fine-grained classification datasets, comparing against 10+ baselines, with extra coverage on OOD detection, decomposed ECE analysis, and BCG stability visualization. The training protocol aligns fully with ACLS to ensure fairness.
Writing Quality: ⭐⭐⭐⭐ Clarifies motivations transparently, with a progressive hierarchy of theoretical proofs $\rightarrow$ design motivation $\rightarrow$ implementation details. The methodology section has slightly high equation density; some derivations could be moved to the appendix to enhance readability.
Value: ⭐⭐⭐⭐ Model calibration is a critical issue in practical deployments. Its plug-and-play setup coupled with zero inference overhead yields strong practical value. The general design philosophy of bidirectional calibration is generalizable to other regularization methodologies.