HEM: a margin-based loss for visual categorisation tasks¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://codeberg.org/mwspratling/HEMLoss
Area: Optimization & Theory
Keywords: Loss Function, Margin-based Loss, Robustness, Continual Learning, Semantic Segmentation
TL;DR¶
Addressing the issues of overconfidence and perpetual weight overwriting in cross-entropy (CE) loss as well as gradient vanishing in traditional multi-class margin (MM) losses, this paper introduces High Error Margin (HEM) loss, which adaptively filters low errors and incorporates class-frequency adjusted margins, substantially outperforming CE and specialized losses across unknown class rejection, adversarial robustness, continual learning, and semantic segmentation.
Background & Motivation¶
In deep neural network (DNN) classification tasks, because classification accuracy is a piecewise constant function with zero gradients almost everywhere, stochastic gradient descent optimization fundamentally relies on smooth surrogate loss functions. For years, cross-entropy (CE) loss has stood as the de-facto standard across computer vision. However, CE exhibits deeply problematic training dynamics: even when the classifier produces the correct prediction with high confidence, the loss value remains far from zero. The resulting non-zero gradients perpetually update the network weights, pushing the target class logit towards positive infinity and non-target logits towards negative infinity. This relentless polarization induces severe overconfidence on out-of-distribution (OOD) or unseen categories, exacerbates catastrophic forgetting in continual learning by continually rewriting previously acquired representations, and completely drowns out minority class features under severe class imbalance. Furthermore, CE loss does not strictly correlate with classification margins—in pathological cases, it can decrease even when prediction confidence degrades or misclassifications worsen.
In contrast, multi-class margin-based losses (such as the Crammer-Singer MM loss) inherently feature a stopping criterion: once the correct class logit exceeds competing logits by a predefined safety margin, the associated error strictly drops to zero, halting weight updates and providing natural safeguards against overconfidence and forgetting. Despite these theoretical merits, traditional MM loss yields standard test accuracy substantially inferior to CE. An in-depth analysis indicates that this degradation stems from the conventional way errors are aggregated: by simply summing or averaging errors across all logits and samples, the gradient magnitude fluctuates wildly across training. At early training epochs, widespread errors create excessively large gradients, whereas near convergence, the vast majority of zero errors drastically dilute the few remaining critical errors. This causes training to stall prematurely before resolving borderline confusions, an effect that becomes increasingly severe as the number of classes scales up.
To resolve the tension between the gradient decay of traditional margin losses and the pathological overconfidence of cross-entropy, this work re-engineers both the error aggregation dynamics and the margin assignment strategy. Core idea: propose High Error Margin (HEM) loss, which adaptively suppresses below-mean errors and averages strictly over non-zero error components to stabilize gradient magnitudes across training, coupled with class-frequency inversely scaled margins to establish a general-purpose margin loss that retains zero-loss robustness while maintaining strong discriminative accuracy.
Method¶
Overall Architecture¶
HEM is designed as an effective drop-in replacement for standard cross-entropy loss in deep neural network training. Given an input sample and its ground-truth target label, the network produces unnormalized classification logits. The pipeline first evaluates the margin violation error for each non-target class relative to the target class; it then dynamically computes the sample-level mean error, truncates sub-mean errors to zero, and calculates the average strictly over non-zero high-error terms; finally, it scales individual class margins inversely with training sample frequency. This optimization flow guarantees that hard negative classes dominate the gradients throughout learning while completely eliminating the gradient dilution caused by easy, zero-error classes.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Network Forward Logits & Ground-Truth Label"] --> B["Stage 1: Zero-Loss Truncation & Safety Margin<br/>Calculate margin violation error for each non-target class"]
B --> C["Stage 2: High Error Adaptive Filtering & Mean Above-Zero Aggregation<br/>Threshold sub-mean errors and average only non-zero errors"]
C --> D["Stage 3: Inverse-Frequency Dynamic Margin Adjustment<br/>Scale margin per class inversely proportional to sample count"]
D --> E["Backpropagate Gradients & Prune Zero-Loss Computational Graph"]
Key Designs¶
1. Zero-Loss Truncation & Safety Margin: Eliminating Overconfidence and Unwarranted Weight Overwriting
To resolve the chronic pathology where CE loss persistently generates non-zero gradients even after samples are correctly classified, HEM introduces an explicit hard margin threshold. Given the network output logit vector \(y\) and the ground-truth target class index \(l\), the margin error \(e_i\) for any non-target class \(i \neq l\) is formulated as: $\(e_i = \max(0, y_i - y_l + \mu_i)\)$ with \(e_l = 0\) for \(i = l\), where \(\mu_i\) denotes a non-negative margin hyper-parameter for class \(i\). This formulation ensures that whenever the target logit \(y_l\) surpasses a competitor \(y_i\) by at least \(\mu_i\), the error drops strictly to zero. This zero-loss cutoff prevents logits from blowing up to extreme values, thereby preserving meaningful confidence scores for unknown class rejection. Furthermore, completely fitted training samples cease to perturb existing network parameters, which naturally enables automatic gradient graph pruning in autograd frameworks, lowering training time and mitigating catastrophic forgetting in continual learning scenarios.
2. High Error Adaptive Filtering & Mean Above-Zero Aggregation: Stabilizing Training Gradient Dynamics
Addressing the severe gradient vanishing problem in standard multi-class margin (MM) loss—where averaging over all candidate classes dilutes the gradient once most errors drop to zero—HEM introduces a two-step adaptive error aggregation scheme. First, the mean error across all \(n\) classes for a given sample is computed and detached from the autograd graph; any error falling below this mean is zeroed out (the thresholding step, thres), concentrating optimization on the most severe violations during early epochs. Second, the loss computes the arithmetic mean exclusively over the remaining non-zero error elements (the mean-above-zero step, maz):
$\(\mathcal{L}_{\text{HEM}} = \frac{\sum_{i=1}^{n} \mathbb{I}\left[e_i \ge \frac{1}{n} \sum_{j=1}^{n} e_j\right] \cdot e_i}{\sum_{i=1}^{n} \mathbb{I}\left[e_i \ge \frac{1}{n} \sum_{j=1}^{n} e_j\right]}\)$
where \(\mathbb{I}[\cdot]\) is the indicator function. In early training iterations where errors are widespread, thresholding forces the network to target the hardest negatives; later in training, when most errors have reached zero, the denominator counts only the remaining active high errors, preventing the loss magnitude from decaying to an ineffective near-zero scale. Across a mini-batch, HEM similarly averages only over samples exhibiting non-zero losses, ensuring consistent and healthy gradient scales throughout the entire training trajectory.
3. Inverse-Frequency Dynamic Margin Adjustment: Parameter-Free Long-Tailed Rebalancing
To prevent majority classes from overwhelming scarce minority class features under severe class imbalance, HEM incorporates class-dependent margin adjustments. Under balanced training conditions, all classes share an identical margin \(\mu_i = \sqrt{M / \sum_{k=1}^n s_k}\) (denoted as HEM-); under imbalanced long-tailed distributions, the margin \(\mu_i\) for class \(i\) is scaled inversely with its training sample frequency \(s_i\): $\(\mu_i = \sqrt{\frac{M}{n \cdot s_i}}\)$ where \(n\) is the total number of classes, \(s_i\) is the number of training samples in class \(i\), and \(M\) is a global scaling hyper-parameter (fixed to 2000 across all experiments, with demonstrated low sensitivity). Because tail classes possess small \(s_i\), they are automatically assigned wider safety margins, obliging the classifier to enforce a larger separation distance around rare categories without requiring artificial architectural modifications or ad-hoc data resampling.
Loss & Training¶
HEM does not introduce fragile hyper-parameter schedules; setting the global hyper-parameter \(M = 2000\) yields robust results across vision benchmarks. Because correctly classified samples produce an exact loss of zero, autograd can prune the corresponding backward computation graph during later training epochs. On TinyImageNet with ResNet-18 (200 epochs), this computational shortcut reduces total training time by approximately 10%. In dense prediction tasks such as Cityscapes semantic segmentation with ResNet-34, HEM similarly delivers a ~4% reduction in training wall-clock time compared to cross-entropy.
Key Experimental Results¶
Main Results¶
The authors evaluated HEM across standard classification benchmarks (MNIST, CIFAR-10, CIFAR-100, TinyImageNet, ImageNet-1k), long-tailed classification, continual learning (PermutedMNIST, SplitMNIST, SplitCIFAR-10/100), and semantic segmentation (CamVid, Cityscapes, SBD, ADE20k), utilizing 19 distinct deep neural network architectures ranging from LeNet to ViT-B/16.
The table below summarizes the average relative performance changes compared to standard cross-entropy (CE) loss across five key benchmark dimensions:
| Evaluation Task / Metric | Cross-Entropy (CE) | LogitNorm (LN) | Logit-adjusted (LA) | DICE | Multi-Class Margin (MM) | HEM (Ours) |
|---|---|---|---|---|---|---|
| Standard Data: Clean Test Accuracy (%) | Baseline (0.00) | -1.25 | 0.00 | -15.42 | -10.85 | -1.19 |
| Standard Data: Common Corruptions Accuracy (%) | Baseline (0.00) | -1.02 | 0.00 | -14.28 | -9.60 | -1.07 |
| Standard Data: Unknown Class Rejection AUROC (%) | Baseline (0.00) | +4.10 | 0.00 | -13.20 | -8.45 | +4.52 |
| Standard Data: AutoAttack Rejection DAR (%) | Baseline (0.00) | +5.31 | 0.00 | -8.10 | -5.70 | +17.17 |
| Long-Tailed Data: Unknown Class Rejection AUROC (%) | Baseline (0.00) | +2.15 | -0.12 | -18.60 | -7.90 | +9.33 |
| Long-Tailed Data: AutoAttack Rejection DAR (%) | Baseline (0.00) | +1.80 | -0.45 | -11.20 | -6.15 | +2.77 |
| Continual Learning: Final Retained Accuracy (%) | Baseline (0.00) | -12.40 | 0.00 | -28.30 | -8.10 | +1.27 |
| Semantic Segmentation: Test mIoU Relative Gain (%) | Baseline (0.00) | -18.50 | -22.10 | -0.30 | -14.20 | +5.31 |
Note: Except for the CE baseline column, values represent average percentage point differences relative to CE. Standard benchmark metrics are averaged across 71 experiments; continual learning covers 80 experiments; semantic segmentation covers 76 experiments.
Ablation Study¶
On CIFAR-10 and CIFAR-100 with ResNet-18 (using margin \(\mu = 0.2\)), an ablation study systematically verifies the individual and joint impacts of the two error aggregation modifications transitioning from MM to HEM (mean and standard deviation across five trials):
| Loss Configuration | CIFAR-10 Clean Accuracy (%) | CIFAR-100 Clean Accuracy (%) | Mechanism Description |
|---|---|---|---|
| MM (Standard Margin Baseline) | 93.79 ± 0.11 | 70.13 ± 0.19 | Standard arithmetic mean over all logit errors |
| + maz | 93.81 ± 0.23 | 74.94 ± 0.35 | Average only non-zero errors, preventing dilution by zeros |
| + thres | 93.78 ± 0.22 | 73.13 ± 0.26 | Threshold errors below sample mean to zero, targeting hard negatives |
| + maz + thres (Full HEM) | 93.84 ± 0.19 | 74.95 ± 0.46 | Combines hard negative focus with stable gradient magnitude (+4.82% over MM) |
Key Findings¶
- Class cardinality dictates the benefit of error aggregation: On 10-class CIFAR-10, MM and HEM perform comparably; on 100-class CIFAR-100, incorporating the mean-above-zero (
maz) mechanism alone boosts accuracy from 70.13% to 74.94% (+4.81%). This empirically proves that as the number of classes grows, gradient dilution from easy zero-error classes in traditional MM loss is the primary failure mode. - Substantial gains in adversarial and OOD robustness: Without requiring adversarial training or custom OOD detection heads, HEM outperforms CE by 17.17% in AutoAttack DAR and achieves a 4.52% boost in unknown class rejection AUROC, even exceeding LogitNorm which was specifically engineered for OOD detection.
- Catastrophic cross-domain failure of specialized losses: LogitNorm degrades sharply when applied to continual learning and semantic segmentation; Logit-adjusted loss underperforms on OOD and segmentation; DICE (the standard segmentation loss) falls behind HEM on segmentation tasks while collapsing on general classification. In contrast, HEM achieved top-1 ranking in 5 out of 10 evaluation benchmarks, emerging as the only truly versatile classification loss.
Highlights & Insights¶
- Revitalizing margin loss for modern deep learning: Conventional wisdom held that margin losses are ill-suited for deep neural networks due to poor clean accuracy. This work demystifies the root cause—showing that it is not the margin principle itself, but the dilution of gradients by zero-error classes during aggregation—and offers an elegant, two-line code solution.
- Cost-free leap in robustness: By simply enforcing bounded margins rather than unbounded cross-entropy logit growth, models gain over 17% higher resilience against AutoAttack without any adversarial data augmentation, highlighting that logit overconfidence is a central driver of adversarial vulnerability.
- Pruning-enabled training acceleration: Leveraging the mathematical property of exact zero loss for well-separated samples, autograd engines can prune backpropagation computation graphs during later training stages, delivering a 4% to 10% speedup for free.
Limitations & Future Work¶
- Slight compromise on clean classification accuracy: On balanced datasets, HEM exhibits a minor 1.19% drop in clean test accuracy compared to CE, presenting a minor trade-off for applications that strictly prioritize raw clean accuracy over safety and robustness.
- Suboptimal margin heuristic for dense pixel prediction: In semantic segmentation, the uniform-margin variant (HEM-) outperforms the class-frequency adjusted variant (HEM), indicating that the inverse-frequency sample counting heuristic designed for image classification does not directly transfer to dense pixel-level distributions.
- Extension to multimodal and language models: The current evaluation is restricted to computer vision (image classification and semantic segmentation). Future investigations should assess whether HEM's zero-loss margin dynamics translate effectively to vision-language models (VLMs) and token-level language generation.
Related Work & Insights¶
- vs Cross-Entropy (CE): CE continuously stretches logit differences and endlessly updates weights even on confident predictions; HEM enforces hard margin truncation, ceasing updates once confidence is adequate, which prevents catastrophic forgetting and overconfidence.
- vs Multi-Class Margin (MM / Crammer-Singer): MM averages over all classes, causing gradient collapse when class counts are large; HEM combines adaptive mean thresholding (
thres) and mean-above-zero (maz) aggregation to maintain a robust gradient scale throughout training. - vs LogitNorm & DICE: Specialized losses excel only within narrow niches and fail catastrophically outside their intended scope; HEM provides an all-around competitive surrogate loss that tops DICE on semantic segmentation while preserving superior OOD robustness.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Pinpoints the exact gradient dilution pathology of classic margin losses in deep learning and introduces a clean adaptive aggregation fix.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely rigorous evaluation across 19 network architectures, benchmark datasets from MNIST to ImageNet, and 10 comprehensive task dimensions.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative detailing the mathematical and intuitive pitfalls of CE and MM losses.
- Value: ⭐⭐⭐⭐⭐ Plug-and-play, zero architectural overhead, offering immediate practical benefits for safe, robust, and continual visual learning.