title: >- [Paper Note] Combining Discrepancy-Confusion Uncertainty and Calibration Diversity for Active Fine-Grained Image Classification description: >- [ECCV 2026][others][Active Learning] To address the failure of traditional active learning in fine-grained classification caused by severe semantic overlap, DECERN combines discrepancy-confusion uncertainty via local feature fusion with calibration diversity balancing local representativeness and global diversity. tags: - ECCV 2026 - others - Active Learning - Fine-Grained Classification - Feature Fusion date: 2026-09-19 content_hash: 5b3e04a780269b90
Combining Discrepancy-Confusion Uncertainty and Calibration Diversity for Active Fine-Grained Image Classification¶
Conference: ECCV 2026
Paper: CVF / ECCV Official Poster
Area: Others (Active Learning / Fine-Grained Classification)
Keywords: active learning, fine-grained image classification, local feature fusion, discrepancy-confusion uncertainty, calibration diversity
TL;DR¶
To tackle the failure of conventional active learning caused by severe inter-class visual overlap in fine-grained image classification, DECERN decouples structural stability and category directionality via local feature fusion, coupled with uncertainty-weighted clustering and calibration diversity to achieve state-of-the-art sample selection under tight annotation budgets.
Background & Motivation¶
The success of deep neural networks relies heavily on abundant annotated datasets. However, in specialized domains such as archaeological dating, biodiversity monitoring, and medical diagnosis, obtaining high-quality fine-grained labels requires scarce domain expertise and incurs prohibitive annotation expenses. Active learning (AL) mitigates this burden by iteratively identifying and querying the most informative instances from an unlabeled pool under a constrained budget. Existing paradigms are predominantly divided into uncertainty-based, diversity-based, and hybrid selection schemes. However, all these methods fundamentally depend on the premise that deep representations can reliably reflect data distribution boundaries.
In fine-grained image classification, where instances belong to subtle subcategories within the same parent category, images exhibit high visual resemblance at early layers and share widespread semantic representations in deep feature spaces. This profound feature overlap impairs traditional AL criteria: uncertainty indicators (e.g., entropy, confidence, margin) fail to trace clean decision boundaries and are easily corrupted by ambiguous pseudo-labels, while diversity techniques (such as CoreSet or K-Means) frequently conflate geometrically proximate yet semantically distinct instances or select intra-class redundancies. Although recent feature-fusion approaches like ALFA-Mix attempt to probe local neighborhoods via convex combinations, they measure sample informativeness strictly by output label inconsistency, overlooking subtle subcategory patterns that span multiple fine-grained classes.
The core tension lies in the conflict between high representation similarity and subtle semantic distinctions. The paper addresses this by actively perturbing features to inspect empirical stability: by injecting local feature perturbations guided by labeled class anchors, the model can observe how severely an instance shifts when structural consistency is disrupted and how resilient its category directionality remains against confusing classes. Core idea: quantify both structural stability (discrepancy uncertainty) and category directionality (confusion uncertainty) through gradient-pooled local feature fusion, and combine uncertainty-weighted clustering with anchor-calibrated diversity to synergistically sample local prototypes and global exploratory instances.
Method¶
Overall Architecture¶
DECERN operates through a two-stage iterative pipeline: (1) discrepancy-confusion uncertainty assessment via local feature fusion to construct a high-uncertainty candidate pool via dynamic thresholding, and (2) uncertainty-weighted clustering followed by calibration diversity sampling to select the final batch of query instances.
At the beginning of each cycle, class anchor prototypes are computed from the current labeled pool. For each unlabeled sample, a pooled-gradient saliency mask isolates discriminative feature dimensions. Local feature fusion is then executed between the sample and all class anchors using confidence-adaptive interpolation weights. The method evaluates structural disruption via cross-entropy discrepancy and directional confusion via entropy, aggregating them into an instance-level uncertainty score. Samples exceeding a skewness-driven dynamic threshold form the candidate subset. Finally, uncertainty-weighted K-Means clustering partitions the candidates, and instances optimizing both cluster-centroid proximity and labeled-anchor divergence are selected for annotation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Unlabeled Samples $z^u$ & Labeled Anchors $z^a$"] --> B["1. Saliency-Guided Local Feature Fusion<br/>Pooled-gradient binary mask identifies key dimensions with confidence-based mixing"]
B --> C["2. Discrepancy-Confusion Uncertainty Scoring<br/>Cross-entropy measures structural shift + Entropy measures directional confusion"]
C --> D["Skewness-Driven Dynamic Thresholding<br/>Filter high-uncertainty candidate pool"]
D --> E["3. Uncertainty-Weighted Clustering & Calibration Diversity<br/>Cluster centroid proximity + Anchor divergence trade-off"]
E --> F["Oracle Annotation & Model Retraining"]
Key Designs¶
1. Saliency-Guided Local Feature Fusion: Decoupling Stability and Directionality
Fine-grained distinctions typically reside in minute local regions (e.g., bill shapes or feather patterns). Global feature interpolation risks obliterating these subtle cues and introducing irrelevant background interference. DECERN addresses this by computing the backpropagation gradient of the predicted pseudo-label, adaptively pooling it into 100 bins, and upsampling the top-\(\eta\) bins back to the original feature dimension to produce a binary saliency mask \(M\). The local feature fusion with the \(j\)-th class anchor \(z_j^a\) is defined as: $$ \phi(z^u, z^a_j; \alpha_j, M) = (1 - M) \odot z^u + M \odot \big((1 - \alpha_j) z^u + \alpha_j z^a_j\big) $$ where the mixing weight \(\alpha_j\) is set adaptively to the model's predicted confidence \(p_j^u\). For the predicted category, a large \(\alpha_j\) imposes high semantic-consistent perturbation to test whether category directionality withstands corruption; for other classes, a small \(\alpha_j\) introduces weak semantic-inconsistent noise to inspect whether the intrinsic semantic structure collapses under slight local pollution.
2. Discrepancy-Confusion Dual Uncertainty with Dynamic Thresholding
Relying on a single metric cannot differentiate whether a sample is unstable due to boundary ambiguity or genuinely lacking discriminative cues. DECERN combines structural stability (discrepancy) and category directionality (confusion). For each class \(j\), using the cosine similarity coefficient \(\beta_j = 1 - \frac{1 + \cos(z^u, z_j^a)}{2}\), the category-level score is computed between the post-fusion prediction \(p_j^m = \mathcal{F}_c(\phi(z^u, z_j^a; \alpha_j, M))\) and reference distributions \(p_j^* \in \{p^u, p_j^b, p_j^w\}\): $$ \mathcal{S}_{dc}(p_j^, p_j^m; \beta_j) = \mathcal{H}(p_j^m)^{1 - \beta_j} + \mathcal{CE}(p_j^, p_j^m)^{\beta_j} $$ Averaging across all classes and theoretical fusion references yields the instance uncertainty score \(\mathcal{S}\). To avoid arbitrary manual proportions across cycles, DECERN dynamically scales candidate selection using score skewness: $$ \zeta = \bar{\mathcal{S}} + \lambda \sigma \frac{\mathbb{E}[(S - \bar{\mathcal{S}})^3]}{\sigma^3} $$ Because skewness naturally reflects model maturity and uncertainty distribution asymmetry, filtering with \(\mathcal{S} \ge \zeta\) dynamically balances candidate quality as training progresses.
3. Uncertainty-Weighted Clustering and Calibration Diversity Sampling
Selecting strictly by uncertainty leads to redundant queries clustered around the same ambiguous boundary, while pure geometric diversity wastes budget on non-informative samples. DECERN first performs K-Means on candidate representations weighted by their uncertainty scores \(\mathcal{S}\), naturally pulling cluster centroids \(z^{\mathcal{C}_k}\) toward informative regions. Within each cluster \(\mathcal{C}_k\), calibration diversity balances local representativeness against global exploratory diversity: $$ x^{s_k} = \arg\max_{x_i \in \mathcal{C}k} \left[ - \xi |z_i^u - z^{\mathcal{C}_k}|_2 + (1 - \xi) \min (1 - \cos(z_i^u, z_j^a)) \right] $$ The first term selects prototypical exemplars near cluster centers to capture intra-class variation cores without querying redundant duplicates. The second term maximizes divergence from all established class anchors, driving the model to explore unfamiliar boundary frontiers and tail distributions.
Loss & Training¶
In each AL iteration, the selected \(B = K \cdot N_c\) samples are labeled by an oracle and appended to \(\mathcal{D}^\ell\). The backbone model (ResNet50 or ViT-Small) is retrained using standard cross-entropy loss with the Adam optimizer (initial learning rate 0.001, cosine learning rate decay, batch size 128) over 8 active learning cycles. Default hyperparameters are set to mask ratio \(\eta = 0.1\), skewness factor \(\lambda = 0.5\), and diversity balance factor \(\xi = 0.8\).
Key Experimental Results¶
Main Results¶
DECERN was evaluated on 7 benchmark fine-grained image classification datasets (Caltech101, BronzeDing, CUB-200-2011, Flowers102, Food101, OxfordIIITPet, StanfordDogs) across 39 distinct settings. The table below summarizes Top-1 accuracy (%) on Caltech101 and BronzeDing using a ResNet50 backbone under different annotation budgets \(B\):
| Dataset & Budget Setting | Random | K-Means | Margin | CoreSet | BADGE | ALFA-Mix | Ours (DECERN) | Gain over runner-up |
|---|---|---|---|---|---|---|---|---|
| Caltech101 (\(B = 1 \cdot N_c\)) | 74.14 | 79.52 | 80.41 | 79.80 | 81.35 | 81.60 | 82.95 ± 0.37 | +1.35% |
| Caltech101 (\(B = 2 \cdot N_c\)) | 82.93 | 86.85 | 87.60 | 87.10 | 88.02 | 88.15 | 88.49 ± 0.37 | +0.34% |
| BronzeDing (\(B = 1 \cdot N_c\)) | 34.20 | 36.80 | 38.50 | 37.10 | 39.80 | 40.20 | 42.15 ± 0.45 | +1.95% |
| BronzeDing (\(B = 2 \cdot N_c\)) | 43.50 | 46.20 | 47.90 | 46.50 | 49.30 | 50.10 | 52.30 ± 0.40 | +2.20% |
(Note: Baseline values reflect convergence metrics in Fig. 2 across 5 random seeds)
Ablation Study¶
Ablations on Caltech101 (ResNet50 backbone) systematically validate the uncertainty metrics and diversity components (Table 1 of original paper):
| Configuration / Variant | Uncertainty | Diversity | \(B = 1 \cdot N_c\) Acc (%) | \(B = 2 \cdot N_c\) Acc (%) | Note |
|---|---|---|---|---|---|
| Full model (DECERN) | \(\mathcal{S}_d + \mathcal{S}_c\) | Weighted + Calibrated | 82.95 ± 0.37 | 88.49 ± 0.37 | full model |
| Diversity only | ✗ | ✓ | 80.73 ± 0.69 | 87.08 ± 0.22 | -2.22% w/o uncertainty candidate filtering |
| Random baseline | ✗ | ✗ | 74.14 ± 0.93 | 82.93 ± 0.47 | baseline without AL guidance |
| w/o \(\mathcal{S}_c\) (Confusion) | Only \(\mathcal{S}_d\) | ✓ | 82.10 ± 0.91 | 88.25 ± 0.42 | removes directional confusion penalty |
| w/o \(\mathcal{S}_d\) (Discrepancy) | Only \(\mathcal{S}_c\) | ✓ | 82.43 ± 0.44 | 88.16 ± 0.42 | removes structural shift penalty |
| w/o Weighted Clustering | \(\mathcal{S}_d + \mathcal{S}_c\) | Unweighted K-Means | 82.39 ± 0.42 | 88.18 ± 0.59 | centroids not biased toward high uncertainty |
| w/o Clustering | \(\mathcal{S}_d + \mathcal{S}_c\) | Direct ranking | 81.27 ± 0.36 | 88.43 ± 0.34 | leads to severe sample redundancy |
| w/o Calibration | \(\mathcal{S}_d + \mathcal{S}_c\) | Cluster center only | 82.94 ± 0.88 | 88.04 ± 0.42 | lacks exploration of global uncharted spaces |
Key Findings¶
- Dual Uncertainty Synergy: Removing both uncertainty terms drops accuracy significantly to 80.73% at \(B = 1 \cdot N_c\). Discarding either discrepancy or confusion individually incurs a 0.5%–0.85% decline, showing that both structural fragility and category ambiguity are vital for fine-grained ranking.
- Sampling Class Balance: As shown in Fig. 3a, standard techniques (CoreSet, NoiseStability, K-Means) suffer severe class sampling imbalance due to local density traps, whereas DECERN maintains an entropy profile that closely mirrors balanced queries, preventing over-querying on dominant classes.
- Superior Runtime Efficiency: In Fig. 3b, DECERN requires under 50 seconds per cycle on Caltech101, outperforming BADGE (~500s), UHerding (>300s), and NoiseStability (>300s) by nearly an order of magnitude while providing superior classification performance.
Highlights & Insights¶
- Saliency-Masked Local Mixing: Rather than applying naive uniform linear blending across all feature dimensions, DECERN uses a pooled backpropagation gradient mask to preserve core structural dimensions and perturb only the most salient sub-features.
- Skewness-Modulated Dynamic Threshold: Adapting candidate selection cutoffs via the empirical skewness of the uncertainty distribution accommodates the evolving difficulty of instances as the model matures over training rounds.
- Synergistic Anchor-Calibrated Diversity: Integrating intra-cluster prototypical compactness with inter-class anchor divergence neatly addresses the classic dilemma between local exploitation and global exploration.
Limitations & Future Work¶
- Scope Constrained to Image Classification: The current formulation is tightly coupled with global feature encoders and discrete category anchors; its applicability to dense visual tasks (e.g., semantic segmentation, object detection) remains unverified.
- Dependence on Initial Feature Representations: The effectiveness of the gradient-pooled mask and anchor centroids relies on a reasonable initial feature space. Under extreme distribution shifts or poorly initialized representations, anchor estimation could become noisy.
- Future Directions: Exploring parameter-efficient fine-tuning (PEFT) integration with large vision-language foundation models and designing adaptive anchor recalibration for extreme long-tailed distributions.
Related Work & Insights¶
- vs ALFA-Mix (CVPR 2022): ALFA-Mix also explores feature interpolation to identify decision boundaries, but relies exclusively on discrete pseudo-label changes, making it prone to label-flipping noise; DECERN evaluates continuous structural shift and directional confusion under local saliency masks.
- vs BADGE (ICLR 2020) & CLUE (ICCV 2021): BADGE computes high-dimensional gradient embeddings that entail steep computational overhead (>10x slower), while CLUE lacks fine-grained inter-class confusion modeling; DECERN achieves superior accuracy with near-optimal wall-clock efficiency.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Clever combination of local feature perturbation, discrepancy-confusion decoupling, and anchor-calibrated diversity]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across 7 fine-grained datasets, 39 experimental settings, and 15 competitive baselines]
- Writing Quality: ⭐⭐⭐⭐ [Clear problem formulation, rigorous mathematical definitions, and comprehensive ablation analysis]
- Value: ⭐⭐⭐⭐ [Provides an effective, highly practical, and computationally efficient active learning framework for real-world fine-grained annotation]