Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

Conference: ECCV 2026
arXiv: 2512.10244
Code: https://github.com/tian1327/SWIFT
Area: Multimodal VLMs
Keywords: Semi-supervised few-shot learning, VLM fine-tuning, Temperature, Auto-annotation, Open data

TL;DR

This paper investigates semi-supervised few-shot learning (SSFSL) from a realistic perspective of auto-annotation. It reveals that directly applying existing SSL methods to fine-tune VLMs fails completely due to "flat" softmax probability distributions generated by VLMs (leading to zero utilization of unlabeled data and insufficient training supervision). The authors propose using two simple temperature parameters (a confidence temperature \(T_{conf}\) and a loss temperature \(T_{loss}\)) to sharpen the softmax output and boost training signals. Combined with a three-stage step-by-step training scheme and external open data retrieval, the proposed method outperforms existing baselines by approximately 5 percentage points across 5 fine-grained benchmarks, even rivaling fully supervised learning.

Background & Motivation

Semi-supervised few-shot learning (SSFSL) naturally matches the practical demand for auto-annotation: given a small number of labeled class samples and a large volume of unlabeled in-distribution data, the model is trained to label the unlabeled data. This setting is particularly crucial in fine-grained classification scenarios (e.g., bird species, aircraft models, satellite imagery land cover) where obtaining large-scale manual annotations is extremely time-consuming and expensive. In recent years, the few-shot learning (FSL) field has fully embraced two major resources: pre-trained vision-language models (VLMs, such as CLIP/OpenCLIP) and open data retrieval (retrieving task-relevant samples from the public pre-training sets of VLMs, like LAION-400M), achieving significant progress. However, existing semi-supervised learning (SSL) methods largely ignore these resources—they either still train from scratch using ImageNet pre-trained backbones or freeze the VLM and perform prompt learning, avoiding full-model fine-tuning. This creates a curious situation where an FSL method that does not use unlabeled data at all (FS-FT) performs better than an SSL method utilizing a large amount of unlabeled data.

Why do existing SSL methods fail on VLMs? Through in-depth empirical analysis, the authors reveal the root cause: VLMs (especially those pre-trained with contrastive learning) generate "flat" softmax probability distributions. Regardless of the input, the model's predicted probabilities across all classes are close to a uniform distribution, with the maximum value often being only 0.2-0.4. This leads to two cascading problems: first, pseudo-labeling-based SSL methods like FixMatch rely on high confidence thresholds (typically 0.8) to filter reliable pseudo-labels, meaning that under flat distributions, all unlabeled samples fall below the threshold, reducing unlabeled data utilization to zero; second, even if the threshold is lowered, the flat distribution provides extremely weak training signals, failing to effectively drive model convergence. Although prior works noticed this phenomenon, they failed to identify its fundamental cause.

The Core Idea of this paper is to use temperature parameters to sharpen the softmax outputs of VLMs. Specifically, a tiny confidence temperature \(T_{conf}=0.01\) is used to "compress" the flat distribution into a sharp, peaky distribution, allowing pseudo-labels to regain high confidence for utilization. Simultaneously, a learnable loss temperature \(T_{loss}\) (a parameter initialized to 0.07) is introduced to boost the strength of the cross-entropy loss training signal. These two temperature parameters work synergistically to make legacy SSL methods like FixMatch highly effective again on VLMs, bringing massive improvements of 14-20 percentage points in 16-shot settings. Furthermore, the authors construct a three-stage training method named SWIFT (Stage-Wise Finetuning with Temperatures), which organically integrates classifier initialization, semi-supervised fine-tuning, and few-shot refinement with the temperature strategies and open data retrieval, ultimately achieving state-of-the-art (SOTA) performance that surpasses existing FSL and SSL methods by about 5 percentage points.

Method

Overall Architecture

SWIFT adopts a three-stage step-by-step training pipeline to progressively fine-tune the vision encoder and the classifier (which is initialized with text embeddings of class names). The first stage trains only the classifier using the labeled few-shot data (with the vision encoder frozen) to provide a better initialization for the subsequent semi-supervised fine-tuning. The second stage jointly fine-tunes both the vision encoder and the classifier using labeled data, unlabeled data, and retrieved open data within a temperature-assisted FixMatch framework. The third stage discards the unlabeled and noisy retrieved data, utilizing only the labeled data for a few epochs of refinement to eliminate the domain shift and class imbalance introduced by the open data.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Small labeled data L<br/>Large unlabeled data U<br/>Open retrieved data R"] --> B["Stage 1: Classifier Initialization"]
    B --> C["Initialize classifier with text embeddings<br/>Train classifier on labeled data L<br/>using Tloss"]
    C --> D["Stage 2: Semi-Supervised Fine-Tuning"]
    D --> E["Temperature-assisted FixMatch<br/>Tconf sharpens pseudo-label confidence<br/>Tloss boosts CE training signal<br/>Joint training with L+U+R"]
    E --> F["Stage 3: Few-Shot Refinement"]
    F --> G["Fine-tune vision encoder + classifier<br/>using only labeled data L<br/>to eliminate domain shift of R"]
    G --> H["Output: Labeled unlabeled data"]

Key Designs

1. Temperature Remedy Mechanism: Synergy Between Confidence and Loss Temperatures

The core obstacle to fine-tuning VLMs for SSFSL lies in the "flat" softmax distribution caused by contrastive pre-training. Directly observing this distribution reveals that for an input image, OpenCLIP yields predicted probabilities for 200 classes that almost all fall between 0.015 and 0.025, with a maximum value of only about 0.2. Consequently, all unlabeled samples fall below the default 0.8 threshold of FixMatch, rendering their utilization exactly zero. An intuitive alternative is lowering the threshold, but this is highly impractical to tune in realistic settings without a validation set, and low thresholds introduce massive amounts of noisy pseudo-labels.

The proposed solution is exceptionally simple: introducing two independent temperature parameters. The confidence temperature \(T_{conf}\) is applied directly to scale the logits of the weakly augmented unlabeled images: \(\mathbf{s}^w = \text{softmax}(\mathbf{W}^T\mathbf{V}(\mathbf{I}^w) / T_{conf})\). When \(T_{conf} = 0.01\), the softmax probabilities originally sitting between 0.015 and 0.025 are sharpened into a peaky distribution near 0 or 1, instantly pushing the maximum confidence above 0.8. This allows a vast amount of unlabeled data to once again satisfy the high-threshold filtering of FixMatch. However, sharpening confidence alone is insufficient; training supervision remains weak under flat distributions, meaning that even if samples pass the threshold, the calculated cross-entropy (CE) loss is too small to drive effective VLM fine-tuning. Thus, a second temperature \(T_{loss}\) (a learnable parameter initialized to 0.07) is introduced to scale the logits during CE loss calculation: \(\mathcal{L}_u = \frac{1}{|U|} \sum \mathbb{1}[max_c(s^c) \geq \sigma] \cdot \ell(\mathbf{W}^T\mathbf{V}(\mathbf{I}^s)/T_{loss}, \hat{y})\). The impact of \(T_{loss}\) is clearly visible in the loss curves: using the default temperature of 1.0 results in almost no decline in training loss, whereas \(T_{loss}=0.07\) leads to a rapid drop in loss, with test accuracy jumping from around 40% to approximately 55%. The paper also finds that making \(T_{loss}\) a learnable parameter (dynamically adjusted starting from 0.07) yields better results than using a fixed value, likely because different training stages require different levels of sharpening.

This design is elegant because instead of inventing new losses or complex architectures, it precisely pinpoints the "temperature mismatch" between VLM pre-training (contrastive learning) and SSL fine-tuning (cross-entropy): VLM pre-training clamps the softmax temperature at 0.01 (necessary for stable multimodal contrastive training), but downstream SSL methods interpret its output logits using a default temperature of 1.0, creating a gulf between the actual and expected probability distributions. Introducing a 0.01 confidence temperature essentially tells the downstream task, "please read my logits using the pre-training temperature."

2. Open Data Retrieval and Three-Stage Step-by-Step Training

Merely making FixMatch effective again is not enough—if only labeled and unlabeled data (\(L\) and \(U\)) are utilized, FixMatch w/ Temp. only reaches 71.2% in the 16-shot setting, which is marginally better than the FSL method FS-FT (69.1%). To push performance further, it is necessary to leverage task-relevant open data \(R\) retrieved from the VLM's public pre-training sets (such as LAION-400M). However, \(R\) inherently carries three challenges: severe class imbalance (some classes have thousands of retrieved images, while others have only dozens), label noise (retrieval is based on text matching where image-text pairs do not strictly correspond to class concepts), and domain shift relative to target task data (retrieved images differ dramatically from fine-grained dataset styles).

Directly co-training these noisy data \(R\) alongside \(L\) and \(U\) yields poor results. The authors design a three-stage pipeline to step-by-step digest these noisy data. Stage 1 performs linear probing on the classifier using only \(L\) (with \(T_{loss}\) boosting the signal) to adapt the text-embedding-initialized classifier to the target data distribution first. Stage 2 is the core: under the temperature-assisted FixMatch framework, \(L\), \(U\), and \(R\) are jointly used for training. Here, a key practical detail is that labeled samples \(L\) and retrieved samples \(R\) are mixed in a single batch to compute the supervised loss \(\mathcal{L}_l\) (calculating standard CE on the noisy labels of \(R\)), while \(U\) calculates the consistency loss \(\mathcal{L}_u\) via FixMatch's pseudo-labeling mechanism. \(R\) is not mixed into the unlabeled set, because its domain shift would disrupt the consistency regularization of FixMatch. Stage 3 completely discards \(U\) and \(R\), performing a few epochs of few-shot fine-tuning using only \(L\) to eliminate the domain shift and noise impact brought by \(R\) during Stage 2. This concept of "boosting with noisy data, then refining to prune noise residuals" aligns with previous work SWAT, but since SWAT lacks \(U\) and the temperature designs, SWIFT achieves significantly larger gains across all stages.

3. Practical Evaluation Protocol Without a Validation Set

In real-world auto-annotation scenarios, labeled data is extremely scarce, making it impossible to carve out a validation set to tune hyperparameters. Therefore, this paper adopts a strict "cross-dataset tuning" protocol: all hyperparameters (learning rate, weight decay, batch size, etc.) are determined on semi-Aves and then directly applied to the other four datasets without any secondary tuning. The temperature hyperparameters are kept identical—\(T_{conf}=0.01\) and \(T_{loss}\) initialized to 0.07 (learnable) across all datasets. Under this setting, SWIFT consistently outperforms previous methods across all 5 datasets, demonstrating the robustness of the temperature strategy (where \(T_{conf}\) yields stable improvements across a wide range from 0.001 to 0.05) and making it far more practical than legacy threshold-tuning methods that are prone to overfitting and require extensive tuning.

Loss & Training

SWIFT employs three types of losses for joint training in Stage 2:

• Supervised Loss \(\mathcal{L}_l\): Computed as CE on labeled data \(L\) and retrieved data \(R\), scaled by \(T_{loss}\).
• Unsupervised Consistency Loss \(\mathcal{L}_u\): For unlabeled data \(U\), softmax probabilities of weakly augmented views are first sharpened by \(T_{conf}\) to generate pseudo-labels \(\hat{y}\). After threshold filtering (\(\sigma=0.8\)), CE is computed on the strongly augmented views' logits scaled by \(T_{loss}\).
• Supervised Retrieval Loss: \(\mathcal{L}_r\) is handled together with \(\mathcal{L}_l\) by mixing \(R\) and \(L\) in the same training batches.

Total loss: \(\mathcal{L} = \mathcal{L}_l + \mathcal{L}_u\)

The learning rate is set to 1e-6 for the vision encoder (small LR to preserve pre-trained features) and 1e-4 for the classifier. The training spans 50 epochs for each of the first two stages, and 10 epochs for Stage 3. AdamW optimizer and cosine annealing scheduler are used.

Key Experimental Results

Main Results

Experiments are conducted on 5 fine-grained classification datasets (semi-Aves, FGVC-Aircraft, Stanford Cars, EuroSAT, DTD) with an OpenCLIP ViT-B/32 backbone.

Setting	Method	4-shot	8-shot	16-shot
FSL	FS-FT (Labeled only)	61.0	65.8	69.1
FSL	SWAT (+ retrieved open data)	67.4	71.0	74.0
SSL	FineSSL (Freeze VLM, prompt tuning)	57.6	64.6	68.9
SSL	FixMatch direct fine-tuning of VLM	39.3	49.9	57.2
SSL	FixMatch + Temperature (Ours)	57.7	65.7	71.2
SSFSL	SWIFT (Ours)	71.5	76.3	79.7
Reference	Fully supervised (U with ground-truth labels)	74.8	75.4	76.0
Reference	Fully supervised + Retrieval	76.0	76.8	77.2

SWIFT outpaces the SOTA FSL method SWAT by approximately 4.1, 5.3, and 5.7 percentage points under 4, 8, and 16-shot settings, respectively. Notably, its 16-shot performance (79.7%) even surpasses the fully supervised baseline (76.0%), demonstrating that the three-stage training and temperature strategies successfully exploit additional signals within the unlabeled data. It is worth noting that directly fine-tuning a VLM with FixMatch performs poorly in the 16-shot setting (57.2%), even falling behind FS-FT (69.1%), which does not use unlabeled data. Conversely, adding temperature (71.2%) not only reverses this trend but also outperforms SWAT (though a gap remains under 16-shot vs. SWAT's 74.0%, FixMatch + Temperature already approaches SWAT's performance under the 4-shot setting).

Ablation Study

Configuration	4-shot	8-shot	16-shot	Description
FixMatch direct fine-tuning of VLM	39.3	49.9	57.2	Baseline, U utilization ≈ 0%
+ Stage 1 (Classifier initialization)	56.3	59.8	62.2	Fine-tune classifier on L with \(T_{loss}\)
+ Stage 2 (Semi-supervised fine-tuning with \(L+U+R\))	68.8	73.3	77.3	FixMatch with \(T_{conf}\) + retrieved data
+ Stage 3 (Few-shot refinement) → SWIFT	71.5	76.3	79.7	Refine with only L to eliminate noise from R
Swap with DebiasPL as the underlying SSL method	73.1	77.4	79.9	SWIFT can be integrated with stronger SSL methods

Key Findings

• Temperature is a key enabler for VLM fine-tuning: Adding temperature to FixMatch directly yields dramatic improvements of 14-20 percentage points, elevating it from worse than FS-FT to substantially beating it. The mechanism of temperature is dissected into two independent dimensions: \(T_{conf}\) increases utilization (jumping from ~0% to 60-80%), and \(T_{loss}\) enhances the supervisory signal (the loss curve transitions from flat to a rapid decline).
• Each of the three stages makes a distinct contribution: Stage 1 contributes about 5-17 percentage points (more prominent in lower-shot settings as classifier initialization has a greater impact when few samples are available). Stage 2 contributes about 6-11 percentage points (the core gain from open data + SSL temperature). Stage 3 contributes about 2-3 percentage points (refining and denoising). The effects of the three stages stack together evidently, and the gains of each stage are consistently replicable across all 5 datasets.
• Generalization validation: SWIFT is equally effective on the DINOv2 backbone and yields an even more pronounced improvement (jumping from 50.2 to 78.2 under 4-shot, a +28 point gain). This proves that the temperature mechanism is not unique to OpenCLIP—flat softmax distributions are a common issue for any contrastively pre-trained model.
• Robustness to hyperparameters: \(T_{conf}\) delivers steady and major improvements within a broad window of [0.001, 0.05], unlike traditional threshold tuning methods which are practically unfeasible without a validation set.
• The divide between VLMs and ImageNet pre-training: Temperature actually harms ImageNet pre-trained ResNet-50 or ViT models (where not using temperature performs best) but is essential for all VLMs (regardless of ResNet or ViT architectures). This confirms that flat softmax distributions stem from the contrastive pre-training objective rather than architectural differences.

Highlights & Insights

• Insight-driven simple solution: The greatest highlight of this work is pinpointing "flat softmax" as the root cause of VLM fine-tuning failures and fixing it with an extremely simple temperature remedy. This is more elegant than designing complex adaptive threshold methods (such as FreeMatch) or specialized SSL methods for new backbones—instead of adding complexity, it restores the temperature match between pre-training and fine-tuning.
• Dual-temperature decomposition: The role of temperature is decoupled into two independent dimensions: confidence temperature (resolving the utilization issue) and loss temperature (resolving the training signal issue), with each validated through careful ablation. This decomposition simplifies the method design and facilitates diagnosing similar issues in other VLM fine-tuning tasks.
• Practical value in validation-free scenarios: Many SSL papers design methods under the assumption of having a validation set for tuning. In contrast, this work strictly follows a validation-free protocol, proving the robustness of the temperature strategy in this demanding set-up, which has direct guiding significance for real-world applications such as auto-annotation.
• Transferable insight: Any downstream task requiring the fine-tuning of contrastively pre-trained models (not just classification) is likely to encounter flat softmax issues. The temperature strategy can be embedded as a general pre-processing step in various existing SSL frameworks, and the paper verifies its compatibility with both FixMatch and DebiasPL.

Limitations & Future Work

• High computational cost: SWIFT employs a three-stage training pipeline and a large volume of retrieved data (500 images per class). Training takes about 8 hours on a single RTX 4090 under the semi-Aves 16-shot setting, which is considerably longer than FS-FT (0.3h) and SWAT (2h). While the performance gains are significant, this represents a trade-off in resource-constrained scenarios.
• Dependency on retrieved data: Open data retrieval has an inherent ceiling. While Tab. 11 shows that increasing from 500 to 1000 images per class still improves performance (suggesting it is not yet saturated), the quality of retrieval (label noise, extent of domain shift) directly affects results. Designing similar SSFSL methods without relying on large-scale open pre-training sets (e.g., when only small-scale in-domain data is available) remains an open problem.
• The cost of missing validation sets: Although a validation-free setting is more realistic, it also means parameters like temperatures cannot be fine-tuned for specific tasks. While the authors mitigate this via broad-range robustness experiments, some extreme tasks may fall outside this robust interval.
• Extreme class imbalance: The class distributions of the 5 benchmark datasets are relatively balanced. Real-world auto-annotation tasks, however, often face long-tailed distributions. Whether the temperature strategy remains robust under extreme imbalance (where pseudo-labels of minority classes could be entirely drowned out by flat distributions) warrants further verification.

Related Work & Insights

• vs. FixMatch / DebiasPL (Traditional SSL methods): Their original implementations assume that the backbone yields sharp softmax distributions, hence utilizing high confidence thresholds. This paper reveals that this assumption breaks down with VLM backbones and restores the effectiveness of these methods by fixing the flat softmax of VLMs using temperature, without altering the algorithmic structure itself. This philosophy of "fixing underlying assumptions rather than rewriting the algorithm" is highly valuable.
• vs. FineSSL (Freeze VLM, prompt tuning): FineSSL avoids fine-tuning difficulties by adapting frozen VLMs to downstream tasks through prompt learning, but SWIFT outperforms it in terms of accuracy (79.7% vs. 68.9% in 16-shot). This indicates that with sufficient data, fine-tuning the VLM remains superior to prompt learning.
• vs. SWAT (FSL + Open data retrieval): Both SWAT and SWIFT utilize retrieved data and a three-step training pipeline, but SWAT does not use unlabeled data and lacks the temperature configuration. SWIFT builds upon retrieved data by leveraging unlabeled data and integrating temperature to make it work, yielding an additional leap of about 5 percentage points.
• vs. FreeMatch (Adaptive threshold SSL): FreeMatch dynamically adjusts the threshold of each class based on the model's learning status rather than relying on a fixed high threshold. Experimental results in the paper (Fig.13) demonstrate that although FreeMatch increases utilization (>60%), its accuracy remains very low due to the lack of training signal reinforcement from \(T_{loss}\). This suggests that boosting utilization alone is insufficient and must be paired with a loss temperature.

Rating

• Novelty: ⭐⭐⭐⭐ The discovery of the root cause (flat softmax) and its minimalist remedy (temperature) construct a strong insight. Although the method itself falls into a simple engineering routine, simplicity itself forms an innovation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 5 benchmarks × 3 shot settings × multiple backbones (OpenCLIP/DINOv2/ImageNet). The ablations go deep into the gains of each phase, the temperature robustness analysis is highly extensive (broad windows, multiple seeds, learnable vs. fixed), and of note is the evaluation of integration across different SSL methods.
• Writing Quality: ⭐⭐⭐⭐⭐ The narrative structure is exceptionally clear and fluent, proceeding from problem discovery to root-cause analysis, proposed solutions, and systematic verification. Fig. 3 visually maps out the causal chain of "flat distribution → low utilization → weak supervision" with extreme clarity, serving as a template for "storytelling with figures."
• Value: ⭐⭐⭐⭐⭐ The practical scenario of SSFSL (auto-annotation) is well-justified. The temperature solution is simple enough to be quickly replicated into any existing SSL codebase, and a major boost of 5 points represents solid progress in the few-shot learning field.

Related Papers

• [CVPR 2026] Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
• [ICLR 2026] Exploring Cross-Modal Flows for Few-Shot Learning
• [CVPR 2026] Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning
• [CVPR 2025] UNEM: UNrolled Generalized EM for Transductive Few-Shot Learning
• [CVPR 2026] Noise-Aware Few-Shot Learning through Bi-directional Multi-View Prompt Alignment