HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/ArianYp/HSFM
Area: Optimization & Theory
Keywords: spurious correlation, worst-group accuracy, bilevel optimization, meta-learning, feature-space augmentation

TL;DR¶

HSFM freezes an ERM-pretrained backbone and treats the feature-space embeddings of a small support set as learnable parameters, optimizing them through a bilevel meta-objective driven by the model's own loss on hard examples (inner loop adapts the linear head, outer loop minimizes hard-set loss); in a few minutes on a single GPU it reaches worst-group accuracy on par with methods that use group labels on Waterbirds and Dominoes, and the resulting feature edits can be decoded back into images via unCLIP to expose the model's spurious dependencies.

Background & Motivation¶

In real datasets, features that have no causal relation to the class label but co-occur with it frequently create spurious correlations — butterfly images on ImageNet usually also contain flowers, and waterbirds in Waterbirds mostly appear against water backgrounds. Deep networks trained with empirical risk minimization (ERM) latch onto these shortcuts and degrade sharply once the correlation no longer holds (minority-group samples); in high-stakes domains such as medical diagnosis the same shortcut problem can lead directly to wrong conclusions. The field typically models the data distribution as a set of groups (combinations of class label and environment attribute, e.g. "waterbird/landbird" × "water/land") and adopts worst-group accuracy, \(\mathrm{WGA}(f)=\min_{g}\mathrm{Acc}_g(f)\), as the primary robustness metric.

A key observation underpins a recent line of work: the backbone trained by ERM actually encodes both core and spurious features, and it is mostly the linear classification head on top that gets captured by the shortcut — freezing the backbone and retraining only the head already recovers a large amount of minority-group accuracy. Yet existing fixes each have a weak spot. GroupDRO directly minimizes the worst-group loss but needs group labels at training time. JTT, AFR, and LfF only use the model's own error signal to reweight existing samples, which keeps them confined to the support of the original training set — they can neither select nor create the minority-group samples that are scarce by construction. Augmentation-based methods such as MaskTune, DaC, DDB, and FFR rely on hand-designed masking/synthesis rules or external generative priors, which is expensive and bounded by generation quality. And DFR, though simple and effective, turns the validation set directly into a balanced retraining set, which is unreliable when the validation set is tiny (MetaShift has only 81 validation samples).

The angle taken here is: since the failure lives in the classification head, and the head's training data is itself something we can move, push the whole augmentation step past the backbone output — instead of synthesizing images, directly optimize the embeddings of the support samples inside the frozen feature space, so that these "edited" features steer the decision boundary toward hard examples when the head is trained; and let the model's own loss on the validation set answer which examples are hard, with no group annotation at all. Core idea: use a bilevel meta-learning framework whose outer objective is the hard-set loss to optimize support-set features as free parameters — the inner loop adapts the linear head on the optimized features, and the outer loop back-propagates through the inner updates to minimize the loss on high-loss examples.

Method¶

Overall Architecture¶

HSFM attaches to a model already trained with ERM and decomposes it into a frozen backbone \(\phi\) and a linear head \((W,b)\): an image first goes through the backbone to give a feature \(h=\phi(x)\), which the head maps to logits. This decomposition is the premise of the whole method — only quantities in feature space are optimizable. Each round alternates two operations: update the feature embeddings of a support set using the loss on "hard" examples, then train the linear head on the updated features. Concretely, a class-balanced support set \(\mathcal{S}\) is first sampled from the training set, and its frozen backbone features serve as the initial values of learnable embeddings \(H=\{h_i\}\); then, in every epoch, the top-\(K_{\text{hard}}\) highest-loss validation samples per class under the current classifier are collected into a hard set \(\mathcal{Q}\); next, \(K_H\) meta steps are performed — each step first adapts the linear head on \(H\) for \(T\) steps (inner loop), then evaluates the adapted head on \(\mathcal{Q}\) and takes the meta-gradient with respect to \(H\) (outer loop); finally the head is trained for \(T\) further steps on the updated \(H\) before the next epoch. The only things actually optimized are a batch of feature vectors and a linear head, and the backbone is invoked once to extract features, so the cost is very low.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["ERM model + train/val sets"] --> B["Sample class-balanced support set<br/>init from frozen features"]
    B --> C["Feature-space support embedding optimization<br/>sample features become learnable"]
    C --> D["Group-label-free hard-set construction<br/>top-loss validation samples per class"]
    D --> E["Bilevel meta-optimization<br/>inner adapts head, outer updates embeddings"]
    E -->|refresh hard set next round| D
    E --> F["Robust classifier"]

Key Designs¶

1. Feature-space support embedding optimization: moving augmentation from pixel space to the backbone output

Conventional augmentation and rebalancing either generate new images or mask/interpolate existing ones, and both require another full backbone training run. HSFM inverts this: it accepts that the backbone already "knows" enough, and defines augmentation at the backbone output — the feature vectors of the support samples start from \(\phi(x_i)\) and are treated as free parameters that are optimized directly; what comes out is the edited sample. The benefits are threefold. First, no backward pass through the backbone is needed: every meta step only propagates through a batch of \(d\)-dimensional vectors and one linear head, which cuts training from hours to minutes. Second, it avoids the quality issues of pixel-space generation and the prior bias of generative models, and sidesteps the question of whether a synthesized feature corresponds to any real image (the method only requires that the edited feature helps the head learn, not that it is realizable). Third, because the optimization lives in feature space, it is not tied to any modality or task assumption — swapping the backbone (ResNet-50 / ViT-B/16 / ConvNeXt / CLIP) only requires re-extracting features once.

2. Group-label-free hard-set construction: replacing group annotation with loss ranking

The great divide among spurious-correlation methods is whether group labels are required. GroupDRO and LISA need them during training, and DFR needs them to balance the validation set into a usable retraining set. HSFM's substitute is strikingly plain: within each class, sort validation samples by the current classifier's cross-entropy loss in descending order, take the top \(K_{\text{hard}}\), and union these per-class top-K sets into the hard set \(\mathcal{Q}\). It can stand in for group labels because under spurious correlation "high loss" and "belongs to a minority group" overlap heavily — an ERM classifier necessarily misclassifies anti-correlated samples such as landbirds on water, so their loss is high and they are picked up by the top-K automatically, without knowing how groups are defined. This yields two practical properties: the hard set is refreshed dynamically and therefore keeps tracking the classifier's current failure modes rather than a set fixed at the start, which fits the fact that the head keeps changing; and it makes no assumption about the form of the spurious correlation (background-label binding, texture shortcut, superimposed synthetic digits all work), because the only criterion is the loss itself.

3. Inner adaptation plus an outer meta-objective: letting augmentation be driven by the model's own failures

How to turn the hard set into an update signal for the support embeddings is where HSFM genuinely parts ways with the reweighting line. Methods such as JTT and AFR use hard examples to reweight existing samples; the samples themselves never move, so the procedure can never escape the support of the original training distribution. HSFM instead uses the hard-example loss as a meta-objective to move where the samples sit in feature space. Formally, the inner loop is standard — starting from the ERM head parameters, it takes \(T\) gradient steps on the learnable support embeddings \(H\) to obtain an adapted head \((W',b')\); the outer evaluation happens on the hard set, whose sample features \(q_j=\phi(x_j)\) are frozen and not optimized:

\[ \mathcal{L}_{\mathrm{out}}(H;\mathcal{Q})=\frac{1}{|\mathcal{Q}|}\sum_{(x_j,y_j)\in\mathcal{Q}}\ell\big(W'q_j+b',\,y_j\big), \qquad H\leftarrow H-\eta\,\nabla_{H}\mathcal{L}_{\mathrm{out}}(H;\mathcal{Q}) \]

where \(\nabla_H\) must back-propagate through the \(T\) inner update steps back to \(H\) (explicit MAML-style meta-gradients, not a first-order approximation). This structure is what makes the method work, and the gain does not come from "emphasizing hard samples" per se: in the controlled ablation, giving the entire training budget to a No-bilevel variant that trains the head directly on the same hard set yields only 44.4 worst-group accuracy when the validation set is cut to 1000 samples (down to 6/3 minority samples), versus 77.2 for HSFM — directly fitting a repeatedly selected hard set amounts to memorizing it. HSFM does not collapse because the support embeddings are always initialized from real training samples and the head is re-adapted every round, so the bilevel structure itself acts as a regularizer: it asks the embeddings to make the head perform well after adaptation, not to perform well on the hard set as such, and the former cannot be achieved by memorization. The paper summarizes this as "the bilevel structure, not hard-sample emphasis, drives the gain," further corroborated by AFR (reweighting on the validation set) trailing HSFM at every validation size.

A Worked Example¶

Take one round on Waterbirds (waterbird/landbird × water/land background). The backbone is an ImageNet-pretrained ResNet-50 already trained with ERM on the training split, then frozen. A class-balanced support set is drawn (some images per class) and their frozen features are copied into the learnable variables \(H\); the top \(K_{\text{hard}}\approx250\) highest-loss validation samples per class form the hard set \(\mathcal{Q}\) — at this point the misclassified "landbirds on land" and "waterbirds on water" enter it automatically because their loss is large. Then \(K_H\) meta steps run: each step adapts the linear head on \(H\) for \(T\ge10\) steps to obtain \((W',b')\), then computes the cross-entropy on the frozen features of \(\mathcal{Q}\) and back-propagates the gradient all the way to \(H\). As a result the handful of embeddings for "landbirds on land" in the support set are pushed toward the water side, while "waterbirds on water" are pushed toward land — exactly the direction shown by the paper's unCLIP visualizations. After \(K_H\) steps the head is trained properly for \(T\) steps on the current \(H\), the hard set is refreshed, and the next round begins.

Loss & Training¶

Both loops use cross-entropy: the inner loss \(\mathcal{L}_{\mathrm{in}}\) is computed on the support embeddings \(\{(h_i,y_i)\}\) with learning rate \(\alpha\), starting from the ERM head parameters \((W^{(0)},b^{(0)})\) and running for \(T\) steps; the outer loss \(\mathcal{L}_{\mathrm{out}}\) is computed on the hard set \(\mathcal{Q}\) with meta learning rate \(\eta\). Each epoch follows a fixed three-step order: refresh the hard set \(\mathcal{Q}\) → \(K_H\) meta steps updating \(H\) → \(T\) steps updating the head on the current \(H\), alternating so as to track the classifier's evolving failure cases. The key hyper-parameters are covered by the ablation: performance is stable across moderate and large \(T\) and \(T\ge10\) suffices, whereas \(T\in\{1,5\}\) degrades sharply (too few inner updates give an unreliable meta-signal; note that in the paper \(T\) denotes both the number of inner-loop updates and the number of ERM updates of the head); \(K_{\text{hard}}\) has a clear sweet spot around 250 — too small and the hard set lacks diversity and gives an unstable signal, too large and easy, high-frequency examples are drawn in and dilute the minority-group signal. Hyper-parameter details and the efficiency discussion are deferred to the appendix (⚠️ the appendix is not part of the accessible full-text cache, so it was not verified).

Key Experimental Results¶

Main Results¶

Four spurious-correlation benchmarks, using the same ImageNet-pretrained ResNet-50 backbone as the baselines (first trained with ERM, then handed to HSFM). The "Group Info" column follows the notation used with DFR for the use of group labels at training/validation time: ✓✓ means validation group labels are used directly during training, ✓ means they are used only for model selection, ✗ means they are not used.

Method	Group Info (train/val)	Waterbirds Worst	Waterbirds Avg	CelebA Worst	CelebA Avg	MetaShift Worst	Dominoes Worst
GroupDRO	✓/✓	91.4±1.1	93.5±0.3	88.9±2.3	92.9±0.2	66.0±3.8	-
LISA	✓/✓	89.2±0.6	91.8±0.3	89.3±1.1	92.4±0.4	59.8±2.3	-
DFR	✗/✓✓	92.3±0.2	93.3±0.5	88.3±1.1	91.3±0.3	72.8±0.6	90.0±0.4
JTT	✗/✓	86.7	93.3	81.1	88.0	64.6±2.3	-
DaC	✗/✓	92.3±0.4	95.3±0.4	81.9±0.7	91.4±1.1	78.3±1.6	89.2±0.1
DDB	✗/✓	93.0±0.1	93.6±0.1	85.8±1.4	87.3±0.7	81.2±0.2	-
Base (ERM)	✗/✗	74.6	90.2	30.6	95.8	64.1	78.6
HSFM (Ours)	✗/✓	93.1±0.1	94.0±0.5	89.2±0.2	90.6±0.5	77.2±0.1	90.4±0.3

For fine-grained classification a different protocol is used: "ERM" means the backbone was first fine-tuned on the dataset's training split with ERM, "Pretrained" means only an ImageNet-pretrained backbone with no dataset-specific fine-tuning; in both cases the backbone is frozen and handed to HSFM. The metric is top-1 accuracy.

Method	Stanford-Cars	CUB	Oxford-Flowers
ERM	83.98	75.01	91.07
DFR-ERM	82.74	72.90	95.98
DFR-Pretrained	32.75	48.33	87.71
HSFM-ERM	83.44	72.83	97.27
HSFM-Pretrained	85.10	72.94	97.10

Ablation Study¶

Controlled CelebA ablation: the same ERM backbone, with the validation set split in half (signal/selection) so that all methods see the same number of samples, sweeping the validation size across rows. "min cnt" is the minority count in the signal/eval halves; the No-bilevel column trains the head directly on the same hard set \(\mathcal{Q}\) without meta-optimization, isolating the contribution of the bilevel structure. The metric is worst-group accuracy.

Val size (min cnt)	DFR (no bal.)	AFR on val	No-bilevel	HSFM
1000 (6/3)	37.2	41.7	44.4	77.2
2000 (9/13)	40.6	73.9	59.4	79.4
3000 (18/20)	41.1	72.4	63.3	86.6
Full (182)	41.1	61.7	79.4	89.0

Key Findings¶

Main results: HSFM achieves the best scores on Waterbirds (93.1) and Dominoes (90.4), and its 89.2 on CelebA is the highest among all methods that do not use group labels during training (LISA's 89.3 is marginally higher but requires group labels at training time). Notably it is unaffected on the synthetic, carefully constructed Dominoes, where the generative-sample-based DDB is inapplicable.
Backbone- and representation-agnostic: with ViT-B/16 the Waterbirds worst-group accuracy rises from 53.9 to 81.5, CelebA from 53.3 to 88.3, and Dominoes from 46.1 to 73.3; with ConvNeXt, Waterbirds goes 82.7 → 92.6 and CelebA 47.2 → 81.1; even on a frozen CLIP ViT-H embedding, Waterbirds improves from 68.4 to 80.5 and CelebA* from 50.0 to 81.0. The gains shrink as the backbone gets stronger, but the direction is consistent.
The gain comes from the bilevel structure, not from hard-sample emphasis: in the ablation, No-bilevel collapses to 44.4 at the smallest validation size (1000, only 6/3 minority samples) while HSFM reaches 77.2; AFR reweighting on the validation set loses to HSFM at every validation size; and DFR without group balancing degrades further (41.1 at the full setting). Together these show that group-label-free meta-optimization that can move beyond the original training support is what matters.
Hyper-parameter sensitivity and data-size dependence: worst-group accuracy is stable for \(T\ge10\) and drops sharply for \(T\in\{1,5\}\); \(K_{\text{hard}}\) peaks around 250 and worsens at both ends. On MetaShift the gain is marginal (77.2, below DaC's 78.3 and DDB's 81.2), which the authors attribute to the mere 81 validation samples in that set — the same reason DFR is unreliable there (72.8). This is precisely where the method's dependence on validation-set size shows.
Asymmetric behaviour on fine-grained tasks: HSFM-Pretrained beats ERM on Stanford-Cars (85.10 vs 83.98), and HSFM-ERM is best on Oxford-Flowers (97.27); on CUB, ERM is best (75.01) while HSFM-Pretrained stays competitive (72.94). By contrast, DFR degrades badly in the Pretrained setting (Cars 32.75, CUB 48.33), indicating that merely refitting a linear head cannot exploit a strong pretrained representation.
Visual evidence: training a linear head on CLIP embeddings and decoding with unCLIP, about 37% of the CelebA decodings and 35% of the Waterbirds decodings flip their predicted attribute after optimization under a Qwen2.5-7B-VL zero-shot attribute classifier, shifting from majority-group configurations (blond female / waterbird on water) toward minority-group ones (blond male / waterbird on land). The feature edits are therefore semantically aligned with the dataset bias rather than random perturbations.

Highlights & Insights¶

Data augmentation moved into the frozen feature space: no image generation and no back-propagation through the backbone — optimizing a batch of \(d\)-dimensional vectors plus a linear head compresses robust training from "generative model + retraining" to "a few minutes on one GPU." This trick transfers wholesale to any setting with a strong pretrained backbone and a lightweight head (few-shot classification, long-tail recognition, domain adaptation).
Loss ranking instead of group labels: no need to know how groups are defined, just take the top-K loss per class. It rests on the empirical fact that high loss ≈ minority group under spurious correlation, costs almost nothing to implement, and is the most reusable engineering point in the paper.
"Good after adaptation" rather than "good right now": the outer objective evaluates the head after \(T\) inner adaptation steps rather than training the head on the hard set directly. The distinction looks small but turns the method from memorizing the hard set into pushing features to a more generalizable location — the gap between 44.4 and 77.2 in the ablation comes almost entirely from here.
The method doubles as a bias-analysis tool: the optimized feature displacements can be decoded back into images with unCLIP, turning "which spurious attribute does the model rely on" from a number into a picture. Compared with the many robustness works that only report WGA, this adds diagnostic value and offers a route to explaining models with generative models that does not depend on prompt engineering.

Limitations & Future Work¶

The limitation the authors acknowledge: the method relies on the frozen backbone producing sufficiently informative features. When that fails, the backbone must be fine-tuned, which raises the computational cost and removes the efficiency advantage.
The dependence on the validation set is downplayed but real: the hard set is drawn from the validation set, and the gain is marginal on MetaShift (81 validation samples); moreover the validation set is used heavily in this process, so strictly speaking the method is no longer a purely post-hoc "head-only" procedure. In addition, group labels are still used at the model-selection stage (the ✗/✓ entry in Table 1) — they are just not needed for training.
Evaluation is limited to image classification. Whether feature-space editing holds for detection, segmentation, or multimodal large models is completely unverified, and the property that "edited features correspond to no real image" may raise new problems in those tasks.
Refreshing the hard set requires a pass over validation losses each round, and this cost grows with the validation set; the paper gives no curve for that growth.
Plausible improvements: per-class top-K is a coarse form of implicit balancing, and when a minority group is extremely rare inside a class, the top-K can be crowded out by other hard samples of the same class — loss-distribution quantiles or a consistency constraint inside the bilevel loop could help. A learnable mixture with DFR-style explicit balancing (estimating group proportions without group labels) is another natural next step.

vs DFR: DFR uses group labels to balance the validation set into a retraining set and then retrains the linear head on it; HSFM builds no balanced set, but identifies high-loss samples and uses their loss as a meta-signal to optimize the support embeddings. The difference is how supervision is used: DFR swaps the data, HSFM changes where the data sits in feature space. DFR is unreliable with a tiny validation set, while HSFM is more stable in that same setting.
vs JTT / AFR / LfF: all three use the model's error signal to reweight existing samples (JTT emphasizes misclassified ones, AFR uses an incorrectness signal, LfF uses the relative difficulty from a biased model), which keeps them inside the support of the original training set; HSFM optimizes the representation directly and can move outside that support, which is why it still works when minority samples are extremely scarce.
vs GroupDRO / LISA: both target the worst group directly but depend on group annotations at training time; HSFM approaches the same level using only loss ranking on the validation set.
vs generative augmentation (DDB / FFR / DaC / MaskTune): these depend on external generative priors or hand-designed rules, are expensive, and are bounded by generation quality and controllability — on a synthetic set like Dominoes they can even fail outright; HSFM's edits are driven entirely by the model's own failure signal with no external prior.
vs dataset distillation / robust dataset distillation: they share the bilevel optimization framework but differ in objective — distillation summarizes the whole training distribution into a compact synthetic set, whereas HSFM replaces nothing and only serves head adaptation inside the frozen feature space, making it lighter and better aligned with the robustness goal.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of bilevel feature editing in a frozen feature space with a group-label-free top-K hard set is clean and distinctive; however, neither bilevel meta-learning nor hard-example mining is new, and the novelty lies mainly in performing both in feature space.
Experimental Thoroughness: ⭐⭐⭐⭐ Four spurious-correlation benchmarks, three fine-grained datasets, four backbones, CLIP, a controlled ablation, and two hyper-parameter sweeps; coverage is broad, but the main table reuses many numbers reported by DaC, and the appendix details on hyper-parameters and efficiency could not be verified here.
Writing Quality: ⭐⭐⭐⭐ The motivation-to-method chain is clear and the "why not reweighting" argument is made explicitly; points deducted because several formulas are corrupted in the PDF text extraction and must be reconstructed by the reader.
Value: ⭐⭐⭐⭐ A few minutes on a single GPU buys competitive or leading worst-group accuracy, plus a bias-visualization tool on the side; both practical value and reproducibility (code released) are good.