IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning¶
Conference: ECCV 2026
Paper: ECCV Full Paper
Code: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/5862.pdf
Area: Medical Imaging
Keywords: self-supervised learning, joint embedding architectures, representation collapse, batch independent, biomedical imaging
TL;DR¶
IConE decouples the anti-collapse mechanism in self-supervised learning from transient mini-batch statistics to a global learnable instance embedding table regularized by an explicit diversity objective, enabling stable pretraining down to a batch size of 1 with superior robustness against severe class imbalance.
Background & Motivation¶
Joint-Embedding Architectures (JEAs) have achieved remarkable success in self-supervised representation learning by enforcing consistency across augmented views of the same instance while preventing representational collapse. However, conventional collapse prevention mechanisms rely fundamentally on mini-batch interactions. Contrastive methods require large negative pools to accurately capture the data distribution, non-contrastive methods depend on empirical sample variance and covariance across the batch, and clustering or distillation methods require stable batch centering or optimal transport over the batch.
In high-dimensional scientific computing and biomedical imaging domains, this reliance creates an unavoidable bottleneck. Large 3D imaging volumes (such as clinical volumetric CT, MRI, or electron microscopy) readily saturate GPU memory, restricting training to tiny batch sizes (\(B=1\) or \(2\)). In such regimes, batch-level statistics become ill-conditioned, negative samples are practically absent, and extreme class imbalance frequently excludes minority classes from individual batches, leading to severe representation degradation. Furthermore, high-dimensional manifolds theoretically require stronger regularization, yet their immense memory footprint prohibits the large batches that conventional regularization demands—creating a fundamental tension between dimensional requirements and computational feasibility.
The critical insight of this work is that the two core geometric desiderata of representation learning—multi-view alignment and global uniformity—can be mathematically decoupled into separate optimization processes. Core idea: shift the anti-collapse mechanism from the dynamic mini-batch to a persistent, learnable dataset-level instance embedding table regularized for hyperspherical uniformity, allowing the encoder to be trained purely via attractive forces toward stable geometric anchors and achieving batch-independent self-supervised learning.
Method¶
Overall Architecture¶
IConE removes the requirement for in-batch negative sampling and batch variance statistics by establishing a dual-space geometric optimization framework. In the persistent parameter space, an auxiliary embedding table indexed by instance IDs spans the entire dataset, maintaining global uniformity via an explicit repulsive penalty. In the representation space, a backbone encoder maps inputs directly to the unit hypersphere \(\mathcal{S}^{d-1}\) without requiring an additional projector network.
During training, a mini-batch of \(B\) unique samples is drawn, and each sample is augmented into \(V\) stochastic views. The encoder maps these views onto normalized representations. Meanwhile, corresponding instance anchors are retrieved from the auxiliary table and row-normalized. The encoder parameters are updated exclusively through attractive gradients (aligning multi-view representations and pulling views toward their instance anchor), whereas the auxiliary table absorbs both the attractive signal from current views and the global dataset-level repulsive regularization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Instances x_i (Batch Size B)"] --> B["Multi-View Augmentation (V Views)"]
B --> C["Backbone Encoder & Hypersphere Projection<br/>z_i^(v) = f(x_i^(v)) / ||f(x_i^(v))||"]
D["Global Instance Embedding Table E (N x d)<br/>Persistent Dataset-Level Geometric Anchors"] --> E["Normalized Instance Anchors e_i"]
C --> F["View-View Consistency Learning<br/>Attractive Multi-View Invariance"]
C & E --> G["View-Instance Alignment Update<br/>Attractive Pull Toward Instance Anchors"]
D --> H["Dataset-Level Diversity Regularization<br/>Gram Matrix Off-Diagonal Repulsion"]
Key Designs¶
1. Persistent Instance Embedding Table: Decoupling Anti-Collapse from Transient Batches To overcome the instability of batch statistics under small-batch regimes, IConE maintains a learnable matrix \(E \in \mathbb{R}^{N \times d}\), where \(N\) denotes the total number of dataset instances and \(d\) represents the feature dimension. The matrix is initialized from a normal distribution \(\mathcal{N}(0, 0.02)\). Each row \(e_i\) serves as a dedicated, persistent geometric anchor for sample \(x_i\), normalized as \(\tilde{e}_i = e_i / \|e_i\|_2\) during similarity computations while unnormalized parameters receive gradient updates. By embedding the anti-collapse constraint directly into persistent model parameters, each sample retains a global coordinate reference frame even when \(B=1\), eliminating the dependence on concurrent batch mates.
2. Dataset-Level Diversity Regularization: Gram Matrix Positive Correlation Penalty To prevent the auxiliary anchors from collapsing to trivial degenerate configurations, an explicit repulsive penalty is imposed across the entire embedding table \(\tilde{E}\). By computing the full Gram matrix \(G = \tilde{E}\tilde{E}^\top\), off-diagonal correlations are penalized using a squared hinge loss: $\(\mathcal{L}_{div} = \frac{1}{N(N-1)} \sum_{i \ne j} \left( \max(0, G_{ij}) \right)^2\)$ Because \(N > d\) typically precludes strict mutual orthogonality, the formulation penalizes only positive correlations. The squared penalty provides gradients proportional to the violation, strongly repelling similar anchors while leaving orthogonal ones unperturbed. Because the table consists of lightweight vectors, the computational overhead is negligible relative to high-dimensional 3D volume forward passes, translating dataset-level repulsion into a deterministic geometric prior.
3. Bidirectional Alignment with Unidirectional Repulsive Gradient Isolation Unlike standard contrastive learning where encoders receive noisy, high-variance repulsive gradients from in-batch negatives, IConE links encoder representations \(z_i^{(v)}\) with persistent anchors \(\tilde{e}_i\) via the cosine alignment objective: $\(\mathcal{L}_{vi} = \frac{1}{B \cdot V} \sum_{i \in B} \sum_{v=1}^V \langle z_i^{(v)}, \tilde{e}_i \rangle\)$ This term drives the encoder outputs toward the instance's anchor while simultaneously pulling the anchor toward the mean representation of its augmented views. Crucially, the encoder parameters \(\theta\) receive only attractive forces (\(\nabla_\theta \mathcal{L} = \nabla_\theta \mathcal{L}_{vi} + \nabla_\theta \mathcal{L}_{vv}\)), shielding the backbone from cross-instance repulsion. All repulsive forces are isolated within the embedding table update (\(\nabla_{e_i} \mathcal{L} = \nabla_{e_i} \mathcal{L}_{vi} + \nabla_{e_i} \mathcal{L}_{div}\)), structurally insulating the encoder from mini-batch size variations.
Loss & Training¶
The overall training objective is a straightforward linear combination without requiring sensitive temperature hyperparameters: $\(\mathcal{L} = \mathcal{L}_{vi} + \mathcal{L}_{vv} + \mathcal{L}_{div}\)$ Here, the multi-view consistency loss \(\mathcal{L}_{vv}\) minimizes the pairwise cosine distance across all stochastic views of the instance: $\(\mathcal{L}_{vv} = \frac{1}{B} \sum_{i \in B} \frac{2}{V(V-1)} \sum_{1 \le m < n \le V} \langle z_i^{(m)}, z_i^{(n)} \rangle\)$ Even at \(B=1\), both \(\mathcal{L}_{vv}\) and \(\mathcal{L}_{vi}\) remain strictly well-defined as long as \(V \ge 2\). Both the encoder parameters \(\theta\) and the auxiliary table \(E\) are trained end-to-end via standard AdamW optimization using learning rate \(\eta\).
Key Experimental Results¶
Main Results¶
On the MedMNIST3D benchmark spanning 6 clinical 3D imaging datasets (Organ3D, Nodule3D, Adrenal3D, Fracture3D, Synapse3D, Vessel3D), representations trained with various self-supervised methods were evaluated via non-parametric k-NN balanced accuracy (%) across batch sizes (\(B \in \{4, 16, 64\}\)):
| Method | B=4 (k=1) | B=4 (k=5) | B=4 (k=20) | B=16 (k=1) | B=16 (k=5) | B=16 (k=20) | B=64 (k=1) | B=64 (k=5) | B=64 (k=20) |
|---|---|---|---|---|---|---|---|---|---|
| BYOL | 54.6 | 53.7 | 49.6 | 58.1 | 56.3 | 49.7 | 58.1 | 55.3 | 53.0 |
| DIET | 55.6 | 55.6 | 54.7 | 55.3 | 55.3 | 55.3 | 57.1 | 57.0 | 54.4 |
| DINO | 55.8 | 54.6 | 50.5 | 57.7 | 54.9 | 51.9 | 57.4 | 56.6 | 54.6 |
| MoCo-v2 | 50.8 | 51.1 | 49.7 | 56.7 | 56.2 | 55.4 | 57.2 | 56.3 | 53.3 |
| SimCLR | 49.9 | 48.4 | 47.7 | 52.0 | 51.6 | 51.8 | 53.1 | 54.5 | 53.9 |
| SwAV | 48.0 | 46.5 | 44.2 | 50.8 | 47.8 | 46.6 | 50.6 | 48.9 | 45.8 |
| VICReg | 50.8 | 47.3 | 45.2 | 55.9 | 56.7 | 55.7 | 59.1 | 57.8 | 57.0 |
| IConE (Ours) | 63.6 | 62.2 | 60.6 | 63.3 | 62.8 | 62.4 | 62.5 | 60.6 | 58.8 |
When extended to supervised representation learning, IConE variants significantly outperform standard Cross-Entropy and Supervised Contrastive Learning (SupCon), particularly in the extreme \(B=1\) setting:
| Method | 2D (B=1) | 2D (B=4) | 2D (B=16) | 2D (B=64) | 3D (B=1) | 3D (B=4) | 3D (B=16) | 3D (B=64) |
|---|---|---|---|---|---|---|---|---|
| Cross-Entropy | 76.20 | 78.17 | 79.87 | 80.32 | 66.61 | 72.70 | 69.65 | 72.85 |
| SupCon | 72.20 | 74.08 | 79.70 | 79.73 | 59.49 | 66.50 | 72.59 | 71.55 |
| IConE-Instance | 76.81 | 78.43 | 79.53 | 79.63 | 68.77 | 75.16 | 76.56 | 75.31 |
| IConE-Class | 78.71 | 82.42 | 82.16 | 82.31 | 69.59 | 74.25 | 77.14 | 76.21 |
Ablation Study¶
Ablation evaluating alternative diversity regularization objectives (VCReg and SIGReg) substituted for the default Gram matrix orthogonality penalty on 2D and 3D benchmarks:
| Regularizer Config | 2D (B=1) | 2D (B=4) | 2D (B=16) | 2D (B=32) | 3D (B=1) | 3D (B=4) | 3D (B=16) | 3D (B=32) |
|---|---|---|---|---|---|---|---|---|
| IConE (Default Orthogonality) | 59.6 | 60.6 | 62.0 | 60.1 | 62.1 | 69.7 | 68.8 | 65.4 |
| IConE-VCReg | 59.3 | 60.8 | 60.4 | 61.2 | 63.0 | 68.8 | 69.1 | 67.9 |
| IConE-SIGReg | 57.3 | 58.3 | 58.5 | 58.4 | 62.9 | 68.4 | 67.5 | 68.0 |
Key Findings¶
- Unprecedented Small-Batch Stability: In 3D benchmarks, while VICReg and SimCLR drop by over 8% and 3% respectively as batch size decreases from 64 to 4, IConE maintains steady performance (rising from 62.5% to 63.6%), exhibiting near-zero Pearson correlation between batch size and accuracy.
- Preservation of High Intrinsic Dimensionality: Intrinsic dimensionality metrics (RankMe, LiDAR, Effective Rank) confirm that baseline representations suffer from severe dimensional collapse under small batches. In contrast, IConE preserves a full singular value spectrum across all batch sizes.
- Implicit Memory Under Class Imbalance: In step-\(k\) class imbalance experiments, when minority classes are subsampled by factors up to 20–25, baselines degrade to 10–20% on minority classes due to frequent mini-batch absence. In contrast, IConE maintains minority accuracy above 40%, because persistent anchors preserve global class separation continuously.
Highlights & Insights¶
- Structural Decoupling of Invariance and Anti-Collapse: By isolating repulsive updates to the auxiliary parameter space, the encoder is optimized exclusively via stable, attractive regressions, bypassing the gradient instability of mini-batch negative sampling.
- Elimination of Temperature Scaling: Unlike InfoNCE-based methods where temperature tuning dictates the sharpness of negative competition, IConE replaces softmax competition with geometric squared-hinge loss, removing a notoriously sensitive hyperparameter.
- Memory Efficiency for Scientific Backbones: By omitting projection heads and relying on a parameter table of merely a few megabytes, IConE frees substantial GPU memory for high-capacity 3D backbones.
Limitations & Future Work¶
- Quadratic Complexity on Massive Datasets: Computing the \(N \times N\) Gram matrix scales quadratically with dataset size \(N\). While trivial for current scientific imaging benchmarks, applying IConE to web-scale datasets requires substituting \(O(N \cdot d^2)\) statistical regularizers like VCReg or SIGReg.
- Dynamic Anchor Allocation: The current parameter table assumes a fixed dataset size known a priori. Extending the architecture to continual learning or streaming data requires dynamic anchor allocation schemes.
Related Work & Insights¶
- vs SimCLR / MoCo: Contrastive learning relies on in-batch negative pairs or momentum memory queues, which suffer from stale feature representations early in training; IConE updates persistent anchors directly via backpropagation without doubling model parameters.
- vs VICReg / Barlow Twins: Non-contrastive methods apply statistical regularization to in-batch feature dimensions, which become ill-conditioned when \(B < 16\); IConE transfers regularization to the dataset dimension \(N\), maintaining valid statistics down to \(B=1\).
- vs DIET: Parametric instance discrimination via cross-entropy enforces multi-view alignment only implicitly through shared labels and relies on softmax competition; IConE incorporates explicit view-view consistency and direct geometric separation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Decoupling multi-view alignment and collapse prevention via a persistent parameter table provides an elegant and effective formulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 2D/3D biomedical benchmarks, natural images, geometric collapse metrics (RankMe/LiDAR/UMAP), and supervised extensions.
- Writing Quality: ⭐⭐⭐⭐⭐ Clearly articulated motivation, rigorous mathematical derivations, and convincing empirical validation.
- Value: ⭐⭐⭐⭐⭐ Highly impactful for memory-constrained domains (volumetric medical imaging, 3D vision, scientific data) where small-batch training is mandatory.