MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment

Conference: ICML 2026
arXiv: 2605.29987
Code: To be confirmed
Area: Model Compression / Representation Learning / Matryoshka Embedding
Keywords: Matryoshka Representations, Embedding Compression, Subspace Alignment, Spectral Isotropy, Self-distillation

TL;DR

This paper proposes MIC, which introduces two geometric regularizations on top of Matryoshka Representation Learning (MRL)—SCR (limiting correlation between prefix/residual subspaces) and SIR (enforcing uniform variance in prefixes and hyperspherical uniformity). These regularizations enable the model to maintain high discriminative power even when truncated to extremely low dimensions such as 16, 32, or 64, significantly outperforming baselines like MRL and ESE.

Background & Motivation

Background: Modern retrieval, semantic search, and clustering rely on dense embeddings. However, while high-dimensional vectors are storage-intensive and computationally expensive, low-dimensional vectors often lack sufficient expressiveness. Matryoshka Representation Learning (MRL, Kusupati 2022) addresses this by nesting multiple low-dimensional sub-vectors within a single high-dimensional vector. By applying InfoNCE supervision simultaneously across multiple truncation dimensions \(\mathcal{M}=\{m_1,\dots,m_k\}\), MRL allows a single model to support "on-demand truncation" across multiple resolutions.

Limitations of Prior Work: MRL only ensures that "truncations are functional" by calculating losses at each prefix dimension, but it lacks mechanisms to constrain the geometric relationship between the prefix and the residual. Empirical observations indicate: - Subspace Redundancy: Features learned by the prefix are highly correlated with the residual (non-zero cross-covariance \(\boldsymbol{\Sigma}_{\mathrm{cross}}\)), meaning low-dimensional prefixes do not compress independent information. - Spectral Collapse / Anisotropy: Prefix feature distributions degenerate into a narrow cone (Ethayarajh 2019), where a few principal components dominate similarity, rendering the remaining dimensions ineffective. - Performance Collapse in Extreme Low Dimensions: Performance drops precipitously when dimensions are reduced from 768 to 16, far exceeding the expected information loss.

Key Challenge: The multi-objective supervision in MRL optimizes for utility but fails to constrain subspace geometry. If the eigenvalues of the prefix covariance matrix decay too rapidly or correlate strongly with the residual, the effective dimensionality becomes much smaller than the arithmetic dimensionality, leading to an actual information capacity far below its theoretical limit.

Goal: To improve structural properties without altering the MRL framework by: (i) making the prefix and residual "complementary rather than redundant," and (ii) ensuring uniform variance across prefix dimensions and a uniform distribution on the hypersphere.

Key Insight: The authors draw inspiration from the "cross-correlation redundance reduction" in Barlow Twins (Zbontar et al. 2021). However, while Barlow Twins performs global decorrelation, it does not account for the "nested subspace" structure unique to MRL. This work applies constraints to the ordered, structured dependency between prefix and residual, while simultaneously addressing anisotropy using hyperspherical uniformity (Wang & Isola 2020).

Core Idea: A "soft + thresholded" cross-correlation penalty (SCR) is used to replace hard orthogonality constraints, combined with a dual regularization (SIR) using the "coefficient of variation of dimensional variance + RBF hyperspherical uniformity" to straighten the spectral properties of the prefix. These are integrated into the MRL self-distillation objective.

Method

Overall Architecture

The backbone remains a standard Transformer encoder \(f_\theta\) outputting hidden states \(\mathbf{H}\in\mathbb{R}^{B\times L\times d_{\mathrm{full}}}\). The training objective includes the original Matryoshka InfoNCE \(\mathcal{L}_{\mathrm{MRL}}\) (summed across all truncation dimensions in \(\mathcal{M}\)). MIC introduces two additions:

1. Layer-wise and Truncation-wise Extraction: Hidden states are extracted and split at truncation point \(d\) into prefix \(\mathbf{H}_{\mathrm{pre}}\in\mathbb{R}^{B\times L\times d}\) and residual \(\mathbf{H}_{\mathrm{res}}\in\mathbb{R}^{B\times L\times d_{\mathrm{res}}}\) (\(d_{\mathrm{res}}=d_{\mathrm{full}}-d\)).
2. Geometric Alignment: For each \((l,d)\) pair, \(\mathcal{L}_{\mathrm{SCR}}^{(l,d)}\) and \(\mathcal{L}_{\mathrm{SIR}}^{(l,d)}\) are calculated and summed to form \(\mathcal{L}_{\mathrm{align}}\).
3. Total Loss: \(\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{MRL}}+\gamma\mathcal{L}_{\mathrm{align}}\).

Regularization is applied only to selected intermediate layers \(L_{\mathrm{align}}\) to avoid interference with the final classification head and because early layers lack sufficient semantic depth.

Key Designs

1. Soft Collapse Regularization (SCR):

   • Function: Decouples the prefix and residual without forcing hard orthogonality, eliminating redundancy between nested subspaces while preserving expressiveness.
   • Mechanism: After mask-aware sequence-wise normalization (\(\tilde{\mathbf{X}}_{\mathrm{pre}}\), \(\tilde{\mathbf{X}}_{\mathrm{res}}\)), it computes the token-wise cross-correlation \(\mathbf{C} \in \mathbb{R}^{d\times d_{\mathrm{res}}}\). A thresholded \(\ell_2\) penalty is applied: \(\mathcal{L}_{\mathrm{corr}}^{(d)}=\frac{1}{d\cdot d_{\mathrm{res}}}\sum_{u,v}\max(0,|C_{u,v}|-\tau_{\mathrm{corr}})^2\). Correlations below the tolerance threshold \(\tau_{\mathrm{corr}}\) are ignored. A variance floor \(\mathcal{L}_{\mathrm{var}}^{(d)}=\max(0,1-\bar\sigma_{\mathrm{pre}})+0.5\max(0,1-\bar\sigma_{\mathrm{res}})\) is added to prevent the model from minimizing correlation by collapsing variance to zero.
   • Design Motivation: Hard orthogonality is too restrictive and might remove meaningful shared information. Thresholding preserves minor correlations (noise/fluctuations) while suppressing major redundancies. The variance floor is a critical engineering patch to prevent "shrink to zero" degenerate solutions.

2. Spectral Isotropy Regularization (SIR):

   • Function: Ensures uniform variance across prefix dimensions and forces embeddings to be uniformly distributed on the unit hypersphere.
   • Mechanism: Uses mean-pooled prefix representations \(\mathbf{Z}^{(d)}\in\mathbb{R}^{B\times d}\). The Coefficient of Variation (CV) loss treats dimensional variances \(v_j\): \(\mathcal{L}_{\mathrm{cv}}^{(d)}=\frac{\sqrt{\frac{1}{d}\sum_j(v_j-\bar v)^2}}{\bar v+\epsilon}\). The Hyperspherical Uniformity loss applies an RBF kernel \(K_{ij}=\exp(-2t(1-S_{ij}))\) to row-normalized vectors to define \(\mathcal{L}_{\mathrm{unif}}^{(d)}=\log(\frac{1}{B(B-1)}(\mathbf{1}^\top\mathbf{K}\mathbf{1}-\mathrm{Tr}(\mathbf{K}))+\epsilon)\).
   • Design Motivation: Standard MRL does not explicitly constrain the spectral distribution of \(\boldsymbol{\Sigma}_{\mathrm{pre}}\), leading to few principal components dominating. The CV loss suppresses rapid eigenvalue decay, while the uniformity loss addresses the "narrow cone" anisotropy characteristic of Transformers.

3. Multi-layer + Multi-truncation Self-distillation Assembly:

   • Function: Distributes SCR/SIR across selected intermediate layers \(L_{\mathrm{align}}\) rather than just the final layer, performing "intra-layer + inter-layer" geometric distillation.
   • Mechanism: \(\mathcal{L}_{\mathrm{align}}=\frac{1}{|L_{\mathrm{align}}||\mathcal{D}|}\sum_{l\in L_{\mathrm{align}}}\sum_{d\in\mathcal{D}}(\mathcal{L}_{\mathrm{SCR}}^{(l,d)}+\mathcal{L}_{\mathrm{SIR}}^{(l,d)})\).
   • Design Motivation: Representation geometry forms throughout the network. Adding regularization only at the final layer fails to correct information distortion in earlier layers.

Loss & Training

The final loss is \(\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{MRL}}+\gamma\mathcal{L}_{\mathrm{align}}\). The training process follows the standard MRL pipeline (shared backbone, simultaneous InfoNCE for all \(m\in\mathcal{M}\)), with the addition of SCR/SIR calculations per step. Backbones tested include TinyBERT-6L, BERT-base, and BGE-M3.

Key Experimental Results

Main Results

Experiments spanned 15+ datasets across Text Classification, NLI, and STS, with truncation dimensions \(\{16, 32, 64, 128, 256, 512, 768\}\). Representative results for the BERT backbone in low-dimensional zones:

Dataset	Dim	Unsup SimCSE	MRL	ESE	MIC	Gain vs ESE
Banking77	16	35.92	46.39	47.01	59.45	+12.44
Banking77	32	54.23	64.90	63.63	75.71	+12.08
Banking77	64	67.78	76.84	76.24	83.05	+6.81
TweetEval	16	48.85	55.96	47.27	56.13	+8.86
STS12 (OOD)	16	47.88	55.13	51.34	60.86	+9.52
STS16 (OOD)	16	50.78	54.78	59.67	63.76	+4.09
SciTail (OOD)	16	68.15	67.45	69.14	73.09	+3.95

In high-dimensional zones (256/512/768), MIC performs on par with or slightly better than baselines. However, gains in low-dimensional zones (16/32/64) are significant (typically +5 to +12 points), precisely where MRL/ESE performance collapses.

Ablation Study

Configuration	Key Finding
Full MIC (SCR + SIR)	Best low-dimensional performance.
w/o SCR	Prefix-residual redundancy increases; performance drops at low dims.
w/o SIR	Anisotropy returns; performance collapses at ultra-low dims.
w/o \(\mathcal{L}_{\mathrm{var}}\)	"Shrink to zero" occurs; dimensional collapse observed.
Hard Orthogonality (\(\tau_{\mathrm{corr}}=0\))	Expressiveness impaired; overall performance decreases.
Last Layer Only	Low-dim gains largely disappear; multi-layer coverage is necessary.

Key Findings

• Higher Compression, Higher Gains: The gap between MIC and baselines is largest at \(d=16\) and nearly vanishes at \(d=768\), proving that SCR/SIR effectively counteract capacity loss during high compression.
• Consistency Across Backbones: The same patterns were observed across TinyBERT-6L, BERT, and BGE-M3.
• Strong OOD Performance: Improvements on OOD datasets (STS12-16, SciTail) often exceed ID improvements, suggesting geometric alignment leads to more transferable representations.
• Variance Floor is Essential: Without \(\mathcal{L}_{\mathrm{var}}\), the model "cheats" the SCR penalty by zeroing out variance.

Highlights & Insights

• Nested Subspace Geometry as a First-Order Problem: Instead of stacking more complex loss objectives, the authors diagnose the root causes of low-dim collapse (redundancy + anisotropy + spectral collapse) and prescribe specific regularizations.
• Combination of Soft Thresholding and Variance Floors: Thresholding prevents over-regularization, while the variance floor prevents trivial solutions. This combination can be generalized to other statistical penalty objectives.
• Dual Spectral Management: Controlling both individual dimensional variance (CV loss) and global distribution (RBF uniformity) effectively treats anisotropy in low-dimensional dense embeddings.

Limitations & Future Work

• Mapping layers to truncation dimensions is fixed and requires re-calibration for different backbone depths/configurations.
• SIR treats all truncation dimensions with equal weight, ignoring the fact that lower dimensions might require more "care."
• Experiments are limited to text tasks; applicability to multimodal or generative scenarios (VLM/Diffusion) remains to be verified.
• Efficiency: Calculating SCR/SIR across \(| \mathcal{D} |\) dimensions for each layer adds overhead; the feasibility for very large models (LLM-scale) needs further study.

Related Work & Insights

• vs MRL: MRL lacks geometric constraints; MIC adds subspace complementarity and spectral uniformity for significant low-dim gains.
• vs ESE: ESE modifies the architecture with "compress-and-express" modules; MIC is a lightweight regularization-only approach that outperforms ESE by 5+ points in low dims.
• vs Barlow Twins: BT is for global decorrelation; MIC adapts cross-correlation for "nested subspaces" with thresholding and variance floors.
• vs SimCSE + Uniformity: MIC explicitly decomposes uniformity into CV and RBF terms to supplement Matryoshka's hierarchical structure.
• vs Whitening: While tradition uses post-processing, MIC internalizes isotropy into the training objective, making low-dimensional prefixes ready for use without extra steps.

Rating

• Novelty: ⭐⭐⭐⭐ While individual components are known, the combination of "soft threshold + variance floor + nested subspaces" is a tailored and effective design for MRL.
• Experimental Thoroughness: ⭐⭐⭐⭐ Extensive testing across 15+ datasets and 3 backbones; OOD coverage is a plus.
• Writing Quality: ⭐⭐⭐⭐ Clear narrative: diagnosis → regularization → anti-degeneration.
• Value: ⭐⭐⭐⭐ Highly valuable for real-world dense retrieval where 5-12 point gains at low dimensions directly translate to significant savings in storage and bandwidth.

Related Papers

• [AAAI 2026] Prototype-Based Semantic Consistency Alignment for Domain Adaptive Retrieval
• [ICML 2026] LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
• [ICML 2026] Event2Vec: Processing Neuromorphic Events Directly by Representations in Vector Space
• [ICML 2026] ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation
• [ICML 2026] Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning