Skip to content

SUGAR: Spectral and Geometry-Aware Alignment for Matryoshka Representation Distillation

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: Knowledge Distillation, Matryoshka Representation Learning, Effective Rank, Grassmannian Manifold, Elastic Representation

TL;DR

SUGAR formalizes the task of Matryoshka Representation Distillation and proposes a multi-objective framework combining spectral effective-rank matching, Grassmannian chordal-distance subspace alignment, and orthogonal band regularization, enabling compact student encoders to retain high accuracy across aggressive dimensional truncations.

Background & Motivation

Dense sentence representations are foundational to modern semantic retrieval, clustering, and search engines. To reconcile conflicting operational budgets—such as low-latency search on edge devices and high-precision re-ranking on server clusters—Matryoshka Representation Learning (MRL) has emerged as an essential paradigm in premier embedding systems such as OpenAI text-embedding-3 and Qwen3-Embedding. By structuring representations into nested, prefix-able slices, MRL enables embeddings to be truncated at inference time without requiring multiple independently trained models.

However, distilling massive MRL teacher models into lightweight student architectures presents severe structural challenges that conventional Knowledge Distillation (KD) frameworks fail to solve. Existing KD techniques operate under a single-scale assumption: they enforce point-wise vector alignment, intermediate token matching, or relational distillation on fixed-dimensional outputs. When applied to multi-scale embeddings, these methods ignore the nested subspace hierarchy, causing low-capacity students to suffer from dimensional collapse and severe anisotropy under aggressive prefix truncation (e.g., \(d=16\) or \(32\)).

The core tension is that Matryoshka embeddings are fundamentally multi-scale linear subspaces rather than isolated point vectors; optimizing vector-level fidelity alone cannot transfer macroscopic information capacity or global subspace orientations across dimensions. Core idea: decompose multi-scale distillation into a dual-level geometric alignment framework that matches spectral capacity via effective-rank (eRank) entropy and aligns global orientations via Grassmannian chordal distances, while anchoring local semantic coordinates and preventing incremental redundancy through orthogonal band regularization.

Method

Overall Architecture

The SUGAR framework compresses a frozen MRL teacher encoder \(f_T\) into a trainable compact student encoder \(f_S\) while preserving the topological structure of its nested subspaces. Given an input sentence \(x\), the teacher and student generate dense representations \(h_T \in \mathbb{R}^{D_T}\) and \(z_S \in \mathbb{R}^{D_S}\). When student dimensionality does not match the teacher (\(D_S \le D_T\)), a trainable linear projection layer \(W \in \mathbb{R}^{D_S \times D_T}\) maps the student embeddings into the teacher's ambient space, yielding \(h_S = z_S W\).

Distillation is computed across a predefined sequence of nested prefix dimensions \(\mathcal{D} = \{d_1, d_2, \dots, d_K\}\) (e.g., \(16, 32, \dots, 768\)). For each prefix slice pair \((s_d, t_d)\), SUGAR applies two complementary levels of geometric alignment: a macroscopic level that enforces intrinsic dimensionality matching through normalized spectral entropy (eRank) and global orientation alignment via Grassmannian manifold chordal distances, paired with a microscopic level that anchors local coordinate fidelity and regularizes consecutive extension bands to remain statistically orthogonal.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Mini-Batch (b)"] --> B["Forward Pass: Teacher Embeddings (b×D) & Student Embeddings (b×d)"]
    B --> C["Linear Projection Layer W: Ambient Space Alignment (b×D)"]
    C --> D["Multi-Scale Prefix Slicing: Extract Prefix Pairs s_d & t_d"]
    D --> E["Effective-Rank Matching: Normalized Spectral Entropy Alignment"]
    E --> F["Grassmannian Subspace Alignment: Thin QR Orthonormal Bases & Chordal Distance"]
    F --> G["Point-wise Cosine Alignment: Local Coordinate Anchoring"]
    G --> H["Orthogonal Band Regularization: Cross-Correlation Frobenius Norm Minimization"]
    H --> I["Unified Multi-Objective Loss: MRL Contrastive Loss & Geometric Distillation"]

Key Designs

1. Effective-Rank Matching: Preserving Spectral Capacity and Intrinsic Dimensionality

Low-capacity student models tend to concentrate variance along a small subset of dominant directions, causing severe underutilization of the available dimensional subspace and leading to anisotropic collapse under prefix truncation. SUGAR remedies this by aligning the effective rank (eRank) between student and teacher batch representations across every prefix dimension. Unlike coordinate regression, effective rank is rotation-invariant and measures the continuous dispersion of variance across the representation space.

For a batch matrix \(H \in \mathbb{R}^{b \times d}\) at prefix dimension \(d\), the eigenvalues of the normalized Gram matrix \(G = \frac{1}{b} H H^\top\) are denoted by \(\{\sigma_k\}\). The normalized spectrum is \(p_k = \frac{\sigma_k}{\sum_j \sigma_j}\), and the effective rank is defined as the exponential of Shannon spectral entropy: $$ \text{eRank}(H) = \exp\left( -\sum_k p_k \log p_k \right) $$ The eRank distillation loss minimizes the L1 distance between the dimension-normalized ranks of student batch \(S_d\) and teacher batch \(T_d\): \(\mathcal{L}_{\text{erank}}^{(d)} = L_1\left( \frac{\text{eRank}(S_d)}{d}, \frac{\text{eRank}(T_d)}{d} \right)\). This continuous regularizer ensures the student mimics the teacher's information dispersion profile and maintains rich capacity across all nested prefixes.

2. Grassmannian Subspace Alignment: Denoised Global Orientation via Chordal Distance

While spectral capacity regularizes intrinsic variance, it is oblivious to the spatial orientation of the underlying manifold. Constraining orientation through rigid point-wise coordinates often overfits low-capacity students to sample-level noise. SUGAR formulates the linear subspace spanned by a batch of embeddings as a point on the Grassmann manifold \(\text{Gr}(r, d)\), aligning the macroscopic orientation of student and teacher subspaces.

To extract robust orthonormal bases, a thin QR decomposition is performed on the transposed batch matrix: \(H^\top = Q R\). To isolate informational directions from numerical noise, only columns of \(Q\) whose corresponding diagonal values satisfy \(|R_{jj}| > 10^{-6}\) are retained, yielding an orthonormal basis \(U \in \mathbb{R}^{d \times r}\) with \(r \le \min(b, d)\). To handle cases where student and teacher batches span subspaces of differing intrinsic dimensionalities (\(r_S \ne r_T\)), SUGAR optimizes a generalized squared chordal distance: $$ \mathcal{L}_{\text{grass}}^{(d)} = r - | U_S^\top U_T |_F^2 $$ where \(r = \min(r_S, r_T)\). The Frobenius norm \(\| U_S^\top U_T \|_F^2\) is analytically identical to the sum of squared cosines of principal angles \(\sum_{i=1}^r \cos^2 \theta_i\), making this loss equivalent to minimizing \(\sum_{i=1}^r \sin^2 \theta_i\). This macroscopic constraint aligns global subspace orientations without overloading student capacity.

3. Orthogonal Band Regularization: Mitigating Inter-Band Semantic Redundancy

Isolated prefix objectives do not explicitly govern the incremental information introduced when expanding from prefix \(d_i\) to \(d_{i+1}\). Without constraints, higher-dimensional bands may duplicate representations already captured by lower prefixes, undermining the compression efficiency of the hierarchy. SUGAR introduces an orthogonal band regularization loss that encourages newly added dimensions to remain uncorrelated with smaller prefixes.

For consecutive dimensions \((d_i, d_{i+1}) \in \mathcal{D}\), the embedding is partitioned into a base prefix \(z_i^{\text{prefix}} = z_{1:d_i} \in \mathbb{R}^{d_i}\) and an extension band \(z_i^{\text{band}} = z_{d_i+1:d_{i+1}} \in \mathbb{R}^{d_{i+1}-d_i}\), with batch matrices \(P_i \in \mathbb{R}^{b \times d_i}\) and \(B_i \in \mathbb{R}^{b \times (d_{i+1}-d_i)}\). After mean-centering and column-wise L2 normalization, the cross-correlation matrix is \(C_i = \bar{P}_i^\top \bar{B}_i\). The regularizer penalizes its normalized squared Frobenius norm across all expansion boundaries: $$ \mathcal{L}{\text{ortho}} = \frac{1}{K-1} \sum \frac{| C_i |}^{K-1F^2}{d_i (d $$ Driving } - d_i)\(C_i \to 0\) forces newly added dimension bands to capture distinct, non-redundant semantic signals, enhancing representational density across the nested hierarchy.

Loss & Training

At the microscopic level, point-wise fidelity is enforced via prefix cosine distance: $$ \mathcal{L}{\text{point-wise}}^{(d)} = \frac{1}{b} \sum) \right) $$ The dimension-specific terms are averaged across all }^b \left( 1 - \cos(s_{d,j}, t_{d,j\(K\) prefix dimensions: \(\mathcal{L}_{\text{obj}} = \frac{1}{K} \sum_{d \in \mathcal{D}} \mathcal{L}_{\text{obj}}^{(d)}\). The unified objective combines the base MRL loss (\(\mathcal{L}_{\text{MRL}}\), instantiated as unsupervised SimCSE contrastive loss across prefixes) with the distillation regularizers: $$ \mathcal{L} = \alpha \mathcal{L}{\text{MRL}} + (1 - \alpha) \left( \mathcal{L}}} + \mathcal{L{\text{erank}} + \mathcal{L} \right) $$ where }} + \mathcal{L}_{\text{ortho}\(\alpha \in [0, 1]\) balances task and distillation objectives (\(\alpha=0.4\) for TinyBERT-6L and BERT-base; \(\alpha=0.5\) for BGE-M3). The student is trained for 5 epochs using AdamW with cosine learning rate scheduling at an initial learning rate of \(2 \times 10^{-5}\) and batch size 16.

Key Experimental Results

Main Results

Evaluations encompass three major NLP task suites: Text Classification (TweetEval, Emotion, Banking77), Natural Language Inference (MRPC, WiC, SciTail), and Semantic Textual Similarity (STS-B, SICK, STS12-16, SickR). Baselines include multi-scale learning algorithms (MRL, ESE) and single-scale knowledge distillation frameworks (SimCSE-unsup, DSKD, CDM, Jasper & Stella, DistillCSE, EMO).

Distillation results for Qwen3-Embedding 0.6B to BERT-base (109M) (Table 3):

Method / Model Dim Emotion (F1) Banking77 (F1) MRPC (Acc) SciTail (Acc) STSB (Spearman) STS15 (Spearman) Overall Avg
Teacher (Qwen3-0.6B) 1024/768 64.13 92.08 74.36 84.76 75.90 79.42 76.06
Student base (no KD) 768 51.08 86.21 68.49 73.84 47.29 60.29 64.03
SimCSE-unsup 768 56.02 89.06 73.91 77.94 69.95 77.06 71.57
DSKD 768 55.10 87.65 71.05 76.20 65.85 72.40 70.04
EMO (SOTA KD) 768 57.85 89.10 73.45 78.20 73.80 78.95 73.63
SUGAR (Truncated) 256 56.59 87.85 73.86 77.03 73.52 79.14 72.93
SUGAR (Truncated) 512 58.56 89.15 74.38 77.00 74.67 79.12 73.80
SUGAR (Full) 768 58.65 89.62 73.80 78.65 74.40 79.25 74.15

Note: All KD baselines are single-scale models evaluated at maximum capacity (\(d=768\)). Truncated SUGAR embeddings at \(d=512\) (and even \(d=256\)) match or exceed full-dimensional state-of-the-art distillation baselines.

Multi-scale performance under aggressive truncation on TinyBERT-6L (Tables 1 & 2 excerpts): - STSB (In-domain): At \(d=16\), SUGAR achieves 59.78 (MRL 60.00 / ESE 57.45); at \(d=32\), SUGAR reaches 67.83 (MRL 63.51 / ESE 60.12, +4.32 / +7.71 gain). - STS13 (Out-of-domain): At \(d=16\), SUGAR attains 62.40 (MRL 55.03 / ESE 53.51, +7.37 / +8.89 gain); at \(d=32\), it reaches 69.54 (MRL 63.48 / ESE 60.46, +6.06 / +9.08 gain). - STS16 (Out-of-domain): At \(d=16\), SUGAR achieves 62.27 (MRL 59.21 / ESE 60.26); at \(d=32\), it reaches 68.10 (MRL 63.45 / ESE 65.63).

Ablation Study

Ablation across distillation components distilling Qwen3-Embedding 4B into BERT-base (Table 4, averaged across all prefix dimensions):

Configuration \(\mathcal{L}_{\text{point-wise}}\) \(\mathcal{L}_{\text{erank}}\) \(\mathcal{L}_{\text{grass}}\) \(\mathcal{L}_{\text{ortho}}\) MRPC (Acc) SciTail (Acc) STS15 (Spearman)
Full SUGAR ✓ ✓ ✓ ✓ 73.66 75.69 76.68
w/o \(\mathcal{L}_{\text{ortho}}\) (No Band Orthogonality) ✓ ✓ ✓ ✗ 72.82 (-0.84) 74.95 (-0.74) 75.73 (-0.95)
w/o \(\mathcal{L}_{\text{grass}}\) (No Grassmannian) ✓ ✓ ✗ ✓ 72.45 (-1.21) 74.68 (-1.01) 75.31 (-1.37)
w/o \(\mathcal{L}_{\text{erank}}\) (No eRank Matching) ✓ ✗ ✓ ✓ 72.14 (-1.52) 74.43 (-1.26) 75.12 (-1.56)
w/o \(\mathcal{L}_{\text{point-wise}}\) (No Point-wise Fidelity) ✗ ✓ ✓ ✓ 71.85 (-1.81) 73.92 (-1.77) 73.54 (-3.14)

Key Findings

  • Every component is essential and mutually reinforcing: Removing point-wise cosine alignment produces the largest performance drop (-3.14 on STS15), demonstrating that local coordinate pinning remains the fundamental anchor. Meanwhile, eliminating macroscopic constraints (\(\mathcal{L}_{\text{erank}}\), \(\mathcal{L}_{\text{grass}}\)) degrades performance by 1.0–1.5 points, proving that balancing spectral entropy and subspace orientation prevents anisotropic collapse.
  • Maximum gains under severe truncation: SUGAR achieves its most dramatic performance improvements over vanilla MRL at low dimensions (\(d=16, 32\)), showing that geometric subspace regularization is vital when representations undergo harsh compression.
  • Scalability across model capacities: The framework generalizes robustly from lightweight TinyBERT-6L up to the large 568M BGE-M3 student (outperforming MRL and ESE on SciTail at \(d=32\) by 85.32 vs 81.74). Upgrading the teacher from 0.6B to 4B consistently boosts downstream quality.

Highlights & Insights

  • Grassmannian Manifold Subspace Distillation: Formulating batch embedding subspaces via thin QR decomposition and measuring distance via generalized chordal distance solves the challenge of unequal intrinsic ranks between student and teacher, aligning global manifold orientations without injecting high-frequency coordinate noise.
  • Dual Anti-Collapse Safeguards: Effective-rank matching enforces variance dispersion across the full dimension spectrum, while orthogonal band regularization suppresses cross-correlation between nested prefix cuts, ensuring each incremental dimension slice contributes genuinely novel semantic information.
  • Zero Inference Overhead: All mathematical complexities—QR decompositions, eigenvalue spectrum computations, and Frobenius norms—are confined to the training objective, preserving seamless, instant prefix truncation during inference.

Limitations & Future Work

  • Dependence on MRL-Trained Teachers: The framework fundamentally requires a high-quality pre-trained MRL teacher; student quality remains structurally bounded by the teacher's geometric properties.
  • Training Computational Footprint: Computing QR decompositions, spectral entropy, and batch-wise cross-correlation matrices introduces moderate training memory and latency overhead (though adding zero cost at inference).
  • Domain Scope: Evaluations are currently focused on English sentence embeddings; extending SUGAR to cross-lingual embedding models, multimodal vision-language representations (e.g., Matryoshka CLIP), and generative LLM hidden states remains open for future research.
  • vs MRL (Kusupati et al., 2022): MRL trains nested representations directly with task or contrastive losses, without transferring knowledge from a pre-trained teacher. SUGAR incorporates spectral and Grassmannian manifold alignment, drastically boosting performance under aggressive prefix cuts.
  • vs Single-Scale KD (EMO, DSKD, DistillCSE): Standard distillation assumes static single-scale representations and degrades when embeddings are truncated. SUGAR formalizes Matryoshka Representation Distillation, aligning macroscopic subspace geometry alongside microscopic coordinate fidelity.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates Matryoshka Representation Distillation for the first time, introducing Grassmannian chordal alignment and spectral entropy matching.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 14 classification, NLI, and STS benchmarks on TinyBERT, BERT-base, and BGE-M3 student backbones.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulations, clear geometric motivations, and lucid prose.
  • Value: ⭐⭐⭐⭐⭐ Highly practical for elastic vector search, cascading retrieval pipelines, and edge-to-cloud embedding deployments.