Rank-Aware Hyperbolic Alignment for Vision–Language Dataset Distillation¶

Conference: ECCV 2026
arXiv: 2606.29464
Code: Yes (Project Page, repository link in original paper)
Area: Multimodal VLM / Dataset Distillation
Keywords: Vision-Language Dataset Distillation, Hyperbolic Geometry, Rank-Aware Alignment, Subspace Decomposition, Optimal Transport

TL;DR¶

Lifts vision-language features to the Lorentz hyperbolic space for contrastive alignment (hITC), decomposes the cross-covariance of real batches via SVD into range (dominant shared directions) and residual (weakly coupled directions) subspaces based on cumulative energy, and matches correlation distributions using Sinkhorn optimal transport with asymmetric regularization. This selectively aligns highly informative shared structures while preserving modality-specific diversity under extreme compression budgets, achieving superior cross-architecture transferability and robustness.

Background & Motivation¶

Vision-language models rely on massive image-text pairs and contrastive objectives. However, as dataset scales grow from millions to billions, concerns regarding privacy, licensing, provenance, and poisoning increasingly restrict how models can be constructed, audited, shared, and deployed. Dataset Distillation (DD) offers a pragmatic alternative: synthesizing a tiny fraction of data to approximate the training signal of the large-scale dataset. This saves computational resources and serves as an auditable proxy when sharing the original pairs is constrained. However, extending distillation from single-modality to image-text pairs is more challenging as the distilled small set must preserve both intra-modality diversity and cross-modal relative ranking structures (i.e., retrieval relationships). Existing VLDD methods generally fall into three categories: trajectory matching (MTT-VL, LoRS, RepBlend), which requires storing expert trajectories at a high cost and inherits teacher architecture biases; generative methods (EDGE), which synthesize data using diffusion priors but lose direct control over the alignment structure; and distribution matching (CovMatch), which efficiently aligns cross-modal second-order moments but still exerts nearly uniform alignment pressure across all feature directions in Euclidean space.

The core problem lies in this "one-size-fits-all" alignment. Vision-language correlations are typically low-rank: a compact shared subspace carries the dominant semantics and coarse-grained relations (which indeed should be cross-modally aligned), while the remaining directions form a weakly correlated residual component that absorbs modality-specific cues, annotation artifacts, and noisy variations. Under tight compression budgets, forcing the alignment of these residual directions suppresses complementary information and harms transferability. Although LoRS relaxes similarity obligations via low-rank decomposition at the similarity level, it does not explicitly control how alignment capacity and structures are allocated. Another overlooked aspect is that multimodal semantics are naturally hierarchical (entity \(\to\) attribute \(\to\) relation). Euclidean geometry provides little inductive bias for such nested structures, particularly when only a tiny set of distilled samples is tasked with carrying the training signals.

This work addresses both issues simultaneously. Core Idea: Lift image-text representations to hyperbolic space to leverage its hierarchical inductive bias, and apply a 'range-residual' rank-aware decomposition on the cross-covariance of real batches. Geodesic alignment is strictly enforced only within the energy-dominant shared range subspace, while the residual subspace is regularized to preserve modality-specific diversity. This explicitly controls what to align and how to align, focusing the limited synthesis capacity on the most valuable structures for alignment.

Method¶

Overall Architecture¶

RAHA is a trajectory-free, geometry-aware vision-language dataset distillation framework. The inputs are large-scale real image-text pairs, and the outputs are a tiny set of differentiable synthetic pairs (images as learnable pixel tensors, texts as learnable input-level token embeddings with fixed masks). During training, the pre-trained encoders are frozen, and only the synthetic data itself is updated via gradients. Each distillation step jointly optimizes two objectives: first, a hyperbolic contrastive loss (hITC) is computed on synthetic pairs to maintain image-text discriminability within the synthetic set; second, rank-aware correlation distillation is applied to transfer the cross-modal relative ranking structure observed in real batches to the synthetic set via range-residual subspace decomposition and optimal transport.

The exact pipeline follows a structured workflow: projected features are first lifted to the Lorentz hyperboloid via the exponential map (where hITC is defined), and then pulled back to the tangent space at the origin via the logarithmic map (performing linear algebra in the tangent space ensures stability). In the tangent space, the cross-covariance of the real batch is computed and decomposed via SVD. A rank \(k\) is adaptively selected based on a cumulative energy threshold \(\rho\), splitting the projected features into range coordinates and residual components. Respectively, similarity matrices are constructed for the two subspaces and converted into row-wise correlation distributions. Entropic regularized Sinkhorn is used to find soft couplings and compute real-to-synthetic matching losses, combined with asymmetric energy regularizations. The final training objective consists of three terms: hITC, range, and residual.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real Image-Text Pairs + Learnable Synthetic Pairs"] --> B["Hyperbolic Lift + hITC<br/>Lorentz Exponential Map<br/>Geodesic Contrastive Alignment for Synthetic Pairs"]
    B --> C["Tangent Space Rank-Aware Decomposition<br/>Covariance SVD with top-k Selection based on Energy"]
    C -->|Dominant Shared Directions| D["Range Correlation Distillation<br/>Sinkhorn OT Matching + Energy Lower Bound Regularization"]
    C -->|Weakly Coupled Complement| E["Residual Correlation Distillation<br/>Sinkhorn OT Matching + Compression Regularization"]
    D --> F["Joint Optimization of Three Loss Terms to Update Synthetic Data"]
    E --> F

Key Designs¶

1. Hyperbolic Contrastive Alignment (hITC): Harnessing Negative Curvature Geometry for Hierarchical Semantics

The limitation of Euclidean contrastive learning (dot product/cosine InfoNCE) is straightforward: it pulls all matching pairs together and pushes all mismatching pairs apart without directional preference, failing to preserve hierarchical variations (e.g., captions being more abstract than images). RAHA addresses this by replacing the standard InfoNCE similarity with the geodesic distance on the Lorentz hyperboloid. The intuition behind using hyperbolic space is not that vision-language data forms a literal tree, but that negative curvature geometry naturally positions generalized concepts closer to the origin while scattering concrete instances outward (echoing MERU's observation that textual concepts are often more general than images). This allows the model to accommodate coarse-to-fine nested relationships with minimal distortion—an inductive bias that Euclidean geometry cannot provide.

Mechanistically, the projected features \(z^v, z^t\) are first lifted to the hyperboloid via the exponential map (utilizing radial scaling with \(\sinh\)) to obtain \(h^v, h^t\) (where curvature \(c\) and scale \(s\) control deformation, collapsing back to Euclidean space as \(c \to 0\)). The geodesic distance between two points is given by the Lorentzian inner product \(\langle u, v\rangle_{\mathcal{L}} = -u_0 v_0 + \bar{u}^\top \bar{v}\):

\[d_c(u,v)=\tfrac{1}{\sqrt c}\,\mathrm{arcosh}\!\big(-c\,\langle u,v\rangle_{\mathcal L}\big)\]

Taking the logits as the negative geodesic distance divided by the temperature \(\tau=0.07\), and computing the bidirectional symmetric cross-entropy yields the hITC loss \(\mathcal{L}_{\mathrm{hITC}}\) (evaluated only on synthetic pairs, without accessing real data). Ablation studies show that hITC alone serves as a strong baseline, demonstrating that 'geodesic InfoNCE' is highly effective for distilling retrieval-capable alignments.

2. Range–Residual Rank-Aware Subspace Decomposition: Unlinking Directional Alignment Demands

This is the core of rank-awareness, directly targeting the limitation where Euclidean methods indiscriminately align all directions. First, hyperbolic features are mapped back to the tangent space at the origin via the logarithmic map (enabling standard linear algebra operations). The cross-covariance \(C_{\mathrm{real}}\in\mathbb{R}^{d\times d}\) of image-text features is calculated for real batches, where each singular value \(\sigma_i\) measures the cross-modal coupling strength along that direction. Crucially, instead of manually setting the rank, the minimum rank \(k\) is adaptively selected once the cumulative squared energy first reaches the threshold \(\rho\) (default 0.95):

\[k=\min\Big\{k':\ \tfrac{\sum_{i=1}^{k'}\sigma_i^2}{\sum_{i=1}^{d}\sigma_i^2}\ge\rho\Big}\]

The top \(k\) left and right singular vectors \(U_k, V_k\) span the range subspace (the dominant shared coupling directions for image and text), onto which tangent features are projected to obtain range coordinates; subtracting the range projection from the original features yields the residual components. This basis is computed from the real batch and applied to both real and synthetic features. This is effective because the range subspace carries the most robust retrieval signals, while the residual subspace absorbs low-energy, unstable, and modality-specific elements. Separating them under tight budgets ensures synthesis capacity is prioritized for the shared structures rather than diluted by weakly correlated directions. The authors verify that \(\rho=0.95\) is optimal (using 1.0 degrades performance by introducing low-energy residual noise), and the selected rank stabilizes within dataset-specific bounds rather than reaching the algebraic upper limit \(\mathrm{rank}(C_{\mathrm{real}})\le B-1\).

3. Sinkhorn OT-based Correlation Distillation + Asymmetric Subspace Regularization: Transferring Real Relative Rankings to the Synthetic Set

While hITC ensures alignment within individual synthetic pairs, it cannot transfer the relative ranking structures inherent in real data. RAHA constructs cross-modal similarity matrices in both range and residual subspaces, which are converted into row-wise probability distributions using correlation temperature \(\tau_r\) (with real features treated as stop-gradient targets). Since there is no predefined one-to-one correspondence between synthetic and real rows, KL divergence is used to construct a cost matrix \(\Gamma\). An entropic regularized optimal transport problem is solved to obtain the soft coupling \(T\) (via Sinkhorn-Knopp iterations), where the transport-weighted cost defines the unidirectional matching loss, averaged bidirectionally to get \(\mathcal{L}_{\mathrm{match}}\). This bypassed the pseudo-correspondence problem of pairing specific synthetic samples to real ones through set-level soft matching.

The regularization for the two subspaces is asymmetric and bidirectional, acting as the pivot for 'selective alignment'. The range side employs an energy lower-bound regularization: it penalizes only when synthetic energy falls below real energy, forcing the synthetic set to preserve at least as much coupling energy as the real data in the top-k subspace, preventing the collapse of dominant shared components:

\[\mathcal L_{\mathrm{reg}}^{\mathrm{range}}=\frac{\max(0,\ e_{\mathrm{real}}-e_{\mathrm{syn}})}{e_{\mathrm{real}}+\epsilon}\]

In contrast, the residual side is subject to a compression regularization: denoting the residual energy ratio as \(r = e_{\mathrm{res}}/(e_{\mathrm{syn}}+\epsilon)\), it uses \(\mathcal{L}_{\mathrm{reg}}^{\mathrm{residual}} = r + \max(0, \, r - 1)\)—where the first term continuously squeezes \(r\) toward zero, and the second term adds an extra penalty if residual energy exceeds range energy (\(r > 1\)). In short, the range is guaranteed not to collapse, while the residual is aggressively compressed to prevent it from dominating. This aligns with the design objective of preserving shared structures while keeping weakly coupled directions in check. Ablations reveal that the residual branch alone is fragile and only functions when anchored on the range branch under compression constraints.

Loss & Training¶

The total loss is a weighted sum of three terms: \(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{hITC}} + \lambda_{\mathrm{range}}\mathcal{L}_{\mathrm{range}} + \lambda_{\mathrm{residual}}\mathcal{L}_{\mathrm{residual}}\), where the range loss is matching + energy lower-bound regularization, and the residual loss is matching + \(\lambda_{\mathrm{comp}} \cdot\) compression regularization. Default hyperparameters are set to \(\lambda_{\mathrm{range}}=0.8\), \(\lambda_{\mathrm{residual}}=0.4\), \(\lambda_{\mathrm{comp}}=0.1\), curvature \(c=1\), scale \(s=1\), temperatures \(\tau=\tau_r=0.07\), energy threshold \(\rho=0.95\), and Sinkhorn parameter \(\varepsilon=0.05\) with 20 iterations. Training follows the online distillation protocol of CovMatch: only synthetic parameters are updated, and encoders are reset to pre-trained weights in each outer loop. The outer loop (default 50 steps) performs gradient updates on the synthetic path, while the inner loop (default 1 step) updates the encoder on real data, allowing the surrogate model's latent geometry to drift slowly to force the synthetic data to generalize across encoder configurations. Numerically, jitter is added to stabilize potentially pathological SVD, and the Sinkhorn cost matrix is mean-normalized to stabilize the effective regularization strength.

Key Experimental Results¶

Main Results¶

Evaluated on three image-text retrieval benchmarks (Flickr8k / Flickr30k / COCO, Karpathy split) using an NFNet + BERT backbone. Distillation budgets \(N \in \{100, 200, 500\}\) represent extreme compression (\(N=100\) is less than 1% of the COCO training set). The metric is the average of bidirectional Recall@{1, 5, 10} (IR/TR/Mean). The table below extracts a representative comparison of Mean (average of IR and TR):

Dataset / Budget	Random	LoRS	CovMatch	RAHA (Ours)	Note
Flickr8k / 100	5.7	9.4	20.4	20.4	Ties with CovMatch at the minimum budget
Flickr8k / 500	15.4	13.5	25.9	30.7	Outperforms clearly as budget increases
Flickr30k / 100	8.6	10.2	22.8	20.7	Slightly underperforms CovMatch at small budget
Flickr30k / 500	22.6	10.9	28.9	32.9	Leads at larger budgets
COCO / 200	5.6	1.7	8.3	10.2	Outperforms
COCO / 500	10.1	4.8	11.2	13.7	Outperforms

The authors honestly conclude that RAHA ties with the strongest distribution matching baseline, CovMatch, at 100 pairs, and consistently outperforms it at 200/500 pairs, without claiming dominance across every extreme compression setting. The appendix further verifies that this advantage scales up at a larger budget (1000 pairs) and on the larger, noisier CC3M-595K-LLaVA dataset (500 pairs, Mean: 5.1 \(\to\) 8.1), completely outperforming the generative baseline EDGE at 1000 pairs. The reason is explicitly discussed: dataset-scale semantics can be efficiently decomposed into range/residual bases, but when there are too few synthetic samples, they cannot fully populate (or "fill up") these semantic patterns, where MTT-based methods might still excel. As the budget increases, RAHA successfully materializes these patterns.

Ablation Study¶

Using Flickr8k \(N=100\) (Full configuration Mean=20.4) as the example:

Configuration	Key Metric (Mean)	Note
Full model	20.4	Full model (hITC + range + residual + regularization)
hITC only	Strong baseline	Geodesic InfoNCE alone is already a strong baseline (Fig.2)
hITC + range	Largest single-component gain	range is the primary carrier of cross-modal coupling
residual only	Weak baseline	Unreliable matching without being anchored on range
Fully Euclidean (eITC + Euclidean matching)	1.4/3.0/2.2	Subspace decomposition almost fails in Euclidean space
Euclidean ITC + Hyperbolic matching (eITC)	18.1/21.7/19.9	Restores most of the performance
Fully Hyperbolic (hITC)	19.0/21.9/20.4	Further improves performance

Key Findings¶

The range branch contributes the most: Adding the range component to hITC yields the largest single-component improvement, proving that the range subspace is the main carrier of cross-modal coupling. The residual branch alone is weak and only brings gains when anchored on the range and subjected to compression regularization—i.e., 'the residual must be explicitly controlled rather than optimized independently'.
Hyperbolic lifting is necessary for subspace decomposition to work: The geometric ablation in Fig.A6(d) is highly telling—placing both contrastive learning and matching in Euclidean space using the same Sinkhorn pipeline yields only a 2.2 Mean. Simply shifting matching to the hyperbolic space (eITC) recovers it to 19.9, and the fully hyperbolic setup (hITC) reaches 20.4. This indicates that the improvement does not stem from simple scalar weights or single loss terms, but from the synergistic interaction between 'selective range-residual supervision \(\times\) hyperbolic geometry'.
Cross-architecture generalization and robustness are where RAHA truly shines (Table 5, Flickr8k): At \(N=200\), average transferability improves from 7.2 to 8.7, consistent across BERT/DistilBERT and various vision backbones; at \(N=500\), transferability reaches 12.7 vs. CovMatch's 8.7. This supports the design intention—distilling correlations inside the rank-adaptive shared range while regulating the residual reduces overfitting to a single encoder's geometry.
Robustness does not consistently improve at the highest budget: Transferability increases strongly with \(N\), but robustness does not necessarily improve at the highest budget. The authors faithfully state this non-monotonic trend.
Qualitatively cleaner results: Images distilled by CovMatch often contain high-frequency artifacts or banding patterns (relying on structured noise to satisfy second-order alignment), whereas RAHA produces cleaner textures and more natural edges, with fewer instances of caption drift to incorrect scenes. The hyperbolic hierarchical bias benefits fine-grained classification the most (CUB +1.9pp, Cars +2.17pp).
Honest disclosure of computational cost: Each distillation step is more expensive than CovMatch (approx. 400s vs. 55s per batch of 64, bottlenecked by the SVD of \(d \times d\) covariance and Sinkhorn iterations, scaling with batch size rather than total synthetic samples; peak VRAM is comparable at ~9.3GB). However, the authors emphasize this as a one-time offline cost, which avoids storing ~18GB of expert checkpoints required by trajectory matching. It can be optimized using randomized/truncated SVD, cached bases, or warm-start Sinkhorn.

Highlights & Insights¶

Making the 'allocation of alignment capacity' an explicitly optimizable geometric problem: The bidirectional asymmetric regularizations—range 'guaranteed not to collapse' + residual 'compressed once sufficient'—is the most elegant design of the paper. It turns 'what direction to align and what to ignore' from an implicit byproduct into an explicit control knob, advancing beyond LoRS which only relaxes similarity soft constraints.
Adaptive rank selection instead of manual \(k\) setting: Ranking based on a cumulative energy threshold \(\rho\) allows the rank to adapt to different data/batches while remaining stable, avoiding the most tedious hyperparameter of low-rank methods. This 'energy-based rank selection' trick is highly transferable to any representation learning scenario that requires separating dominant and residual subspaces.
Explaining modality gap not as pure misalignment through hyperbolic geometry: It allows structured radial modality disparities (texts are more general \(\to\) closer to the origin), treating the modality gap as an abstract hierarchical discrepancy rather than an error to eliminate. The triangle inequality of 'radial difference \(\Delta r \le\) hyperbolic distance of matching pairs' presented in Fig.A5 is a remarkably clean geometric diagnostic of whether pair semantics sit at compatible depths.
Reusable paradigm: The round-trip 'lifting to hyperbolic space \(\to\) pulling back to tangent space for linear algebra (SVD/projection) \(\to\) pushing back to alignment objectives' offers a highly portable template for tasks that want to leverage non-Euclidean geometries but are tightly coupled with standard linear algebra techniques.

Limitations & Future Work¶

Heavy computational overhead: Each distillation step is nearly an order of magnitude more expensive than Euclidean statistical matching (SVD + Sinkhorn scaling with batch size). Although it is a one-time offline cost, the authors acknowledge that truncated/randomized SVD, cached bases, and warm-start Sinkhorn are needed to alleviate this—currently a trade-off for carrying 'structure awareness'.
Dependency on hierarchical structure assumptions: RAHA's rank-aware prior works best when shared signals concentrate in dominant subspaces. If helpful signals are scattered across many weak directions, over-compressing the residual may prune task-relevant information, resulting in limited gains on weakly hierarchical datasets (explicitly listed as a limitation).
Bounded by teacher encoders: As with all multimodal distillation, performance is capped by the representation capacity of pre-trained encoders, and degrades under domain drift or noisy captions.
Suboptimal performance at extremely small budgets: At 100 pairs, it merely ties with CovMatch and even slightly underperforms on Flickr30k. Having too few samples to fill out the decomposed structures is a genuine weakness; its advantages only emerge at 200+ pairs.
No guarantee of fairness: Since the range retains dominant directions, harmful correlations could be preserved if they happen to lie within the dominant singular directions. The authors list 'structure-aware distillation + explicit bias auditing' as a future direction.

vs. CovMatch (Strongest distribution-matching baseline): Both perform trajectory-free distribution matching and jointly train the text encoder. However, CovMatch directly aligns cross-covariance and feature statistics in Euclidean space, applying uniform pressure to all directions, which is highly effective under minimal budgets. RAHA shifts this to the hyperbolic space + range/residual decomposition with row-wise correlation distribution matching, explicitly controlling the allocation of alignment capacity. The cost is a slower training speed, but it yields superior transferability/robustness and cleaner synthetic images, with the advantage scaling up as budget increases.
vs. LoRS (Low-rank similarity): LoRS also identifies the low-rank nature of vision-language correlations, but only handles low-rank decomposition at the similarity matrix level, without explicitly separating dominant shared directions from weak residual directions or incorporating geometric/hierarchical biases. RAHA elevates the low-rank concept into a representation-space subspace decomposition combined with hyperbolic geometry, controlling the alignment structure itself rather than mere similarity values.
vs. MTT-VL / RepBlend (Trajectory matching): These rely on matching training trajectories, requiring the storage of massive expert checkpoints (~18GB) and inheriting teacher architecture biases. RAHA is trajectory-free, saves storage, and dominates at medium-to-large budgets. RepBlend's solution to modality collapse is orthogonal to RAHA and could in principle be combined.
vs. EDGE (Generative): EDGE utilizes Stable Diffusion priors to synthesize images along with generating discrete captions, trading direct control over alignment structures for scalability. RAHA does not rely on pre-trained generative models and instead focuses on retrieval-relevant shared structures, outperforming EDGE across all three datasets at the 1000-pair budget.
vs. Hyperbolic frameworks like HDD / MERU: HDD focuses on single-modality hyperbolic centroid matching, and MERU reveals interpretable radial layouts on the Lorentz model. RAHA is the first to introduce explicit range-residual decomposition inside the hyperbolic space to implement 'selective alignment of shared directions + suppression of uninformative residuals' in extreme-compression image-text distillation.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The combination of hyperbolic geometry, rank-aware range/residual decomposition, and asymmetric regularization explicitly objectifies 'alignment capacity allocation' as an optimizable goal, presenting a highly novel perspective.
Experimental Thoroughness: ⭐⭐⭐⭐ Coverage spans across three benchmarks \(\times\) multiple budgets, alongside cross-architecture, robustness, classification, cost, and reproducibility analyses. The geometric ablation is exceptionally clean. However, it does not dominate at small budgets, and experiments use the classic NFNet+BERT configuration instead of stronger contemporary VLMs.
Writing Quality: ⭐⭐⭐⭐⭐ Clear correspondence between the dual motivations (low-rank and hierarchical structures) and the designs. The appendix honestly documents the bidirectional regularization, the hyperbolic round-trip, actual costs, and reproducibility.
Value: ⭐⭐⭐⭐ Offers a geometry-aware path toward data-efficient and auditable VLM training. The 'energy-threshold rank selection + dominant/residual bidirectional regularization' is highly transferable, yet the computational overhead and dependency on strong hierarchical assumptions limit its generic applicability.