Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://huggingface.co/SchuefflerLab/Pathryoshka-B
Area: Model Compression
Keywords: Multi-teacher knowledge distillation, nested embeddings, computational pathology, feature alignment, cropping augmentation

TL;DR¶

Pathryoshka compresses global and spatial knowledge from three large pathology foundation models into a prefix-truncatable student, achieving a median accuracy of 85.6 across ten classification benchmarks with 86M parameters while reducing encoding and feature-storage costs.

Background & Motivation¶

Computational pathology often divides a whole-slide image into numerous tiles, encodes them with a foundation model, and then trains a classifier or slide-level aggregator. Deployment cost therefore depends not only on parameter count per prediction, but also on how many tiles must be encoded and how often their high-dimensional representations are stored and accessed. Virchow2, UNI2-h, and H-optimus-1 provide strong representations, but their hundreds of millions to billions of parameters and wide embeddings increase both computation and storage demands. Replacing the backbone with a small model does not guarantee preservation of its morphological knowledge; shortening features alone does not automatically reduce backbone forward-pass cost.

Knowledge distillation and Matryoshka representation learning address these two different problems. The former teaches a small student to imitate a large teacher, while the latter makes a long vector's prefixes meaningful after truncation, but conventional approaches usually address only one side. A single-teacher student such as H0-mini may also inherit weaknesses on particular tissue types, whereas different pathology teachers have different strengths. GPFM demonstrates the potential of combining teachers, but does not make substantial capacity reduction together with variable-length embeddings its central objective.

This paper treats every short prefix as a representation that must receive distillation supervision, instead of deleting dimensions arbitrarily after training. Training signals come from teachers' CLS tokens and spatial patch tokens, without requiring the DINO/iBOT prototype heads often omitted from public checkpoints. Existing pathology models can thus serve as fixed knowledge sources while deployment retains only one compact student. Core Idea: compress model capacity through multi-teacher distillation and representation length through multilevel prefix supervision, while separating spatially aligned and non-aligned crops to preserve local morphology and global robustness.

Method¶

Overall Architecture¶

The input consists of unlabeled H&E pathology tiles, each producing a shared spatial crop and model-specific independent spatial crops during training. The student and three frozen teachers encode these views into global CLS tokens and local patch tokens. Student representations are sliced into nested prefixes and mapped into each teacher's feature space by teacher-specific heads, with joint global and spatial distillation objectives. The final output is the student's own representation, not a concatenation of teacher features; downstream applications can select either the full vector or a shorter prefix.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Unlabeled H&E tile"] --> Crop["Dual-view cropping"]
    Crop --> Student["Student encoder<br/>CLS and patch tokens"]
    Crop --> Teachers["Frozen teachers<br/>Virchow2 / UNI2-h<br/>H-optimus-1"]
    Student --> Nested["Nested prefix supervision"]
    Nested --> Align["Multi-teacher dual-branch alignment"]
    Teachers -.->|Training targets only| Align
    Align -.->|Update student and heads| Student
    Nested -->|Select prefix at inference time| Output["Classification / retrieval / segmentation"]

The alignment heads and teachers belong to the training branch, rather than being additional models required for downstream deployment. The nested structure does not dynamically remove Transformer layers: short prefixes mainly save storage and downstream processing, while encoding savings come from the smaller student backbone. The paper primarily reports Pathryoshka-B and also evaluates the smaller Pathryoshka-S, with different capacity and computation trade-offs.

Key Designs¶

1. Dual-view cropping: apply spatial supervision only to corresponding regions

A shared spatial crop is first generated from the original tile, so the student and all teachers receive content with the same location and geometric transformation. Each model then receives independent H&E stain perturbations and visual augmentations such as blur, allowing the same tissue structure to have different color appearances. Both CLS and patch tokens are compared here because spatial locations still correspond, avoiding supervision that treats different tissues as the same region. Each crop covers 25% to 100% of the original tile area, uses an aspect ratio from [0.9, 1.1], and includes random horizontal and vertical flips. The result is resized to 224×224 pixels, so changing the crop extent also changes the relative scale of tissue in the input.

Each model additionally generates another independent spatial crop with visual augmentation. These views may cover different regions, so only their global CLS tokens are aligned, without enforcing positionwise patch correspondence. The intuition is to preserve tissue-level semantics as the observed region and apparent magnification change, rather than demand pixelwise invariance of local texture. The contribution is therefore not simply more random cropping, but coupling the applicability of each loss to whether the views are spatially aligned. This rule has a boundary: in heterogeneous tiles, whether different regions should share similar global representations still depends on their morphological composition.

2. Nested prefix supervision: short vectors carry a complete semantic objective during training

Ordinary distillation only asks the full student vector to match its teacher, allowing information to spread across all dimensions without guarantees after truncation. Pathryoshka-B supervises the full representation and progressively halved prefixes; Section 3.2 gives the following training dimensions:

\[ \mathcal{M}=\{768,384,192,96,48\}. \]

These are neither five independent encoders nor five disjoint vector segments, but five lengths sharing the beginning of the same vector. The earliest dimensions participate in objectives at every length, while later dimensions serve only longer representations, encouraging prefixes to become independently useful. For every teacher and prefix length, separate CLS and patch projection heads avoid forcing different input widths through one fixed-width head. All prefixes align to the teacher's full target space; teacher features are not truncated to the same length. Short student representations must therefore retain enough information for their training-time heads to recover relevant structure in the teacher representation.

After training, selecting a 48-dimensional prefix requires neither backbone retraining nor a teacher forward pass. However, explicit training down to 48 dimensions does not guarantee reliability at every shorter length; results at 12 dimensions and other stronger truncations are empirical evaluations. Likewise, this particular 768-dimensional nesting set belongs to the detailed B-model description and should not be assigned to the S model without evidence. The contribution is to make the compression budget part of representation training, not to claim that every downstream task can use the shortest prefix losslessly.

3. Multi-teacher dual-branch alignment: combine global semantics with local morphology

The teachers are Virchow2, UNI2-h, and H-optimus-1, trained on data from different institutions, with no single teacher winning every task in Table 1. On 10,000 training images, the authors report teacher-pair CKA values of 0.64 to 0.71 as evidence that their representations are related but not identical. This does not establish that every teacher improves every task, but supports selecting sources with both compatible and complementary information. All teachers remain frozen while the student and projection heads are updated, so the large pathology models do not need to be reoptimized.

Each projection head is a three-layer MLP whose input width matches the student prefix and whose output width matches the target teacher embedding. The intermediate layer also uses the teacher embedding width, avoiding another narrow hidden bottleneck beyond the student prefix itself. The CLS branch uses a cosine similarity loss for tile-level semantic direction, while the patch branch uses mean squared error for spatial representations. Before the patch loss is computed, each teacher's patch features are standardized using current-batch channel means and standard deviations to mitigate numerical scale differences. Teachers receive equal weights; the main text does not use a gate that dynamically estimates teacher reliability by task or sample. Complementary knowledge is therefore combined through a shared student and multiple alignment objectives, not teacher voting at inference time.

A Worked Example¶

Consider a 448×448-pixel tile extracted from an H&E slide containing glands and surrounding connective tissue. A shared crop might retain a gland region that all models observe under different stain perturbations, activating both global and local losses. Independent crops can change the proportion of glands and background; training then aligns only CLS semantics instead of imposing incorrect local correspondence. The student's 768-dimensional representation receives supervision from three teachers through heads for its 768-, 384-, 192-, 96-, and 48-dimensional prefixes. For patch retrieval at deployment, the system can store 48-dimensional prefixes and compare queries and database patches in the same feature space without storing teacher outputs. At the same numerical precision and excluding indexing overhead, a 48-dimensional vector occupies 1/16 of the raw storage of a 768-dimensional vector; this is a dimensional relationship, not an end-to-end system measurement. For slide-level classification, tile CLS features still require a downstream aggregator such as ABMIL; the student does not directly turn a whole slide into a diagnosis.

Loss & Training¶

The identifiable core of Equation (8) is a sum over teachers and prefix lengths, retained below with consistent typesetting:

\[ \mathcal{L}_{\mathrm{cls}}=\sum_t\sum_{m\in\mathcal{M}}\mathcal{L}_{\mathrm{cos}}\!\left(g_s^{(t,m)}(z_{s,1:m}),z_s^{(t)}\right). \]

The student CLS prefix passes through its teacher- and scale-specific head before comparison with the teacher CLS; the spatial branch uses MSE against standardized patch targets. Losses on shared and independent views follow the earlier correspondence rule; spatial loss must not be applied indiscriminately to every crop. Several cached equations contain displaced characters, especially the standardization and patch-loss expressions, so guessed repairs are not presented as exact author equations. The main text also does not fully specify total-loss coefficients, optimizer, learning rate, or training epochs, and these hyperparameters are not invented here.

The unlabeled training set contains 158,233 H&E whole-slide images from one institution and one scanner, yielding 243 million tiles. The authors identify tissue with CLAM and sample at most 2,250 tiles of 448×448 pixels per slide, stopping earlier when insufficient tissue is available. Sampling proportions at 10×, 20×, and 40× magnification are 20%, 40%, and 40%, respectively, to cover different morphological scales. Stain augmentation perturbs hematoxylin and eosin channels separately, alongside flipping and Gaussian blur, to mitigate the narrow training source. Public datasets are explicitly excluded from student training to avoid merely memorizing teachers' representations of already-seen evaluation images. This strengthens the student evaluation but does not automatically rule out overlap between test data and the pretraining data of every teacher or baseline.

Key Experimental Results¶

Main Results¶

Table 1 (page 10) uses default eva settings: a single linear layer for eight patch tasks and ABMIL for two slide tasks, all using the corresponding model's CLS features. Values are means over five runs with the variation reported in the source table; multiclass tasks use classification accuracy and binary tasks use balanced accuracy, in percent. The following excerpt retains key model columns; the median is taken over ten dataset scores, not over per-dataset improvements.

Dataset	UNI2-h	H0-mini	GPFM	Pathryoshka-B
BACH	90.7 ±1.1	79.2 ±0.5	83.0† ±0.4	90.8 ±0.3
BRACS	66.1 ±0.9	60.6 ±0.6	62.8† ±0.5	65.3 ±0.2
BreakHis	85.9 ±0.3	74.7 ±0.9	76.0† ±0.5	87.1 ±0.7
CRC	96.5 ±0.2	96.1 ±0.2	95.2† ±0.1	96.7 ±0.1
Gleason	77.4 ±0.3	77.9 ±0.5	76.6 ±0.6	78.7 ±0.6
MHIST	82.6 ±0.3	79.0 ±0.2	81.4 ±0.1	84.0 ±0.2
Patch Camelyon	95.1 ±0.1	94.2 ±0.1	94.4 ±0.0	94.1 ±0.1
CAMELYON16	93.8 ±1.3	93.1 ±1.7	94.9† ±2.6	95.0 ±1.0
PANDA	76.8 ±1.1	78.1 ±0.4	79.1† ±0.9	77.0 ±0.8
UniToPatho	54.0 ±0.3	49.2 ±0.3	49.7† ±0.1	54.9 ±0.1
Median	84.3	78.6	80.3	85.6

† indicates that the corresponding model trained on that dataset, potentially overstating generalization. Pathryoshka-B's median exceeds H0-mini and GPFM by 7.0 and 5.3 percentage points, respectively, but it does not win every task. For example, it trails UNI2-h on BRACS and Patch Camelyon, and trails H0-mini and GPFM on PANDA.

Table 2 (page 11) measures efficiency on an RTX 3090 with FP16 and batch size 32, reporting throughput mean and standard deviation over 500 batches.

Model	Parameters M	FLOPs G	Throughput img/s
H-optimus-1	1,135	591.8	82 ±1
UNI2-h	681	360.7	136 ±1
H0-mini	86	44.6	842 ±4
GPFM	303	155.6	278 ±1
Pathryoshka-S	22	11.1	2338 ±14
Pathryoshka-B	86	44.6	842 ±4

Parameter counts follow Table 2; Figure 1 contains different labels such as 87M, which are not mixed into exact comparisons. The prose describes roughly 11 times the throughput of H-optimus-1, but dividing the tabulated 842 by 82 gives approximately 10.3; these are not the same exact numerical claim.

Ablation Study¶

Table 3 (page 12) compares the B model with and without the proposed random-cropping strategy under the classification protocol; representative tasks and full-benchmark aggregates are retained below.

Dataset or aggregate	With cropping	Without cropping
BACH	90.8 ±0.3	90.0 ±0.4
BreakHis	87.1 ±0.7	85.5 ±0.6
MHIST	84.0 ±0.2	85.1 ±0.2
CAMELYON16	95.0 ±1.0	91.6 ±1.5
PANDA	77.0 ±0.8	77.0 ±1.2
Ten-task average	82.4	81.6
Ten-task median	85.6	85.3

Cropping improves average accuracy by 0.8 percentage points and CAMELYON16 by 3.4 points, but reduces MHIST by 1.1 points. The prose states a median improvement of 0.2, whereas the displayed Table 3 values differ by 0.3; this note preserves the original table and flags the inconsistency without correcting it.

Key Findings¶

Nested representations benefit more than classification. Table 5 (page 15) reports Pathryoshka-B ConSep Dice of 63.6 ±0.3 at full width and 62.6 ±0.2 at 1/16 width; MoNuSAC changes from 63.8 ±0.7 to 61.3 ±0.8.
These are declines of 1.0 and 2.5 Dice points, respectively; the prose's 1.6% and 3.9% are relative decreases, not absolute percentage-point losses.
Section 4.2 evaluates six multiclass patch datasets with k-NN using k=10 and cosine similarity: the student uses prefixes, while baselines randomly sample dimensions over five runs. This is not a comparison against dedicated compression methods such as PCA.
Page 11 reports a one-sided Wilcoxon test against UNI2-h with p=0.042, but statistical superiority across ten benchmarks does not establish superiority for every clinical task or patient population.

Highlights & Insights¶

Model size and feature size are separately controllable deployment costs. The framework addresses both without treating storage savings from short vectors as backbone encoding acceleration.
Whether crops are aligned determines whether local supervision is valid, which matters more than simply strengthening augmentation. Image distillation with changing views likewise requires checking the supervision granularity first.
Teacher-specific heads accommodate knowledge with different dimensions and distributions in one student objective. Their benefit is removing the teacher ensemble from deployment, not guaranteeing complete preservation of every teacher's knowledge.

Limitations & Future Work¶

The authors acknowledge the need for broader clinical utility evaluation. Public classification, retrieval, and segmentation results do not replace prospective clinical validation, calibration, or real workflow assessment.
The private training set is large but comes from one institution and one scanner. Stain and scale augmentation can mitigate appearance changes without establishing coverage of all cross-hospital domain shifts.
Student inference throughput does not represent the total cost of multi-teacher training. Missing training budgets and complete hyperparameters also limit independent reproduction from the main text.
The cache does not include cited supplementary Tables S4 and S6, preventing verification of detailed low-dimensional classification curves and single-teacher ablation numbers; no strongest-teacher contribution is inferred from them.
Reader recommendation: compare learned dimensionality reduction, alternative teacher combinations, and external institutions to disentangle nested supervision, teacher diversity, and training-data scale.

Versus H0-mini / DINOv2 distillation: the method targets available CLS and patch tokens instead of requiring teachers' DINO/iBOT prototype heads, while adding multiple teachers and truncatable prefixes.
Versus GPFM: both combine pathology teacher knowledge, but this paper more explicitly targets student capacity reduction and flexible feature width; differences under distinct training data cannot all be attributed to one component.
Versus AM-RADIO: the approach builds on multi-teacher representation aggregation and combines it with pathology view augmentation and nested prefix training, extending aggregation toward deployable compression.
Versus Matryoshka / Franca: nested representations provide part of the foundation, but here multilevel objectives connect to teacher-specific global and spatial heads rather than merely training a low-dimensional downstream classifier.
Transferable direction: when domain teachers expose only intermediate features, multilevel prefix distillation could support a single budget-adjustable encoder. This is a research direction suggested by the structure, not a validated cross-domain result.

Rating¶

Novelty: 4/5. The combination of multiple teachers, nested supervision, and crop-dependent supervision has a clear purpose, with innovation primarily in integration and adaptation.
Experimental Thoroughness: 4/5. Multiple downstream tasks and efficiency metrics are covered, but clinical validation and verifiable training details remain incomplete.
Writing Quality: 3/5. The main argument and ablations are clear, but some numerical claims differ between prose and tables, and cached equations are damaged.
Value: 4/5. The method is useful for large-scale pathology tile encoding and feature storage, with benefits depending on deployment budgets and target tasks.