Skip to content

Scaling Whole-Slide Pathology Foundation Model Pretraining with Billions Off-the-Shelf Tokens

Conference: ECCV 2026
Paper: ECCV Official Link
Area: Medical Imaging
Keywords: Computational Pathology, Whole Slide Image (WSI), Foundation Model, Self-Supervised Learning, Token Retention

TL;DR

To break the dilemma between information collapse in tile-level [CLS] tokens and prohibitive quadratic compute of all spatial tokens in gigapixel pathology, this paper introduces a training-free greedy diversity retention module (TokRet) and a block-sparse long-context pretraining framework (PathTokScale), scaling pretraining to 2 billion tokens with only 10% of the WSI volume required by prior SOTA foundation models.

Background & Motivation

Whole slide image (WSI) analysis is foundational for computational pathology, supporting clinical diagnostics, tumor grading, patient survival analysis, and biomarker discovery. However, individual WSIs exhibit gigapixel resolutions (billions of pixels per slide) alongside limited annotations, spurring the widespread adoption of weakly supervised multiple instance learning (MIL) and self-supervised foundation models (FMs). Established frameworks (e.g., UNI, Virchow, GigaPath, and TITAN) partition slides into tiles, run pretrained vision Transformers (ViTs) on patch tokens, and universally extract the task-agnostic [CLS] token as the summary embedding for each tile to maintain computational feasibility.

Unfortunately, exclusive reliance on [CLS] tokens introduces a severe representation bottleneck. Recent empirical findings reveal that scaling tile-level ViTs to larger architectures yields only marginal gains, and naively concatenating [CLS] with average-pooled spatial tokens fails to recover subtle, localized pathological morphologies. Pathology diagnosis fundamentally hinges on localized morphological patterns and tissue heterogeneity that are easily smoothed out by global pooling. While keeping all spatial tokens across a slide would preserve these crucial details, the quadratic attention complexity of Transformers renders slide-level sequence lengths (hundreds of thousands of tokens) computationally prohibitive. Crucially, neighboring spatial tokens within local tiles exhibit substantial semantic redundancy, indicating that a carefully selected sparse subset can retain nearly the entire feature distribution.

Departing from the conventional paradigm of blindly expanding slide volume or parameter counts, this paper investigates scalability along the effective token dimension. Core idea: formulate intra-tile spatial token retention as a combinatorial Max-Min Diversity Problem, solve it via a training-free greedy selection module (TokRet), and scale slide-level self-supervised pretraining to two billion tokens using a block-sparse long-context Transformer with 2D-RoPE.

Method

Overall Architecture

The PathTokScale pipeline comprises hierarchical WSI partitioning, training-free greedy spatial token retention, block-routed long-context Transformer encoding, and region-level multi-view self-distillation (DINOv2). WSIs are first decomposed into non-overlapping macro-regions (\(7168 \times 7168\) pixels) and tiles (\(448 \times 448\) pixels), with patch ViTs generating spatial tokens. TokRet then prunes redundant spatial tokens within each tile to preserve the top 10% most diverse and representative tokens. Finally, an efficient Transformer with coarse-to-fine block attention models long-range contextual relationships across tens of thousands of tokens under a self-supervised objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["WSI Input<br/>Hierarchical partitioning into regions & tiles"] --> B["Tile ViT Encoding<br/>Extract all spatial patch tokens"]
    B --> C["TokRet Greedy Diversity Retention<br/>Iterative Max-Min distance subset selection"]
    C --> D["Block-Sparse Routing Transformer<br/>Coarse block scoring + Top-M fine attention"]
    D --> E["DINOv2 Multi-View Self-Distillation<br/>Student-teacher alignment on augmented views"]
    E --> F["Downstream Adaptation<br/>Subtyping / Survival / Gene Mutation"]

Key Designs

1. TokRet Greedy Token Retention: Training-Free Compression via Combinatorial Diversity Rather than relying on heuristic self-attention scores that fluctuate across layers and tasks, TokRet formulates intra-tile token retention as the Max-Min Diversity Problem (MMDP) from combinatorial optimization. Given a candidate token set \(T\) with size \(m\), the objective is to find a subset \(\tilde{T}\) of target size \(\tilde{m}\) (retaining ~10% of tokens) that maximizes the minimum pairwise cosine distance among selected tokens:

\[\tilde{T} = \arg\max_{C \subset T, |C|=\tilde{m}} \min_{\gamma, \omega \in C} d(\gamma, \omega)\]

where \(d(\gamma, \omega)\) denotes cosine distance. To avoid combinatorial explosion, TokRet adopts an efficient greedy approximation: pairwise token distances are precomputed once via matrix multiplication. The process seeds \(\tilde{T}\) with the token that maximizes average distance to all others, guaranteeing a globally distinct starting point. In each subsequent step, it identifies the remaining candidate that maximizes the minimum distance to the tokens already selected into \(\tilde{T}\), continuing until \(|\tilde{T}| = \tilde{m}\). This guarantees that the retained tokens span the peripheral boundary of the original embedding distribution, effectively preserving rare morphological variants without requiring gradient fine-tuning.

2. Block-Routed Sparse Long-Context Encoding: Eliminating Quadratic Complexity for 200k Tokens Even after TokRet compresses each tile by 90%, an entire WSI generates 100k to 200k retained tokens, exceeding standard Transformer context windows. PathTokScale incorporates a block-sparse attention architecture inspired by Mixture-of-Block-Attention (MoBA). The retained token sequence is partitioned into non-overlapping blocks \(\{B_1, \dots, B_K\}\) of size 512. For each query token \(q_i\), the model first computes a coarse relevance score against each block's mean key representation:

\[\tilde{s}_{i,k} = \mathbf{q}_i^\top \text{mean}(\mathbf{K}_k)\]

The router selects only the top \(M\) most relevant blocks (with \(M=4\)) to participate in full token-level scaled dot-product attention, lowering attention complexity to near-linear. Furthermore, two-dimensional Rotary Position Embeddings (2D RoPE) encode relative spatial coordinates. All retained tokens belonging to the same tile share identical coordinate indices, letting the frozen tile ViT capture internal patch geometry while RoPE allows seamless length extrapolation from \(7168 \times 7168\) pretraining regions to arbitrary whole-slide dimensions during downstream inference.

3. PathTokScale Self-Supervised Pretraining: Billion-Token Slide-Level Self-Distillation At the slide and region levels, the architecture is pretrained using DINOv2 self-distillation. Two augmented views \(T_r^{(1)}\) and \(T_r^{(2)}\) are generated in token space via random cropping, horizontal flipping, and feature-space Gaussian perturbations. These views are processed by a student network and an exponential-moving-average (EMA) momentum teacher network, training the student to match the teacher's output distribution. By extracting informative spatial tokens from the open-source HistAI cohort and ~10k private slides, PathTokScale scales pretraining to over 2 billion tokens—a 20-fold increase in effective tokens over existing WSI slide-level models—without requiring massive proprietary slide collections.

Key Experimental Results

Main Results

On slide-level classification (BRACS tumor subtyping, BCNB ER biomarker prediction, TCGA-LUAD TP53 mutation prediction) and disease-specific survival analysis across five TCGA cohorts, PathTokScale outperforms conventional MIL, modern state-space Mamba baselines, and multi-hundred-thousand-slide foundation models.

Table 1: Slide-Level Classification Performance (Extracted from Table 1 of the original paper)

Method BRACS (F1) BRACS (AUC) BCNB ER (F1) BCNB ER (AUC) TCGA-LUAD TP53 (F1) TCGA-LUAD TP53 (AUC)
ABMIL (UNI-1) 0.692±0.03 0.875±0.02 0.754±0.08 0.857±0.07 0.646±0.04 0.701±0.07
TransMIL (UNI-1) 0.592±0.03 0.859±0.02 0.690±0.06 0.810±0.08 0.608±0.04 0.674±0.08
GMMamba (UNI-1) 0.670±0.04 0.893±0.03 0.784±0.08 0.873±0.07 0.678±0.04 0.734±0.05
PathVQ (UNI-1) 0.730±0.02 0.902±0.01 0.782±0.07 0.872±0.05 0.684±0.03 0.733±0.06
GigaPath 0.677±0.03 0.862±0.03 0.791±0.07 0.875±0.06 0.670±0.04 0.738±0.06
TITAN 0.696±0.04 0.891±0.01 0.813±0.06 0.905±0.05 0.702±0.03 0.765±0.06
UNI-2 (ABMIL) 0.698±0.03 0.887±0.02 0.770±0.07 0.861±0.06 0.661±0.03 0.729±0.06
Ours + UNI-1 0.779±0.03 0.914±0.03 0.801±0.05 0.890±0.06 0.714±0.03 0.763±0.04
Ours + Conch-1.5 0.767±0.05 0.910±0.05 0.809±0.07 0.898±0.05 0.725±0.05 0.771±0.03

Table 2: Patient Disease-Specific Survival Prediction (C-Index, Extracted from Table 2 of the original paper)

Model Category / Name BRCA CRC BLCA UCEC KIRC Average C-Index
ABMIL (UNI-1) 0.633±0.06 0.612±0.08 0.540±0.07 0.671±0.08 0.691±0.08 0.629
TransMIL (UNI-1) 0.612±0.07 0.684±0.06 0.595±0.06 0.695±0.08 0.671±0.10 0.651
PANTHER (UNI-1) 0.758±0.06 0.665±0.10 0.612±0.07 0.757±0.10 0.716±0.10 0.702
CHIEF 0.737±0.04 0.680±0.08 0.599±0.02 0.758±0.10 0.736±0.06 0.702
GigaPath (170k+ slides) 0.687±0.08 0.628±0.08 0.589±0.05 0.779±0.10 0.751±0.07 0.687
TITAN (330k+ slides) 0.713±0.04 0.710±0.11 0.657±0.05 0.789±0.09 0.774±0.06 0.729
Ours + UNI-1 0.712±0.07 0.699±0.09 0.635±0.04 0.769±0.06 0.767±0.09 0.716
Ours + Conch-1.5 0.717±0.04 0.719±0.07 0.655±0.05 0.784±0.08 0.778±0.07 0.731

Ablation Study & Key Findings

Comprehensive ablations analyze token selection methods, retention capacity \(K\), and pretraining token scaling: - Comparison of Token Selection Strategies (Fig. 3a): On BRACS tumor subtyping, vanilla [CLS] reaches ~0.70 accuracy; [CLS] concatenated with average pooling yields ~0.70; spatial average pooling achieves 0.68 (\(4 \times 4\)) and 0.70 (\(7 \times 7\)); attention-magnitude Top-16 and Top-64 reach 0.72 and 0.74 respectively. In contrast, TokRet Top-20 scores 0.76 and TokRet Top-32 achieves ~0.77, proving that maximum diversity selection captures distribution extremes far more effectively than heuristics or averaging. - Sensitivity on Retention Number \(K\) (Fig. 4): Severe pruning (\(K \le 4\), with Top-2 scoring 0.64 and Top-4 scoring 0.66) falls behind the [CLS] baseline, as extreme sparsity disrupts spatial structural coherence. Performance scales rapidly once \(K \ge 8\) and reaches an optimal trade-off plateau at \(K=20 \sim 32\). - Pretraining Token Scaling Law (Fig. 3b): Increasing pretraining token counts from 0.1B ([CLS] baseline, ~0.70 on BRACS) to 0.8B (~0.74), 1.6B (~0.77), and 2.0B (~0.79) produces monotonic performance gains, empirically confirming a scaling law across token capacity in computational pathology.

Highlights & Insights

  • Formulating Token Selection as Combinatorial Diversity: Casting patch token pruning as a Max-Min Diversity Problem avoids gradient backpropagation and attention-score drift, achieving robust, parameter-free feature coverage.
  • Scaling Tokens Over Slide Collections: Demonstrates that extracting off-the-shelf spatial tokens from ~30k publicly available slides matches or exceeds foundation models trained on 170k–330k slides, breaking the bottleneck of scarce medical data.
  • Hardware-Friendly Sparse Long-Context Design: Integrating block-routed sparse attention with 2D-RoPE allows a lightweight 6-layer Transformer to process up to 200k tokens per slide with near-linear complexity.

Limitations & Future Work

  • Shared Intra-Tile Positional Coordinates: All retained tokens originating from a single tile share the same \((x, y)\) coordinate, relying on ViT feature representations to implicitly preserve intra-tile spatial arrangements. Finer hierarchical coordinate modeling could be explored.
  • Pairwise Distance Matrix Footprint: Computing the initial \(m \times m\) pairwise distance matrix across tens of thousands of tiles can introduce transient memory peaks during large-batch preprocessing.
  • Absence of Multimodal Alignment: The framework focuses exclusively on self-supervised vision tokens; integrating retained spatial tokens with diagnostic text reports represents a promising direction for pathology VLMs.
  • vs GigaPath / TITAN: Both GigaPath and TITAN rely on massive proprietary data (170k and 330k slides) to train slide-level Transformers over tile [CLS] tokens. PathTokScale demonstrates that the primary bottleneck lies in [CLS] information collapse rather than slide volume; by mining spatial tokens, PathTokScale achieves superior or on-par performance using only 10% of their slide data.
  • vs PathVQ: PathVQ attempts to retain spatial details by compressing feature dimensions via discrete vector quantization (VQ-VAE), which requires dedicated training and introduces reconstruction loss. TokRet uses training-free diversity retention on uncompressed representations, preserving high-fidelity features while achieving better downstream accuracy and efficiency.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Introduces combinatorial max-min diversity optimization to spatial token retention in pathology without heuristic tuning]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across diagnostic subtyping, biomarker prediction, gene mutation, and 5 cancer survival cohorts with complete token scaling analyses]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, compelling empirical motivation, and elegant architectural design]
  • Value: ⭐⭐⭐⭐⭐ [Offers a highly practical paradigm for scaling pathology foundation models under limited data budgets]