PyraE2E: Enhancing End-to-End WSI Analysis via Cross-Scale Super-Resolution¶

Conference: ECCV 2026
Paper: ECCV Original
Area: Medical Imaging
Keywords: Whole Slide Image, End-to-End Learning, Super-Resolution, Multiple Instance Learning, Cross-Scale Self-Supervised Learning

TL;DR¶

To resolve the critical dilemma in end-to-end whole slide image (WSI) analysis where computational constraints necessitate low-magnification inputs and random sampling—causing severe morphological loss and sparse supervision—PyraE2E leverages the native multi-resolution WSI pyramid to formulate an auxiliary cross-scale super-resolution task, paired with an embedded cluster-score sampling module and a lightweight partitioned encoder, achieving state-of-the-art diagnostic performance under strictly controlled computational cost.

Background & Motivation¶

Whole slide images (WSIs) span gigapixel scales, yet clinical weak-supervision frameworks typically provide only a single slide-level diagnostic label. The predominant paradigm relies on a two-stage multiple instance learning (MIL) pipeline: a frozen encoder pretrained on natural images (e.g., ImageNet) extracts patch-level feature vectors offline, which are subsequently aggregated into slide-level predictions by a trainable MIL pooling module. However, the persistent domain gap between natural scene distributions and complex histopathology morphologies yields suboptimal patch representations, forming an architectural bottleneck that restricts downstream clinical utility. In theory, end-to-end learning directly aligns the feature extractor with downstream diagnostic objectives to eliminate this domain gap. Yet backpropagating gradients across gigapixel images under practical GPU memory limits introduces formidable computational challenges.

To operate within realistic memory budgets, contemporary end-to-end computational pathology methods commonly resort to two major compromises: restricting inputs to low-magnification small patches (e.g., 5×) with uniform random patch sampling, and relying strictly on sparse slide-level classification labels across deep encoder-aggregator backbones. Low-magnification inputs severely blur high-frequency subcellular structures such as nuclear atypia, chromatin patterns, and fine stromal boundaries, while unguided random sampling frequently overlooks isolated, focal micro-metastatic regions. Although auxiliary self-supervised objectives such as contrastive learning or distillation have been explored, they operate primarily at coarse instance-level or slide-level feature spaces. Conversely, masked autoencoding offers dense pixel-level supervision but incurs prohibitive computational overheads at the WSI scale, preventing its integration into fully end-to-end training loops.

To break this computational deadlock, the authors recognize an intrinsic yet underutilized data structure: histopathology slides are natively stored as multi-resolution image pyramids. A low-resolution (LR) patch at low magnification is computationally lightweight to process but sacrifices fine-grained details, whereas its spatially aligned high-resolution (HR) counterpart at high magnification preserves rich cytological morphology. Rather than paying the prohibitive cost of directly encoding high-magnification inputs, the model can retain low-magnification patches as its sole computational input while employing the naturally aligned high-magnification patches as dense reconstruction targets. Core Idea: turn the intrinsic multi-resolution WSI pyramid into a dense pixel- and frequency-domain super-resolution self-supervision signal, combined with an embedded cluster-score sampling mechanism and a lightweight global–local partitioned encoder to capture high-magnification morphology under low-magnification computational budgets.

Method¶

Overall Architecture¶

The core philosophy of PyraE2E is to strictly restrict the network inputs during both training and inference to low-cost low-magnification (5×) patches, using high-magnification (20×) patches exclusively as self-supervised reconstruction targets during training. The pipeline executes in a closed online loop: first, an embedded Cluster-Score Sampling module efficiently identifies a representative and informative subset of LR patches directly from the low-magnification pyramid level. Second, a lightweight Global–Local Partitioned (GLP) encoder processes the sampled LR patches via partitioned channels to capture both global histological context and local high-frequency textures. The extracted shared feature representations are then routed to dual task heads: a super-resolution (SR) reconstruction branch supervised by joint spatial and Fourier-domain losses against paired HR patches, and a slide-level prediction branch that applies spatial attention pooling before feeding an attention-based MIL classifier. Finally, instance attention scores and feature vectors are dynamically fed back to refresh the online sampling state.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Low-Magnification WSI Pyramid Level (5×)"] --> B["Cluster-Score Sampling<br/>Exploration-Exploitation Dynamic Quota Allocation"]
    B --> C["Global–Local Partitioned (GLP) Encoder<br/>Channel Splitting & Multi-Scale Fusion"]
    C --> D["Cross-Scale Super-Resolution Branch<br/>PixelShuffle Reconstruction of Aligned 20× Patch"]
    C --> E["Slide-Level Prediction Branch<br/>Attention Pooling & MIL Aggregation"]
    D --> F["Pixel L1 Loss + 2D-FFT Frequency Loss"]
    E --> G["Slide Cross-Entropy Loss + Cluster-Wise KL Regularizer"]
    E -.->|Feedback Attention Weights & Instance Embeddings| B

Key Designs¶

1. Embedded Cluster-Score Sampling: Closed-Loop Instance Selection Balancing Exploration and Exploitation

Heuristic patch selection solely driven by raw attention weights is vulnerable to severe confirmation bias, causing sampling distributions to rapidly collapse into a narrow subset of high-scoring patches while starving morphologically diverse tissue regions. Conversely, uniform random sampling frequently fails to capture sparse, focal diagnostic lesions. PyraE2E introduces an embedded online sampling mechanism that mirrors the multi-armed bandit exploration-exploitation trade-off within the end-to-end pipeline without external preprocessing. For a per-slide sampling budget \(K\), the module allocates \(K_e\) slots for uniform random exploration over unseen instances \(U\) and reserves \(K_r = K - K_e\) slots for exploitation over the seen set \(S\). The system dynamically maintains an online state tuple \((a_i, \mathbf{f}_i, s_i, c_i)\) for each instance, recording its attention score, instance feature vector, sampling frequency counter, and semantic cluster assignment. Online \(k\)-means clustering is adaptively executed every \(T_c\) epochs over seen features to refresh cluster identities. During exploitation, cluster quotas \(q_c\) are assigned via a largest-remainder formulation based on both cluster population \(n_c\) and average attention \(\bar{a}_c\). Following a warm-up phase of \(T_w\) epochs to bypass initial attention volatility, instances within each cluster are prioritized by a seen-penalized utility ranking:

\[r_i = \text{Norm}(a_i) \cdot \frac{1}{\sqrt{s_i + 1}}\]

This penalty explicitly suppresses over-sampled redundant patches via the inverse square-root of the visitation counter, forcing the network to discover under-represented diagnostic morphology across tissue phenotypes. Updated features and attention scores smoothly refresh instance states via exponential moving average (EMA), forming an active closed-loop selection policy.

2. Global–Local Partitioned Encoder: Channel-Partitioned Multi-Scale Representation Modeling

Standard heavy vision backbones (e.g., Vision Transformers) incur prohibitive GPU memory activations when backpropagating through long instance chains. PyraE2E introduces the lightweight Global–Local Partitioned (GLP) encoder, which processes shallow convolution features \(F_0\) through \(N=8\) cascaded refinement blocks. Each block couples a Global–Local Fusion Unit (GLFU) and a Partitioned Spatial Mixer (PSM), both protected by residual skip connections. Within the GLFU, feature channels are split into equal global and local sub-streams. The global stream is spatially compressed via strided max-pooling with downsampling factor \(s\), processed by a \(3\times3\) depthwise convolution to capture broad histological architectural context, and dynamically modulated by channel-wise variance gating before bilinear upsampling. The local stream preserves full spatial resolution and passes through a channel-expanding inverted bottleneck convolution to extract cellular contours and nuclear boundaries. Subsequently, the Partitioned Spatial Mixer (PSM) restricts \(3\times3\) spatial convolution mixing to only a subset fraction \(p\) of channels, while leaving the remaining channels untouched for efficient identity propagation. This architectural synergy allows the GLP encoder to attain comprehensive contextual and fine-grained representation capabilities with an exceptionally compact parameter footprint of only 2M parameters.

3. Cross-Scale Super-Resolution Supervision: Dense Pixel- and Frequency-Domain Guidance

To enable low-magnification inputs to internalize high-magnification cytological details, PyraE2E harnesses the spatial alignment of the WSI pyramid. A single 5× low-magnification patch precisely corresponds in physical coordinates to a 20× high-magnification region. The encoder's fused feature map \(\mathbf{z} = F_0 + F_N\) is routed to an SR reconstruction head comprising a \(3\times3\) convolution and a PixelShuffle upsampling layer with scale factor \(r=4\), generating a super-resolved patch \(\hat{\mathbf{x}}^{\text{SR}}\) of size \(2048\times2048\) from a \(512\times512\) input. To prevent standard \(L_1\) pixel regression from generating over-smoothed reconstructions that obscure fine chromatin textures and nuclear margins, supervision is coupled in both spatial and 2D Fast Fourier Transform (FFT) frequency domains:

\[\mathcal{L}_{\text{PIX}} = \|\hat{\mathbf{x}}^{\text{SR}} - \mathbf{x}^{\text{HR}}\|_1, \quad \mathcal{L}_{\text{FFT}} = \|\mathcal{F}(\hat{\mathbf{x}}^{\text{SR}}) - \mathcal{F}(\mathbf{x}^{\text{HR}})\|_1\]

The frequency-domain constraint penalizes spectral discrepancies across high-frequency bands, compelling the low-magnification encoder to encode fine-grained cytological markers directly into its latent representations. High-magnification targets are accessed strictly during training; test-time inference requires only low-magnification inputs, imposing zero extra runtime overhead.

Loss & Training¶

The overall training loss optimizes classification and reconstruction branches jointly in an end-to-end fashion:

\[\mathcal{L}_{\text{total}} = \alpha \mathcal{L}_{\text{PIX}} + \beta \mathcal{L}_{\text{FFT}} + \gamma \mathcal{L}_{\text{CE}} + \delta \mathcal{L}_{\text{KLD}}\]

where \(\mathcal{L}_{\text{CE}}\) denotes slide-level cross-entropy classification loss, and \(\mathcal{L}_{\text{KLD}}\) is a cluster-wise Kullback-Leibler divergence regularizer that encourages balanced attention weight distributions across instances within the same semantic cluster, directly preventing attention collapse. The framework is trained using loss weights \((\alpha, \beta, \gamma, \delta) = (1.0, 0.05, 0.5, 0.5)\). Each slide samples \(K=160\) low-resolution patches (5×) per epoch, with a warm-up period \(T_w=30\) epochs and clustering refresh interval \(T_c=10\) epochs. The number of clusters \(c \in \{2, 4, 6, 8\}\) adapts dynamically based on cumulative seen instance counts.

Key Experimental Results¶

Main Results¶

PyraE2E was thoroughly evaluated on three public cancer WSI diagnostic benchmarks from The Cancer Genome Atlas: TCGA-RCC (renal cell carcinoma subtyping, 3 classes), TCGA-NSCLC (non-small cell lung cancer subtyping, 2 classes), and TCGA-BRCA (breast cancer subtyping, 2 classes), benchmarked against conventional two-stage MIL pipelines, foundation models, self-supervised methods, and end-to-end architectures.

Encoder	Method	E2E	TCGA-RCC ACC (%)	TCGA-RCC AUC (%)	TCGA-NSCLC ACC (%)	TCGA-NSCLC AUC (%)	TCGA-BRCA ACC (%)	TCGA-BRCA AUC (%)
ResNet	ABMIL	✗	89.1 ± 0.7	98.4 ± 0.4	88.3 ± 2.1	95.8 ± 1.4	88.9 ± 1.5	92.0 ± 3.3
ResNet	CLAM	✗	90.5 ± 0.9	98.7 ± 0.1	89.9 ± 2.1	95.5 ± 2.2	88.7 ± 1.2	93.5 ± 1.7
ResNet	TransMIL	✗	90.2 ± 0.9	97.9 ± 0.5	85.7 ± 1.9	93.7 ± 1.3	87.9 ± 1.8	91.9 ± 0.8
ResNet	2D Mamba	✗	90.6 ± 1.5	98.3 ± 0.9	89.6 ± 1.7	96.2 ± 1.3	90.1 ± 1.7	94.0 ± 1.4
ResNet	SimCLR-X	✗	90.2 ± 1.7	97.7 ± 0.3	90.1 ± 2.2	96.7 ± 1.3	89.2 ± 2.3	92.9 ± 1.6
ResNet	C2C	✓	90.3 ± 0.7	98.0 ± 0.2	88.8 ± 1.7	94.8 ± 1.1	88.0 ± 1.9	88.6 ± 0.8
ResNet	Streaming	✓	90.2 ± 1.1	97.9 ± 0.2	90.2 ± 1.1	96.2 ± 0.4	88.0 ± 1.1	88.0 ± 2.9
ResNet	ABMILX (E2E)	✓	91.3 ± 1.3	98.1 ± 0.1	90.2 ± 1.0	96.3 ± 0.3	87.0 ± 3.9	90.1 ± 4.5
UNI (FM)	ABMILX	✗	91.5 ± 0.9	99.0 ± 0.2	90.1 ± 2.6	96.1 ± 2.1	93.2 ± 0.8	95.1 ± 0.5
CONCH (FM)	ABMILX	✗	92.8 ± 0.6	99.3 ± 0.3	91.8 ± 1.0	97.0 ± 0.6	92.0 ± 0.3	94.7 ± 0.9
GLP (Ours)	PyraE2E-L	✓	90.6 ± 2.0	97.8 ± 0.6	91.4 ± 0.8	97.5 ± 0.7	90.0 ± 0.7	92.5 ± 1.0
GLP (Ours)	PyraE2E-X	✓	93.1 ± 1.2	98.3 ± 0.2	92.0 ± 1.4	97.6 ± 0.4	90.5 ± 1.3	93.5 ± 0.7

Ablation Study¶

Ablations on TCGA-NSCLC using the ABMILX aggregator decouple the individual contributions of sampling strategies and loss components:

Variant / Module	Cluster Guide	Attn. Guide	Embedded (Emb.)	Loss Weights \((\alpha, \beta, \gamma, \delta)\)	ACC (%)	AUC (%)	Overhead per Epoch (Time)
Random Sampling Baseline	✗	✗	✓	\((1.0, 0.05, 0.5, 0.5)\)	89.9 ± 0.8	96.3 ± 1.3	0.0 m
Cluster Only (Embedded)	✓	✗	✓	\((1.0, 0.05, 0.5, 0.5)\)	90.7 ± 0.9	97.0 ± 0.7	+3.3 m
External Cluster (C2C style)	✓	✗	✗	\((1.0, 0.05, 0.5, 0.5)\)	90.5 ± 0.7	96.9 ± 0.4	+8.1 m
External Attn. (CDSR style)	✗	✓	✗	\((1.0, 0.05, 0.5, 0.5)\)	91.1 ± 1.0	97.1 ± 0.6	+9.6 m
Full Sampling (Default)	✓	✓	✓	\((1.0, 0.05, 0.5, 0.5)\)	92.0 ± 1.4	97.6 ± 0.4	+4.2 m
Half SR Weight (\(0.5\times \text{SR}\))	✓	✓	✓	\((0.5, 0.025, 0.5, 0.5)\)	91.2 ± 1.3	97.2 ± 0.6	-
Double SR Weight (\(2\times \text{SR}\))	✓	✓	✓	\((2.0, 0.10, 0.5, 0.5)\)	91.8 ± 1.2	97.5 ± 0.4	-
Without Frequency Loss (w/o FFT)	✓	✓	✓	\((1.0, 0.0, 0.5, 0.5)\)	86.2 ± 1.8	93.6 ± 0.9	-
Without Attention Regularizer (w/o KLD)	✓	✓	✓	\((1.0, 0.05, 0.5, 0.0)\)	90.5 ± 1.4	96.2 ± 0.6	-

In computational efficiency assessments conducted on a single NVIDIA A800 GPU, PyraE2E achieves superior balance: requiring only 2M parameters (versus 307M for UNI, 90M for CONCH, and 21M–28M for ResNet baselines), executing inference in 3.1 s/slide (compared to 17.0 s for UNI and 7.8 s for CONCH), and completing training in 1.1 days without extensive multi-node GPU pretraining. Super-resolution image quality across the \(5\times \to 20\times\) task attains \(25.1\text{ dB}\) PSNR and \(0.67\) SSIM on TCGA-NSCLC.

Key Findings¶

The 2D-FFT frequency loss \(\mathcal{L}_{\text{FFT}}\) is paramount for high-level classification accuracy. Removing the frequency loss causes ACC to plunge from 92.0% to 86.2% and AUC to drop by 4.0%, proving that spatial \(L_1\) loss alone suffers from spectral smoothing, whereas Fourier constraints force the backbone to preserve crucial high-frequency histological features.
Embedded cluster-score sampling delivers both accuracy and runtime advantages over external pre-computed sampling pipelines. While external clustering (+8.1 min/epoch) and external attention (+9.6 min/epoch) incur heavy computational latency, PyraE2E's embedded module incurs only +4.2 min/epoch while boosting ACC to 92.0%.
Loss sensitivity experiments across a \(0.5\times \sim 2.0\times\) range of super-resolution weights demonstrate steady AUCs (97.2% to 97.6%), confirming that the cross-scale reconstruction objective acts as a robust regularizer rather than destabilizing task-specific classification gradients.

Highlights & Insights¶

Zero-Annotation Multi-Scale Self-Supervision: Exploits the innate multi-resolution pyramid structure of digital slide files as free pixel-level supervision, circumventing manual pixel annotations and resolving the sparse supervision bottleneck of end-to-end WSI learning.
Compute-Bounded High-Resolution Representation: Feeds exclusively low-magnification small patches into the encoder to keep memory and runtime footprint strictly bounded, while enabling a tiny 2M-parameter backbone to capture cellular-level morphology comparable to heavy pathology foundation models.
Closed-Loop Anti-Collapse Sampling: Seamlessly incorporates multi-armed bandit exploration-exploitation principles with online clustering and seen-penalized utility scoring, maintaining broad tissue diversity while concentrating capacity on diagnostic lesions.

Limitations & Future Work¶

Dependence on Pyramid Alignment Integrity: The cross-scale pixel and frequency supervision hinges upon exact spatial correspondence across magnification layers. In legacy scanners or heavily compressed formats with spatial interpolation drift, reconstruction targets may introduce spatial artifacts.
Exploration of Larger Magnification Jumps: Current experiments focus on \(5\times \to 20\times\) (\(4\times\) upscaling). Extending this paradigm to larger cross-scale steps (e.g., \(2.5\times \to 40\times\)) or progressive continuous pyramids could unlock full-slide context modeling with minimal patch fragmentation.
Evaluation on Subcellular Molecular Biomarkers: Evaluations have centered primarily on anatomical cancer subtyping. Future work should evaluate its effectiveness on fine-grained diagnostic targets such as microvascular invasion, spatial gene expression profiling, and patient survival analysis.

vs C2C / Streaming (Standard End-to-End WSI Baselines): Prior end-to-end works primarily explore gradient checkpointing or memory streaming under sparse slide labels, while suffering from severe representation degradation due to low-magnification inputs. PyraE2E resolves this fundamental information bottleneck by coupling active online sampling with dense cross-scale super-resolution self-supervision.
vs UNI / CONCH (Pathology Foundation Models): Large foundation models require training on millions of tissue patches across heavy compute clusters and typically operate as static, frozen feature extractors during downstream tasks. PyraE2E demonstrates that a compact 2M end-to-end model trained on target datasets can achieve competitive classification performance at a fraction of the parameter and inference cost.
vs CDSR (Super-Resolution for Histopathology): Prior SR applications treat resolution enhancement as an offline pre-processing pipeline or cascaded feature generator. PyraE2E uniquely embeds super-resolution as an auxiliary multi-task training regularizer, enhancing backbone representations during backpropagation while discarding the SR head at test time to incur zero extra latency.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Creatively transforms the native multi-resolution WSI pyramid into an auxiliary cross-scale super-resolution objective to conquer end-to-end computational bottlenecks.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three TCGA benchmarks with exhaustive ablations on sampling mechanisms, loss components, runtime efficiency, and reconstruction fidelity.
Writing Quality: ⭐⭐⭐⭐⭐ The narrative is clear, technically cohesive, and supported by rigorous math and intuitive diagrams.
Value: ⭐⭐⭐⭐⭐ Establishes a highly practical and efficient design pattern for resource-constrained, high-accuracy gigapixel pathology modeling.