Verifying Cancer Segmentation in Vision Transformers via Internal Concepts

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/deep-real/CancerSegFailure
Area: Medical Imaging
Keywords: cancer segmentation, sparse autoencoders, mechanistic interpretability, internal concepts, feeling of error

TL;DR

By demonstrating that Vision Transformers exhibit a latent "Feeling of Error" (characterized by fewer active internal concepts with weaker magnitudes during prediction failures), this paper extracts hierarchical Sparse Autoencoder concepts across network depths to achieve highly accurate, interpretable failure detection and patch-level false positive correction without sacrificing segmentation quality.

Background & Motivation

In clinical oncology, Vision Transformers (ViTs, such as MedSAM) have shown exceptional promise for automated tumor and organ delineation. However, their clinical integration is constantly hindered by the risk of silent failure: even high-performing models frequently generate anatomically plausible yet erroneous masks that risk missed lesions or unnecessary biopsies. Radiologists, operating under intense clinical workloads, cannot feasibly inspect and verify every automated segmentation mask by hand. Conventional failure detection relies heavily on output-level confidence proxies—such as maximum softmax probability, predictive entropy, or energy scores. Yet these output-level statistics suffer from two fundamental bottlenecks: first, they are uninterpretable and fail to explain why an error occurred; second, enforcing failure rejection via output thresholds leads to a severe sensitivity–quality tradeoff, where catching more failures inadvertently flags borderline, clinically acceptable predictions, causing drastic drops in overall Dice Similarity Coefficients (e.g., a 20-point gain in failure detection F1 incurs an 8–10 point drop in segmentation DSC).

This empirical dilemma raises a fundamental question: Do AI models internally "know" when they are wrong? In cognitive psychology, humans possess an intuitive "Feeling of Error" (FOE)—a spontaneous sense of cognitive uneasiness arising from conflict detection during reasoning that flags potential mistakes. When investigating the inner representations of ViT segmentation backbones, the authors discovered a striking latent signature: when segmentation fails, the model internally activates significantly fewer interpretable concepts with markedly lower activation magnitudes compared to successful cases (\(p < 10^{-5}\), Cohen's \(d = 1.8\)). This indicates that an analogous FOE is indeed naturally encoded within the model's latent processing states, even when output probabilities appear deceptively confident.

The paper's angle of attack is to bypass black-box output logits and inspect the internal reasoning of ViT models directly. Using Sparse Autoencoders (SAEs), dense neural activations are decomposed into an overcomplete dictionary of disentangled, clinically grounded concepts across representational depths. The core idea is to extract hierarchical internal concepts across ViT layers via patch-level Sparse Autoencoders, capture non-linear failure interaction patterns with an interpretable classifier, and selectively prune false-positive patches to reconcile failure detection with high segmentation accuracy.

Method

Overall Architecture

The proposed framework decouples failure diagnosis into two coordinated stages: multi-layer internal concept extraction and classifier-driven failure diagnosis. Given an input medical image (e.g., prostate MRI or pancreatic CT), it is passed through a frozen ViT segmentation backbone (e.g., MedSAM). Rather than pooling features across the whole image, patch-level latent activations are extracted from early, middle, and deep Transformer layers. Independent Sparse Autoencoders project these dense representations into sparse concept activation vectors. These layer-wise vectors are concatenated into a holistic failure representation and fed into a lightweight classifier (XGBoost), which outputs both image-level failure risk and patch-level diagnostic importance. Finally, a patch retention mechanism leverages the local failure scores to selectively prune false-positive mask regions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Medical Image<br/>MRI / CT slice"] --> B["Frozen ViT Backbone<br/>Extract Multi-layer Patch Embeddings"]
    B --> C["Patch-level Hierarchical Concept Extraction<br/>BatchTopK SAEs across Shallow / Mid / Deep Layers"]
    C --> D["Internal Concept Interaction Classifier<br/>XGBoost for Failure Prediction & Attribution"]
    D --> E["Patch-level Selective Failure Correction<br/>Threshold-guided Rejection of False Positive Patches"]
    E --> F["Auditable Segmentation Mask & Explanations"]

Key Designs

1. Patch-level Hierarchical Concept Extraction: Preserving Spatial Granularity Across Layers

Because cancer segmentation failures can stem from diverse factors spanning low-level visual artifacts (motion, noise, poor contrast) to high-level anatomical confusion (ambiguous organ margins, morphological irregularities), the model samples representations across the entire backbone depth: \(L = \{1, 3, 5, 7, 9, 11\}\). Unlike classification models that pool tokens into a global representation, segmentation is a dense pixel-level task where spatial structure is essential. Therefore, independent Sparse Autoencoders are trained directly on local patch embeddings \(H_l(x) \in \mathbb{R}^{h \times w \times d}\).

Each SAE encoder \(E_l: \mathbb{R}^d \to \mathbb{R}^D\) transforms dense embeddings into an overcomplete dictionary (\(D = 1536 > d\)). Concatenating sparse concept activations across sampled layers constructs a holistic failure representation: $\(c(x) = \bigoplus_{l \in L} z_l(x) = \bigoplus_{l \in L} E_l(h_l(x)) \in \mathbb{R}^{|L| \cdot D}\)$ Expert radiologist evaluation confirmed that these unsupervised SAE concepts align closely with genuine clinical structures (e.g., peripheral zone, transition zone, gland margin, and tumor core/edge). In failure cases, tumor-specific concept activations vanish or drop substantially, providing a mechanistic fingerprint of model uncertainty.

2. Internal Concept Interaction Classifier: Learning Non-linear Signatures and Attribution

A single isolated concept is often insufficient to flag a failure, as clinical errors typically arise from conflicting concept co-activations (for instance, an abnormal intensity concept firing without supporting gland margin or tumor boundaries). The authors construct a failure dataset \(\mathcal{D} = \{(c_i, y_i)\}_{i=1}^N\) based on ground-truth Dice thresholds (\(y_i = 1\) when \(\text{Dice} \le \tau\)) and train a gradient-boosted decision tree (XGBoost) \(g: \mathbb{R}^{|L| \cdot D} \to [0, 1]\).

XGBoost is uniquely suited for this setting: it excels with high-dimensional sparse inputs, natively captures complex non-linear concept interactions across different layers, and outputs intrinsic feature importance scores \(w \in \mathbb{R}^{|L| \cdot D}\). By projecting top importance scores back onto corresponding SAE concepts and their spatial heatmaps, the system provides transparent diagnostic justifications to clinicians, indicating exactly which anatomical concepts triggered the failure alarm.

3. Patch-level Selective Failure Correction: Circumventing the Sensitivity-Quality Tradeoff

Conventional confidence-based rejection methods discard the entire predicted mask once an image is flagged as faulty, which irreversibly eliminates true-positive regions and causes overall Dice scores to plummet. Because the proposed framework operates on localized patch concepts, failure correction is cast as a patch retention mechanism.

For each spatial patch \((i, j)\) in the initial predicted mask \(\hat{m}\), its local failure probability \(g(c(x)_{ij})\) is evaluated against a rejection threshold \(\eta\): $\(\hat{m}_{\text{corrected}}(i, j) = \hat{m}(i, j) \cdot \mathbf{1}\left[ g(c(x)_{ij}) < \eta \right]\)$ This surgery-like selective filtering purges false-positive regions while retaining high-confidence true-positive tumor tissue, effectively breaking the traditional Pareto tradeoff between failure detection sensitivity and segmentation accuracy.

Loss & Training

1. Backbone Fine-tuning: The MedSAM backbone is adapted for prompt-free automatic segmentation and fine-tuned per cancer dataset using Focal Loss (\(\alpha = 0.97, \gamma = 2\)) and Adam optimizer (\(1 \times 10^{-4}\) weight decay, exponential learning rate scheduling).
2. SAE Optimization: With the backbone frozen, SAEs are trained using the BatchTopK strategy at learning rate \(1 \times 10^{-4}\) for 5 epochs, enforcing sparsity \(S = 8\) and dictionary size \(D = 1536\).
3. Classifier Training: XGBoost is fitted on the failure representation dataset \(\mathcal{D}\) using regularized binary cross-entropy loss with standard tree-complexity constraints.

Key Experimental Results

Main Results

The framework was evaluated on prostate cancer (PI-CAI, Prostate158) and pancreatic cancer (PanTS). Table 1 presents the zero-shot failure detection performance when models trained on PI-CAI are deployed directly to the unseen Prostate158 dataset without any fine-tuning. While output confidence baselines completely collapse (AUROC near random chance at ~50%), the internal concept-based detector demonstrates outstanding domain generalization.

Table 1: Zero-shot failure detection on unseen Prostate158 dataset (trained on PI-CAI, original Table 1)

Method	FPR95 (↓)	AUROC (↑)	AUPR (↑)
MaxProb	100.0%	50.0%	64.4%
MeanProb	100.0%	52.6%	65.1%
Entropy	97.8%	53.1%	65.6%
Energy	100.0%	48.0%	63.6%
Ours	64.4%	73.6%	74.1%

On the Pareto frontier analysis (original Fig. 5), baseline methods display a strict tradeoff between F1 and DSC. The proposed method surpasses this Pareto frontier, delivering +8.7 points in F1 and +6.2 points in DSC on PI-CAI, and +9.2 points in F1 with +2.3 points in DSC on PanTS.

Ablation Study

Table 2: Ablation study on SAE architecture (sparsity \(S\) and dictionary size \(D\), original Table 2)

Dictionary Size \(D\)	Sparsity \(S=4\) (DSC / F1)	Sparsity \(S=8\) (DSC / F1)	Sparsity \(S=16\) (DSC / F1)
\(D = 1536\) (\(768 \times 2\))	49.7% / 60.3%	50.1% / 63.5%	49.9% / 62.5%
\(D = 3072\) (\(768 \times 4\))	50.1% / 58.4%	49.7% / 57.4%	49.6% / 62.6%

Table 3: Ablation on concept source depth (original Table 3)

Layers (\(L\))	Layers Included	Segmentation DSC (↑)	Failure Detection F1 (↑)
Early	Layer 1, 3	47.5%	44.5%
Mid	Layer 5, 7	48.8%	50.4%
Deep	Layer 9, 11	49.6%	57.6%
All	Layer 1, 3, 5, 7, 9, 11	50.1%	63.5%

Table 4: Ablation on failure detection classifier architecture (original Table 4)

Classifier	Segmentation DSC (↑)	Failure Detection F1 (↑)
Logistic Regression	47.8%	59.2%
Decision Tree	48.5%	58.7%
Random Forest	44.6%	57.0%
XGBoost (Ours)	50.1%	63.5%

Key Findings

• Deep layers provide high diagnostic discriminability, while multi-depth integration is optimal: Deep layer concepts (Layers 9 and 11) capture critical semantic abstractions, achieving an F1 score of 57.6% on their own. Combining them with mid-level anatomical context and early-level texture details yields the highest overall performance (63.5% F1 and 50.1% DSC).
• Compact SAE dictionaries fit medical imaging better: Unlike NLP models requiring dictionary sizes exceeding 10,000, medical ViTs perform best with \(D = 1536\) and \(S = 8\). Overly large dictionaries dilute feature density and harm downstream tree-based aggregation.
• Minimal inference latency and memory overhead: On an NVIDIA RTX A6000 GPU (1024x1024 resolution), MedSAM base inference requires 126.3 ms. Adding all 6 SAE encoders adds only 4.6 ms (+3.7%), and the complete end-to-end pipeline (including SAE decoders and XGBoost) runs at 176.5 ms (~5.66 images/s) with an additional GPU memory footprint of only 60 MB (+1.8%).

Highlights & Insights

• Empirical Confirmation of "Feeling of Error" in ViTs: The work provides rigorous statistical evidence that Vision Transformers exhibit an internal latent signature during prediction failures: failure states show markedly fewer active concepts with suppressed amplitudes (\(p < 10^{-5}\), Cohen's \(d = 1.8\)).
• Surgical False-Positive Pruning: By retaining patch-level concept vectors, the framework identifies and subtracts local false-positive regions without discarding surrounding true positives, directly boosting clinical utility (e.g., lifting patient case DSC from 74.1 to 86.5).
• Non-semantic Internal Signatures: The most predictive failure features are not limited to human-annotated clinical terms; unsupervised SAE discovery captures subtle machine-level processing artifacts that correlate strongly with impending errors.

Limitations & Future Work

• Correlation Rather Than Causality: The framework relies on statistical co-activation rather than verified causal pathways. Incorporating causal mediation analysis or causal abstraction would enable interventional steering and debugging.
• Cancer-Specific SAE Models: Current SAE dictionaries are trained separately per cancer type. Developing a universal medical SAE spanning multiple anatomies (brain, liver, lung, pancreas) represents a key next step.
• Handling Missing Modalities: In clinical workflows with incomplete imaging sequences (e.g., missing diffusion-weighted MRI), extending internal concept analysis could detect modality-specific information loss and guide adaptive fusion.

Related Work & Insights

• vs Output Uncertainty Metrics (MaxProb, Entropy, Energy Score): Output metrics offer no qualitative explanations and degrade overall segmentation quality when thresholded; internal concept monitoring provides localized, interpretable, and actionable failure signals.
• vs Concept Bottleneck Models (CBMs): CBMs require costly predefined concept annotations and alter the forward architecture; this approach functions entirely post-hoc on pre-trained, frozen segmentation networks without architecture modification.
• vs Medical Vision-Language Models (VLMs): Medical VLMs constrain internal representations to human language vocabularies; SAEs uncover both clinical anatomical concepts and critical non-semantic failure signals.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Groundbreaking perspective connecting cognitive "Feeling of Error" with mechanistic interpretability via SAEs for medical failure detection.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain zero-shot evaluation, granular layer ablations, classifier benchmarking, and concrete runtime latency profiling.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, elegant conceptual framing, and intuitive visualizations.
• Value: ⭐⭐⭐⭐⭐ Offers a practical, non-invasive, and clinically auditable safety wrapper for safety-critical medical image segmentation.

Related Papers

• [CVPR 2026] EEGiT: Teaching Vision Transformers to Understand the EEG signal
• [ICML 2026] Scaling Vision Transformers for Functional MRI with Flat Maps
• [CVPR 2026] MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy
• [CVPR 2026] Keep It Frozen: Domain-Routed Conditional Residual Modulation for Multi-Domain Vision Transformers
• [ECCV 2026] PhenoLIP: Phenotype Guided Medical Vision–Language Pretraining