SeekFlow: Synergizing Radiology and Pathology Foundation Models for Precision Oncology via Knowledge-Guided Evidence Flow¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/HKU-MedAI/SeekFlow
Area: Medical Imaging
Keywords: Multimodal Fusion, Flow Matching, Vision-Language Foundation Model, Evidential Theory, Precision Oncology
TL;DR¶
To tackle scale mismatch and topological degeneracy between macro-scale radiology and micro-scale pathology foundation models, SeekFlow builds a clinical prototype-anchored evidence manifold, employs knowledge-guided continuous flow matching to achieve topology-preserving feature transport, and integrates hierarchical evidential aggregation via Dempster-Shafer theory, achieving state-of-the-art performance across tumor grading, staging, and survival prediction.
Background & Motivation¶
Precision oncology critically depends on multidisciplinary tumor boards, where clinicians integrate macroscopic anatomical context from radiology (e.g., MRI, CT) with microscopic histopathological details from whole-slide images (WSIs). Propelled by recent breakthroughs in biomedical vision-language foundation models (VLFMs, such as CONCH and BiomedCLIP), unimodal feature extraction now encapsulates rich domain-specific semantics. However, training an omni-modal foundation model capable of natively processing both radiology and pathology remains computationally prohibitive and is severely constrained by the scarcity of paired multi-scale clinical cohorts. Consequently, post-hoc multimodal fusion of frozen, heterogeneous unimodal experts remains the predominant and practical paradigm.
Conventional cross-modal fusion strategies suffer from two fundamental bottlenecks. First, static mapping induces topological degeneracy. Microscopic cellular morphology and macroscopic anatomical structures differ by orders of magnitude in spatial resolution, causing their latent representations to reside on topologically disjoint manifolds. Standard operations (such as cross-attention or linear projection) forcibly project heterogeneous tokens into a shared space, violently conflating unaligned semantic neighborhoods, corrupting intra-modal geometric structures, and triggering catastrophic feature collapse. Second, black-box synergies suffer from severe semantic entanglement. Existing methods typically condense multimodal inputs into an uninterpretable, monolithic vector, failing to disentangle shared biological manifestations (e.g., tumor necrosis) from modality-specific hallmarks (e.g., mitotic figures in pathology versus mass effect or edema in radiology), which hampers clinical transparency.
Bridging this vast multi-scale divide requires replacing violent static projections with a smooth, structure-preserving transport process. Core idea: reformulate multimodal fusion as a knowledge-guided continuous flow matching transport problem onto a unified evidence manifold anchored by clinical guidelines, and synthesize cross-modal evidence via Dempster-Shafer theory to enable uncertainty-aware decision-making.
Method¶
Overall Architecture¶
SeekFlow comprises three core phases: Knowledge-Routed Evidence Manifold construction, Knowledge-Guided Evidence Flow transport, and Uncertainty-Aware Evidential Synthesis. First, structured clinical concepts extracted from expert medical guidelines (e.g., WHO classification criteria and AJCC staging manuals) by an LLM anchor a shared manifold parameterized by learnable Gaussian prototypes; a Bayesian knowledge router assigns visual tokens to shared or modality-specific prototypes. Next, optimal transport provides micro-targets while conditional flow matching learns a continuous, time-dependent vector field, transporting visual tokens along smooth ODE trajectories while preserving their intrinsic manifold topology. Finally, a hierarchical Dempster-Shafer evidential aggregator quantifies epistemic uncertainty across prototypes, dynamically suppresses ambiguous evidence, and synthesizes a robust patient-level representation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Multimodal Input<br/>Radiology MRI/CT + Pathology WSI"] --> FE["Frozen Foundation Encoders<br/>BiomedCLIP + CONCH"]
FE --> D1["1. Knowledge-Routed Evidence Manifold: Disentangled Anchoring"]
D1 --> D2["2. Knowledge-Guided Evidence Flow: Continuous Flow Matching Transport"]
D2 --> D3["3. Hierarchical Evidential Synthesis: Dempster-Shafer Aggregation"]
D3 --> OUT["Downstream Precision Oncology<br/>Tumor Grading / Staging / Survival Analysis"]
Key Designs¶
1. Knowledge-Routed Evidence Manifold: Disentangled Anchoring
To resolve modality asymmetry, SeekFlow establishes a unified geometric evidence manifold parameterized by \(K\) learnable Gaussian prototypes \(\mathcal{P} = \{\mathcal{N}(\mu_k, \text{diag}(\sigma_k^2))\}_{k=1}^K\). The prototype space is explicitly partitioned into shared prototypes \(\mathcal{P}_{\text{shared}}\) (capturing cross-modal traits such as necrosis) and modality-specific sets \(\mathcal{P}_{\text{spec}}^{(m)}\) (capturing pathology-specific mitotic activity or radiology-specific peritumoral edema), while retaining several fully learnable prototypes to ensure representational adaptability. Structured visual descriptions elicited from WHO and AJCC guidelines via an LLM (e.g., Gemini) are encoded by modality VLFMs to serve as knowledge prior embeddings \(t_k^{(m)}\).
During routing, projected visual tokens \(\tilde{x}_i^{(m)}\) compute cosine similarities against clinical concepts \(\ell_{\text{prior}}\) and merge with an MLP router to yield assignment logits \(\ell_{i,k}^{(m)} = \mathcal{R}_\phi^{(m)}(\tilde{x}_i^{(m)})_k + \ell_{\text{prior},i,k} / \tau_p\). Rather than competitive softmax pooling, SeekFlow applies a Sigmoid gating mechanism \(w_{i,k}^{(m)} = \sigma(\ell_{i,k}^{(m)}) \cdot A_{k,m}\) (governed by binary modality mask \(A\)), allowing a single token to concurrently provide evidence for multiple co-occurring pathological phenomena. Prototype collapse is prevented via an orthogonal diversity loss \(\mathcal{L}_d\), while arbitrary deviations from clinical priors are penalized by a KL-divergence regularization term \(\mathcal{L}_k\):
2. Knowledge-Guided Evidence Flow: Continuous Flow Matching Transport
To prevent the geometric topological destruction caused by discrete static mappings, SeekFlow formulates token transformation as continuous transport governed by an ordinary differential equation (ODE) vector field. To circumvent the quadratic computational overhead of dense global matching, training follows an Expectation-Maximization framework. In the E-step, each token \(i\) samples a target prototype \(k_i \sim \text{Categorical}(w_i)\), and \(J\) support points \(z_j\) are drawn from \(\mathcal{N}(\mu_{k_i}, \sigma_{k_i}^2)\). By defining a cost matrix coupling normalized Euclidean distance and Mahalanobis density consistency \(C_{ij} = C_{ij}^{\text{dist}} + (w_i - (1 - d_{Mj}))^2\), entropic optimal transport is solved via Sinkhorn iterations to compute a barycentric target \(z_{i,k_i}^* = \sum_j \gamma_{ij} z_j / (\sum_j \gamma_{ij} + \epsilon)\), transporting salient features toward the prototype dense core while directing ambiguous tokens to the periphery.
In the M-step, a conditional flow matching network \(v_\theta\) conditioned on timestep \(t\), modality identifier \(c_m\), and prototype embedding \(c_k\) via Adaptive Layer Normalization (AdaLN) regresses along the linear geodesic path \(x_t = (1-t)\tilde{x}_i + t z_{i,k_i}^*\):
During inference, a Top-\(K\) Gumbel-Sigmoid activation mask filters active tokens, and an Euler ODE numerical solver integrates the learned continuous vector field \(\hat{x}_{i,k} = \tilde{x}_i + \int_0^1 v_\theta(x_t, t \mid c_m, c_k) dt\), establishing a smooth diffeomorphism that preserves intra-modal manifold topology.
3. Hierarchical Evidential Synthesis: Dempster-Shafer Aggregation
Due to gigapixel whole-slide imaging generating thousands of patch tokens compared to a modest number of radiological slices, naive flat pooling causes pathology to completely drown out radiology. SeekFlow designs a two-tier aggregation strategy grounded in Dempster-Shafer Theory (DST). At the intra-modal stage, gated attention scores are combined with routing weights \(w_{i,k}^{(m)}\) to yield modality-specific prototype summaries \(h_k^{(m)} = \sum_i (\alpha_{i,k}^{(m)} \cdot w_{i,k}^{(m)}) \hat{x}_{i,k}^{(m)}\).
At the cross-modal consensus stage, a light MLP with Softplus activation estimates non-negative evidence masses \(e_k^{(m)}\), deriving belief degrees \(b_k^{(m)} = e_k^{(m)} / (e_k^{(m)} + 1)\) and global epistemic uncertainty \(U_k = 1 / (\sum_m e_k^{(m)} + 1)\). The fused representation dynamically weights modalities by their reliability: \(h_k^{\text{fused}} = \sum_m \frac{b_k^{(m)}}{\sum_{m'} b_k^{(m')} + \epsilon} h_k^{(m)}\), naturally accommodating missing modalities through near-zero belief assignments. When passing the concatenated fused sequence into a self-attention module to synthesize the patient-level token, an uncertainty-biased attention mask is injected:
This bias explicitly penalizes unconfident or conflicting prototypes, and an evidential entropy loss \(\mathcal{L}_e = -\frac{1}{K}\sum_{k=1}^K \log(1 - U_k + \epsilon)\) drives the model to aggregate robust, cross-modal consensus.
Loss & Training¶
SeekFlow is trained end-to-end under the joint objective function:
Where \(\mathcal{L}_{\text{task}}\) utilizes cross-entropy loss for tumor grading/staging and negative Cox partial log-likelihood for survival prediction. Loss weights are configured as \(\lambda_f = 0.1, \lambda_k = 0.5, \lambda_d = 1.0, \lambda_e = 0.01\) for classification tasks, and \(\lambda_f = 1.0, \lambda_k = 1.0, \lambda_d = 1.0, \lambda_e = 0.01\) for survival analysis. Training utilizes the Adam optimizer with learning rate \(1 \times 10^{-4}\), weight decay \(1 \times 10^{-5}\), latent dimension \(D=128\), and runs for 20 epochs.
Key Experimental Results¶
Main Results¶
SeekFlow was rigorously evaluated on three paired multimodal clinical cohorts: TCGA Glioma (WSI + MRI, 3-class grading and LGG/GBM survival analysis), Gastric Cancer (WSI + CT, survival analysis), and Chondrosarcoma (WSI + MRI, 3-class grading and 3-class staging). All experiments adopt 10-fold cross-validation with an 8:1:1 split, reporting mean and standard deviation.
Table 1: Classification Performance (Tumor Grading & Staging)
| Method Type | Method | TCGA Glioma Grading (AUC↑ / ACC↑) | Chondrosarcoma Staging (AUC↑ / ACC↑) | Chondrosarcoma Grading (AUC↑ / ACC↑) |
|---|---|---|---|---|
| Unimodal Baselines | Radiology-Specific | 0.797±0.048 / 0.591±0.053 | 0.772±0.136 / 0.574±0.126 | 0.658±0.120 / 0.488±0.181 |
| Unimodal Baselines | ABMIL (Pathology) | 0.865±0.031 / 0.688±0.056 | 0.611±0.121 / 0.478±0.232 | 0.903±0.104 / 0.723±0.165 |
| Unimodal Baselines | TransMIL (Pathology) | 0.891±0.035 / 0.709±0.062 | 0.651±0.146 / 0.425±0.161 | 0.911±0.086 / 0.722±0.130 |
| Multimodal Fusion | Concat | 0.878±0.036 / 0.712±0.061 | 0.663±0.155 / 0.434±0.166 | 0.887±0.130 / 0.725±0.186 |
| Multimodal Fusion | MCAT | 0.882±0.024 / 0.697±0.069 | 0.719±0.105 / 0.531±0.164 | 0.924±0.069 / 0.782±0.148 |
| Multimodal Fusion | MOTCat | 0.886±0.027 / 0.682±0.065 | 0.681±0.140 / 0.502±0.159 | 0.918±0.053 / 0.809±0.166 |
| Multimodal Fusion | SurvPath | 0.890±0.016 / 0.706±0.038 | 0.686±0.122 / 0.488±0.102 | 0.890±0.081 / 0.761±0.115 |
| Multimodal Fusion | M4Survive | 0.882±0.030 / 0.718±0.053 | 0.714±0.141 / 0.501±0.219 | 0.914±0.103 / 0.785±0.123 |
| Multimodal Fusion | PIBD | 0.886±0.049 / 0.709±0.055 | 0.661±0.149 / 0.545±0.146 | 0.877±0.084 / 0.735±0.154 |
| Multimodal Fusion | CTF | 0.882±0.034 / 0.736±0.058 | 0.776±0.147 / 0.562±0.270 | 0.853±0.094 / 0.735±0.142 |
| Ours | SeekFlow (Ours) | 0.901±0.011 / 0.739±0.055 | 0.820±0.087 / 0.595±0.128 | 0.931±0.077 / 0.832±0.127 |
Table 2: Survival Analysis Performance (C-Index↑, mean ± std)
| Method Type | Method | TCGA-LGG | TCGA-GBM | Gastric Cancer |
|---|---|---|---|---|
| Unimodal Baselines | Radiology-Specific | 0.529±0.116 | 0.536±0.048 | 0.576±0.047 |
| Unimodal Baselines | ABMIL (Pathology) | 0.645±0.194 | 0.496±0.077 | 0.677±0.064 |
| Unimodal Baselines | CLAM (Pathology) | 0.669±0.164 | 0.452±0.083 | 0.726±0.051 |
| Unimodal Baselines | TransMIL (Pathology) | 0.730±0.199 | 0.488±0.097 | 0.722±0.056 |
| Multimodal Fusion | Concat | 0.683±0.169 | 0.484±0.084 | 0.697±0.046 |
| Multimodal Fusion | MCAT | 0.696±0.182 | 0.524±0.069 | 0.592±0.091 |
| Multimodal Fusion | MOTCat | 0.709±0.198 | 0.519±0.107 | 0.719±0.056 |
| Multimodal Fusion | SurvPath | 0.722±0.181 | 0.505±0.071 | 0.715±0.071 |
| Multimodal Fusion | PIBD | 0.686±0.153 | 0.550±0.124 | 0.697±0.052 |
| Multimodal Fusion | M4Survive | 0.752±0.144 | 0.485±0.044 | 0.679±0.061 |
| Multimodal Fusion | CTF | 0.713±0.179 | 0.525±0.096 | 0.721±0.031 |
| Ours | SeekFlow (Ours) | 0.793±0.099 | 0.563±0.062 | 0.740±0.059 |
Ablation Study¶
A comprehensive ablation study across all tasks evaluates the macro-averaged impact of architectural modules (Full Model baseline: AUC 0.884, ACC 0.722, C-Index 0.699).
Table 3: Ablation Study on Architecture and Mechanisms (Macro-Averaged Across Tasks)
| Ablation Category | Configuration Variant | AUC ↑ | ACC ↑ | C-Index ↑ | Note |
|---|---|---|---|---|---|
| Full Model | SeekFlow (Full Model) | 0.884 | 0.722 | 0.699 | Complete 3-stage pipeline |
| (1) Evidence Manifold | w/o Evidence Anchors | 0.868 (-1.6%) | 0.687 (-3.5%) | 0.641 (-5.8%) | Removes clinical prototype anchoring |
| (1) Evidence Manifold | All Shared Prototypes | 0.833 (-5.1%) | 0.671 (-5.1%) | 0.673 (-2.6%) | Prevents modality-specific semantic disentanglement |
| (1) Evidence Manifold | All Specific Prototypes | 0.870 (-1.4%) | 0.718 (-0.4%) | 0.654 (-4.5%) | Destroys cross-modal shared semantic substrate |
| (1) Evidence Manifold | No Knowledge Prior | 0.867 (-1.7%) | 0.710 (-1.2%) | 0.674 (-2.5%) | Pure data-driven routing drifts from clinical concepts |
| (1) Evidence Manifold | Full Knowledge Prior | 0.867 (-1.7%) | 0.685 (-3.7%) | 0.617 (-8.2%) | Static priors lack dataset-adaptive tuning |
| (2) Evidence Flow | Flow Matching ⇒ Optimal Transport | 0.567 (-31.7%) | 0.507 (-21.5%) | 0.501 (-19.8%) | Discrete OT causes severe topological collapse |
| (2) Evidence Flow | Euclidean Distance Only | 0.872 (-1.2%) | 0.715 (-0.7%) | 0.668 (-3.1%) | Omits Mahalanobis density alignment |
| (3) Evidential Synthesis | Flat Aggregation | 0.815 (-6.9%) | 0.663 (-5.9%) | 0.577 (-12.2%) | Pathology tokens drown out radiology features |
| (3) Evidential Synthesis | Standard Gated Pooling | 0.878 (-0.6%) | 0.717 (-0.5%) | 0.656 (-4.3%) | Lacks router guidance in intra-modal aggregation |
| (3) Evidential Synthesis | Multi-Head Attention | 0.837 (-4.7%) | 0.701 (-2.1%) | 0.663 (-3.6%) | Ignores epistemic uncertainty, vulnerable to noise |
Key Findings¶
- Continuous flow transport prevents topological collapse: The most dramatic ablation occurs when continuous flow matching is replaced by discrete static optimal transport, which causes AUC to crash by 31.7% (0.884 → 0.567) and C-Index to fall to 0.501 (near random guessing). This empirically proves that discrete mappings induce catastrophic manifold collapse across multi-scale features, whereas continuous diffeomorphic flows preserve latent geometric topologies.
- Hierarchical evidential synthesis cures token volume disparity: Replacing hierarchical evidential synthesis with flat token aggregation causes a 12.2% drop in C-Index. Since gigapixel WSIs contribute orders of magnitude more tokens than radiology scans, DST-based hierarchical fusion is essential to prevent pathology tokens from overwhelming radiological context.
- Backbone and LLM agnostic: Substituting pathology backbones between CONCH and MUSK, radiology backbones between BiomedCLIP and GenMedCLIP, or knowledge-elicitation LLMs across Gemini 3.0 Pro, GPT-5.5, and Grok 4.3 produces minimal metric fluctuation (AUC remains between 0.877 and 0.884). This verifies that SeekFlow's gains stem from its principled continuous transport framework rather than single-model idiosyncrasies.
Highlights & Insights¶
- Multimodal fusion as a continuous transport problem: Rather than forcing disparate embeddings together via heuristic attention or concatenation, SeekFlow formulates cross-modal alignment as continuous vector field integration, mathematically guaranteeing topology preservation.
- Clinically interpretable disentangled manifold: Anchoring the latent space to clinical concepts extracted from WHO/AJCC guidelines uncouples shared traits from modality-specific hallmarks, shedding light into previously opaque deep learning representations.
- Dempster-Shafer theory dynamically manages clinical conflict: By quantifying epistemic uncertainty at each clinical prototype, the model naturally downweights conflicting or unconfident evidence and gracefully handles missing imaging modalities without pipeline retraining.
Limitations & Future Work¶
- Admitted limitation: Evaluation is currently restricted to bi-modal radiology-pathology setups; non-imaging omics modalities (e.g., bulk RNA sequencing, spatial transcriptomics, somatic mutations) have not yet been integrated into the continuous flow framework.
- Inference overhead: Solving the ODE vector field via numerical Euler integration introduces modest computational latency compared to direct linear projection; future exploration into 1-step rectified flow distillation could accelerate inference speed.
- Broader transferability: The concept of knowledge-anchored prototype manifolds combined with continuous flow matching can be readily adapted to other multi-scale robotics or autonomous driving tasks where heterogeneous sensors (e.g., LiDAR, high-res cameras, radar) exhibit extreme scale and density disparity.
Related Work & Insights¶
- vs MCAT / MOTCat: MCAT aggregates pathology patches using genomic queries via asymmetric cross-attention, while MOTCat applies discrete optimal transport plans. Both operate on static coordinates, which struggle with extreme cross-scale geometric divergence; SeekFlow introduces continuous flow matching that warps distributions via smooth velocity fields.
- vs PIBD: PIBD adopts prototypical information bottlenecks for disentanglement but retains uninterpretable black-box latent vectors; SeekFlow directly grounds prototypes in auditable clinical guidelines.
- vs M4Survive / CTF: Existing foundation model adaptation frameworks rely on cross-modal adapters or bi-directional fine-tuning, risking encoder over-fitting; SeekFlow keeps foundation model weights entirely frozen, ensuring modular plug-and-play deployment.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates multimodal medical fusion as a continuous flow matching transport problem anchored by clinical prototypes.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across 3 clinical cancer cohorts, 4 distinct tasks, rigorous 10-fold cross-validation, and extensive ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid explanation of the mathematical underpinnings, topological degeneracy challenges, and evidential synthesis.
- Value: ⭐⭐⭐⭐⭐ Delivers an interpretable, robust, and state-of-the-art methodology for integrating multi-scale foundation models in clinical oncology.