Histopathology Multi-modal Embedding for Pathology Composed Retrieval¶

Conference: ECCV 2026
arXiv: 2502.07221
Paper: Project Page
Code: None
Area: Medical Imaging
Keywords: Pathomics Retrieval, Multimodal Embedding, MLLM Adaptation, Composed Retrieval, Contrastive Learning

TL;DR¶

HOMIE proposes a model-agnostic two-stage adaptation framework to transform any generative MLLM into a pathology retrieval expert. In the first stage, pure-text LoRA training is employed to teach the LLM a discriminative metric space (resolving task mismatch). In the second stage, native resolution processing, stain augmentation, and a progressive knowledge curriculum inject pathomorphological priors (resolving domain mismatch). On the newly proposed PCR composed retrieval benchmark, a 2B parameter version significantly outperforms 7B specialized pathology MLLMs and dual-encoder models.

Background & Motivation¶

Clinical deployment of pathology AI faces barriers of trust: traditional supervised models are "black boxes" lacking interpretability, while generative large models carry hallucination risks that are clinically unacceptable. The retrieval-based paradigm is naturally suited for clinical settings—the system retrieves similar cases from a database based on the pathologist's query to serve as a reference for "computational consultation," allowing doctors to make independent decisions based on evidence and preserving clinical autonomy.

Existing pathology multimodal models (such as CONCH, MUSK, PathoCLIP, etc.) are entirely based on dual-encoder architectures and can only perform simple image-to-text or text-to-image retrieval. However, real-world clinical queries are inherently composed and interleaved—for example, "this H&E slice + 'find the zoomed-in views of atypical areas'"—and dual encoders lack a mechanism for deep fusion of interleaved multimodal inputs, resulting in an architectural mismatch. When forced to perform composed retrieval using vector addition or Reciprocal Rank Fusion (RRF), these models degrade to shallow visual matching, completely ignoring textual instructions and incorrectly retrieving the original image itself.

Intuitively, the deep fusion architecture of MLLMs is naturally capable of handling interleaved inputs. However, direct deployment faces two new mismatches: (1) Task Mismatch: The latent space of MLLMs is optimized for generation, lacking the discriminative metric structure required for retrieval. Even the SOTA pathology MLLM (PathoR1-7B) achieves only a 14.7% Recall@1 on composed retrieval. (2) Domain Mismatch: General MLLMs lack the capability to interpret subtle cytological morphology and staining artifacts in pathology.

To this end, this paper formally defines the Pathology Composed Retrieval (PCR) task and proposes the HOMIE framework. Core Idea: Through a model-agnostic systematic adaptation process, task mismatch and domain mismatch are resolved sequentially in two stages, turning any generative MLLM into a pathology retrieval expert capable of producing unified multimodal embeddings.

Method¶

Overall Architecture¶

The goal of HOMIE is: given a query composed of arbitrarily interleaved pathology images, text, and videos, to output a unified dense embedding vector for similarity retrieval in a candidate database. The framework contains three core components: a vision encoder \(f_v\) (SigLIP-2 ViT), an MLP projector \(f_p\), and an LLM \(f_\varphi\). The visual input is encoded and projected to obtain the visual token sequence \(h_v = f_p(f_v(V))\), which is fed into the LLM along with text tokens \(h_t\). Finally, an Explicit One-word Limitation (EOL) prompting strategy—appending "Summarize above ... in one word:" at the end of the input—forces the LLM to compress all multimodal context into the last-layer hidden state of the <emb> token, serving as the unified any-modal embedding \(E = f_\varphi(h_v, h_t)\).

Training proceeds sequentially in two stages, first overcoming the task mismatch, and then resolving the domain mismatch.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Query<br/>Image+Text+Video"] --> B["Vision Encoder<br/>SigLIP-2 ViT<br/>Native Resolution+2D-RoPE"]
    B --> C["MLP Projector"]
    A --> D["Text Tokens"]
    C --> E["LLM Backbone<br/>Qwen3-VL etc."]
    D --> E
    E --> F["EOL Prompting Strategy<br/>'in one word:' → &lt;emb&gt;"]
    F --> G["Any-modal Embedding Vector"]
    G --> H["Candidate Database<br/>Cosine Similarity Retrieval"]
    I["Stage 1: Pure-Text LoRA<br/>NLI+MedNLI+MedMCQA"] -.->|"Train First"| E
    J["Stage 2: Pathology-Specific Tuning<br/>Stain Aug+Native Resolution+Progressive Course"] -.->|"Train Second"| B
    J -.->|"Train Second"| E

Key Designs¶

1. EOL Prompting Strategy: Extracting Dense Embeddings from Causal LLMs

Unlike the [CLS] token in BERT, which naturally aggregates semantic information using bidirectional attention, LLMs utilize causal attention. Standard pooling (e.g., averaging across all tokens) performs poorly in causal LLMs because earlier tokens lack future context, which dilutes the global representation. The EOL strategy explicitly appends "in one word:" at the end of the prompt, forcing the LLM to compress all preceding multimodal context into the hidden state of the <emb> token. For pure image inputs, it uses "<image> Summarize above image in one word: <emb>"; for interleaved image and text inputs, it uses "<image1><text1>... Summarize above image and sentence in one word: <emb>", and similarly for text and video. Ablations indicate that competitive results can still be achieved without the EOL constraint (showing that training has already taught the model to aggregate semantics at the final token), but retaining this "in one word" semantic bottleneck yields the best performance (79.8% Recall@1 vs. 78.0% without prompt) because it further suppresses residual generation noise.

2. Two-Stage Adaptation: Sequentially Resolving Task Mismatch and Domain Mismatch

This is the most critical design decision of HOMIE—rather than training text adaptation and pathology adaptation together, they are strictly serialized. Stage 1 (Pure-Text Retrieval Adaptation): The vision encoder and projector are frozen, and LoRA (rank=64) is applied only to the LLM. The model is trained on pathology-related text pairs (NLI + MedNLI + 15k pathology QA pairs filtered from MedMCQA). This stage forces the LLM to map semantically similar text to adjacent embedding spaces, building the discriminative metric structure required for retrieval before the model has ever seen an image. Stage 2 (Pathology-Specific Course Tuning): Initialized from the Stage 1 checkpoint, the vision encoder and projector are unfrozen, and the entire model is fine-tuned (with LoRA rank increased to 128) on pathology image-text pairs. The key to this sequential design is allowing the LLM to first learn "what makes a good retrieval embedding" before injecting domain knowledge, preventing domain data from disrupting the learning of the retrieval metric space.

3. Pathology-Specific Tuning Trio: Native Resolution + Stain Augmentation + Progressive Course

Stage 2 is not just simple multimodal fine-tuning; it incorporates three designs specifically tailored to the characteristics of pathology images. (1) Native Resolution Processing: Unlike dual encoders that force all images to be downscaled to a fixed low resolution (e.g., 336x336), HOMIE utilizes SigLIP-2 ViT to dynamically adjust the image size to multiples of the base patch size (14x14) and preserves spatial information with 2D-RoPE, ensuring the model can analyze multi-scale fine-grained morphological details. In ablation studies, forcing a standard low-resolution input caused Image-to-Text retrieval on Bookset to plummet by 8.9%. (2) Stain Augmentation and Normalization: Pathology images suffer from significant color variations caused by different laboratory staining protocols. Introducing RandStainNA stain augmentation forces the model to learn stain-invariant morphological representations. (3) Progressive Knowledge Course: Rather than simply mixing datasets, the model is first allowed to establish fundamental morphological priors on PathGen-1.6M (which emphasizes tissue-cell morphology and spatial organization), and then learns to associate morphology with high-level multimodal knowledge containing diagnostic information on PathCap and filtered Quilt-1M. Removing the progressive course was the most detrimental individual operation in the ablation studies (dropping Video-to-Text from 30.7% to 18.4% and Image-Text to Image from 52.7% to 46.7%).

4. Data Bootstrap Filtering: Purifying Web-scraped Data

The Quilt-1M dataset, primarily sourced from social media, contains a large number of noisy image-text pairs, and direct training on the mixture harms performance. HOMIE adopts a bootstrapping strategy: a base model is first trained on the unfiltered Quilt-1M, and then used to calculate similarity scores for all image-text pairs. Pairs below the threshold \(\lambda=0.1\) are discarded, ultimately retaining about 500k high-quality pairs. Ablation studies show that removing data filtering drops Video-to-Text from 30.7% to 23.4%, verifying the importance of high-quality aligned data for constructing a robust embedding space.

A Complete Example: Image+Text to Image Retrieval¶

An example of walking through the pipeline with Image-Text to Image retrieval \((q^i, q^t) \to c^i\) on Bookset: The query consists of a source pathology image and relational text (e.g., an H&E slice of rectal adenocarcinoma + "find the zoomed-in views of atypical areas"), and the goal is to retrieve the correct target image.

Vision Encoding: The source image is fed into SigLIP-2 ViT at its native resolution (e.g., 1148px), dynamically scaled to a multiple of 14, and encoded with position information via 2D-RoPE, outputting a sequence of visual tokens.
Text Encoding: The relational text ("find the zoomed-in views of atypical areas") is tokenized into text tokens.
LLM Fusion: Visual tokens and text tokens are interleavingly fed into the LLM, undergoing deep fusion through the causal attention layers adapted by LoRA—at this point, the LLM has already learned to map semantically similar content to close positions in Stage 1 and has acquired pathomorphological priors in Stage 2.
EOL Compression: The prompt is appended with "Summarize above image and sentence in one word: ", and the last-layer hidden state of the <emb> token is extracted as the unified embedding.
Retrieval: The cosine similarity between this embedding and all target image embeddings in the candidate database is computed, and the Top-K results are returned after ranking.

In comparison with the dual-encoder CONCH: it can only independently encode the source image, the candidate target images, and the text, and then perform vector addition or RRF fusion. Consequently, CONCH ignores textual instructions, degrades to visual matching, and directly retrieves the source image itself (yielding a Recall@1 of 43.7%), while HOMIE correctly executes spatial-semantic composed reasoning (achieving a Recall@1 of 52.7% with the 8B version).

Loss & Training¶

Both stages employ the InfoNCE contrastive loss. Given a batch size \(B\), the embedding of query \(q_i\) is trained to be close to the embedding of its positive sample \(c_i\) and far from other negative samples:

\[\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B}\log\left[\frac{\exp\left[\cos(q_i, c_i) / \tau\right]}{\sum_{j=1}^{B}\exp\left[\cos(q_i, c_j) / \tau\right]}\right]\]

Training Configuration: 8x H100 GPUs, BF16 precision, FlashAttention-2, DeepSpeed ZeRO-2. Stage 1: 2 epochs, global batch size 576, learning rate \(2\times10^{-4}\), cosine decay, 0.03 warmup, LoRA rank=64/\(\alpha\)=128, with the vision encoder and projector frozen. Stage 2: Initialized from Stage 1, 2 epochs, global batch size 384, learning rate \(1\times10^{-4}\), LoRA scaled to rank=128/\(\alpha\)=256, unfreezing the vision encoder and projector to capture domain-specific features.

Key Experimental Results¶

Main Results¶

HOMIE comprehensively dominates all baselines across five composed retrieval tasks on the PCR Benchmark. The 2B version alone reaches a 75.2% Recall@1 (Multi-Image to Text), which is over 30 percentage points higher than the best dual-encoder Patho-CLIP-L (44.3%) and over 60 percentage points higher than the best pathology MLLM PathoR1-7B (14.7%).

Model	Params	Multi-Img→Text (Bookset) R@1	Img+Text→Img (Bookset) R@1	Img+Text→Text (Quilt-VQA) R@1	Video→Text (Videopath) R@1
Patho-CLIP-L (Add)	~300M	44.3	40.7	22.1	-
MUSK (Add)	~1B	43.0	47.9	34.1	-
CONCH (Add)	~1B	41.8	43.7	9.7	-
PathoR1-7B	7B	14.7	18.2	14.2	19.3
LamRA-7B	7B	5.8	7.4	23.6	2.5
HOMIE (Qwen3-VL-2B)	2B	75.2	49.8	34.4	22.1
HOMIE (Qwen3-VL-8B)	8B	79.8	52.7	35.8	30.7

Ablation Study¶

All experiments are based on the Qwen3-VL-8B backbone, removing each component one by one to observe the change in Recall@1.

Configuration	Multi-Img→Text R@1	Img+Text→Img R@1	Img+Text→Text R@1	Video→Text R@1	Key Findings
HOMIE (Full)	79.8	52.7	35.8	30.7	Full model
w/o Progressive Course	76.3 (-3.5)	46.7 (-6.0)	30.9 (-4.9)	18.4 (-12.3)	Largest drop, verifying the necessity of morphology prior to diagnostic reasoning
w/o Data Filtering	79.5 (-0.3)	51.1 (-1.6)	34.6 (-1.2)	23.4 (-7.3)	Noisy data significantly harms video retrieval
w/o Stain Augmentation	77.7 (-2.1)	52.1 (-0.6)	35.1 (-0.7)	28.4 (-2.3)	Stain invariance is more critical for multi-image tasks
w/o Native Resolution	79.5 (-0.3)	46.3 (-6.4)	29.0 (-6.8)	23.8 (-6.9)	Fixed low resolution severely harms tasks requiring fine-grained details
EOL Prompt Robustness
No Prompt	78.0 (-1.8)	51.8 (-0.9)	34.3 (-1.5)	29.1 (-1.6)	Training enables the model to compress semantics at the final token
w/o "one word"	78.5 (-1.3)	51.5 (-1.2)	34.8 (-1.0)	30.0 (-0.7)	Weak prompt remains competitive
HOMIE (EOL)	79.8	52.7	35.8	30.7	"in one word" semantic bottleneck suppresses residual noise, yielding optimal results

Key Findings¶

Progressive Course Contributes the Most: Its removal causes Video-to-Text to drop sharply by 12.3 percentage points, indicating that the model indeed needs to establish a morphological foundation first before learning diagnostic reasoning, rather than simply mixing datasets.
Native Resolution is Crucial for Fine-Grained Tasks: Under a fixed low-resolution input, both Image-to-Text and Image+Text-to-Text drop by approximately 6-7 percentage points, whereas the "coarse-grained" Multi-Image to Text drops by only 0.3 percentage points—the latter likely compensates for resolution loss through multi-image complementarity.
Framework Generalization: Applying the HOMIE adaptation pipeline to various MLLM backbones such as Qwen2.5-VL-7B, HuatuoGPT-V-7B, Lingshu-7B, and PathoR1-7B consistently drives substantial improvements. Crucially, the pathology-specialized backbone (PathoR1-7B \(\to\) HOMIE 78.5%) continuously outperforms general-purpose backbones (Qwen2.5-VL-7B \(\to\) HOMIE 77.2%), validating the synergistic effect of domain priors + HOMIE.
Simple Retrieval Improves Instead of Dropping: HOMIE also comprehensively outperforms specialized dual encoders (such as MUSK and Patho-CLIP) in traditional Image-to-Text / Text-to-Image retrieval, showing that composed reasoning capability is not achieved at the expense of basic performance.
Modality Gap Significantly Reduced: UMAP visualization shows that HOMIE's modality gap \(\|\Delta\|_{gap}=0.325\), which is much lower than that of pathology CLIP models (\(\geq 0.571\)), verifying that the two-stage adaptation successfully maps vision and text to a shared metric space rather than adjacent disjoint territories.

Highlights & Insights¶

Three-"Mismatch" Diagnostic Framework: Breaking down the challenges into three tiers—architectural mismatch, task mismatch, and domain mismatch—with each tier addressed by a specific solution. This "diagnosis-prescription" paper structure is highly exemplary. Conducting domain transfer by systematically analyzing why existing methods fail and then tailoring solutions is far more persuasive than simply compiling modules.
Two-Stage Sequential Design: Stage 1 (pure-text retrieval adaptation) and Stage 2 (pathology multimodal tuning) are strictly serialized rather than jointly trained. The underlying insight is that the learning of the retrieval metric space and the acquisition of domain knowledge can interfere with each other if mixed. This sequential design of "learning to retrieve first, then learning pathology" can be transferred to other scenarios requiring domain-specific retrieval capabilities (e.g., legal document retrieval, satellite imagery retrieval).
Simplicity of the EOL Prompting Strategy: High-quality dense embeddings are extracted from causal LLMs simply by appending "in one word:" to the end of the prompt, without altering the model architecture or adding special tokens, making it highly lightweight. Ablations show that the model performs well even without EOL, indicating that the training process itself has already taught the model to compress semantic information at the final token—EOL serves more to align inference behavior with training representation rather than acting as the sole source of core capability. This finding, in turn, validates the effectiveness of the two-stage training.
Application of Bootstrapped Data Cleaning in Medicine: Using the model itself to score and filter noisy data is common in the natural image domain (BLIP-style), but validating its effectiveness on pathology image-text pairs is a practical contribution. Setting the threshold to 0.1 is quite conservative, indicating that Quilt-1M indeed contains a significant volume of low-quality pairs.
Scalability of the Framework-Agnostic Backbone: HOMIE remains effective when adapted to various MLLMs, with performance monotonically increasing alongside the backbone's capability. This implies that HOMIE can directly benefit from the emergence of stronger MLLMs, ensuring a robust lifecycle.

Limitations & Future Work¶

Composed Query Data Sourced from GPT-5 Rewriting Rather Than Real Clinical Logs: The paper candidly admits that the composed queries in the PCR Benchmark are reconstructed from existing datasets using GPT-5, which might have a distribution shift from the querying habits of real pathologists. The authors plan to collaborate with hospitals to collect actual clinical query logs.
Modality Limitations: Currently, HOMIE only fuses vision (image + video) and text, while modern comprehensive pathological diagnosis increasingly integrates genomics, transcriptomics, and patient clinical history. Future plans include expanding to multi-omics data.
High Computational Demand for Training: Although the inference stage requires only a 16-24GB GPU and about 50ms/query, the training stage requires 8x H100 GPUs—still a barrier for institutions with limited resources. The paper mentions exploring INT4/INT8 quantization and KV-cache optimization.
Trained Solely on Public Data: Although this lowers the barrier to data acquisition, it may limit the model's performance on rare cases. The paper does not discuss the fine-grained performance differences of the model across various cancer types.
Unexplored Joint Training with Pathology Report Generation: HOMIE focuses on retrieval, but if the LLM's generative capability is fine-tuned concurrently (similar to the joint objective of CoCa), it might endow report generation functionalities while maintaining retrieval power, further enhancing clinical utility.

vs. CONCH / MUSK (Pathology Dual Encoders): Based on CLIP/CoCa/BEiT3 architectures, these models utilize independent encoders for images and text, limiting inputs to a single image or a single text segment. HOMIE replaces the dual-encoder architecture with a unified MLLM, naturally supporting interleaved multimodal inputs. While CONCH/MUSK excel in simple retrieval (trained on a fixed 336x336 size), they degrade severely facing composed queries. HOMIE also surpasses them in simple retrieval, proving that native resolution + pathology-specific training does not compromise fundamental capabilities.
vs. E5-V / VLM2Vec / GME (General Multimodal Embedding): These works adapt MLLMs into retrieval models but focus on general domains. They perform extremely poorly on the PCR Benchmark (E5-V achieves only 2.3% R@1), exposing the domain mismatch—the vision encoder of general models cannot interpret pathomorphology. The core difference of HOMIE lies in adding pathology-specific adaptation layers (stain augmentation, native resolution, progressive course) and completing adaptation using only public data, bypassing the need for expensive pathology instruction-tuning data.
vs. Direct Pathology MLLMs (PathoR1, etc.): PathoR1-7B possesses rich pathology priors but achieves only 14.7% Recall@1 on PCR—generative capabilities do not automatically translate into retrieval capabilities. The fundamental contribution of HOMIE is bridging this gap: Stage 1 pure-text training forces the MLLM to learn a discriminative metric space. This insight is universally applicable: any generative foundation model must undergo dedicated metric space adaptation to be effectively utilized for retrieval.

Rating¶

Novelty: ⭐⭐⭐⭐ Formulates the PCR task and proposes a systematic adaptation framework for the first time. The triple-"mismatch" analytical perspective is innovative, and the application of the EOL strategy in the context of two-stage training possesses high originality.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 10+ dual-encoder baselines, 9 MLLM baselines, 5 backbone generalization experiments, complete ablations (components + prompt robustness), dual zero-shot composed and simple retrieval scenarios, UMAP visualizations, and qualitative case studies. The experimental design is exceptionally rigorous and comprehensive.
Writing Quality: ⭐⭐⭐⭐ The diagnosis framework of the three mismatches is clear and robust. The methodology is explained thoroughly, and the failure analysis visualization in Figure 1 directly supports the motivation.
Value: ⭐⭐⭐⭐ Pathology composed retrieval addresses real-world clinical needs. As a model-agnostic framework, HOMIE stands to benefit continuously from the advancement of stronger MLLMs. Its methods (two-stage adaptation, progressive course, EOL embedding extraction) are highly transferable to other professional retrieval scenarios such as legal or remote sensing retrieval.