MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution

Conference: ECCV2026
Paper: ECCV Paper
Area: Medical Imaging
Keywords: latent diagnostic memory, anatomical priors, counterfactual rewards, knowledge distillation, medical visual question answering

TL;DR

MedSynapse-V compresses anatomical priors from a medical segmentation encoder into continuous memory, trains the model to use that memory through region-masking counterfactual rewards, and distills it into an autonomous branch without the external encoder, achieving a reported seven-benchmark average of 59.6 on offline medical evaluations, not evidence of clinical effectiveness.

Background & Motivation

Medical vision-language models must connect boundaries, tissue textures, and spatial relationships to a particular question, not merely recognize objects. Explicit chain-of-thought can organize an answer into fluent text without ensuring that every step relies on visual evidence: an early mistake about lesion location may simply propagate through the remaining explanation. The paper motivates continuous memory through the limited expressiveness of discrete words for subtle visual changes and the possible weakening of visual evidence during long generation. This is a modeling argument, not an established general law of cognition.

Adding retrieved text or learnable prompts alone does not establish whether a model uses medical evidence. External knowledge can occupy context while being bypassed, and a general latent reasoning method need not acquire anatomically meaningful representations. The paper therefore asks how segmentation pretraining can inform answer generation and how the resulting model can eventually dispense with its auxiliary encoder, rather than asking how to generate more explanation.

Core Idea: condense anatomical priors into case-dependent continuous memory, use masking interventions to test the model's reliance on memory evidence, and transfer the prior-conditioned teacher's output distribution to autonomously generated memory. The term "clinical intuition" is the authors' analogy for rapid pattern retrieval; it does not mean that the model possesses a physician's experience or judgment.

Method

Overall Architecture

The input is a medical image and a clinical question, and the output is a diagnosis-related answer. Training proceeds through Meta Query for Prior Memorization (MQPM), Causal Counterfactual Refinement (CCR), and Intrinsic Memory Transition (IMT). These stages address where memory comes from, how the model uses it, and how deployment avoids dependence on external priors.

Early in training, frozen MedSAM3 supplies spatial features, which a memory sampler converts into continuous vectors matching the VLM hidden dimension. The vectors appear after the question encoding and before answer generation, participate in self-attention, and are not emitted as readable words. The final deployment branch generates memory from the VLM's own visual features, retaining a lightweight autonomous memory module while removing MedSAM3 and the prior-memory sampling pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Medical image and question"] --> MQPM["Meta Query for<br/>Prior Memorization"]
    Prior["Frozen MedSAM3"] --> MQPM
    MQPM --> CCR["Causal Counterfactual<br/>Refinement"]
    CCR --> IMT["Intrinsic Memory<br/>Transition"]
    IMT --> Output["Autonomous memory and answer<br/>MedSAM3 removed at inference"]

The arrows represent training-stage progression, not a requirement to execute the entire training chain for every inference case. Memory here is an input-dependent hidden representation, not a persistent database of patient cases or a mechanism for accumulating experience online.

Key Designs

1. Meta Query for Prior Memorization: turn spatial priors into usable context

MedSAM3's final feature map contains spatial information learned through segmentation pretraining. The paper flattens it into a feature sequence and uses learnable meta-query probes as queries, with spatial features as keys and values, to aggregate information through cross-attention. The default uses 16 probes and a two-layer cross-attention Transformer sampler with output dimension 4096, matching the Qwen3-VL-8B hidden state.

These are not manually named "boundary probes" or "texture probes." The authors use such pathological semantics to illustrate possible learned attention patterns; this does not establish a fixed, validated interpretation for every vector. Their operational role is to select compact representations from many spatial locations instead of inserting the entire segmentation feature map into the language model.

The continuous vectors enter the attention context shared with visual, question, and answer tokens, allowing generation to access priors without first translating every visual detail into words. Randomly initialized vectors could nevertheless disturb the original model. Stage I therefore freezes the VLM and MedSAM3 and trains only the memory sampler with reference-answer next-token prediction, establishing a usable representation interface before refinement.

2. Causal Counterfactual Refinement: reward answer paths that rely on diagnostic regions

After alignment, the model could still answer through existing visual or linguistic shortcuts and treat memory as unused padding. CCR freezes the memory sampler and VLM backbone and updates only VLM LoRA adapters. It samples 4 candidate answer trajectories per example and uses GRPO with group-relative rewards. The directly trained behavior is therefore how the model uses memory through attention, not the sampler itself.

The counterfactual condition uses high-confidence diagnostic-region masks from MedSAM3: corresponding spatial features are zeroed, and the same frozen sampler constructs intervened memory. The original image and question remain fixed. The comparison concerns support for the same answer trajectory under original versus intervened memory. The accuracy reward checks whether the final answer matches the reference, while the counterfactual reward encourages genuine dependence on information contributed by the masked regions.

The composite relationship is intact in the source:

\[ R(o)=\lambda_{\mathrm{acc}}r_{\mathrm{acc}}(o)+\lambda_{\mathrm{causal}}r_{\mathrm{causal}}(o). \]

The two weights are 1.0 and 0.5, respectively. The paper interprets a positive counterfactual reward as evidence that the original memory contributes to the current diagnosis. However, the extracted trajectory-probability expression is corrupted. This note does not guess its summation, normalization, or probability-ratio details and does not present it as an implementation-ready equation.

"Refinement" also needs a precise reading. Although the narrative discusses removing redundancy and retaining useful components, the method gives no explicit operation that deletes a memory slot, and the sampler remains frozen. The defensible interpretation is that the policy learns to emphasize diagnostically relevant memory evidence. Region masking measures model dependence under an artificial intervention, not automatically causal identification of disease mechanisms; erroneous masks and unusual zeroed features may also affect the signal.

3. Intrinsic Memory Transition: learn the teacher's behavior rather than copy memory coordinates

The post-CCR teacher still obtains privileged memory from MedSAM3 and the sampler. IMT adds an autonomous student branch whose two-layer MLP plus LayerNorm memory module operates on the VLM's own visual encoding features. Teacher and student share VLM backbone parameters; the distinction is whether the backbone receives memory derived from external priors or the autonomous module.

Training first samples an answer trajectory from the student, then compares teacher and student full-vocabulary next-token distributions under identical history prefixes, aligning them across generation positions. The divergence is generalized Jensen-Shannon divergence with mixture coefficient 0.5. Its extracted equations are also damaged, so this note retains the verifiable training semantics without reconstructing the exact loss expression.

The supervision does not require coordinate-wise equality between memory vectors; it encourages similar predictive behavior. The student receives guidance on prefixes it actually visits and can learn from probability information beyond the final correct token. The teacher is a fixed target, and gradients update only the autonomous memory module. That module remains at deployment, so the method is not literally free of additional components.

A Worked Example

Consider "Is there a pulmonary nodule in this CT image?" as a process illustration, not an additional measured case or numerical prediction. During training, MedSAM3 supplies spatial features, 16 probes condense memory, and the VLM reads this continuous context before answering. Warmup uses the reference answer to establish semantic alignment.

CCR produces 4 candidate trajectories for the same input. Training then masks features from candidate diagnostic regions, generates intervened memory, and compares support for each trajectory under the two memory conditions. A correct answer independent of these regions can receive a different composite reward from a correct answer that uses regional information. This still does not guarantee that the mask selected the lesion correctly.

IMT subsequently trains the student to produce memory and answers without MedSAM3 while the teacher supplies distributional supervision on the student's prefixes. For a new image at deployment, only the VLM and autonomous memory module run; region interventions, teacher inference, and GRPO updates are no longer performed.

Loss & Training

Stage I uses 50K PubMedVision image-text pairs covering radiology and pathology, a learning rate of \(2\times10^{-4}\), and 3 epochs. Stage II combines 3K closed-ended OmniMedVQA examples with 1K open-ended examples from the SLAKE and PathVQA training sets, totaling approximately 4K across 8 imaging modalities.

Stage II uses LoRA rank 64 on all attention layers, a GRPO clipping coefficient of 0.2, rollout batch size 32, and maximum generation length 1024 tokens. The implementation details specify 200 training steps, but Figure 7 extends to 2000 and the surrounding text discusses changes near step 900. These accounts conflict, so the actual training budget cannot be determined from them.

Stage III reuses Stage II data and samples one student trajectory per example at each gradient update, with learning rate \(1\times10^{-4}\) for 3 epochs. The damaged CCR and IMT equations still require checking against the original typesetting or implementation, but the central sequence remains understandable: reward memory dependence, then distill predictive behavior.

Key Experimental Results

Main Results

The following subset comes from source Table 1. MMMU* denotes Health & Medicine, MedXpert denotes MedXpertQA-MM, and GMAI denotes GMAI-MMBench. The four VQA tasks report accuracy (%); the latter two benchmarks use their official protocols. Average reproduces the paper's seven-benchmark column, not clinical accuracy pooled across all cases. The source states that results average five independent runs, but the table does not show variability.

Method	VQA-RAD	SLAKE	PathVQA	PMC-VQA	MMMU*	MedXpert	GMAI	Seven-benchmark average
Qwen3-VL-8B	58.6	66.2	55.4	42.5	48.3	22.1	47.2	48.6
MMedExpert-R1-7B	65.2	72.8	58.1	56.8	57.3	27.5	52.1	55.7
IVT-LR, adapted to the same 8B backbone	62.3	70.1	56.2	47.8	50.4	23.5	43.1	50.5
MedSynapse-V-8B, encoder retained	75.6	81.4	66.2	59.8	62.7	29.4	54.8	61.4
MedSynapse-V-8B, IMT	74.2	79.8	64.8	58.5	61.4	26.8	51.6	59.6

IMT exceeds MMedExpert-R1 by 3.9 percentage points in the seven-benchmark average, but not on every task: MedXpert is 26.8 versus 27.5, and GMAI is 51.6 versus 52.1. The encoder-retaining variant leads each benchmark in the table; its conclusion should not be transferred wholesale to the deployment branch.

Ablation Study

The following results come from source Table 2, whose average covers only the five displayed tasks. Consequently, the full model's 67.7 does not conflict with the main table's seven-benchmark average of 59.6. Apart from the encoder-retaining row, results represent inference after IMT without the external encoder.

Config	VQA-RAD	SLAKE	PathVQA	PMC-VQA	MMMU*	Five-benchmark average
Zero-shot Qwen3-VL-8B, no memory	58.6	66.2	55.4	42.5	48.3	54.2
MQPM → SFT → IMT	63.2	71.2	60.4	49.8	51.3	59.2
Skip MQPM, direct RL → IMT	57.4	64.3	54.8	43.1	44.7	52.9
Accuracy reward only	70.1	75.8	61.2	54.3	56.6	63.6
MQPM → CCR → IMT, full model	74.2	79.8	64.8	58.5	61.4	67.7
MedSAM3 encoder retained	75.6	81.4	66.2	59.8	62.7	69.1
Randomly initialized encoder → IMT	56.4	64.0	55.1	41.2	43.3	52.0

Key Findings

Adding the counterfactual reward raises the five-benchmark average from 63.6 to 67.7, a 4.1-point gain supporting its usefulness within this training setup, not independently proving clinical causal reasoning. Skipping MQPM yields 52.9, below the zero-shot 54.2. The nearby source wording "barely above zero-shot" contradicts the table, which is followed here.

Removing the encoder costs 1.4 points on the five-benchmark average and 1.8 points on the seven-benchmark average, indicating a larger effect on the additional difficult benchmarks. Probe-count analysis qualitatively supports a setting near the default 16; increasing capacity is not necessarily beneficial. Fine-grained curve values are not read from the damaged extraction.

Latency values are omitted: implementation details use ms/sample, Table 2 labels ms/token, and the efficiency analysis also uses s/sample. These units must not be mixed or used to directly infer a speedup.

Highlights & Insights

The most reusable distinction is between possessing expert features and actually using them. Warmup makes the features readable, while the counterfactual reward makes their use advantageous. This adds a behavioral constraint beyond simply attaching an external encoder to a VLM.

IMT transfers predictive distributions rather than memory coordinates. This suggests a route for other visual tasks with expensive training-time experts and lightweight deployment requirements, but the student's degradation on difficult subsets deserves evaluation alongside its average score.

Limitations & Future Work

The evidence comes from offline medical question answering and multimodal benchmarks, not prospective clinical trials, patient outcomes, or real-workflow evaluation established here. Benchmark improvements do not establish diagnostic safety, clinical effectiveness, or replacement of physicians. External-center testing, calibration, abstention, and human-AI collaboration evaluation remain necessary.

Region masking depends on MedSAM3 mask quality, and zeroing can create out-of-distribution features. Random-region and area-matched intervention controls would help distinguish reliance on the correct lesion from sensitivity to any feature disruption. Clustering in a t-SNE plot of latent memory also cannot independently establish causal faithfulness.

The 200/2000-step conflict, corrupted CCR/IMT equation extraction, and mixed latency units limit exact reproduction. Main-table methods differ in scale and training background, and the five-run results lack accompanying variance, so every performance gap cannot be attributed to a single module.

The authors propose extensions to longitudinal analysis, multimodal report generation, and larger differential diagnosis spaces. These are future directions; current results do not establish working cross-time memory updates or real clinical multi-evidence integration.

Related Work & Insights

Compared with medical CoT reinforcement learning methods such as Med-R1 and MedVLM-R1, this work emphasizes the memory condition and its regional dependence rather than longer explicit reasoning. Shorter outputs can reduce generation work without making answers more interpretable or auditable.

Compared with general latent reasoning methods such as Coconut and IVT-LR, the approach explicitly borrows medical segmentation priors and adds counterfactual calibration. Adaptations with the same backbone and data provide a more direct comparison, but do not establish that all latent reasoning tasks require an anatomical encoder.

Compared with RAG or input-prefix methods, the key difference is transferring the effect of external priors into an autonomous module so deployment no longer retrieves external cases. It consequently lacks the explicit source tracing of an updatable knowledge base, motivating future interfaces for auditing memory evidence.

Rating

• Novelty: 4/5. Anatomical priors, regional counterfactual rewards, and autonomous memory distillation form a coherent training pipeline.
• Experimental Thoroughness: 3/5. Multiple benchmarks and stage ablations are useful, but statistical reporting, reproducibility, and external clinical validation remain limited.
• Writing Quality: 3/5. The method is understandable, but conflicting training budgets and efficiency units weaken reproducibility.
• Value: 4/5. A research route for training-time expert assistance and lightweight deployment, not evidence of clinical readiness.