MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Edisonhimself/MediRound
Area: Medical Imaging / Multimodal VLM
Keywords: multi-round dialogue, entity-level reasoning, medical segmentation, historical masks, quality correction

TL;DR¶

MediRound trains a medical dialogue model on MR-MedSeg to reference previously segmented entities, then improves masks passed across rounds with the Judgment & Correction Mechanism (JCM), increasing overall validation cIoU from 55.8 to 58.9.

Background & Motivation¶

Traditional medical segmentation usually answers where an organ is, while text-guided methods let users describe the target in language. Reasoning segmentation methods such as LISA further use a multimodal large language model to interpret implicit instructions and pass target semantics to a segmenter. Medical education, however, often proceeds by inspecting a structure and then asking where its blood flows or what the corresponding structure on the other side is. The referent is not merely a noun in the conversation: it is a specific image region returned earlier. A model retaining only text may know the relationship between atria and ventricles without identifying the particular entity the user means.

SegLLM has explored multi-round segmentation in natural images, but medical scenarios additionally require knowledge of anatomical hierarchy, lesion attachment, and physiological relationships. The proposed MEMR-Seg task therefore couples cross-round entity relationships with pixel-level outputs, rather than presenting a sequence of independent segmentation requests. A further difficulty is that training can provide correct historical masks, whereas deployment must reference the model's own predictions. If an earlier round segments the wrong entity, both the reference crop and its location can shift, propagating the error through later references. A multi-round system must therefore maintain visual context that future questions can trust, not merely answer the current question.

The authors address both data and inference: they construct medical entity-relation dialogues and explicitly supply the appearance and location of the referenced entity. Instead of passing every current output directly to future rounds, the model first estimates the reliability of its segmentation semantics. Core Idea: convert historical masks into referenceable entity evidence, and selectively correct segmentation features according to predicted quality before their masks become references for subsequent rounds.

Method¶

Overall Architecture¶

Inputs comprise the same medical image, the current question, existing question-answer history, and a historical mask selected by the user. Outputs are a textual response containing [SEG] and a binary mask for the current target. Entity-Relation Data Construction supplies training dialogues, Historical Entity Fusion supports cross-round understanding, and Judgment & Correction protects the segmentation outputs that later rounds will reference. LLaVA-Med-v1.5-Mistral-7B handles image-language understanding, while MedSAM provides image encoding and mask decoding. The original image serves different purposes in the two visual paths: global semantics for the language model and dense features for recovering boundaries in MedSAM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Existing medical images and annotations"] --> Data["Entity-Relation<br/>Data Construction"]
    Data -.->|Training supervision| Fusion["Historical Entity Fusion"]
    Input["Image, current question,<br/>dialogue history and reference mask"] --> Fusion
    Fusion -->|Current SEG feature| JCM["Judgment & Correction"]
    JCM --> Decoder["MedSAM decoding<br/>Current mask and textual response"]
    Decoder -->|Mask selected in a later round| Fusion

Data construction in the diagram is a preparation step, not a generation service invoked during each inference round. Historical Entity Fusion receives ground-truth historical masks during training and previously predicted masks during testing. Judgment & Correction is trained separately after the base model and conditionally activates its correction branch during free-running multi-round inference. LLaVA-Med supplies the textual response; JCM modifies the feature used for mask decoding rather than regenerating the entire dialogue.

Key Designs¶

1. Entity-Relation Data Construction: supervise references to previous results

MR-MedSeg builds on the curated images, masks, and labels of SA-Med2D-20M, starting from a candidate pool of 118K images and 569K masks. These are candidate-pool sizes, not necessarily counts of distinct images in the final training set. The authors first manually select subdatasets and medical entities suitable for multi-round interaction, then construct relationships between entities. The five relationship types cover organ-lesion dependency, anatomical hierarchy, organ/tissue attributes, spatial relationships, and strong inferential relationships. For example, identifying a lesion attached to an organ and identifying its contralateral counterpart require different knowledge, not simply alternative organ names. The resulting dataset covers 168 medical entities and 9 medical imaging modalities.

Relationship generation combines human annotation with GPT-5 and distinguishes whether a particular image is required. Stable anatomical or physiological attributes can be generated without image context, whereas image-dependent relationships such as relative positions explicitly use the corresponding image. GPT-5 rewrites each entity relationship into 50–80 semantically equivalent templates, after which entities are inserted manually and the dialogues are further refined. Three inspectors with medical expertise screen the data to reduce relationship errors and unnatural phrasing. Training, validation, and testing contain 174,934, 1,270, and 1,273 conversations, respectively, totaling 177,477. Template expansion broadens linguistic coverage but is not equivalent to collecting the same number of authentic student interactions.

2. Historical Entity Fusion: specify both the appearance and location of the referent

The current round does not necessarily reference the immediately preceding round; the user selects an existing round as the reference. If the fourth round references the second, the model obtains an original-image crop corresponding to the second-round mask and its bounding-box coordinates in the original image. The crop passes through the LLaVA-Med vision encoder and projection layer, while a separate linear layer encodes the box. The box encoder maps 4-dimensional coordinates to 4096 dimensions so that spatial information can be combined with the language model's input representations. The crop provides local entity appearance, whereas the box preserves global location, addressing distinct information gaps. The model also retains the original image, current question, and complete dialogue history, rather than performing segmentation solely on the crop.

These inputs are combined for LLaVA-Med, which generates a textual answer and emits [SEG] when a mask is needed. The final-layer hidden representation at that token becomes the semantic representation of the current target. Separately, the MedSAM image encoder extracts dense original-image features, and its mask decoder combines them with target semantics to produce a mask in full-image coordinates. The historical crop therefore helps determine which object to segment; it does not restrict the output to the old mask's spatial extent. This distinction matters for requests about a contralateral organ, whose target may lie entirely outside the reference region. When no mask is referenced, the algorithm permits reference round 0, avoiding an artificial history for the first question.

3. Judgment & Correction: intervene before an error becomes a future reference

The base model uses teacher forcing, obtaining reference regions from ground-truth masks during training. Testing is autoregressive and free-running, with each round restricted to earlier predictions; JCM addresses this difference in reference quality. Rather than applying morphological repairs directly to pixels, JCM introduces two lightweight MLPs operating on the [SEG] feature. The judgment module predicts the quality of the mask that the uncorrected feature would produce, while the correction module learns a feature better suited to decoding. Predicted quality \(q\) lies in \([0,1]\); increasing threshold \(\beta\) sends more features through correction. The following rule summarizes Algorithm 1 on page 8, with \(f\) denoting MedSAM image features and \(h_c\) the current segmentation semantic feature:

\[ \widehat M = \begin{cases} \operatorname{Dec}(f,h_c), & q>\beta,\\ \operatorname{Dec}(f,\operatorname{MLP}_C(h_c)), & q\leq\beta. \end{cases} \]

Selective correction avoids unconditionally changing already-correct features; the threshold controls modification of the segmentation representation, not binarization of pixel probabilities. The authors select \(\beta=0.6\) through validation-set threshold ablations. The corrected current mask enters the collection of reference candidates, so later rounds may benefit from both improved references and their own correction. This does not rerun every previous round or ask an external language model to reinterpret the question. If a historical entity is severely mismatched, lightweight feature correction may still fail to restore the intended reference; the paper offers no guarantee of eliminating such errors.

A Worked Example¶

Figures 1 and 4 use cardiac cavities to illustrate cross-round references; this walkthrough explains the process rather than treating the illustration as a quantitatively annotated test case. The first round requests the right atrial blood cavity and saves its mask as instance [1] in the dialogue. The second asks for the cavity receiving blood from instance [1], using its crop and location to infer the connected ventricle. The third returns to the first round to find the other atrium, showing that references need not form a strictly adjacent sequence. The fourth asks for the other ventricle relative to instance [2], so the actual reference is the second-round mask, not the third-round result. If the second-round feature has low predicted quality, JCM corrects it before decoding, potentially improving the entity evidence available to the fourth round. The example requires anatomical relationships, reference resolution, and pixel localization together, rather than simply replacing a relational phrase with a fixed organ name.

Loss & Training¶

Base-model training freezes the MedSAM image encoder and LLaVA-Med vision encoder, while adapting the language model with LoRA. The mask decoder and box encoder are fully trainable, and textual answers receive autoregressive cross-entropy supervision. Mask supervision combines pixelwise BCE and Dice losses with weights of 2.0 and 0.5, respectively. After base training, all MediRound components are frozen, and only the JCM judgment and correction MLPs are trained. The original feature takes both an uncorrected decoding path and a corrected path; the former supplies the quality target and the latter receives mask supervision. The quality label is not a manually assigned good/bad category, but overlap between the uncorrected mask and ground truth:

\[ Q=\operatorname{IoU}(\widetilde M,M) =\frac{|\widetilde M\cap M|}{|\widetilde M\cup M|}. \]

The judgment module minimizes BCE between prediction \(q\) and soft target \(Q\); the correction branch uses the same BCE and Dice mask losses described above. This trains the judgment module to estimate original-output quality, not directly to predict how much correction will improve it. Base training uses 4 RTX A6000 GPUs with 48 GB each, a per-device batch size of 15, and approximately 1.5 days; JCM requires another approximately 0.5 days. Optimization uses AdamW, a learning rate of 0.0003, WarmupDecayLR with 100 warm-up iterations, and LoRA rank 8. The judgment and correction MLP dimensions are [256, 512, 512, 1] and [256, 512, 512, 256], respectively. The text does not fully specify how the language model's final-layer representation interfaces with these 256-dimensional features, so the intermediate implementation projection should not be guessed. Equations (1)–(6) have multiple extraction artifacts in the cache; this note explains losses and branching from the prose and algorithm without presenting reconstructed damaged expressions as exact author equations.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1 on page 11, retaining its percentage scale and aggregating all dialogue rounds in MR-MedSeg. Validation columns report overall performance, while testing is split into regular and hard cases; the text describes the latter only as more challenging cases. SegLLM is fine-tuned on MR-MedSeg, whereas Human-Thinking and external-MLLM combinations are evaluated zero-shot, so their training budgets are not equivalent.

Method	Val Dice	Val gIoU	Val cIoU	Test regular Dice	Test hard Dice
Human-Thinking + MediSee	43.0	33.5	35.4	45.6	45.1
Qwen3-VL + MediSee	42.7	34.0	34.5	45.2	37.8
SegLLM-13B	38.9	31.2	39.1	45.3	33.0
MediRound + READ	53.9	45.1	54.6	60.3	47.2
MediRound	55.8	46.5	55.8	61.0	48.1
MediRound + JCM	58.4	49.0	58.9	63.3	50.3

The cleanest mechanism comparison is MediRound with versus without JCM: validation Dice increases by 2.6 points and cIoU by 3.1 points. The available experimental prose does not detail the aggregation definitions of gIoU and cIoU, so their labels are retained without reinterpreting gIoU as generalized box IoU from object detection.

Ablation Study¶

Table 5 on page 13 compares reference-mask representations on the MR-MedSeg validation set as an input ablation of base MediRound.

Reference crop	Reference box	Dice	gIoU	cIoU
Disabled	Disabled	41.9	32.5	38.4
Enabled	Disabled	55.4	45.8	56.0
Enabled	Enabled	55.8	46.5	55.8

Adding the crop increases cIoU from 38.4 to 56.0, indicating that explicit historical visual entities matter beyond retaining more text turns. Adding the box improves Dice and gIoU but changes cIoU from 56.0 to 55.8; the authors' general description of the configuration as best does not establish superiority on every metric. Table 2 on page 12 reports validation cIoU by round, allowing inspection of whether JCM only improves early interactions.

Method	Round 2	Round 3	Round 4	Round 5	Round 6	Round 7	Round 8
MediRound	50.2	60.8	55.9	58.8	63.7	64.7	46.1
MediRound + JCM	52.6	63.6	59.7	64.6	69.5	66.3	54.8

Key Findings¶

Round 8 cIoU rises from 46.1 to 54.8, an 8.7-point gain supporting the usefulness of correction in longer dialogues.
Baseline scores are not monotonically decreasing across rounds, and sample composition may differ, so Table 2 alone is not a causal experiment on error propagation.
In the single-round evaluation of Table 3 on page 13, MediRound achieves Dice 62.1 versus MediSee's 61.2, but their cIoU scores are 66.3 and 70.8, respectively; this does not support universal superiority.
Figure 9 on page 15 surveys 15 medical students, reporting practicality 3.80, helpfulness 3.93, convenience 4.33, and potential 4.07 on a 1–5 scale; these are subjective ratings, not a learning-outcome trial.

Highlights & Insights¶

A historical mask is both a previous answer and a visual entity that can be referenced later. Reintroducing its crop and location provides more direct evidence than text-only history.
JCM judges a pre-decoding semantic feature using supervision derived from final-mask IoU. This connects lightweight quality estimation to the downstream pixel task.
Correction occurs before the current output becomes future context, so benefits may continue along reference dependencies. This could inform other stateful visual interactions, although cross-task transfer is not evaluated here.

Limitations & Future Work¶

Data largely comes from medical-relation templates, GPT-5 rewriting, and human screening; ambiguity, follow-up questions, and errors in real user language still require validation.
The text reports conversation splits without fully specifying patient-level or source-image-level separation; dialogue counts alone do not establish the absence of correlated-sample leakage.
Both the base model and JCM use teacher forcing during training, so correction does not eliminate the protocol mismatch with free-running testing.
Feature adjustment cannot guarantee detection of an incorrect entity, absent target, or false medical relationship; educational deployment still needs human confirmation and abstention mechanisms.
Multimodal scale, cross-domain generalization, and authentic learning benefits need further assessment; a 15-person satisfaction survey cannot establish clinical reliability or improved knowledge acquisition.

Compared with SegLLM: both use cross-round segmentation entities, but MediRound adds medical-relation data, medical backbones, and JCM for error propagation; it is not the first general multi-round segmentation approach.
Compared with LISA and MediSee: these methods establish interfaces between language reasoning and segmentation, while the current task makes targets depend on selected historical masks, beyond single-round competence.
Compared with MedSAM: MedSAM supplies pixel localization, while MediRound handles conversational references and entity semantics; ground-truth box-guided results are not language-reasoning baselines of equivalent difficulty.
Compared with MTurn-Seg: the authors acknowledge concurrent work on a bilingual multi-turn medical dialogue segmentation benchmark, so MediRound should be positioned as a specific advance in medical entity-level reference tasks and methods.

Rating¶

Novelty: 4/5, a clear combination of medical entity-relation tasks and selective feature correction, although the basic multi-round interface follows earlier work.
Experimental Thoroughness: 4/5, with overall, per-round, backbone, and input analyses, but without authentic learning-outcome trials or stronger generalization validation.
Writing Quality: 4/5, a clear task and inference pipeline, with some implementation interfaces and metric definitions insufficiently specified.
Value: 4/5, a useful research direction for interactive medical segmentation, but not sufficient evidence for direct educational or clinical decision-making deployment.