Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation¶

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/4207
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/4937.pdf
Code: https://github.com/Yundi218/Concept-to-Pixel
Area: Model Compression (knowledge distillation; applied to medical image segmentation)
Keywords: knowledge distillation, universal medical segmentation, disentangled concepts, dynamic convolution, geometric consistency

TL;DR¶

C2P distills model-generated medical semantics into semantic tokens, supervises geometric tokens with mask attributes, and uses them to generate image-specific convolution kernels and geometry-consistency fusion weights, achieving 88.22% average Dice across eight datasets with one model and no manual prompts or reference images, but without robust transfer to every structural category.

Background & Motivation¶

Universal medical segmentation aims to handle ultrasound, MRI, CT, pathology, and other images with one parameter set rather than maintaining a separate model for each task. The difficulty extends beyond color and noise: similar pixel appearances can represent different tissues, so joint optimization on mixed datasets can conflate incompatible appearance rules. Interactive segmentation models can use points or boxes to specify a target, but still require manual input. In-context methods such as UniverSeg and Spider instead define the task through reference image–mask pairs, making deployment dependent on reference quality and support-set preparation.

C2P separates the questions of target shape and location from the meaning of texture under a particular imaging condition. The former can be computed from annotated masks and shared across modalities; the latter requires medical concepts and modality context. Adding unconstrained learnable tokens would not guarantee this division of labor. The paper therefore assigns them verifiable geometric regression targets and medical-text alignment targets, then makes both representations participate in pixel prediction rather than remain auxiliary training branches.

Core Idea: turn medical knowledge into explicitly supervised geometric and semantic tokens, using semantic distillation for appearance understanding and geometric supervision for structural constraints, then let those concepts directly control image-specific segmentation boundaries and prediction self-checks.

Method¶

Overall Architecture¶

The input is a medical image and the outputs are foreground and background segmentations. Test time requires no points, boxes, reference masks, or newly generated text. Two types of supervision are prepared offline: geometric attributes extracted from ground-truth masks, and structured descriptions generated for training images by a multimodal large language model and encoded with PubMedBERT. During training, Concept Formulation and Modality Injection creates tokens with distinct roles, Bidirectional Concept Interaction connects them to visual features, and the Concept-Guided Dynamic Head produces the masks.

At inference, the same model can also process perturbed views such as flips, with Geometry-Aware Consensus deciding which predictions deserve to be combined. “Prompt-free” describes the deployment interface; it does not mean training uses neither textual knowledge nor pixel annotations. Semantic distillation retains offline model knowledge inside the segmentation network rather than invoking the large model online for segmentation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Medical image<br/>offline supervision during training"] --> B["Concept Formulation and Modality Injection"]
    B --> C["Bidirectional Concept Interaction"]
    C --> D["Concept-Guided Dynamic Head"]
    D --> E["Geometry-Aware Consensus"]
    E --> F["Final segmentation mask"]

Key Designs¶

1. Concept Formulation and Modality Injection: share shape knowledge without forcing appearance knowledge to be shared

Geometric tokens [GEO] are not uninterpreted latent variables: each receives explicit attribute supervision. The paper uses 9 attribute categories—bounding box, area, perimeter, aspect ratio, compactness, centroid, eccentricity, orientation, and solidity—computed offline from training masks. Area and centroid constrain size and location, while the remaining attributes add contour and shape information. These targets make geometric representations accountable to actual structure, reducing the chance of treating an imaging-specific brightness pattern as target identity. However, cross-modality sharing does not imply applicability to every topology: priors such as compactness remain limited by the shapes of training targets.

Semantic tokens [SEM] learn a different type of evidence. Qwen3-VL-Plus generates descriptions of training images across 9 dimensions: morphology, margin definition, internal texture, surrounding interaction, boundary distinctness, malignancy risk, pathological inference, differential reasoning, and predicted diagnosis. PubMedBERT encodes them as 768-dimensional embeddings for semantic alignment. These are model-generated training targets, not manually confirmed diagnostic reports. Because concepts such as hypoechogenicity depend on modality, the Style-Content Fusion Module (SCFM) extracts channel-wise means and standard deviations from shallow E2 features and globally average-pooled content from deep E5 features. A gated MLP fuses them and injects modality information only into semantic tokens. Geometric tokens do not receive this modality bias, giving structural sharing and appearance differentiation separate entry points.

2. Bidirectional Concept Interaction: let concepts read the image and let the image receive concept constraints

If semantic tokens only memorized offline text and geometric tokens only memorized average training shapes, neither would describe the current case. C2P flattens and projects deep E5 features, adds learnable two-dimensional positional encoding, concatenates the two token groups, and applies multilayer bidirectional cross-attention. Concept tokens first serve as queries over visual keys and values, allowing representations of margins, position, and related properties to collect evidence from the current image. Residual connections and feed-forward networks update them from global learnable priors into sample-specific concepts.

The direction then reverses: visual features query the updated concept tokens. This lets pixel features incorporate structural and semantic constraints, suppressing background that has local texture responses but does not match the target concepts. The two attention steps are not disconnected branches. The first adapts concepts to the case; the second returns that adaptation to the image, so decoding retains spatial evidence while receiving high-level guidance. The positional encoding preserves visual layout and is not an additional source of geometric supervision.

3. Concept-Guided Dynamic Head: turn concepts into the weights that classify pixels

A static segmentation head shared across imaging conditions may still force different cases into fixed pixel-classification rules. The Concept-Guided Dynamic Head, corresponding to the paper’s Token-Guided Dynamic Head (TGDH), does not simply attach a classifier to the tokens. It first decodes the interacted visual features into high-resolution D2 features, then globally average-pools D2 to obtain an image query. Two parallel cross-attention paths use this query to aggregate the final geometric and semantic tokens separately, allowing the current image to determine how much each type of concept should contribute.

The geometric aggregate, semantic aggregate, and image query enter a kernel-generation MLP together. It generates separate foreground and background convolution parameters, which are applied to D2 to produce pixel probabilities for both classes. The head is dynamic because its parameters are regenerated for each image, not because a task identifier selects a predefined classifier. Retaining the image query matters: kernel weights can incorporate overall image context alongside abstract concepts, while high-resolution D2 preserves the spatial details needed for boundaries.

4. Geometry-Aware Consensus: downweight unreliable views using disagreement between two prediction paths

For multiple perturbed views of an image, the network predicts both segmentation masks and area and centroid values regressed from geometric tokens. The system also computes area and centroid directly from each mask, comparing the target expected by the concept branch with the target drawn by the pixel branch. Greater disagreement yields a lower confidence weight. The paper uses an exponentially decaying penalty combining area and centroid-location differences, plus additional false-positive suppression when a foreground prediction contradicts a near-zero regressed area.

The final prediction is a weighted fusion rather than the equal averaging used in standard test-time augmentation. The following fusion relation is given in the paper, where \(M_i\) is a view mask participating in fusion and \(w_i\) is its geometric-consistency weight:

\[ M_{\mathrm{final}}=\frac{\sum_i w_i M_i}{\sum_i w_i}. \]

This mechanism measures internal self-consistency, not ground-truth accuracy. If both prediction paths inherit the same incorrect shape prior, they can agree while being wrong. It should therefore be understood as outlier-view suppression rather than an independent measure of clinical trustworthiness. Operators in the cached weight equation are corrupted by text extraction, so unverified coefficients, thresholds, and an exact penalty formula are not reconstructed here.

Loss & Training¶

The overall objective jointly trains pixel prediction, geometric regression, and semantic alignment:

\[ \mathcal{L}_{\mathrm{total}}=\lambda_{\mathrm{seg}}\mathcal{L}_{\mathrm{seg}}+\lambda_{\mathrm{geo}}\mathcal{L}_{\mathrm{geo}}+\lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}. \]

The segmentation loss uses structure and Dice losses for both foreground and background. The structure term increases boundary-region weights using the difference between the mask and its \(31\times31\) average-pooled version, then combines weighted Focal and IoU terms; the Focal parameter is \(\gamma=2.0\). Geometric attributes receive mean squared error supervision against normalized targets. The detailed semantic loss is described as cosine embedding alignment. The earlier generic phrase “contrastive loss” does not justify assuming an InfoNCE objective with negative samples.

The implementation section specifies a ConvNeXt-B backbone, \(384\times384\) inputs, Adam, batch size 8, 50 training epochs, an initial learning rate of \(10^{-4}\), and cosine decay. However, the ablation section describes the baseline backbone as ConvNeXtV2 without fully clarifying the version relationship. The main text also does not specify the numerical weights of the three loss terms. The available file contains the complete main paper and references but not the cited appendix, so preprocessing configurations and additional hyperparameters are not invented here.

Key Experimental Results¶

Main Results¶

Joint training covers 8 datasets and 7 modalities. Average Dice averages the dataset-level results; values are percentages and higher is better. The following selection from Table 1 distinguishes specialized models, with one model per task, from universal models, with one model across tasks. A better average does not imply winning every individual task.

Method	Setting	Average Dice ↑	Polyp Dice ↑	COVID Dice ↑	AMDSD Dice ↑	BTD Dice ↑
nnUNetV2	Specialized	86.68	90.47	79.44	88.01	86.32
RWKV-UNet	Specialized	87.38	91.41	81.41	85.65	85.15
Spider	Universal	87.32	92.40	83.14	83.21	84.66
SR-ICL	Universal	87.25	92.22	82.92	84.88	85.37
C2P	Universal	88.22	93.13	83.91	86.27	85.73

C2P improves average Dice over Spider by 0.90 percentage points and over nnUNetV2 by 1.54 percentage points, but remains below nnUNetV2 on AMDSD and BTD. The following zero-shot results are selected from Table 2; all values are Dice (%) ↑. C2P adds no new tokens, generates no test-time text, and uses no support images. Spider uses 64 reference images and SR-ICL uses self-referencing refinement, so the resource conditions differ.

Dataset	Transfer type	Spider (64 references)	SR-ICL (self-reference)	C2P (no references)
TN3K	New dataset, seen ultrasound modality	74.24	70.59	75.14
ACDC	New dataset, seen MRI modality	63.26	59.95	71.87
BUSBRA	New dataset, seen ultrasound modality	85.16	80.70	83.39
GlaS	New dataset, seen pathology modality	56.42	53.78	53.99
Montgomery	Unseen X-Ray modality	83.89	67.00	87.60
ISBI EM	Unseen electron microscopy modality	64.63	83.21	26.79

Table 2 reports 75.14 for TN3K, whereas another passage reports 75.15; this note consistently uses the table value. The table caption also incorrectly calls ultrasound unseen. According to the data setup, ultrasound participates in training; the additional unseen modalities are X-Ray and electron microscopy.

Ablation Study¶

The following selection from Table 3 reports average Dice (%) ↑ across the eight datasets. Rows represent different training or inference configurations, not a fully symmetric set of one-component removals from the final model.

Config	Average Dice ↑	Note
No concept tokens, static head	82.38	Joint-training baseline
Semantic tokens only, static head	87.15	Semantic supervision and guidance alone
Geometric tokens only, static head	87.31	Geometric supervision and guidance alone
No concept tokens, dynamic head	87.19	The dynamic head alone also brings a substantial gain
Both token types, dynamic head, no augmentation	87.79	Full training architecture
Both token types, dynamic head, standard TTA	88.16	Multiview averaging
Both token types, dynamic head, Geometry-Aware Consensus	88.22	Full inference configuration

Key Findings¶

Geometric tokens with a static head improve on the baseline by 4.93 percentage points; semantic tokens with a static head improve it by 4.77 percentage points. However, a dynamic head without concepts reaches 87.19, so the entire gain cannot be attributed to semantic distillation.
Geometry-Aware Consensus adds only 0.06 percentage points over standard TTA, while standard TTA adds 0.37 percentage points over the unaugmented configuration. Most gains come from the training architecture rather than the final reweighting.
Zero-shot advantages depend on structure: ACDC and Montgomery are strong results, BUSBRA does not surpass Spider, and ISBI EM lags substantially. t-SNE and attention visualizations support the proposed division of representational roles but do not establish causality.

Highlights & Insights¶

Geometric supervision serves two purposes: defining token meaning during training and checking masks through the same attribute system at inference. Reusing this signal directly connects self-checking to the training objective instead of requiring a separately trained quality estimator.
The large model participates in offline knowledge preparation, while deployment uses the segmentation network. Removing inference prompts should therefore be distinguished from eliminating training costs: the former is supported by the experiments, whereas the latter is not the paper’s claim.

Limitations & Future Work¶

The authors explicitly acknowledge that compact-structure priors do not suit elongated neural structures in electron microscopy, and propose broadening training to vessels and neurons. The 26.79% Dice on ISBI EM directly demonstrates this boundary rather than a negligible degradation.
The main tables provide neither repeated-run variance nor significance testing, and no independent ablation of semantic-report quality control is shown. The 0.06-percentage-point inference gain and potential errors in generated semantics therefore warrant caution.
These experiments concern image foreground/background segmentation. They do not establish arbitrary-organ multiclass segmentation, consistency across three-dimensional volumes, or clinical safety. Backbone naming and unavailable appendix configurations also limit reproduction from the current text alone.

vs Spider / UniverSeg: Reference image–mask pairs help define the task, whereas C2P internalizes task-related knowledge as supervised concepts. This removes reference-set preparation at test time but loses the flexibility of redefining out-of-distribution targets through reference examples.
vs SR-ICL: SR-ICL uses self-reference and iterative refinement; C2P uses agreement between geometric and mask branches to weight multiple views. Both include inference-stage processing, so being reference-free does not by itself establish lower latency for C2P.
vs nnUNetV2: C2P achieves a higher cross-task average with one model, while nnUNetV2 retains advantages on AMDSD and BTD. Universal-model benefits do not replace task-specific evaluation.

Rating¶

Novelty: 4/5 — A coherent combination of semantic distillation, explicit geometric supervision, a dynamic head, and geometric self-checking; the individual operators are not entirely new.
Experimental Thoroughness: 4/5 — Covers multiple modalities, external datasets, and informative failure cases, but statistical stability and some configurations remain insufficiently documented.
Writing Quality: 3/5 — The main methodological story is clear, with minor inconsistencies in numbers, modality labels, and backbone naming that require checking.
Value: 4/5 — Offers a universal segmentation path without test-time prompts and makes the applicability boundary of structural priors visible.