Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models¶

Conference: ECCV 2026
arXiv: 2511.14900
Code: https://github.com/l593191569/Skin-R1
Area: Multimodal VLM / Medical Imaging
Keywords: Dermatological Diagnosis, Vision-Language Models, Clinical Reasoning, Reinforcement Learning, Textbook Knowledge

TL;DR¶

Skin-R1 proposes a three-stage training framework: first, it automatically constructs a hierarchical perception and differential diagnosis reasoning trajectory dataset, SkinRationale, from authoritative dermatology textbooks; next, Supervised Fine-Tuning (SFT) is utilized to initialize the model's clinical reasoning capabilities; finally, Group Relative Policy Optimization (GRPO) reinforcement learning, paired with a hierarchical reward design, generalizes the reasoning capability to large-scale, sparsely annotated datasets, comprehensively outperforming existing Med-VLMs across multiple dermatological diagnosis benchmarks.

Background & Motivation¶

Background: Dermatological diagnosis is fundamentally a visual reasoning task—clinicians must interpret subtle visual patterns, compare them against hierarchical disease classification taxonomies, and perform differential diagnosis (DDx) to distinguish visually similar conditions. While auxiliary diagnostic systems based on Multimodal Vision-Language Models (VLMs) have recently shown potential, existing methods still face three core challenges. First is data heterogeneity: real-world dermatology datasets are highly inconsistent in diagnostic label taxonomies and conceptual annotations—SkinCon provides only 29 conceptual annotations and a benign/malignant binary classification, whereas DermNet covers over 600 diseases but lacks structured conceptual annotations. Second is the lack of expert-level reasoning supervision: accurate diagnosis requires hierarchical reasoning that respects disease taxonomies and differential diagnosis capabilities, yet existing datasets rarely provide clinically interpretable annotations of reasoning processes. Third is the limited scalability of training paradigms: models trained on small-scale, densely annotated data struggle to generalize to large-scale, sparsely annotated data.

Key Challenge: Existing Med-VLMs either rely on small-scale, high-quality annotated data (strong reasoning capability but limited scale) or on large-scale, weakly annotated data (large scale but lacking reliable reasoning supervision). There is an irreconcilable contradiction between these two paradigms—arising from the absence of a unified framework that can both acquire clinical reasoning patterns from authoritative sources and effectively generalize to heterogeneous datasets.

Key Insight: The unique insight of this work is that textbook-level diagnostic reasoning knowledge, though small in quantity, is of exceptionally high quality and can serve as a "seed" for reasoning capabilities. An RL training framework can then transfer this reasoning capability from densely annotated data to sparsely annotated data. Combining the two can overcome the previous trade-offs.

Core Idea: Constructing a three-stage training pipeline: first, automatically generating a hierarchical and differential-diagnostic reasoning trajectory dataset, SkinRationale, based on authoritative dermatology textbooks; next, establishing a foundation for clinical reasoning via SFT; and finally, generalizing the reasoning capability to large-scale, sparsely annotated datasets using the GRPO reinforcement learning framework combined with a hierarchical diagnostic precision reward.

Method¶

Overall Architecture¶

The training pipeline of Skin-R1 consists of three stages. Stage 1 extracts three types of complementary knowledge from the 9th edition of the authoritative dermatology textbook Fitzpatrick's Dermatology (Part 20: Neoplasia, 399 pages in total) using an automated five-stage knowledge extraction pipeline: diagnostic case triplets (clinical image, textual reasoning, diagnostic label), a differential diagnosis graph (DDx graph, connecting clinically confusable diseases), and a hierarchical taxonomy tree. Based on this knowledge, 2,020 diagnostic reasoning trajectories of three types are constructed. Stage 2 performs SFT on these trajectories, enabling the model to learn clinically consistent diagnostic reasoning patterns. Stage 3 employs GRPO reinforcement learning alongside a composite reward function (format reward, hierarchical depth-aware reward, malignancy discrimination reward) to generalize the reasoning capabilities from densely annotated data to large-scale, sparsely annotated datasets.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Textbook Knowledge Source<br/>Fitzpatrick 9e"] --> B["Stage 1: SkinRationale Construction<br/>Diagnostic Cases + DDx Graph + Taxonomy Tree"]
    B --> C["Reasoning Trajectory Dataset<br/>2,020 Trajectories Across 3 Categories"]
    C --> D["Stage 2: SFT<br/>Language Model Mimics Clinical Reasoning"]
    D --> E["Grounded SFT Model"]
    E --> F["Stage 3: GRPO RL<br/>Hierarchical Reward Generalization"]
    H["Large-Scale Sparsely Annotated<br/>Dermatology Datasets"] --> F
    F --> G["Final Skin-R1 Model"]

Key Designs¶

1. Textbook-Knowledge-Driven Reasoning Trajectory Generation: Addressing the Scarcity of High-Quality Reasoning Data

Initiating reliable reasoning trajectories from a cold start is a key obstacle. Existing methods rely on large reasoning models (such as QwQ-32B) to generate reasoning trajectories, but model-generated reasoning can contain errors, leading to model collapse or hallucination propagation. Skin-R1's solution is to directly extract structured clinical knowledge from authoritative textbooks. First, an automated five-stage pipeline—consisting of image/text extraction, image clustering filtering (two-level hierarchical k-means), text alignment filtering (spatial proximity + regex), diagnostic rule extraction (gpt-4.1-mini generation of rule-like statements), and DDx/classification tree refinement—is used to extract 220 diagnostic cases \((I_i, r_i, d_i)\), a DDx graph containing 211 nodes and 245 edges (covering 61.36% of cases), and a hierarchical disease taxonomy containing 458 nodes and 473 edges from Fitzpatrick's Dermatology. Based on this knowledge, three types of reasoning trajectories are constructed: Type 1 (220 trajectories, direct mapping of image \(\rightarrow\) diagnosis + reasoning), Type 2 (900 trajectories, adding DDx comparative reasoning to Type 1 while keeping the final diagnosis unchanged), and Type 3 (900 trajectories, correcting the diagnosis after adding DDx comparative reasoning), totaling 2,020 trajectories. The DDx pairings for Types 2 and 3 use a hierarchical fallback strategy (prioritizing DDx graph neighborhood queries \(\rightarrow\) querying taxonomy tree parent node neighborhood if there are no results \(\rightarrow\) querying child nodes) to ensure high coverage. These three types of trajectories progressively cover different scenarios of diagnostic reasoning from simple to complex, providing rich reasoning supervision signals for SFT while the textbook source mitigates the risk of error propagation found in purely model-generated reasoning.

2. Hierarchical Reward Design: Generalizing RL to Sparsely Annotated Data

This is the core innovation of the paper. The authors designed a composite reward function \(R_{\text{total}} = R_{\text{format}} + R_{\text{gran}} + R_{\text{malignancy}}\) for GRPO. The format reward \(R_{\text{format}}\) is a binary score (1/0) indicating whether the output contains the required tags, ensuring structural consistency in model responses. The hierarchical depth-aware reward \(R_{\text{gran}}\) is the key breakthrough: if the predicted diagnosis \(\hat{\ell}\) lies on the correct classification path \(\mathcal{P}\), the reward is \(0.75 \times i^{*}/L\) (where \(i^{*}\) is the matched depth and \(L\) is the total path depth); otherwise, it is 0. This means the model receives a higher reward for selecting a correct fine-grained disease (e.g., "superficial spreading melanoma") than for selecting only a correct coarse-grained category (e.g., "melanoma"), thereby encouraging more precise hierarchical reasoning. The malignancy discrimination reward \(R_{\text{malignancy}}\) assigns a 0.25 reward if the ternary classification of malignant vs. benign vs. precancerous/in-situ lesion is correct, reinforcing basic classification from a clinical risk perspective. The elegance of this design lies in the fact that the hierarchical depth reward directly encodes the structural information of the disease taxonomy tree into the RL training signal, enabling the model to reason within the disease hierarchical space rather than performing simple single-label classification, thereby generalizing better to unseen disease types.

3. Two-Stage Training Strategy Combining GRPO and Grounding SFT: Resolving the Lack of Clinical Knowledge in RL-Only Methods

Recent works (such as MedVLM-R1 and Med-R1) have demonstrated that applying GRPO directly to Med-VLMs can elicit reasoning behaviors without SFT; some even claim that SFT might induce memory shortcuts. This paper, however, presents a different conclusion supported by empirical experiments. The authors' strategy is to first perform SFT on large-scale reasoning trajectories (using standard autoregressive loss) to endow the model with foundational clinical reasoning capability, and then employ GRPO reinforcement learning to amplify and generalize these capabilities. The objective function of GRPO is \(\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_x[\frac{1}{K}\sum_{j=1}^K \min(\rho_j A_j, \text{clip}(\rho_j, 1-\varepsilon, 1+\varepsilon)A_j)] - \beta \text{KL}(\pi_{\theta} \| \pi_{\text{ref}})\), where \(A_j = (r_j - \mu)/\sigma\) is the group-normalized advantage function. Ablation studies clearly indicate that although pure RL outperforms the base model (ID average improves from 0.4367 to 0.5843), it remains significantly inferior to the full SFT+RL pipeline (0.5843 vs 0.6385). This proves that high-quality, grounded SFT is a prerequisite for effective RL. The divergence from Med-R1's findings stems from the fact that Med-R1's SFT data lacked clinical reasoning, whereas Skin-R1's textbook-derived reasoning trajectories supply authentic reasoning knowledge.

An Example: Granularity-Aware Reward Calculation Workflow¶

Given the ground-truth label "Superficial Spreading Melanoma", its classification path is: Neoplasm \(\rightarrow\) Melanocytic \(\rightarrow\) Melanoma \(\rightarrow\) Superficial Spreading Melanoma, with path depth \(L=4\). If all four hierarchical nodes are options and the model selects "Melanoma" (depth \(i^{*}=3\)), then \(R_{\text{gran}} = 0.75 \times 3 / 4 = 0.5625\). If the model selects the correct fine-grained "Superficial Spreading Melanoma" (\(i^{*}=4\)), it receives the maximum reward of 0.75. If it selects a node not on the path, the reward is 0. This gradually decaying reward design encourages the model to search through hierarchical space rather than performing simple multiple-choice selection, enhancing reasoning generalization.

Loss & Training¶

The SFT stage uses the standard autoregressive loss \(\mathcal{L} = -\sum_{i=1}^n \log \pi_{\theta}(x_i \mid x_{<i}, I, p)\) and is trained for 4 epochs on the 2,020 reasoning trajectories. The RL stage employs GRPO with group size \(K=4\) and a sampling temperature of 1.0. The base model is Qwen2.5-VL-7B-Instruct. Both SFT and RL are fine-tuned using LoRA (lora_r=64, lora_alpha=32, lora_dropout=0.1). The learning rates are \(3\times 10^{-5}\) for SFT and \(1\times 10^{-5}\) for RL, both with a batch size of 16. Training is conducted on two NVIDIA A100 40GB GPUs. Greedy decoding is used during inference with a maximum image resolution of 448×448.

Key Experimental Results¶

Main Results¶

Task	Dataset	Metric	Skin-R1	Best Baseline	Gain
ID Disease	BCN20000	Acc	0.6345	0.4911（Qwen2.5-VL-7B）	+0.1434
ID Disease	HAM10000	Acc	0.7214	0.5162（Qwen2.5-VL-7B）	+0.2052
ID Disease	PAD-UFES-20	Acc	0.6573	0.6009（InternVL3-8B）	+0.0564
ID Disease	Derm12345	Acc	0.6608	0.4121（LLaVA-v1.6-13B）	+0.2487
ID Disease	ID Average (6)	Acc	0.6385	0.4430（InternVL3-8B）	+0.1955
OOD Disease	OOD Average (5)	Acc	0.7171	0.6887（MedGemma-4B）	+0.0284
Lesion Condition	Lesion Average	Acc / F1	0.6928 / 0.4287	0.6491 / 0.3654（Qwen3-VL-32B）	+0.0437 / +0.0633

On In-Distribution (ID) diagnostic tasks, Skin-R1 achieves an average accuracy of 0.6385, outperforming the runner-up baseline InternVL3-8B (0.4430) by 19.55 percentage points, showing an extremely significant improvement. For Out-Of-Distribution (OOD) generalization, Skin-R1's average of 0.7171 also leads all baselines. On malignant/benign/precancerous classification, Skin-R1 achieves both the highest accuracy (0.6928) and Macro-F1 (0.4287), indicating balanced performance across different categories. Notably, multiple baseline models exhibit severe class bias in lesion classification—MedGemma-4B is heavily biased toward the "precancerous/in-situ" category (7,860 out of 8,390 predictions belong to this category), while LLaVA-Med-7B is biased toward "benign", whereas Skin-R1 does not show similar extreme biases.

Ablation Study¶

Configuration	ID Average Acc	OOD Average Acc	Description
Qwen2.5-VL-7B (Base)	0.4367	0.5027	Base model, ~30% output formats invalid, preventing answer extraction
+ SFT	0.4743	0.5153	Format alignment is significantly improved after SFT, with only ~2% invalid outputs
RL without SFT	0.5843	0.5674	Pure RL shows noticeable improvement, but remains far below the full pipeline
RL with standard reward	0.6379	0.6386	RL with standard reward (without hierarchical depth awareness)
Skin-R1 (1500 steps)	0.6385	0.7171	Full pipeline, achieving the largest OOD improvement

Key findings from the ablation study: First, the SFT stage is a critical prerequisite—SFT not only enhances diagnostic performance (ID: 0.4367 to 0.4743) but, more importantly, substantially improves output format alignment (reducing invalid extraction rates from ~30% to ~2%), removing formatting barriers for subsequent RL. Second, although RL without SFT outperforms the base model (ID: 0.5843 vs 0.4367), the full pipeline (ID: 0.6385) is significantly superior, showing that high-quality grounding SFT is essential for effective RL, and that pure RL lacks a clinical knowledge foundation. Third, the hierarchical depth reward provides the largest advantage in OOD scenarios (0.6386 to 0.7171), proving that hierarchical reward signals indeed help the model learn generalizable representations of disease classification reasoning, rather than overfitting to the training set label distribution.

Key Findings¶

The format alignment and reasoning initialization provided by the SFT stage serve as the foundation for subsequent RL; RL primarily contributes generalization capability, particularly in OOD scenarios.
The introduction of the hierarchical depth reward further boosts OOD accuracy from 0.6386 to 0.7171 (+0.0785), representing the single largest contribution within the entire framework, validating the effectiveness of encoding the disease taxonomy tree structure into RL rewards.
In the Fitzpatrick skin type evaluation, slight performance degradation occurs in types II, V, and VI (minimum 0.4634 vs. maximum 0.5494), suggesting potential demographic distribution bias in the training data.
Skin-R1 achieves the highest accuracy in targeted DDx evaluations, proving that the three types of reasoning trajectories (especially Type 2 and Type 3) indeed enhance the model's ability to distinguish easily confusable diseases.

Highlights & Insights¶

A Complete Closed Loop from Textbook Knowledge Distillation to RL Training: The most elegant aspect of this work is not an isolated innovation but the construction of a complete knowledge pipeline—automatically extracting structured knowledge from textbook PDFs \(\rightarrow\) constructing reasoning trajectories \(\rightarrow\) SFT grounding \(\rightarrow\) RL generalization. The design of each phase addresses the needs of the subsequent one, creating a cohesive, closed loop rather than a collection of disjointed modules.
Valuable Evidence Offered to the Debate of "Whether RL Will Replace SFT": Recent efforts like Med-R1 advocate that direct RL (without SFT) can yield superior results. Through meticulous ablation studies, this work demonstrates that the outcome depends on the quality of reasoning supervision data—when SFT data originates from authoritative textbooks rather than model generation, SFT acts as a necessary prerequisite for RL. This finding holds instructive value for the broader Med-VLM domain.
Elegant and Transferable Design of the Hierarchical Depth Reward: Encoding the disease taxonomy tree structure directly into the RL reward is simple yet effective, significantly boosting the model's OOD generalization. This paradigm can easily transfer to other clinical diagnostic tasks with hierarchical label systems (e.g., pathological grading, imaging staging) and even broader hierarchical categorization tasks.
Structured Design of Three Types of Reasoning Trajectories: Type 1 (direct mapping), Type 2 (unmodified DDx), and Type 3 (modified DDx) cover diverse clinical decision-making scenarios from simple to complex, safeguarding basic diagnostic accuracy while ensuring the model learns to execute differential diagnosis and correct itself under uncertainty.

Limitations & Future Work¶

Both the reward signal and the evaluations are based on answer-level metrics (multiple-choice accuracy). The correctness and faithfulness of reasoning trajectories are analyzed only qualitatively, lacking quantitative evaluation. In open-ended clinical scenarios, future work must establish confidence calibration, selective abstention mechanisms under high uncertainty, and severity-aware error analysis for malignant cases.
The disease taxonomy tree and the DDx graph are constructed offline and remain static during training. Since these structures are also utilized to formulate distractors in evaluations, the ID assessments partially reflect consistency with fixed clinical knowledge (though the OOD benchmark OmniMedVQA remains decoupled from this). A static design also limits the capacity to adapt to dataset-specific or evolving clinical ontologies.
The framework is currently instantiated only on Qwen2.5-VL-7B; its transferability to alternative base models (including domain-specific medical backbones) remains unverified. Furthermore, the observed slight accuracy drop on Fitzpatrick II/V/VI skin types implies a potential demographic bias in the training distributions.
The outcome-driven objective function does not explicitly reward the correctness of intermediate reasoning steps. Defining and validating process-level reward signals for clinical reasoning remains an open challenge.

vs MedVLM-R1 / Med-R1: These methods directly apply GRPO to VLMs without SFT to elicit reasoning behaviors. Skin-R1 demonstrates that this occurs due to their lack of high-quality reasoning data—when textbook-grade reasoning trajectories are available, SFT+RL significantly outperforms pure RL, showing that RL should not be viewed as an alternative to SFT, but rather as a tool to amplify existing reasoning capabilities.
vs SkinVL-PubMM: As a dermatology-specific VLM, SkinVL-PubMM uses manual annotations for grounding SFT but lacks DDx awareness and hierarchical perception capabilities. Skin-R1’s three types of reasoning trajectories provide more structured supervision for clinical reasoning, while training on sparsely annotated data makes the model more generalizable.
vs MedGemma: MedGemma incorporates a mix of grounding SFT, distillation, and RL reward training, yielding a degree of hierarchical perception capability, yet it lacks DDx awareness. Skin-R1 explicitly models confusion relationships among diseases via the DDx graph, making it more reliable in differential diagnosis scenarios.

Rating¶

Novelty: ⭐⭐⭐⭐ The three-stage framework combining textbook knowledge distillation and RL generalization is a first systematic attempt in the Med-VLM domain. Encoding the taxonomy tree structure into RL rewards as a hierarchical depth-aware reward is highly original, and the design of the three types of reasoning trajectories is cleverly executed.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The experiments are exceptionally comprehensive, covering 6 ID datasets + 5 OOD datasets, component-wise ablation studies for SFT/RL/reward designs, targeted evaluations for DDx and hierarchical reasoning, Fitzpatrick skin type fairness analyses, and comparison tables covering 13 baseline models across 6 categories.
Writing Quality: ⭐⭐⭐⭐ The motivation is clear, the methodology is presented in a well-structured hierarchy, and the appendix details the complete pseudocode and hyperparameter configurations. The description of the three stages in the main text is highly readable, though the explanation of distractor generation strategies in the multi-choice VQA evaluation framework could be more detailed in the main body.
Value: ⭐⭐⭐⭐⭐ This work provides vital empirical evidence for the design choices of Med-VLM training paradigms (SFT vs. RL vs. SFT+RL); the design of the hierarchical depth-aware reward is readily transferable; and dermatological auxiliary diagnosis has clear, robust clinical utility.