Meerkat2: Bridging Global Reasoning and Local Knowledge for Efficient Medical AI¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/collections/aigensciences/meerkat2
Area: LLM Reasoning
Keywords: Medical AI, Clinical Reasoning Distillation, Verbosity-Decoupled Distillation, Medical Law Synthesis, Mechanistic Interpretability
TL;DR¶
Meerkat2 decouples clinical reasoning transfer from high-fidelity local statute learning across an 837K-example mixture and introduces Verbosity-Decoupled Distillation (VDD) to cancel internal gradient updates toward excessive output length, matching frontier models on U.S. and Korean clinical licensing benchmarks while slashing token generation costs by over 50%.
Background & Motivation¶
Large language models are increasingly recognized as viable clinical assistants capable of aiding differential diagnosis, medical information retrieval, and doctor-patient communication. Because clinical records and medical inquiries inherently carry strict patient privacy and regulatory restrictions, healthcare institutions have created strong demand for open-source medical models that can run locally on hospital infrastructure. However, the vast majority of existing open medical LLMs remain predominantly English-centric, failing to reflect the local legal, pharmacological, and institutional systems that govern practice outside the United States. Adapting a model to non-English healthcare ecosystems cannot be treated as a mere translation problem: regional guidelines and national medical statutes require verbatim factual memorization, and conventional teacher distillation frequently hallucinate or alter statutory boundaries.
Simultaneously, distilling complex medical reasoning introduces a severe efficiency bottleneck into practical clinical workflows. While transferring long chains of thought (CoT) from frontier reasoning models substantially boosts problem-solving accuracy, student models inevitably internalize the teacher's verbose, repetitive response style as their default behavior. In high-pressure clinical decision-support environments, unnecessary verbal padding introduces non-trivial latency and recurring cognitive friction, violating the operational imperative that clinical AI must deliver concise, timely insights precisely at the moment of decision making.
The central tension lies in transferring sophisticated, knowledge-intensive clinical reasoning without forcing compact student models to inherit wasteful verbosity or inaccurate local legal approximations. The paper addresses this dilemma by introducing knowledge-matched supervision paired with a preventative training strategy. Core idea: decouple clinical reasoning distillation from direct, source-grounded statute synthesis, while introducing Verbosity-Decoupled Distillation (VDD) to condition long rationales on explicit prompt instructions and cancel parameter gradients toward verbose generation.
Method¶
Overall Architecture¶
Meerkat2 establishes a comprehensive data curation and training pipeline across four distinct supervision streams: clinical reasoning distillation, statutory question-answer synthesis, open-ended biomedical inquiry, and general-capability replay. Clinical reasoning is distilled from frontier teacher models (DeepSeek-V4-Pro and Qwen3-235B-A22B-Thinking) over verified multi-choice benchmarks using Verbosity-Decoupled Distillation. Local legal knowledge is synthesized directly from 12 official Korean medical statutes into self-contained multiple-choice questions with 500+ word rationales. After filtering open biomedical QA datasets and injecting general multi-turn instruction datasets to safeguard against catastrophic forgetting, the resulting 837K mixture is fine-tuned onto Qwen3 architectures (14B, 32B, and 30B-A3B MoE) using loss masking on empty thinking blocks to enforce structured internal reasoning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Source Input Corpus<br/>(Clinical Exams / Statutes / Open Web)"] --> B["Knowledge-Decoupled Data Mixture<br/>(837K Multi-Task Dataset)"]
B --> C["Source-Grounded Statute Synthesis<br/>(Direct Chunking of 12 Korean Laws)"]
B --> D["Verbosity-Decoupled Distillation VDD<br/>(Dual CoT Trajectories + Prompt Binding)"]
C --> E["Verbose Head Gradient Cancellation<br/>(Negative Parameter Update on Verbose Heads)"]
D --> E
E --> F["Efficient Medical Model Family<br/>(Meerkat2 14B / 32B / 30B-A3B)"]
Key Designs¶
1. Knowledge-Decoupled Data Mixture: Curating an 837K multi-task dataset aligned with disparate knowledge properties
Medical AI assistants must concurrently exhibit deep clinical diagnostic reasoning, exact institutional statutory memory, broad biomedical literacy, and robust multi-turn conversational adherence. Training exclusively on clinical examination questions leads to catastrophic forgetting of general dialogue capabilities, while naive teacher distillation fails on statutory memorization. To address this, the authors curate an 837,555-example mixture partitioned into a medical domain subset (668,778 examples) and a general domain subset (168,777 examples). The medical component incorporates 592K concise and verbose clinical CoT trajectories (from MedQA, KorMedMCQA, and MedBooks-18), 57K filtered open-domain biomedical QA conversations (from NaturalReasoning, OpenThoughts, and WebInstruct), and 18.7K statutory QA pairs. The general replay component strategically incorporates five subsets from SmolTalk2 and Tulu-3—covering system prompt adherence, multilingual understanding across 22 languages, multi-turn dialogue context handling, instruction rewriting, and safety refusal—ensuring specialized medical post-training does not degrade general conversational versatility.
2. Source-Grounded Statute Synthesis: Eliminating distillation hallucination via direct synthesis from official legal corpora
Healthcare practice is bound to specific legal frameworks including the Medical Service Act, Pharmaceutical Affairs Act, Regional Public Health Act, and Narcotics Control Act. When large teacher models are prompted to generate reasoning for medical legal questions, they exhibit unacceptably high error rates because foreign or general models lack granular knowledge of regional statutory clauses. Instead of relying on unreliable teacher distillation, the authors synthesize question-answer pairs directly from official legal source documents. Using a recursive text splitter with 3,000-character chunk sizes and 1,000-character overlaps across 12 Korean medical statutes, an LLM prompt synthesizes up to 8 independent multiple-choice questions per chunk accompanied by detailed solutions containing 500+ words of explanatory notes. As evaluated by the legal fidelity framework LF-Eval, synthesized data attains a legal accuracy score of 98.7, vastly exceeding the 79.0 achieved by standard teacher distillation rationales.
3. Verbosity-Decoupled Distillation (VDD): Attributing length to prompt constraints to prevent default verbosity inheritance
In clinical reasoning distillation, long teacher rationales convey richer diagnostic logic than concise answers, raising MedQA accuracy from 83.9% to 85.4% on a 14B student. However, the student unconditionally adopts the teacher's verbose presentation style, ballooning output length from 299.9 tokens to 669.3 tokens. To decouple reasoning quality from verbose response habits, VDD incorporates both concise and verbose CoTs in the training distribution, but attaches an explicit verbosity instruction to all verbose instances via the system prompt (e.g., "Give me a very detailed, long-form answer"). Under causal language modeling, the generation of verbose tokens is attributed to the presence of the system prompt constraint rather than the intrinsic reasoning trajectory. During standard evaluation where no verbose prompt is present, the model retains the high diagnostic accuracy of detailed CoTs while generating compact, clinically practical rationales averaging only 309.9 tokens (a 54% reduction relative to verbose distillation).
4. Verbose Head Gradient Cancellation: Mechanistic explanation of prompt-conditioned preventative training
To explain why a system prompt instruction effectively suppresses verbosity internalization, the authors conduct mechanistic interpretability on Qwen3-4B attention heads. By computing the activation difference between paired long and short responses across layers, they isolate the layer-wise verbose direction vector: $\(v_{\text{attn}}^{l} = \bar{r}_{\text{attn}}^{l,\text{long}} - \bar{r}_{\text{attn}}^{l,\text{short}}, \quad \hat{v}_{\text{attn}}^{l} = \frac{v_{\text{attn}}^{l}}{\|v_{\text{attn}}^{l}\|}\)$ The contribution of each individual attention head \(h\) at layer \(l\) is measured via: $\(\bar{C}_{h}^{l,\text{long}} = \langle \bar{A}_{h}^{l} W_{o}^{l,h}, \hat{v}_{\text{attn}}^{l} \rangle\)$ This reveals that verbose behavior is driven by a sparse set of attention heads (such as L7H1, L24H24, and L33H4). Using a first-order Taylor approximation of the loss gradient: $\(\Delta \bar{C}_{h}^{l,\text{long}} \approx -\eta \|\bar{A}_{h}^{l}\|_{2}^{2} \cdot \frac{\partial L}{\partial \bar{O}_{h}^{l}} \hat{v}_{\text{attn}}^{l}\)$ The empirical analysis demonstrates that unconditioned verbose CoTs push parameters heavily along the verbose direction (+251.2 aggregated across all heads). In contrast, attaching the verbosity instruction flips the gradient sign to negative (-37.1), mirroring the negative parameter shift observed in concise distillation (-63.0). Activating the verbose behavior via the prompt fulfills the generative objective, effectively cancelling the loss gradient that would otherwise bake verbosity into the student's base weights.
Loss & Training¶
All Meerkat2 variants are fine-tuned across 8 NVIDIA A100 GPUs using DeepSpeed ZeRO-3. The training setup utilizes an 8,192 maximum sequence length, an AdamW optimizer with learning rate \(5 \times 10^{-5}\), and a global batch size of 256 over a single training epoch. To support the hybrid thinking capability of Qwen3 backbones, loss masking is applied over the user prompt and empty <think></think> blocks. This penalizes the model for skipping its reasoning phase and ensures it generates explicit step-by-step thinking content by default unless explicitly commanded otherwise.
Key Experimental Results¶
Main Results¶
On MedQA (U.S. clinical licensing) and KorMedMCQA (Korean medical licensing), Meerkat2 variants dramatically surpass their respective base models and demonstrate competitive parity with proprietary frontier models.
| Model Family | Active Params | MMLU | KMMLU | MedQA (Instant) | MedQA (Thinking) | KorMedMCQA-Doctor (Instant EN) | KorMedMCQA-Doctor (Instant KO) |
|---|---|---|---|---|---|---|---|
| Qwen3-14B | 14B | 82.0 | 63.8 | 76.0 | 83.7 | 75.6 | 67.1 |
| hari-q3-14B | 14B | 80.8 | 61.2 | 75.3 | 85.8 | 76.6 | 69.9 |
| Meerkat2-14B (Ours) | 14B | 81.4 | 62.6 | 85.2 | 89.7 | 86.4 | 88.5 |
| MedGemma-27B | 27B | 83.2 | 63.8 | 84.6 | — | 86.2 | 81.8 |
| Qwen3-30B-A3B | 3B (MoE) | 83.1 | 65.2 | 81.2 | 85.5 | 77.9 | 72.4 |
| Meerkat2-30B-A3B (Ours) | 3B (MoE) | 81.9 | 63.6 | 87.3 | 88.9 | 88.5 | 87.6 |
| Qwen3-32B | 32B | 83.4 | 69.5 | 79.2 | 87.4 | 82.1 | 79.3 |
| Solar Pro 2 | 32B | 86.2 | 73.7 | 84.4 | 89.2 | 87.8 | 87.8 |
| Meerkat2-32B (Ours) | 32B | 83.9 | 67.1 | 87.5 | 90.3 | 87.6 | 89.4 |
| Qwen3-235B-A22B | 22B (MoE) | 88.6 | 75.9 | 89.8 | 92.9 | 88.7 | 91.0 |
| GPT-5.1 | Frontier | 90.9 | 77.5 | 93.2 | 96.4 | 92.2 | 94.0 |
Ablation Study¶
1. Reasoning Efficiency and VDD Dynamics (Table 6)
| Model Setting | MedQA Acc (Avg Tokens) | KorMedMCQA EN (Avg Tokens) | KorMedMCQA KO (Avg Tokens) |
|---|---|---|---|
| Qwen3-14B Base | 76.0% (492.2) | 75.5% (435.6) | 72.4% (567.4) |
| + Verbose CoTs | 85.4% (669.3) | 82.9% (693.5) | 80.6% (965.5) |
| + Concise CoTs | 83.9% (299.9) | 82.0% (236.7) | 81.1% (350.8) |
| + VDD (Ours) | 85.2% (309.9) | 82.5% (243.6) | 82.1% (358.3) |
2. Legal Data Synthesis vs. Teacher Distillation across Professional Subsets (Table 5)
| Model & Legal Data Strategy | Dentist | Doctor | Nurse | Pharmacist |
|---|---|---|---|---|
| Qwen3 14B Base | 50.0% | 57.7% | 52.9% | 55.7% |
| + Distillation | 48.3% | 57.7% | 50.0% | 49.2% |
| + Statute Synthesis (Ours) | 56.7% | 71.2% | 69.1% | 57.4% |
| Qwen3 30B-A3B Base | 56.7% | 61.5% | 54.4% | 59.0% |
| + Distillation | 50.0% | 55.8% | 50.0% | 50.8% |
| + Statute Synthesis (Ours) | 56.7% | 73.1% | 69.1% | 59.0% |
| Qwen3 32B Base | 53.3% | 63.5% | 52.9% | 60.7% |
| + Distillation | 51.7% | 57.7% | 57.4% | 54.1% |
| + Statute Synthesis (Ours) | 68.3% | 84.6% | 77.9% | 70.5% |
Key Findings¶
- Reversal of the Non-English Performance Penalty: While base Qwen3-14B suffers an 8.5%p degradation when tested on Korean licensing questions versus English ones (67.1% vs 75.6%), Meerkat2-14B completely reverses this gap, scoring 88.5% in Korean (+2.1%p higher than English).
- Extreme Efficiency with Sparse MoE: Meerkat2-30B-A3B operates with only 3B active parameters per token, yet its KorMedMCQA Doctor score (88.5% English, 87.6% Korean) matches the 235B dense-equivalent frontier model Qwen3-235B-A22B (88.7% / 91.0%), providing an exceptional compute-to-accuracy Pareto frontier.
- Superiority of Source Synthesis over Distillation: Teacher distillation on legal subsets yields near-zero or negative gains due to subtle legal hallucinations. In contrast, direct synthesis from statutory text improves Doctor legal accuracy from 63.5% to 84.6% and Nurse legal accuracy from 52.9% to 77.9% on the 32B model.
- Transferability across Architectures: Fine-tuning Gemma3-4B and Phi4-mini with this exact data mixture yields dramatic score increases (Gemma3-4B improves from 50.8% to 72.2%, outperforming Google's MedGemma-4B by 14.7%p), proving the framework is model-agnostic.
Highlights & Insights¶
- Gradient-Level Grounding for Prompt-Based Decoupling: Rather than treating prompt engineering as an empirical heuristic, the paper uses mechanistic interpretability and first-order Taylor expansion to prove that prompt-level behavioral elicitation mathematically cancels parameter updates along the verbose direction.
- De-homogenization of Medical Training Data: The work departs from uniform distillation by classifying knowledge into deep causal reasoning (suitable for teacher CoT distillation) versus strict statutory compliance (demanding document-grounded synthesis), establishing a principled template for localized AI adaptation.
- Zero-Shot Temporal Generalization: Evaluated on the 89th Korean Medical Licensing Examination (KMLE 2025)—conducted strictly after the training knowledge cutoff—Meerkat2-32B achieves 89.0% accuracy in Korean, outperforming Qwen3-235B-A22B (88.4%) and demonstrating robust out-of-distribution clinical reasoning.
Limitations & Future Work¶
- Reliance on Massive Frontier Teachers: Generating initial CoT candidate pools and filtering multi-turn conversations still necessitates millions of inference tokens from proprietary or massive open models (DeepSeek-V4-Pro and Qwen3-235B-A22B), presenting high upfront compute requirements.
- Limited Scale of Physician Interactive Trials: Although validated on DiagnosisArena and HealthBench Hard, extensive clinical trials assessing multi-turn diagnostic interviews and bedside decision support across multiple hospital specialties remain constrained by the cost of clinical expert time.
- Jurisdiction-Specific Construction Costs: The current legal synthesis pipeline is specifically calibrated for the South Korean legal code; porting Meerkat2 to other legal jurisdictions requires manual identification and chunking of corresponding national public health acts.
Related Work & Insights¶
- vs MedGemma / MEDITRON: Earlier open medical LLMs rely heavily on continued pre-training over PubMed papers and textbook PDFs, which is compute-intensive and lacks explicit reasoning optimization. Meerkat2 demonstrates that high-quality, knowledge-matched instruction tuning and verbosity control provide a much more parameter-efficient route to state-of-the-art clinical proficiency.
- vs HuatuoGPT-o1 / Med-RLVR: While reinforcement learning with verifiable rewards is gaining traction in medical domains, RL can encourage excessive chain-of-thought exploration that incurs heavy serving costs. Meerkat2 proves that supervised distillation with VDD achieves comparable reasoning performance while strictly maintaining compact, clinically viable generation lengths.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Novel VDD framework with rigorous gradient cancellation proof via mechanistic interpretability]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations on MedQA, KorMedMCQA, future-dated KMLE 2025, open clinical benchmarks, and mechanistic ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional narrative clarity, thorough appendices, and intuitive visual explanations]
- Value: ⭐⭐⭐⭐⭐ [Provides a definitive blueprint for building localized, high-efficiency clinical reasoning models for hospital deployment]