Dental-TriageBench: Benchmarking Multimodal Reasoning for Hierarchical Dental Triage¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Medical LLM
Keywords: Dental Triage, Multimodal Reasoning, Hierarchical Multi-label Classification, Chain-of-Thought, Omission Risk
TL;DR¶
Dental-TriageBench introduces the first expert-annotated multimodal benchmark for hierarchical dental triage from paired chief complaints and orthopantomograms (OPGs). Benchmarking 21 MLLMs against three junior dentists demonstrates that frontier models substantially trail human clinicians on fine-grained referral planning, exhibiting severe omission-heavy under-coverage on complex multi-specialty cases.
Background & Motivation¶
Dental triage functions as a safety-critical gatekeeping process at the primary point of clinical access, where clinicians must determine the appropriate specialty referral pathways before definitive chairside examinations can be performed. In routine outpatient workflows, patient chief complaints describe localized subjective symptoms and immediate care intentions but often fail to uncover asymptomatic concurrent pathologies or specify anatomical complexity. Conversely, orthopantomograms (OPGs) display panoramic structural findings across the dentition and surrounding maxillofacial anatomy. Triage is therefore inherently multimodal, requiring clinicians to cross-reference patient-reported concerns with radiographic evidence, detect multiple concurrent dental issues, and translate them into a coherent referral plan governed by clinical protocols.
Despite rapid advancements in dental AI for panoramic radiograph perception—such as tooth detection, numbering, and visual question answering—existing benchmarks evaluate descriptive perception or open-ended QA rather than authentic clinical triage. In clinical triage, decision errors carry severe asymmetric costs: modest over-referral primarily increases downstream specialist review workload, whereas under-referral leaves active pathological conditions unmanaged and delays critical interventions. Current multimodal large language models (MLLMs) can generate clinically fluent rationales yet fail to demonstrate comprehensive referral coverage across complex, long-tailed, and multi-problem patient presentations.
To bridge this critical translational gap, the authors design a benchmark rooted in authentic outpatient dental workflows. The core idea is to introduce Dental-TriageBench, pairing de-identified chief complaints and OPGs with expert consensus labels and clinical reasoning trajectories across 8 coarse-grained domains and 22 fine-grained treatment categories, benchmark 21 frontier MLLMs against human junior dentists, and systematically diagnose model vulnerabilities through a clinician-grounded error taxonomy.
Method¶
Overall Architecture¶
Dental-TriageBench formalizes multimodal dental triage as a hierarchical multi-label classification task with clinical rationale generation. Each input case comprises a textual chief complaint \(X_{\text{text}}\) and a panoramic radiograph \(X_{\text{img}}\). Conditioned on this multimodal evidence, the model must output a multi-hot binary label vector \(\mathbf{y} \in \{0, 1\}^{|L|}\) across the \(|L| = 22\) fine-grained treatment categories (which aggregate into 8 coarse-grained specialty domains) alongside a natural language rationale \(R\) explaining the clinical routing rationale. The benchmark pipeline integrates authentic case de-identification, independent double-blind senior dentist annotation with consensus adjudication, a two-level hierarchical taxonomy, and an LLM-as-judge failure attribution workflow:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Clinical Input<br/>Chief Complaint + OPG Radiograph"] --> B["Dual-Expert Annotation & Adjudication<br/>Double-blind review and consensus adjudication"]
B --> C["Hierarchical Triage Taxonomy<br/>8 coarse specialty domains ↔ 22 fine-grained categories"]
C --> D["Zero-Shot Multimodal Inference<br/>Structured JSON prediction and rationale generation"]
D --> E["Clinician-Grounded Failure Attribution<br/>Cross-family multi-judge majority voting"]
Key Designs¶
1. Authentic Outpatient Curation and Expert-Authored Golden Reasoning Trajectories: Anchoring Realistic Clinical Workflows
Unlike synthetic QA or crowdsourced datasets, Dental-TriageBench curates 246 authentic patient cases retrospectively from university-affiliated dental teaching hospital outpatient archives. The dataset retains the true clinical distribution where multi-label referrals predominate and rare categories appear in natural proportions. Two licensed senior dentists independently evaluated each de-identified complaint–OPG pair in a custom annotation interface before any mutual discussion. Pre-adjudication inter-rater reliability was high (Cohen's \(\kappa = 0.83\)), with discrepancies occurring in only ~5% of cases. The senior clinicians conducted consensus discussions to settle conflicting labels and author comprehensive golden reasoning trajectories that delineate the exact clinical justification for every positive referral.
2. Hierarchical Treatment-Oriented Triage Taxonomy: Decoupling Departmental Routing from Resource Allocation
Dental triage is fundamentally distinct from definitive pathological diagnosis; chairside diagnostic tests (e.g., pulp vitality or periodontal probing) cannot be performed on radiographs alone. To reflect realistic gatekeeping functions, the taxonomy defines two clinical levels: - Coarse-grained domains (8 specialties): Maps high-level departmental routing across Periodontology (Perio), Cariology, Endodontics (Endo), Crowns, Bridges, Dentures, Implants, and Oral & Maxillofacial Surgery (OMFS). - Fine-grained treatment categories (22 labels): Stratifies referrals by clinical complexity and resource requirements. For instance, in Endodontics, anterior/premolar teeth (simpler root canal morphology suitable for standard appointment slots) are separated from molars (intricate root canal anatomy requiring endodontic specialists, extended chair time, and dental microscopes). Similarly, simple exodontia is distinguished from minor oral surgical procedures (e.g., complex impacted wisdom teeth requiring bone removal or mucoperiosteal flaps).
3. Clinician-Grounded Root-Cause Failure Taxonomy: Dissecting Decision Failures via Hierarchical Error Attribution
Aggregate set-matching metrics fail to reveal whether errors stem from perceptual oversights or decision boundary violations. The authors formalize five mutually exclusive, clinically grounded failure dimensions assigned via a strict top-to-bottom hierarchy: 1. Hallucination (D5): The model hallucinates symptoms, anatomical teeth, or lesions unsupported by either the complaint or radiograph. 2. Chief Complaint Neglect (D1): The model completely ignores the primary complaint or symptom expressed by the patient. 3. OPG Finding Omission (D2): The model completely omits clinically significant radiographic abnormalities documented in the expert rationale (e.g., failing to mention bone loss, retained roots, or impacted teeth). 4. OPG Misinterpretation (D3): The model identifies the lesion site but misjudges its severity or pathological depth (e.g., classifying pulp-invading deep caries as superficial). 5. Protocol Misalignment (D4): The model perceives and describes the pathology with complete accuracy (no D1, D2, or D3 errors), yet maps it to an incorrect referral label under clinical guidelines.
To scale error analysis across all model outputs, a cross-family panel (Gemini-3-Pro, GPT-5.4, and GLM-5) operates via majority vote, achieving 94.6% agreement against senior dentist annotations on 100 validation cases.
Loss & Training¶
The benchmark investigates the zero-shot capabilities of frontier MLLMs. Models receive a standardized clinical instruction prompting output strictly in JSON format with binary indicator keys for the 22 categories and a concise reasoning string (\(\le 200\) words). A two-stage parsing pipeline permits up to three retries for malformed JSON; persistent syntax errors are resolved using a frozen formatting model (Gemini-2.0-Flash) that preserves semantic content without modifying clinical predictions.
Key Experimental Results¶
Main Results¶
The authors benchmark 21 MLLMs (9 proprietary, 8 open-source, and 4 medical-domain models) alongside a human baseline formed by three licensed junior dentists.
| Model / Baseline | Fine-grained Macro-F1 | Fine-grained Macro-Recall | Fine-grained Micro-F1 | Fine-grained Exact-Match | Coarse-grained Macro-F1 | Coarse-grained Micro-F1 | Coarse-grained Exact-Match |
|---|---|---|---|---|---|---|---|
| Human Baseline (Junior Dentists) | 0.402 | 0.431 | 0.525 | 0.083 | 0.562 | 0.727 | 0.272 |
| Proprietary MLLMs | |||||||
| Gemini-3-Flash | 0.302 | 0.310 | 0.459 | 0.049 | 0.488 | 0.656 | 0.220 |
| Gemini-3-Pro | 0.294 | 0.262 | 0.447 | 0.118 | 0.493 | 0.641 | 0.224 |
| Claude-Opus-4.7 | 0.288 | 0.298 | 0.420 | 0.077 | 0.463 | 0.634 | 0.171 |
| GPT-5.5 | 0.284 | 0.242 | 0.422 | 0.098 | 0.437 | 0.581 | 0.195 |
| Claude-Sonnet-4.5 | 0.269 | 0.275 | 0.350 | 0.024 | 0.442 | 0.565 | 0.130 |
| GPT-5.2 | 0.272 | 0.239 | 0.404 | 0.041 | 0.398 | 0.576 | 0.215 |
| Kimi-K2.5 | 0.249 | 0.204 | 0.348 | 0.053 | 0.393 | 0.537 | 0.187 |
| Grok-4 | 0.232 | 0.187 | 0.329 | 0.094 | 0.392 | 0.529 | 0.187 |
| Qwen-VL-Max-Latest | 0.222 | 0.206 | 0.283 | 0.037 | 0.398 | 0.517 | 0.183 |
| Open-source MLLMs | |||||||
| Ministral-3-14B-Instruct-2512 | 0.189 | 0.158 | 0.247 | 0.029 | 0.327 | 0.453 | 0.163 |
| MiMo-VL-7B-RL | 0.125 | 0.093 | 0.175 | 0.020 | 0.266 | 0.386 | 0.146 |
| Qwen2.5-VL-7B-Instruct | 0.101 | 0.077 | 0.148 | 0.024 | 0.191 | 0.301 | 0.102 |
| InternVL3_5-8B | 0.088 | 0.056 | 0.093 | 0.016 | 0.176 | 0.196 | 0.045 |
| GLM-4.1V-9B-Thinking | 0.072 | 0.050 | 0.086 | 0.016 | 0.133 | 0.182 | 0.069 |
| Medical MLLMs | |||||||
| MedGemma-27b-it | 0.121 | 0.094 | 0.173 | 0.029 | 0.266 | 0.357 | 0.130 |
| Lingshu-7B | 0.076 | 0.056 | 0.115 | 0.012 | 0.172 | 0.277 | 0.094 |
| HuatuoGPT-Vision-7B | 0.074 | 0.058 | 0.101 | 0.004 | 0.121 | 0.188 | 0.061 |
| MedGemma-1.5-4b-it | 0.059 | 0.036 | 0.068 | 0.008 | 0.116 | 0.140 | 0.041 |
Ablation Study¶
Modality ablations demonstrate that combining textual complaints and panoramic radiographs is strictly required, verifying that \(\text{F1}_{\text{Fusion}} > \max(\text{F1}_{\text{Text}}, \text{F1}_{\text{Image}})\).
| Evaluated Model | CC-only Micro-F1 | OPG-only Micro-F1 | Multimodal (CC+OPG) | Case-level \(\Delta \text{F1} > 0\) | Case-level \(\Delta \text{F1} > 0.1\) | Case-level \(\Delta \text{F1} > 0.2\) |
|---|---|---|---|---|---|---|
| Claude-Sonnet-4.5 | 0.282 | 0.305 | 0.350 | 17.1% | 13.0% | 11.0% |
| Gemini-3-Flash | 0.385 | 0.412 | 0.459 | 26.0% | 17.5% | 10.2% |
| GPT-5.2 | 0.298 | 0.327 | 0.404 | 35.4% | 34.1% | 27.6% |
Key Findings¶
- Persistent Human–Model Performance Gap: On the primary 22-label task, the highest-ranking frontier model (Gemini-3-Flash, Macro-F1 0.302) trails junior dentists (Macro-F1 0.402) by 0.100 (5,000 bootstrap resample 95% CI: [0.059, 0.140], strictly above 0). The deficit is especially pronounced on Macro-Recall (0.310 vs. 0.431), indicating that clinicians excel at preserving referral breadth.
- Underperformance of Open and Medical MLLMs: Open-source foundation models lag significantly behind proprietary offerings; Ministral-3-14B is the top open model at 0.189 Macro-F1. Specialized medical models fail to surpass generalist baselines (MedGemma-27B achieves only 0.121 Macro-F1), showing that broad biomedical visual-instruction tuning does not confer specialized dentomaxillofacial reasoning capabilities.
- Collapse on Multi-Domain Complexity and Omission Bottleneck: 78.9% of outpatient cases require multi-domain referrals (average 2.60 positive domains). While junior dentists maintain stable Micro-F1 across complexity tiers (0.548 on single-domain cases vs. 0.565 on 5+ domain cases), MLLMs experience severe degradation (Gemini-3-Pro plunges from 0.586 to 0.357; GPT-5.2 from 0.461 to 0.280). Error taxonomy analysis indicates that D2 OPG Finding Omission represents the single most dominant failure mode across all models. MLLMs consistently predict narrower referral sets, incurring high omission risk that jeopardizes patient safety.
Highlights & Insights¶
- Authentic Outpatient Triage Realism: Moves beyond academic classification and generic VQA by framing triage around actionable clinical referrals, preserving natural class imbalances and multi-morbidity patterns.
- Rigorous Root-Cause LLM-as-Judge Framework: The mutually exclusive D1–D5 clinical taxonomy provides an interpretable audit trail that differentiates visual omission from protocol rule confusion, yielding a reusable methodology for auditing medical AI reasoning.
- Unveiling Asymmetric Omission Hazards: Demonstrates that MLLMs exhibit an unsafe "narrow referral bias"—sacrificing necessary clinical referrals under uncertainty. This highlights the necessity of designing coverage-aware loss functions, finding-first reasoning chains, and selective deferral mechanisms.
Limitations & Future Work¶
- Single-Institution Cohort: The 246 cases originate from a single university dental hospital, ensuring consistent imaging protocols and adjudication standards but lacking external validation across multi-center hardware and regional triage guidelines.
- Absence of Chairside Physical Findings: Inputs are restricted to intake complaints and panoramic radiographs, excluding vital clinical observations (e.g., electric pulp tests, thermal responsiveness, BPE periodontal probing depth) necessary for definitive diagnostics.
- Sparse Long-Tail Categories: Several fine-grained categories (such as resin-bonded bridges or multi-tooth implants) possess fewer than 10 positive cases, limiting granular statistical claims for rare tail classes.
Related Work & Insights¶
- vs. MMOral / OralGPT / DentVLM: Prior dental multimodal models concentrate on radiological reporting or isolated tooth condition recognition; Dental-TriageBench tackles integrated gatekeeping decisions requiring multi-label referral planning under clinical protocols.
- vs. Emergency Triage Benchmarks (e.g., TriageAgent, ClinicBench): Traditional triage benchmarks primarily evaluate text-based systemic acuity scoring (e.g., ESI levels); Dental-TriageBench targets multimodal specialized dental routing coupling high-resolution panoramic radiology with patient narratives.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First benchmark formalizing reasoning-driven multimodal dental triage with expert-authored clinical trajectories
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking of 21 MLLMs against 3 junior dentists with extensive difficulty stratifications and modality ablations
- Writing Quality: ⭐⭐⭐⭐⭐ Clinically grounded, methodologically sound, and clearly articulated clinical protocols and error taxonomy
- Value: ⭐⭐⭐⭐⭐ Exposes critical omission vulnerabilities in clinical AI decision-making, providing essential benchmarks for safe healthcare deployment