EMER-Guard: Protocol-Grounded Verification for LLM Emergency Instructions¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/qq0518/emer-guardbench
Area: Safety
Keywords: Emergency Instruction Verification, Protocol Evidence Grounding, Asymmetric Risk Evaluation, Counterfactual Benchmark, Multi-Task Representation Learning
TL;DR¶
Addressing the asymmetric real-world costs of LLM-generated emergency instructionsโwhere missed hazards cost lives and over-refusal blocks urgent shelter guidanceโthis paper introduces EMER-GuardBench (a 69k-sample leakage-aware counterfactual benchmark across 10 disaster types and 6 hazard mechanisms), EGIS (an evidence-conditioned, protocol-supervised verifier), and SWMR (Severity-Weighted Miss Rate), reframing safety from prompt-based generation to auditable, protocol-grounded verification.
Background & Motivation¶
As large language models increasingly serve as core public information infrastructure, disaster-response organizations, public health platforms, and municipal agencies are exploring their use for real-time evacuation routing, sheltering directives, and hazard mitigation advice. However, in high-stakes emergency management, standard aggregate accuracy collapses as a viable safety objective: missing a lethal instruction and inappropriately suppressing a correct emergency measure incur drastically asymmetric human costs. A false negative can lead residents to undertake fatal actionsโsuch as opening windows during a raging wildfire or wading into moving floodwatersโwhile a false positive in the form of over-refusal denies citizens vital survival directives during the golden evacuation window. Standard safety guardrails such as OpenAI Moderation and Llama Guard are structurally unequipped for this mission because they target broad objectionable content categories rather than domain-specific operational safety rules, causing them to either fail against plausibly phrased fatal advice or aggressively block legitimate distress queries.
Prior research on emergency NLP and context-aware safety has made strides toward situational awareness, yet it predominantly emphasizes instruction generation rather than post-hoc verification against authoritative regulatory guidance. Knowledge-grounded generation frameworks like E-KELL and reconnaissance-based agentic pipelines such as MoRA-RAG concentrate upstream on producing decisions, but regardless of generator architecture, downstream text generation remains prone to subtle hallucinations that invert evacuation vectors, corrupt chemical exposure thresholds, or omit critical life-safety prerequisites. Crucially, existing evaluation regimes suffer from severe data leakage where identical action semantics are shared across splits, inflating apparent performance and blinding auditors to models that rely on superficial lexical shortcuts. When frontier LLMs act as zero-shot judges, their over-refusal rates spike past 20% to 54%, revealing that prompting alone cannot reconcile sensitivity to hazards with safe acceptance of benign instructions.
To resolve this impasse, the authors execute a threefold methodological inversion: shifting the task from advice generation to post-hoc verification against cited authoritative protocols; shifting the role of evidence from inference-time prompt decoration to training-time representation shaping; and shifting safety evaluation from scalar benchmark scores to a vector surface that makes residual life-safety risk explicit alongside over-refusal. Core idea: construct an instruction-group-disjoint counterfactual benchmark (EMER-GuardBench) across 10 disasters and 6 hazard mechanisms, develop an evidence-conditioned multi-task verifier (EGIS) jointly optimizing hazard detection, mechanism attribution, severity estimation, support verification, and counterfactual ranking, and evaluate systems via Severity-Weighted Miss Rate (SWMR) to expose trade-offs hidden by scalar metrics.
Method¶
Overall Architecture¶
The EMER-Guard verification workflow formalizes emergency instruction auditing as a structured prediction task within a pipeline spanning authoritative guideline retrieval, counterfactual perturbation, evidence-conditioned multi-task encoding, and asymmetric-cost decision gating with human escalation. Given a disaster scenario context \(c\), cited authoritative protocol evidence \(e\) (sourced from FEMA, CDC, WHO, OSHA, and the Red Cross), and candidate instruction text \(x\), the verifier predicts \((y, m, s, e_{\text{sup}})\), where \(y \in \{0, 1\}\) denotes hazard status, \(m \in \mathcal{M}\) identifies the specific hazard mechanism, \(s \in \{1, 2, 3, 4\}\) estimates the ordinal hazard severity level, and \(e_{\text{sup}}\) verifies whether the candidate instruction is supported or contradicted by the cited protocol passage.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Scenario c + Protocol Evidence e + Candidate Instruction x"] --> B["Leakage-Aware Benchmark EMER-GuardBench<br/>10 Disasters ร 6 Counterfactual Edit Mechanisms"]
B --> C["Multi-Task Verifier EGIS<br/>BGE-M3 Backbone with Frozen Lower / Tuned Upper Layers"]
C --> D["Joint Inference Heads<br/>Hazard y + Mechanism m + Severity s + Support e_sup"]
D --> E["Asymmetric-Cost Metric SWMR<br/>Severity-Weighted Miss Rate Read Jointly with ORR"]
E --> F["Output: Pareto-Dominant Triage Decision & Human Escalation"]
Key Designs¶
1. Leakage-Aware Benchmark EMER-GuardBench: Eliminating Semantic Shortcut Learning via Instruction-Group Disjointness Conventional safety benchmarks partition data randomly or by row-level hash, permitting paraphrased variants of the exact same action verbs and syntactic patterns to leak across training and test partitions. EMER-GuardBench grounds its data in authoritative safety protocols across 10 disaster scenarios (earthquake, flood, wildfire, hurricane/tornado, extreme heat, winter storm, tsunami, hazmat spill, public health emergency, and active threat). Safe atomic instructions are systematically perturbed via six controlled minimal-edit hazard mechanisms: Action Reversal (\(m_1\), reversing recommended actions), Temporal Corruption (\(m_2\), altering timing parameters), Quantity/Threshold (\(m_3\), corrupting safe buffer distances or dosages), Condition Mismatch (\(m_4\), swapping triggering prerequisites while preserving surface actions), Critical Omission (\(m_5\), removing vital protective steps), and False Assurance (\(m_6\), inserting unwarranted safety guarantees). Crucially, EMER-GuardBench enforces an instruction-group-disjoint split: all paired counterfactuals, contexts, and surface wrappers derived from a base instruction family reside strictly within a single partition (60,000 train, 3,000 val, 6,000 test, with balanced 50/50 safe/hazard labels). It supplements this with a separate 1,000-sample benign-stress set composed entirely of safe instructions laden with urgent vocabulary to audit over-refusal.
2. Multi-Task Verifier EGIS: Conditioning Representations on Protocol Supervision Rather than Prompt Decoration
Appending evidence passages directly into prompts of general LLMs often dilutes attention and exacerbates false positives, while naive binary sentence classifiers fail to capture fine-grained risk nuances. EGIS builds upon BGE-M3 (an XLM-RoBERTa-large backbone with 568.3M parameters), encoding sequence pairs as [CLS] Evidence [SEP] Instruction [SEP]. To preserve base semantic representations and prevent catastrophic overfitting, EGIS freezes the embedding layer and the bottom 12 Transformer layers, fine-tuning only the upper 12 layers (152.7M trainable parameters). It mounts four specialized classification and regression heads: a risk-weighted hazard classifier (\(y\)), a 6-class mechanism identification head (\(m\)), a 4-level ordinal severity estimator (\(s\)), and an auxiliary evidence-support head (\(e_{\text{sup}}\)). During training, EGIS optimizes a pairwise margin ranking objective across paired counterfactual safe/hazardous batches, actively forcing the hazard score of the minimally edited hazardous variant above that of its safe counterpart by a predefined margin \(\delta\), instilling high sensitivity to semantic hazard shifts.
3. Asymmetric-Cost Metric SWMR: Exposing Residual Risk on a Multi-Objective Operational Frontier In life-critical settings, treating a false negative on an immediately fatal directive the same as a miss on a low-severity guideline distorts safety assessments, while single-scalar scores obscure aggressive over-refusal. The paper formulates the Severity-Weighted Miss Rate (SWMR) as a transparent, dataset-derivable proxy for residual hazard mass: $$ \text{SWMR} = \frac{\sum_{i \in \text{FN}} s_i}{\sum_{i \in \text{hazard}} s_i} $$ where \(s_i \in \{1, 2, 3, 4\}\) denotes the ground-truth severity level of instruction \(i\), and \(\text{FN}\) denotes the set of false-negative hazardous instructions. By construction, SWMR must always be read jointly with the Over-Refusal Rate (ORR, false positive rate on gold-safe items) across both the IID test set and the benign-stress set, because a degenerate system can achieve \(\text{SWMR} = 0.000\) simply by blocking every query. On a comprehensive sweep of 364 thresholded LLM-judge operating points, EGIS establishes a strictly Pareto-dominant frontier, achieving superior Harm F1 with minimal over-refusal.
Loss & Training¶
EGIS optimizes a composite multi-task loss function combining five objectives: $$ \mathcal{L}{\text{total}} = \mathcal{L}}} + \lambda_{\text{mech}} \mathcal{L{\text{mech}} + \lambda}} \mathcal{L{\text{sev}} + \lambda}} \mathcal{L{\text{sup}} + \lambda $$ }} \mathcal{L}_{\text{pair}\(\mathcal{L}_{\text{hazard}}\) is a risk-weighted binary cross-entropy loss equipped with label smoothing \(\epsilon = 0.05\) and positive hazard weighting \(\alpha = 0.25\); \(\mathcal{L}_{\text{mech}}\) is cross-entropy over the six hazard mechanisms; \(\mathcal{L}_{\text{sev}}\) is smooth \(\text{L}_1\) regression over ordinal severity ranks; \(\mathcal{L}_{\text{sup}}\) is cross-entropy over evidence-support relations; and \(\mathcal{L}_{\text{pair}} = \max(0, \delta - (P(y=1|x_{\text{hazard}}, e) - P(y=1|x_{\text{safe}}, e)))\) enforces a pairwise margin \(\delta = 0.5\) across matched counterfactual pairs. EGIS is optimized with AdamW at a learning rate of \(2 \times 10^{-5}\) using linear warmup (ratio 0.1) and linear decay, an effective batch size of 32 (micro-batch 4 with 8 gradient accumulation steps), bfloat16 mixed precision, and gradient checkpointing on two NVIDIA RTX 4090 D GPUs.
Key Experimental Results¶
Main Results¶
Evaluated on the 6,000-example IID test split and the 1,000-example benign-stress set under fixed decision threshold \(\tau = 0.5\):
| Model Architecture | Configuration | Benign-Stress ORR (%) โ | IID Test ORR (%) โ | Harm F1 (%) โ | Counterfactual Consistency CC (%) โ | Severity-Weighted Miss Rate SWMR โ |
|---|---|---|---|---|---|---|
| DeepSeek-V3.2 | 0-shot default prompt | 12.5 | 27.3 | 86.5 | 85.0 | 0.029 |
| DeepSeek-V3.2 + protocol | 0-shot with protocol evidence | 10.9 | 34.5 | 83.5 | 84.8 | 0.035 |
| DeepSeek-V3.2 (strong) | 6-mechanism rubric + 6-shot CoT | 12.8 | 52.2 | 78.7 | 79.4 | 0.012 |
| DeepSeek-V3.2 (max prompt) | Rubric + CoT + protocol evidence | 18.7 | 44.2 | 78.7 | 77.5 | 0.062 |
| Kimi-K2.5 | 0-shot default prompt | 10.1 | 23.9 | 88.6 | 83.6 | 0.014 |
| Kimi-K2.5 (strong) | 6-mechanism rubric + 6-shot CoT | 14.0 | 54.0 | 78.0 | 89.4 | 0.017 |
| GLM-5 | 0-shot default prompt | 10.2 | 53.7 | 78.8 | 82.0 | 0.000 |
| Qwen3-8B (Ours) | Logit-calibrated JSON next-token | 0.7 | 17.3 | 79.6 | 89.7 | 0.223 |
| Retrieval + NLI | 0-shot mDeBERTa-v3 entailment | 54.2 | 71.2 | 66.2 | 71.9 | 0.155 |
| XLM-R Seq. Clf. | Text-only sequence classifier | 10.0 | 20.7 | 85.3 | 96.8 | 0.088 |
| XLM-R + Evidence | Binary classifier with evidence | 3.2 | 14.6 | 88.0 | 94.6 | 0.085 |
| Qwen3-8B + LoRA (Ours) | 8.19B backbone fine-tuned on train | โ | 12.5 | 87.8 | โ | โ |
| EGIS w/o Evidence | BGE-M3 ablation without evidence | 10.0 | 20.1 | 87.1 | 91.7 | 0.063 |
| EGIS (Full, Ours) | BGE-M3 multi-task protocol verifier | 0.0 | 10.4 | 89.9 | 94.4 | 0.088 |
Ablation Study¶
To verify whether the benefits of protocol grounding generalize across model scales and architectures, a \(2 \times 2\) cross-backbone ablation was conducted with BGE-M3 (568M) and mDeBERTa-v3-base (86M):
| Backbone Architecture | Parameters | Evidence Input | Harm F1 (%) โ | Macro F1 (%) โ | IID Test ORR (%) โ | Benign-Stress ORR (%) โ | Matthews Corr. MCC โ |
|---|---|---|---|---|---|---|---|
| BGE-M3 | 568M | No (w/o Evidence) | 87.1 | 86.3 | 20.1 | 10.0 | 0.733 |
| BGE-M3 | 568M | Yes (Full EGIS) | 89.9 | 89.9 | 10.4 | 0.0 | 0.798 |
| \(\Delta\)(BGE-M3) | โ | Evidence Gain | +2.8 | +3.6 | -9.7 | -10.0 | +0.065 |
| mDeBERTa-v3-base | 86M | No | 87.6 | 87.0 | 17.5 | 0.0 | 0.744 |
| mDeBERTa-v3-base | 86M | Yes | 88.6 | 88.0 | 16.4 | 0.0 | 0.764 |
| \(\Delta\)(mDeBERTa) | โ | Evidence Gain | +1.0 | +1.0 | -1.1 | 0.0 | +0.020 |
Key Findings¶
- The Prompted LLM Over-Refusal Trap: Elaborate few-shot prompting, detailed hazard rubrics, and chain-of-thought scaffolds do not fix prompt-based LLM judges; instead, they shift decision thresholds uniformly toward hyper-defensiveness. DeepSeek-V3.2 and Kimi-K2.5 achieve 98%+ hazard recall under rubric+CoT prompting, but precision drops to ~65%, driving IID over-refusal above 52%โ54%. GLM-5's nominal \(\text{SWMR} = 0.000\) represents a pathological boundary condition achieved by rejecting 53.7% of legitimate instructions.
- Evidence Acts via Representation Shaping, Not Inference-Time Lookup: Inference-time causal perturbation experiments (dropping, shuffling, or swapping cross-scenario evidence, as well as synthetic contradictory injections in Appendices N, Q, and U) alter EGIS predictions by only 1.2% to 1.4% (statistically indistinguishable from irrelevant placebos, \(z = -0.279, p = 0.780\)). Crucially, safe rows exhibit zero flips under contradictory evidence injection. This proves EGIS does not execute naive RAG-style string matching at inference time; rather, protocol evidence structures the geometry of the decision boundary during supervised training.
- Architecture and Multi-Task Design Beat Raw Scale: The 14.4ร larger Qwen3-8B+LoRA model (8.19B parameters, consuming 5ร the training compute) reaches 87.8% Harm F1 and 12.5% IID ORR, remaining strictly inferior to the 568M EGIS (89.9% / 10.4%). Scaling parameter count without counterfactual margin ranking and multi-task auxiliary supervision fails to close the joint performance gap.
- Severity-Stratified Miss Analysis: On the IID test split, EGIS achieves 94.0% recall on critical-risk (\(s=4\)) hazards (missing only 340 severity mass) and 90.3% recall on high-risk (\(s=3\)) hazards. Its aggregate SWMR of 0.088 is dominated by misses on medium-risk (\(s=2\)) hazards (recall 37.7%), demonstrating that the model effectively eliminates catastrophic, life-threatening safety breaches.
Highlights & Insights¶
- First-Class Evaluation Integrity: EMER-GuardBench moves beyond superficial token randomization by enforcing instruction-group-disjoint partitioning, ensuring the model is evaluated on entirely unseen action semantics rather than memorized templates.
- Reframing Safety as a MissโOver-Refusal Frontier: Demonstrates that safety is an operational trade-off surface rather than a single-scalar leaderboard metric, proving that none of the 364 threshold-tuned LLM-judge operating points Pareto-dominates EGIS.
- Training-Time Shaping Over Inference Decoration: Uncovers the critical empirical insight that pairing small specialized models with protocol supervision during training provides stronger verification guarantees and dramatically lower false-positive rates than prompting massive foundation models.
Limitations & Future Work¶
- Domain Shift Under Heavy Paraphrasing: While EGIS generalizes robustly across structured field formats (staying within \(\pm 2.2\) percentage points of IID Harm F1 across neighborhood posts, field notes, chatbot queries, and draft alerts), deep colloquial paraphrasing into informal social media style causes Harm F1 to drop to 64.1% and ORR to rise to 25.0%.
- Vulnerability to Condition Mismatch (\(m_4\)): EGIS achieves its lowest mechanism F1 on condition mismatch (51.6%), because the surface action remains identical to the safe guideline while only the triggering environmental condition is subtly altered (e.g., opening windows for gas leak vs. loud noise), requiring explicit state-machine and causal reasoning beyond dense semantic embeddings.
- Future Directions: Integrating dynamic temporal retrieval with jurisdictional freshness gates, expanding beyond 10 US-centric regulatory bodies to international disaster management standards, and augmenting training data with structurally evidence-dependent samples (5%โ10%) to foster true inference-time RAG adaptability.
Related Work & Insights¶
- vs Llama Guard & OpenAI Moderation: General guardrails focus on coarse-grained toxic content and fail on nuanced procedural safety directives; EMER-GuardBench focuses on protocol-grounded, context-dependent emergency verification with zero benign-stress over-refusal.
- vs E-KELL & MoRA-RAG: Knowledge-graph and agentic RAG systems focus on generating emergency decisions; EGIS provides the complementary post-hoc verification layer indispensable for catching subtle generator hallucinations before dissemination.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering protocol-grounded counterfactual verification benchmark, reframing safety evaluation as an asymmetric-cost Pareto frontier.
- Experimental Thoroughness: โญโญโญโญโญ Exemplary rigor spanning 69,000 examples, 364 LLM-judge Pareto points, cross-backbone ablations, causal evidence perturbations, and independent blind expert audits.
- Writing Quality: โญโญโญโญโญ Clear, precise, methodologically reflective, and candidly transparent about negative causal results and failure modes.
- Value: โญโญโญโญโญ Crucial foundation for deploying safe, auditable LLM applications in disaster response and life-critical public infrastructure.