EMER-Guard: Protocol-Grounded Verification for LLM Emergency Instructions¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/qq0518/emer-guardbench
Area: Information Extraction
Keywords: Emergency instruction verification, protocol-grounded auditing, counterfactual benchmark, asymmetric risk evaluation, multi-task representation learning
TL;DR¶
Addressing the severe trade-off between life-threatening instruction errors and excessive refusal in emergency NLP, this paper constructs EMER-GuardBench—a 69,000-example counterfactual auditing benchmark with instruction-group-disjoint splits—and presents EGIS, an evidence-conditioned multi-task verifier that delivers 89.9% Harm F1 while eliminating benign-stress false positives entirely under the Severity-Weighted Miss Rate (SWMR) metric.
Background & Motivation¶
As large language models (LLMs) transition into critical public-facing information infrastructure for disaster response, evacuation guidance, and public health advisories, conventional scalar safety benchmarks have become fundamentally insufficient. In high-stakes crisis scenarios such as earthquakes, hazardous chemical spills, wildfires, and epidemic outbreaks, safety errors carry heavily asymmetric human costs: failing to intercept a hazardous instruction (e.g., advising residents to open windows during a wildfire, drawing toxic smoke indoors, or recommending a 30-minute delay before tsunami evacuation) directly endangers human life. Conversely, over-defensively suppressing or refusing a completely sound, urgent survival instruction (such as drop-cover-and-hold during an earthquake) deprives citizens of critical self-rescue guidance when standard communication lines are overwhelmed. Neither failure mode is captured by a single aggregate accuracy score.
Existing safety guardrails fail to resolve this operational dilemma. General-purpose moderation models such as OpenAI Moderation and Llama Guard are trained to flag broad toxic content categories like hate speech, violence, and self-harm, leaving them blind to subtle, domain-specific emergency hazard mechanisms like inverted action timing or modified dosage thresholds. Meanwhile, zero-shot LLM judges suffer from acute over-refusal when presented with high-pressure emergency prompts; under the influence of defensive safety instructions and chain-of-thought scaffolds, frontier models arbitrarily inflate recall at the expense of precision, frequently rejecting over 50% of legitimate emergency actions. Furthermore, prior disaster NLP literature has overwhelmingly focused on situational awareness, entity extraction, or open-ended instruction generation, overlooking the complementary necessity of auditable, post-hoc verification against authoritative safety protocols.
To protect public safety while maintaining operational availability, emergency NLP must transition from ungrounded refusal heuristics to protocol-conditioned, auditable verification. Core idea: reframe emergency instruction auditing as a structured protocol-grounded verification task, instantiate it via the instruction-group-disjoint counterfactual benchmark EMER-GuardBench, and deploy the multi-task, evidence-supervised verifier EGIS to establish a strict Pareto-dominance frontier over frontier LLM judges across Harm F1, over-refusal rate, and Severity-Weighted Miss Rate (SWMR).
Method¶
Overall Architecture¶
Under the protocol-grounded verification formulation, the input comprises a disaster scenario description \(c\), a candidate emergency instruction \(x\), and authoritative safety protocol evidence \(e\) curated from FEMA, CDC, WHO, OSHA, or Red Cross guidelines. Rather than generating text or performing unconstrained retrieval, the system executes structured verification: it outputs a tuple \((y, m, s)\), where \(y \in \{0, 1\}\) represents the binary hazard classification (hazardous vs. safe), \(m \in \mathcal{M}\) identifies one of six fine-grained hazard mechanisms when \(y=1\), \(s \in \{1, 2, 3, 4\}\) denotes an ordinal hazard severity rating, and \(r \in \{\text{supported}, \text{contradicted}\}\) captures the evidence-support status relative to the cited protocol.
The core verifier, EGIS, is built upon the BGE-M3 multi-task encoder (an XLM-RoBERTa-large backbone with 568.3M parameters). The model formats input as [CLS] Evidence [SEP] Instruction [SEP] to establish dense cross-attention between authoritative guidelines and candidate operational steps. To preserve general multilingual representation while preventing catastrophic forgetting, the embedding layer and the lower 12 Transformer layers are kept frozen, while the upper 12 layers and task-specific classification heads (152.7M trainable parameters) are fine-tuned end-to-end. Training is simultaneously guided by a pairwise margin ranking loss across paired counterfactual instances to explicitly separate the latent score of hazardous edits from their safe origins.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Scenario c + Instruction x + Protocol Evidence e"] --> B["Counterfactual Benchmark Construction<br/>6-mechanism minimal edits with group-disjoint splits"]
B --> C["Protocol-Conditioned Joint Encoder<br/>Freeze lower 12 layers of BGE-M3 and encode joint context"]
C --> D["Multi-Task End-to-End Prediction Heads<br/>Jointly predict hazard, mechanism, severity, and evidence support"]
D --> E["Counterfactual Margin Ranking Optimization<br/>Enforce score separation between paired safe and hazardous edits"]
E --> F["Cost-Sensitive Operating Decision<br/>Balance SWMR residual hazard against zero over-refusal"]
Key Designs¶
1. Counterfactual Benchmark Construction: Eliminating Action-Level Shortcut Leakage
To prevent models from memorizing surface-level action phrases across splits, EMER-GuardBench enforces rigorous counterfactual curation across 10 representative disaster scenarios (earthquake, flood, wildfire, hurricane/tornado, extreme heat, winter storm, tsunami, hazmat spill, public health emergency, and active threat). Starting from verified safe atomic instructions, the benchmark applies targeted edit operators spanning six explicit hazard mechanisms: - \(m_1\) Action Reversal (reversing a safe action, e.g., "stay indoors" \(\to\) "go outside"); - \(m_2\) Temporal Corruption (altering timing or delays, e.g., "evacuate immediately" \(\to\) "wait 30 minutes"); - \(m_3\) Quantity/Threshold Alteration (modifying physical safety margins, e.g., "stay \(\ge\) 330 ft" \(\to\) "stay \(\ge\) 3 ft"); - \(m_4\) Condition Mismatch (altering the triggering precondition while preserving action surface form, e.g., "if you smell gas, open windows" \(\to\) "if you hear loud noises, open windows"); - \(m_5\) Critical Omission (removing life-preserving protective actions); - \(m_6\) False Assurance (inserting unwarranted safety guarantees).
Each edit generates a minimal-edit counterfactual pair. The core benchmark comprises 69,000 examples (60,000 train, 3,000 val, 6,000 test with balanced 50/50 safe/hazardous labels) plus a distinct 1,000-sample safe-only benign-stress evaluation set containing urgent linguistic cues. Crucially, splits are partitioned at the instruction-group level, ensuring zero lexical overlap of base action semantics across training and test splits.
2. Protocol-Conditioned Joint Encoder: Structuring Latent Space via Authoritative Rules
EGIS couples candidate instructions with 60 curated authoritative summary passages derived from official disaster response agencies. By feeding evidence passages alongside the candidate text into the BGE-M3 backbone, the model leverages cross-attention to evaluate factual consistency directly against regulatory standards. Freezing the bottom 12 layers while tuning the top 12 layers allows the network to adapt its upper semantic representations toward rule verification without degrading core linguistic structure, ensuring evidence acts as training-time inductive supervision rather than transient inference-time prompt decoration.
3. Multi-Task End-to-End Prediction Heads: Enforcing Semantic Regularization
Rather than relying on an isolated binary hazard classifier, EGIS branches into four concurrent heads: hazard classification \(L_{\text{hazard}}\) (weighted cross-entropy with risk factor \(\alpha=0.25\) and label smoothing \(\epsilon=0.05\)), hazard mechanism identification \(L_{\text{mech}}\) (6-way classification), severity estimation \(L_{\text{sev}}\) (ordinal regression across 4 tiers), and evidence support verification \(L_{\text{support}}\) (supported vs. contradicted). These auxiliary objectives enforce semantic consistency across different risk dimensions; on the full test set, the support head achieves 99.93% joint agreement with the hazard head, stabilizing decision boundaries and substantially reducing false-positive rates on safe instructions.
4. Counterfactual Margin Ranking Optimization: Enforcing Sharp Local Separation
Leveraging the paired counterfactual structure of EMER-GuardBench, EGIS optimizes a pairwise margin ranking objective every 2 micro-steps. Given a safe base instruction \(x_{\text{safe}}\) and its corresponding hazardous edit \(x_{\text{hazard}}\), the ranking loss enforces a minimum confidence gap \(\delta = 0.5\) between their predicted hazard probabilities:
$\(\mathcal{L}_{\text{rank}} = \max\left(0, \delta - \left(P_{\text{hazard}}(x_{\text{hazard}}) - P_{\text{hazard}}(x_{\text{safe}})\right)\right)\)$
This objective ensures that the model remains highly sensitive to localized semantic corruptions (e.g., swapping a single word like close to open), driving Counterfactual Consistency (CC) on unseen actions to 94.4%.
5. Cost-Sensitive Operating Decision: Severity-Weighted Miss Rate (SWMR) Discipline
To explicitly quantify the asymmetric life-safety costs of false negatives, the authors introduce the Severity-Weighted Miss Rate: $\(\text{SWMR} = \frac{\sum_{i \in \text{FN}} s_i}{\sum_{i \in \text{Hazard}} s_i}\)$ where \(s_i \in \{1, 2, 3, 4\}\) denotes the severity ordinal of hazard \(i\). SWMR measures the proportion of unmitigated hazard mass. The authors show that while extremely conservative prompt-based LLM judges (such as GLM-5) can achieve an apparent SWMR of 0.000, they do so only by blocking 53.7% of legitimate emergency instructions. Reading (Harm F1, ORR, SWMR) as a vector reveals that EGIS successfully intercepts 94.0% of critical hazards (\(s=4\)) while restricting IID over-refusal to 10.4% and benign-stress false positives to 0.0%.
Loss & Training¶
EGIS is trained using AdamW with an initial learning rate of \(2 \times 10^{-5}\), weight decay of 0.01, and a linear warmup schedule over 10% of total steps followed by linear decay. Sequence lengths are capped at 256 tokens, using per-device micro-batch size 4 with 8 gradient accumulation steps (effective batch size 32). The system is trained on 2 NVIDIA RTX 4090 D GPUs (24GB) using bfloat16 mixed precision and gradient checkpointing for 3 epochs with early stopping on validation Macro F1 (patience 2), taking roughly 12 GPU hours.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on the full 6,000-example IID test set (3,000 safe, 3,000 hazardous) and the 1,000-example benign-stress set, comparing EGIS against leading frontier zero-shot LLM judges, calibrated local models, zero-shot NLI, and same-backbone supervised baselines:
| Model / Evaluator | Model Size | Protocol Evidence | Benign-Stress Over-Refusal (ORRbenign↓) % | IID Over-Refusal (ORRIID↓) % | Harm F1 (HF1↑) % | Counterfactual Consistency (CC↑) % | Severity-Weighted Miss Rate (SWMR↓) |
|---|---|---|---|---|---|---|---|
| DeepSeek-V3.2 (0-shot) | - | No | 12.5 | 27.3 | 86.5 | 85.0 | 0.029 |
| DeepSeek-V3.2 + Protocol (0-shot) | - | Yes | 10.9 | 34.5 | 83.5 | 84.8 | 0.035 |
| DeepSeek-V3.2 (Rubric + 6-shot CoT) | - | No | 12.8 | 52.2 | 78.7 | 79.4 | 0.012 |
| DeepSeek-V3.2 (Rubric + CoT + Evidence) | - | Yes | 18.7 | 44.2 | 78.7 | 77.5 | 0.062 |
| Kimi-K2.5 (0-shot) | - | No | 10.1 | 23.9 | 88.6 | 83.6 | 0.014 |
| Kimi-K2.5 (Rubric + 6-shot CoT) | - | No | 14.0 | 54.0 | 78.0 | 89.4 | 0.017 |
| GLM-5 (0-shot) | - | No | 10.2 | 53.7 | 78.8 | 82.0 | 0.000 |
| Qwen3-8B (Logit-Calibrated, 0-shot) | 8.19B | No | 0.7 | 17.3 | 79.6 | 89.7 | 0.223 |
| Retrieval + NLI (0-shot mDeBERTa) | 86M | Yes | 54.2 | 71.2 | 66.2 | 71.9 | 0.155 |
| XLM-R Seq. Classifier | 568M | No | 10.0 | 20.7 | 85.3 | 96.8 | 0.088 |
| XLM-R + Evidence (Binary) | 568M | Yes | 3.2 | 14.6 | 88.0 | 94.6 | 0.085 |
| Qwen3-8B + LoRA SFT | 8.19B | Yes | - | 12.5 | 87.8 | - | - |
| EGIS w/o Evidence | 568M | No | 10.0 | 20.1 | 87.1 | 91.7 | 0.063 |
| EGIS (Full, Ours) | 568M | Yes | 0.0 | 10.4 | 89.9 | 94.4 | 0.088 |
Cross-Backbone Evidence Ablation¶
The ablation study across BGE-M3 (568M) and mDeBERTa-v3-base (86M) proves that evidence conditioning provides framework-level improvements across different model scales:
| Backbone | Evidence Input | Harm F1 (HF1↑) % | Macro F1 % | IID Over-Refusal (ORRIID↓) % | Benign-Stress Over-Refusal (ORRbenign↓) % | Matthews Corr. (MCC↑) |
|---|---|---|---|---|---|---|
| BGE-M3 (568M) | No | 87.1 | 86.3 | 20.1 | 10.0 | 0.733 |
| BGE-M3 (568M) | Yes | 89.9 | 89.9 | 10.4 | 0.0 | 0.798 |
| Δ (BGE-M3) | - | +2.8 | +3.6 | -9.7 | -10.0 | +0.065 |
| mDeBERTa (86M) | No | 87.6 | 87.0 | 17.5 | 0.0 | 0.744 |
| mDeBERTa (86M) | Yes | 88.6 | 88.0 | 16.4 | 0.0 | 0.764 |
| Δ (mDeBERTa) | - | +1.0 | +1.0 | -1.1 | 0.0 | +0.020 |
In leave-one-scenario-out out-of-distribution evaluation, EGIS averages 86.5% Harm F1 and 0.049 SWMR across 10 held-out disaster scenarios, achieving 100% detection and 0.000 SWMR on Wildfire, Winter Storm, Public Health, and Tsunami.
Key Findings¶
- Complex prompting severely inflates over-refusal in LLM judges: Providing frontier models (DeepSeek-V3.2, Kimi-K2.5) with detailed rubric definitions and 6-shot chain-of-thought demonstrations pushes hazard recall above 98%, but causes precision to collapse to ~65% and doubles over-refusal to 52.2%–54.0%. Over-refusal is a structural training problem that cannot be prompt-engineered away.
- Evidence operates as training-time representation shaping, not inference-time retrieval: Stress tests perturbing inference evidence (dropping passages, shuffling within scenarios, swapping across scenarios, truncating 50%, or injecting synthetic contradictory rules) moved EGIS predictions by at most 0.9 pp in Harm F1, with flip rates under 1.4% (statistically indistinguishable from random noise). This proves that EGIS internalizes protocol constraints into latent feature geometry during training rather than performing shallow lexical matching at runtime.
- Architecture and multi-task loss outperform raw parameter scaling: The 14.4\(\times\) larger Qwen3-8B + LoRA baseline trained on the exact same 60k examples reached only 87.8% Harm F1 and 12.5% ORR, trailing EGIS by 2.1 pp on both axes despite 5\(\times\) greater training compute, underscoring the critical value of pairwise counterfactual ranking and multi-task supervision.
Highlights & Insights¶
- Instruction-Group-Disjoint partitioning as a gold standard: Enforcing zero overlap in action semantics between training and test sets eliminates shortcut learning on common emergency verbs, ensuring evaluations measure genuine semantic understanding.
- Debunking scalar metric illusions via SWMR Pareto frontiers: The paper exposes how GLM-5's "perfect" SWMR = 0.000 is attained by arbitrarily rejecting more than half of all valid life-saving advice, demonstrating that safety benchmarks must be analyzed along a multi-dimensional Pareto frontier.
- High-efficiency, deployable verification: Achieving state-of-the-art performance with a 568M parameter model trained in 12 GPU hours offers public emergency management infrastructure an auditable, low-latency, and zero-false-positive verification component.
Limitations & Future Work¶
- Unresolved failure on Condition Mismatch (\(m_4\)): When safe and hazardous instructions share identical surface actions and differ solely in the triggering condition, 4-judge consensus drops to 20.9% and EGIS achieves only 51.6% F1, highlighting the limitation of bi-encoder text classification for complex causal reasoning.
- Distribution shift under informal social media paraphrases: On a five-register probe, DeepSeek-generated colloquial social media paraphrases (containing slang and abbreviations) caused EGIS Harm F1 to drop to 64.1% and ORR to increase to 25.0%, showing that synthetic benchmarks must be further augmented with noisy, human-authored crisis text.
- Static protocol assumption: The current setup assumes authoritative protocols are pre-filtered and non-conflicting, leaving dynamic multi-jurisdictional legal conflicts and real-time guideline versioning for future operational extensions.
Related Work & Insights¶
- vs. Llama Guard & OpenAI Moderation: While general guardrails classify broad categories of content toxicity, they lack domain knowledge of operational safety standards and exhibit high over-refusal on legitimate crisis actions. EGIS grounds verification directly in authoritative FEMA/CDC protocols.
- vs. CASE-Bench & OR-Bench: Existing safety benchmarks evaluate general context sensitivity and over-refusal in conversational LLMs; EGIS translates these insights into life-critical emergency management, pairing counterfactual evaluation with asymmetric loss penalties.
- vs. E-KELL & MoRA-RAG: Prior disaster NLP focuses on generating decisions using knowledge graphs or multi-agent RAG. EMER-Guard establishes the complementary, indispensable line of defense: post-hoc, protocol-grounded verification.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers protocol-grounded emergency instruction verification with leakage-aware counterfactual splits and the SWMR metric]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across 4 LLM judges, 8B supervised baselines, cross-backbone ablations, LOO generalization, and expert human audits]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional clarity, rigorous methodological self-critique, and exemplary negative-result transparency]
- Value: ⭐⭐⭐⭐⭐ [Critical safety infrastructure for real-world LLM deployment in public health and disaster relief]