Skip to content

Visual Compliance via Executable Safety Rule Entailment in Multimodal LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Safety
Keywords: Multimodal Safety, Neuro-Symbolic Visual Grounding, Rule Entailment, Executable Safeguards, Scene Graph Reasoning

TL;DR

Addressing the limited adaptability and unexplainable reasoning of end-to-end safeguards over complex multimodal regulations, this paper proposes GUARDEN, an executable safety rule entailment framework that compiles safety rules into executable proposition trees and grounds them bottom-up via scene graphs, improving F1 by 9.8 points and suppressing spurious violations by over 80% across four real-world regulatory domains.

Background & Motivation

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have broadened the scope of automated content moderation from superficial keyword matching and coarse risk filtering toward nuanced, context-dependent visual safety reasoning. However, in high-stakes domains governed by formal standards—such as the FDA Food Code in food hygiene, the NYC Building Code in construction safety, Meta Community Standards in platform governance, and the UK CAP Code in advertising compliance—real-world risk patterns evolve rapidly. Regulatory policies are characterized by hierarchical conditions, legal exceptions, and cross-modal interdependencies. Mainstream end-to-end safeguards (e.g., LlamaGuard and LlavaGuard) conceptualize safety as a static taxonomy; adapting to revised policies or emerging edge cases necessitates expensive expert annotation and continual retraining, rendering them chronically slow to respond to regulatory shifts.

The fundamental tension stems from the inability of monolithic end-to-end models to perform verifiable, deductive logical reasoning over complex regulatory logic, causing them to succumb to two severe failure modes. First, models suffer from Visual Sensitivity Bias: provocative visual elements (such as blood, protest crowds, or property damage) trigger excessive heuristic alarm, causing models to declare violations while ignoring explicit regulatory prerequisites. Second, models exhibit Rule-Induced Bias: the normative, high-stakes framing inherent in safety rules (such as "terrorist organizations" or "violence and incitement") imposes strong contextual priors, biasing the model toward false positives even when images represent benign news reporting or compliant practices. Existing inference-time alternatives—whether direct prompting, latent representation alignment (e.g., SafeCLIP), or unconstrained agentic tool exploration (e.g., ViperGPT, CompAgent)—flatten the logical hierarchy of rules into holistic prompts or unconstrained execution paths, failing to preserve strict compositional operators such as disjunctions, conjunctions, and negations.

To break this deadlock, this paper argues for reframing visual compliance assessment from an implicit perceptual classification task into a formal problem of visual regulatory entailment. An effective safeguard must externalize complex regulations into explicit, inspectable symbolic logic while grounding elementary assertions in concrete, objective visual evidence, thereby insulating regulatory verdicts from emotional or visual priors. Core idea: formulate multimodal compliance verification as executable rule entailment, compiling safety regulations into atomic proposition trees and executing them bottom-up through scene graph instantiation and tool-aware visual grounding to achieve robust, interpretable, zero-shot compliance auditing.

Method

Overall Architecture

GUARDEN (Guarding by Safety Rule Entailment) formalizes visual safety assessment as deciding whether an image \(I\) logically entails a safety rule \(R\), written as \(I \models R\). Drawing upon logical atomism and formal semantics, any safety regulation can be decomposed into a finite set of atomic visual propositions \(P = \{p_1, \dots, p_n\}\) linked by an explicit logical form \(\Phi(P)\), such that \(I \models R \iff \Phi(p_1, \dots, p_n)\). Each atomic proposition \(p_i\) represents an objective assertion regarding visual evidence. GUARDEN decouples this verification into two distinct phases: offline Safety-Rule Compilation, which externalizes text regulations into executable proposition trees, and test-time Scene-Grounded Execution, which constructs a visual scene graph from the image to resolve atomic leaves bottom-up, supported by Tool-Aware Visual Grounding whenever perceptual tools encounter out-of-distribution targets.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Image I + Safety Rule R"] --> B["Safety-Rule Compilation<br/>Decompose-verify loop generates proposition tree"]
    B --> C["Scene-Grounded Execution<br/>Extract entities/relations & evaluate bottom-up"]
    C -->|Low confidence/Abstraction gap| D["Tool-Aware Visual Grounding<br/>Closed-loop alignment queries substitute keywords"]
    D -->|Update substitute cache| C
    C --> E["Output: Symbolic compliance verdict & verification trace"]

Key Designs

1. Safety-Rule Compilation: Policy complexity and normative bias -> Closed-loop goal decomposition into executable proposition trees -> Neutralizes rule-induced false positives and enables zero-shot regulatory adaptation
Presenting raw, normatively charged regulatory text directly to VLMs inevitably injects emotional and cautionary priors that trigger widespread false alarms. GUARDEN overcomes this by introducing an offline compilation pipeline that externalizes regulations into executable proposition trees \(T_R = (\phi_R, \{T_r\}_{r \in \text{sub}(R)})\), where internal nodes represent primitive logical operators \(\phi \in \{\land, \lor, \neg\}\) and terminal leaves represent atomic propositions \(p \in P\). Atomic propositions are restricted to two canonical, value-neutral structures: object-state predicates \((o, \sigma, \emptyset)\) asserting that an object \(o\) exhibits state \(\sigma\), and relational predicates \((o_1, \rho, o_2)\) asserting a spatial or physical relationship \(\rho\) between objects \(o_1\) and \(o_2\). Compilation proceeds via a top-down goal decomposition loop between a decomposer \(D\) and an iterative verifier \(V\). In trial \(t\), \(D\) proposes a logical operator and candidate sub-rules conditioned on prior feedback \(\eta_r^{(t-1)}\); \(V\) evaluates the proposal, returning a quality score \(s_r^{(t)}\), targeted refinement critique \(\eta_r^{(t)}\), and an atomicity flag \(a_r^{(t)} \in \{\text{True}, \text{False}\}\). Refinement repeats until the score saturates or leaf atomicity is certified. This converts subjective legal mandates into objective, value-neutral checks whose logical dependencies are explicitly preserved, allowing rule modifications or new regulatory introductions to be handled purely by recompiling the symbolic tree without model retraining.

2. Scene-Grounded Execution: Contextual visual disconnect and unexplainable reasoning -> Dual-stage agenda selection and geometric scene graph instantiation -> Delivers verifiable deductive proofs for safety verdicts
Even with an executable proposition tree, prompting a standard VLM to directly answer boolean queries over raw images remains vulnerable to spatial hallucinations and contextual misinterpretations. GUARDEN grounds the compiled tree by constructing a dedicated image scene graph \(G(I) = (V(I), E(I))\), transforming the logical tree into an image-grounded instance \(T_R^{(I)} = T_R[p \mapsto p \in G(I) : p \in P]\). Execution operates in two distinct stages: during the Selection stage, a lightweight selector VLM \(M_{sel}\) parses the leaf propositions against the image to establish a perception agenda containing required target objects \(O_{req}^{(I)}\) and candidate relations \(\Gamma_{req}^{(I)}\); during the Grounding stage, open-vocabulary detector SAM 3 and Google Document OCR locate bounding boxes and extract embedded text, while Depth Anything 3 provides metric monocular depth. Relational edges in \(\Gamma_{req}^{(I)}\) are resolved deterministically using formal geometric, overlap, and depth-ordering rules (spanning topological containment inside, proximity near, physical contact adjacent, vertical support on, vertical alignment above/below, and visual occlusion covering). Leaves in the proposition tree query the instantiated scene graph nodes and edges for truth values, propagating boolean states up the tree to produce a conclusive, auditable verdict with an end-to-end evidence chain.

3. Tool-Aware Visual Grounding: Perceptual foundation model failure on abstract and specialized legal terms -> Iterative closed-loop substitute keyword alignment -> Recovers omitted visual evidence and amortizes inference latency
Off-the-shelf vision foundation models (such as SAM 3 or open-vocabulary detectors) are bounded by their web-scale pre-training distributions, creating severe blind spots when confronted with legal terminology. They encounter an Abstraction Gap when rules reference overarching categories (such as "unmistakably recognized food containers") and a Lexical Gap when facing specialized statutory jargon (such as "egress pathway obstruction"), causing detector confidence to fall below the validation threshold (\(\tau = 0.8\)) and introducing silent, fatal omissions into \(G(I)\). GUARDEN resolves this via Tool-Aware Visual Grounding: whenever target object \(o\) fails detection with confidence \(c_o < \tau\), a grounding VLM \(M_{ground}\) inspects the failed target, prior proposals, detector responses, and raw image to iteratively generate detector-compatible substitute keywords \(k_o^{(t+1)}\). Once a proposed keyword achieves detection confidence exceeding \(\tau\) and \(M_{ground}\) confirms semantic fidelity, the accepted substitute is stored in a persistent substitute cache \(C[o]\). Across streaming evaluations, this compositional cache amortizes the query cost, allowing subsequent occurrences of statutory concepts to hit cached proxies immediately, keeping end-to-end deployment latency bounded between 8.6 and 11.6 seconds per image.

Loss & Training

GUARDEN operates as a non-parametric, inference-time neuro-symbolic framework and requires no domain-specific fine-tuning on safety violation datasets. All underlying vision backbones (SAM 3, SigLIP 2, Depth Anything 3, Google Document OCR) remain frozen. Offline Safety-Rule Compilation is executed using GPT-5.4 as the decomposer and verifier, while test-time online execution is powered by Gemma 4 26B (or alternatively Qwen 3.5 27B / GPT-5.4-mini). In the appendix, the authors benchmark a supervised LoRA fine-tuning baseline (rank 16, learning rate \(1\times 10^{-4}\), AdamW optimizer, bfloat16 precision across two NVIDIA RTX A6000 48GB GPUs, taking 4.5 to 7.0 hours to train to convergence) on Food Safety and Construction Safety. Empirical results show that even fully converged domain-fine-tuned models underperform the training-free Base + GUARDEN architecture on unseen, fine-grained safety rules.

Key Experimental Results

Main Results

Evaluation is conducted on SAFETYVISIONBENCH, spanning Food Safety (FDA Food Code), Construction Safety (NYC Building Code), Platform-Content Safety (Meta Community Standards), and Advertising-Content Safety (UK CAP Code), containing 24,000 real-world violation cases and 10,000 challenging hard-safe counterfactuals generated via deletion, addition, swapping, and attribute reversal. Gemma 4 26B serves as the inference VLM.

Method Food Safety (Recall / F1) Construction Safety (Recall / F1) Platform Safety (Recall / F1) Advertising Safety (Recall / F1) Mean F1 Gain
Direct Prompting 34.5±2.4 / 45.4±2.3 72.0±1.6 / 78.6±1.5 16.5±3.6 / 26.6±4.4 57.5±1.3 / 57.1±0.8 Baseline (51.9)
ViperGPT (Surís et al., 2023) 73.1±0.4 / 72.5±0.4 84.0±0.0 / 74.0±0.3 43.9±1.6 / 53.3±4.0 60.6±0.3 / 45.5±0.7 +9.2
ETA (Ding et al., 2025) 83.7±0.8 / 82.6±0.2 95.8±1.2 / 83.6±1.2 39.7±4.3 / 52.0±7.5 73.4±0.3 / 57.4±1.2 +14.8
SafeCLIP (Zhao et al., 2025) 82.9±1.2 / 81.6±0.2 79.0±0.0 / 84.2±0.7 41.8±2.1 / 53.5±2.1 71.9±0.9 / 55.2±1.7 +14.5
CLUE (Wang et al., 2025) 66.0±0.8 / 71.7±0.6 87.1±1.0 / 72.1±0.7 53.2±2.5 / 57.9±2.6 55.3±1.2 / 44.2±0.3 +9.6
CompAgent (Ghosh et al., 2026) 76.8±1.9 / 77.9±1.0 91.6±1.8 / 80.6±1.2 54.9±3.2 / 59.5±2.1 64.3±2.4 / 53.7±1.4 +16.0
GUARDEN (Ours) 89.3±1.2 / 84.4±3.5 92.9±0.8 / 86.4±0.1 70.0±4.0 / 69.0±2.6 69.8±0.5 / 74.9±3.6 +26.8 vs DP (+9.8 vs SOTA)

Ablation Study

The contribution of each individual component in GUARDEN is evaluated by removing one module at a time across all four domains using Gemma 4 26B (mean F1-score over three runs):

Configuration Food Safety F1 Construction F1 Platform F1 Advertising F1 Relative Mean Drop & Impact
GUARDEN (Full) 84.4 ± 3.5 86.4 ± 0.1 69.0 ± 2.6 74.9 ± 3.6 Reference Performance (Mean F1: 78.7)
w/o Relation Grounding 77.7 ± 3.6 52.0 ± 7.6 45.5 ± 2.8 44.9 ± 13.4 -30.1% (Severe collapse in spatial compliance, e.g. egress obstruction)
w/o Verifier Loop 74.6 ± 3.1 62.2 ± 15.7 46.3 ± 1.8 45.1 ± 12.8 -27.4% (Syntactic ambiguities and flipped operators degrade logic trees)
w/o Tool-Aware Grounding 76.2 ± 4.6 75.5 ± 6.8 64.6 ± 2.9 46.5 ± 12.8 -16.5% (Lexical/abstraction gaps induce widespread missing visual evidence)
w/o Scene-Grounded Execution 75.9 ± 1.4 80.5 ± 4.4 64.7 ± 4.6 53.4 ± 9.6 -12.8% (Direct VLM answering over raw images suffers from context drift)

Key Findings

  • Superiority in Contextually Complex Domains: GUARDEN delivers its most substantial gains in domains heavily governed by contextual exceptions and normative text—outperforming the best baseline by 17.5 F1 points in Advertising Safety (74.9 vs 57.4) and 9.5 F1 points in Platform Safety (69.0 vs 59.5).
  • Criticality of Spatial Relational Grounding: Discarding relational predicates causes F1 in Construction Safety to plummet by 34.4 points (86.4 \(\to\) 52.0), proving that spatial relationships (e.g., whether scaffolding blocks a fire exit) are the bedrock of physical regulatory compliance and cannot be approximated by isolated entity tags.
  • Drastic Reduction in Spurious Biases: In platform safety, Direct Prompting exhibits a 15.5% spurious violation rate on benign images containing provocative cues (blood, hate symbols, protests), whereas GUARDEN slashes this to 3.2% (an 80.4% reduction). On normative rules like "Dangerous Organizations", false positives collapse from 52.8% to 7.8% (an 85.2% reduction).
  • Outperforming Supervised Fine-Tuning: On Food Safety and Construction Safety, LoRA-fine-tuned Gemma 4 26B achieves 81.2 and 89.1 F1 respectively, whereas training-free Base + GUARDEN achieves 88.7 and 91.1 F1, proving that externalized rule entailment scales far better than parameter memorization.

Highlights & Insights

  • Paradigm Shift to Executable Regulatory Entailment: Moving away from black-box holistic toxicity scoring, GUARDEN treats compliance as deductive logic over scene graphs. Converting subjective legal text into neutral atomic checks eliminates the emotional and normative bias common in VLMs.
  • Adaptive Proxy Alignment for Tool Gaps: The closed-loop grounding alignment agent bridges lexical and abstraction gaps between formal legal prose and vision foundation models without modifying underlying weights, with cached proxies amortizing runtime overhead.
  • Instant Adaptation to Evolving Regulations: When statutory standards evolve (e.g., the UK CAP Code shifting from a relative youth appeal standard to an absolute strong-appeal threshold), updating GUARDEN requires only recompiling the symbolic proposition tree in seconds, entirely bypassing retraining.

Limitations & Future Work

  • Strict Dependence on Rule Quality and Precision: An audit reveals that 86% of compilation errors stem from inherently ambiguous statutory phrasing (e.g., "excessive", "small", "adequate"), indicating that statutory rules lacking quantifiable thresholds pose challenges for automated compilation.
  • Absence of Case-Law Analogical Reasoning: The framework is bounded by deductive entailment over explicit text rules. While augmenting the selection stage with a lightweight precedent retriever yields consistent 0.8–2.4 F1 improvements, integrating deep case-based analogical reasoning remains an open challenge.
  • Multi-Stage Latency Overhead: Despite parallelizing perceptual calls to achieve 8.6–11.6s per image, the pipeline remains substantially slower than sub-second end-to-end classifiers, limiting its direct deployment in real-time, high-throughput social feed filtering.
  • vs ViperGPT (Surís et al., 2023): ViperGPT generates arbitrary Python code to orchestrate vision APIs, often failing due to syntax bugs or infinite loops when parsing intricate legal clauses. GUARDEN enforces a disciplined, verified proposition tree that guarantees bounded, interpretable evaluation.
  • vs CLUE (Wang et al., 2025): CLUE converts rules into flat visual preconditions and relies on token logits for verification. GUARDEN preserves complete boolean logic structures (nested conjunctions, disjunctions, negations), beating CLUE by 30.7 F1 points in advertising compliance.
  • vs CompAgent (Ghosh et al., 2026): CompAgent utilizes an unconstrained ReAct agentic loop prone to wandering in complex visual scenes. GUARDEN utilizes an explicit perception agenda derived from the compiled tree, achieving higher precision with fewer redundant tool invocations.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers a rigorous neuro-symbolic executable rule entailment formulation for multimodal regulatory compliance]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmark covering four major statutory codes, rigorous bias audits, ablations, and cross-model evaluations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear formal semantics, cohesive narrative structure, and comprehensive empirical presentation]
  • Value: ⭐⭐⭐⭐⭐ [Provides a practical, auditable, and adaptable blueprint for deploying automated compliance safeguards in high-stakes domains]