Skip to content

Beyond Single-Policy: Evaluating Composed Organization-Specific Policy Alignment in LLM Chatbots

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/FudanSELab/COPAL
Area: Dialogue Systems
Keywords: organization-specific policies, composed policy alignment, chatbot evaluation, handling contracts, interaction patterns

TL;DR

Addressing the critical gap where real-world chatbot queries naturally span multiple organizational rules, COPAL introduces an automated evaluation framework driven by four empirical interaction patterns and target facets paired with explicit handling contracts, revealing an average composed-policy error rate of 33.1% across 9 frontier LLMs.

Background & Motivation

Large language model (LLM) chatbots are increasingly deployed across high-stakes organizational settings such as healthcare, finance, public administration, and customer service. In these production environments, chatbot behavior is strictly governed by organization-specific policies that delineate permissible and prohibited outputs. For example, a healthcare chatbot may be permitted to offer general appointment scheduling and descriptive product nutrition information, yet strictly barred from providing diagnostic opinions or individualized medical advice. Evaluating whether chatbots consistently adhere to these customized boundaries is critical to preventing compliance violations, erroneous user guidance, and institutional liability.

However, existing policy-alignment benchmarks evaluate systems using a single-policy paradigm, testing one isolated policy constraint at a time. This assumption diverges fundamentally from production reality: real users formulate requests around their immediate goals rather than organizational policy boundaries, meaning a single natural query frequently activates multiple distinct policies. An audit of 1,000 human-labeled real-traffic interactions from deployed chatbots reveals that 47.6% of queries involve more than one policy, and these multi-policy queries exhibit failure rates nearly three times higher than single-policy queries. Passing every single-policy test individually provides no guarantee of compliance when a model must jointly satisfy conflicting, nested, or conditional constraints within one unified response. For instance, when asked "I have diabetes. Which lunch option is best for me?", a chatbot often eagerly satisfies the permitted product description request while simultaneously breaching the medical advice constraint.

Translating this insight into a scalable evaluation pipeline presents two core challenges: first, the combinatorial explosion of policy subsets makes unguided pairing generate mostly implausible scenarios; second, evaluating composed responses directly against full policy inventories forces the judge to recompose the boundaries dynamically, confounding judging errors with chatbot failures. Core idea: systematically construct composed-policy test queries guided by four empirically derived interaction patterns and diagnostic target facets, pairing each query with an explicit handling contract that pre-specifies what to provide and avoid to enable reliable evaluation.

Method

Overall Architecture

Given an organization's profile and its natural-language policy inventory, COPAL constructs a compact, high-diagnostic suite of composed-policy test queries paired with verifiable handling contracts. The pipeline operates in four sequential stages: 1. Policy Grounding: parsing unstructured policy texts into structured clauses with normalized operational effects; 2. Composition Construction: searching the combinatorial space using trigger compatibility and scope overlap, matched to four recurring interaction patterns; 3. Composed Query Generation: instantiating selected combinations into natural, adversarial user requests via target facets, and compiling explicit handling contracts; 4. Policy-Handling Evaluation: mapping target model responses into observed handling sets and scoring them deterministically against the contracts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Organization Profile & Policy Inventory<br/>Natural-language policy documents"] --> B["Policy Grounding<br/>Trigger-Scope-Effect tuples"]
    B --> C["Composition Construction<br/>Feasible subsets via 4 interaction patterns"]
    C --> D["Composed Query Generation<br/>Facet-guided queries with handling contracts"]
    D --> E["Target Chatbot Execution<br/>Response under organization system prompt"]
    E --> F["Policy-Handling Evaluation<br/>Deterministic checking of Required and Forbidden sets"]
    F --> G["Evaluation Results & Diagnostics<br/>One-sided failure profiling and facet atlas"]

Key Designs

1. Policy Grounding: Structured Clause Representation and Effect Normalization

Natural-language organizational policies frequently leave trigger conditions implicit or bundle multiple obligations into loose phrasing. To make diverse policies semantically comparable and composable, COPAL grounds each policy rule into an operational clause tuple: $\(r = (\phi_r, \omega_r, \epsilon_r)\)$ where \(\phi_r\) denotes the trigger condition activating the rule (e.g., user health inquiry, account status, dialogue context), \(\omega_r\) defines the governed response scope (e.g., product attributes, personal diagnosis, payment fields), and \(\epsilon_r\) represents the normalized operational effect. Drawing from PBSuite taxonomy, effects are normalized into six functional categories: PERMIT, PROHIBIT, REQUIRE-GATE, DISCLOSE, WITHHOLD, and ROUTE. This formal decomposition isolates the operative constraints required for subsequent multi-clause alignment.

2. Pattern-Guided Composition Construction

Pairing arbitrary clauses yields an intractable combinatorial space where the vast majority of combinations never co-occur in practice. COPAL prunes this space via a two-stage filter. First, candidate clause sets must exhibit trigger compatibility (satisfiable within a single coherent user intent) and scope overlap (addressing semantically related facets of that intent). Second, the paired effects must instantiate one of four recurring interaction patterns derived from real-world empirical interactions: - Scope Restriction: balancing PERMIT/DISCLOSE against PROHIBIT/WITHHOLD by providing the licensed sub-request while cleanly omitting the prohibited portion; - Prerequisite Gating: enforcing REQUIRE-GATE prior to PERMIT/DISCLOSE by requiring identity verification or consent before releasing governed information; - Selective Disclosure: mediating between DISCLOSE and WITHHOLD by releasing authorized record fields while strictly withholding protected attributes; - Workflow Transfer: navigating PERMIT/CONTINUE versus ROUTE by terminating default autonomous fulfillment and handing off to authorized human or specialized channels upon encountering specific criteria.

3. Facet-Conditioned Query Generation and Explicit Handling Contracts

To prevent test queries from collapsing into bland or easily resolved requests, COPAL guides LLM generation using target facetsโ€”pre-specified failure pressures tailored to each interaction pattern (e.g., boundary overreach, over-refusal, or semantic leakage for scope restriction; pre-gate leakage or skipped gates for prerequisite gating). Simultaneously, COPAL solves the evaluator ambiguity problem by generating an explicit handling contract alongside each query: - \(R_q\): the required handling set defining necessary content or actions that the response must provide; - \(F_q\): the forbidden handling set defining prohibited disclosures or actions that the response must avoid.

4. Contract-Based Policy-Handling Evaluation and Error Attribution

During response scoring, the LLM judge does not inspect the full, noisy policy inventory. Instead, it maps the model output \(M(q)\) to an observed handling set \(H_M(q)\) and verifies contract satisfaction directly. A response incurs an error (\(e_q = 1\)) if any required handling is missing or any forbidden handling is triggered: $\(e_q = \mathbb{I}\left( R_q \not\subseteq H_M(q) \;\lor\; F_q \cap H_M(q) \neq \emptyset \right)\)$ The overall score is given by \(\text{Err}(M, Q) = \frac{1}{|Q|}\sum_{q\in Q} e_q\). Fixing the evaluation standard prior to judging prevents the judge from having to dynamically reconstruct multi-policy boundaries, decoupling evaluator reasoning failures from actual model deficiencies.

Key Experimental Results

Main Results

The primary benchmark evaluates 9 served commercial and open-weight models across 30 simulated enterprise company worlds comprising 882 policy rules and 900 selected composed queries (totaling 8,100 judgments). Additionally, a paired single-policy contrast evaluates 5 models on 2,315 single-clause projections to isolate composition-induced degradation.

Model Overall Composed Error (%) Scope Restriction (%) Prerequisite Gating (%) Selective Disclosure (%) Workflow Transfer (%) Single-Policy Projection Error (%) Composition Gap (pp)
GPT-5.5 15.89 13.19 17.88 13.14 21.98 2.94 +12.95
Gemini 3.1 Pro 18.33 14.29 26.26 20.00 17.03 2.42 +15.91
GLM-5.1 25.33 19.51 39.66 21.14 26.92 - -
Claude Sonnet 4.6 27.00 21.15 35.75 25.71 31.32 3.11 +23.89
Kimi K2.6 28.89 24.73 39.66 29.71 25.82 - -
Qwen3.5 42.89 37.91 48.04 48.00 42.86 - -
Doubao-Seed-2.0-pro 43.89 37.36 56.98 44.00 43.96 3.46 +40.43
MiniMax-M2.7 46.67 40.93 56.42 50.29 45.05 - -
DeepSeek-V3.2 49.33 46.43 49.72 50.86 53.30 4.15 +45.18

Ablation Study

The ablation experiments investigate the diagnostic yield of COPAL's construction pipeline under a strictly matched final budget of 12 test items per company world across two probe models. In addition, an evaluation oracle validity audit on 200 human-labeled query-response pairs verifies the necessity of contract-based judging.

Construction Configuration / Judge Setting Target Metric Score Key Takeaway
Construction Pipeline Ablation Matched budget: 12 items/company
Raw-policy planning Probe Error Rate 4.03% Unstructured generation produces generic, non-diagnostic queries
Clause-only planning Probe Error Rate 7.50% Structural grounding alone lacks targeted interaction tension
w/o facet query generation Probe Error Rate 10.69% Removing target facets under-samples critical boundary failure modes
Full COPAL Probe Error Rate 31.94% Full pipeline concentrates sharp, multi-policy compliance risks
Judge Validity Audit (200 Human Gold Labels) Gemini 3 Flash evaluator
Full Policy Inventory Agreement / F1 56.5% / 40.8% Recomposing boundaries dynamically causes severe recall collapse (26.1%)
Handling Contract (COPAL) Agreement / F1 80.5% / 80.4% Contract-based checking delivers high precision (95.2%) and minimal false alarms (4.7%)

Key Findings

  • Drastic Composition-Induced Performance Collapse: Across paired evaluations, models achieve low error rates on isolated single policies (2.42% to 4.15%), but their error rates escalate sharply to 15.89%โ€“49.33% when identical rules are jointly activated in a composed request. This establishes that failures stem from handling policy interactions rather than single-policy ignorance.
  • Predominance of One-Sided Failures: Error attribution reveals that chatbots rarely fail symmetrically; instead, they satisfy one side of the contract while violating the other. Models either satisfy the permitted request and leak restricted information (e.g., pre-gate leakage at 55.2%, semantic leakage at 47.4%), or exhibit over-refusal by rejecting the entire query due to the presence of a restricted topic.
  • Prerequisite Gating as the Primary Bottleneck: Prerequisite gating proves to be the most challenging pattern for 7 of the 9 evaluated models, with error rates surpassing 56% for several architectures, highlighting widespread difficulty in withholding information until verification constraints are satisfied.

Highlights & Insights

  • Exposing the Hidden Multi-Policy Blind Spot: Systematically identifies and formalizes composed-policy alignment as an under-tested failure source in enterprise LLM deployment, shifting the paradigm from single-rule compliance to realistic multi-constraint negotiation.
  • Explicit Handling Contracts Decouple Judging Uncertainty: By binding deterministic inclusion and avoidance criteria to each query during synthesis, COPAL eliminates evaluator-side policy reconstruction errors, increasing judging F1 from 40.8% to 80.4%.
  • Actionable Blueprint for Chatbot Alignment: The four interaction patterns (scope restriction, prerequisite gating, selective disclosure, workflow transfer) offer concrete training guidelines for synthetic data generation and RLHF preference modeling in organizational dialogue systems.

Limitations & Future Work

  • Single-Turn Evaluation Context: The current benchmark focuses predominantly on single-turn request-response pairs, omitting dynamic state tracking across long multi-turn sessions with evolving user authorizations.
  • Simulated Environment Abstraction: Enterprise company worlds represent controlled approximations that do not fully capture backend API tool executions, enterprise database state, or real-time permission switches.
  • Conservative Judge False Negatives: While the contract-based judge achieves high precision (95.2%), it exhibits a 30.4% false-negative rate against human annotations, occasionally overlooking subtle semantic leakages.
  • vs PBSuite (Varshney et al., 2025): While PBSuite pioneeringly generated industry-specific policy worlds, its evaluations test multi-turn adherence to individual policies; COPAL builds on this environment foundation to investigate combinatorial policy tensions within individual queries.
  • vs Compass (Choi et al., 2026): Compass examines organizational policy conflicts and retrieval; COPAL focuses directly on generative response-boundary adherence, testing whether models can deliver partial compliance without overreaching.
  • vs SORRY-Bench & OR-Bench (Xie et al., 2025; Cui et al., 2025): Standard safety refusal benchmarks assess binary compliance vs. refusal on harmful queries; COPAL targets legitimate organizational workflows where the compliant response is nuanced selective service rather than blanket refusal.

Rating

  • Novelty: โญโญโญโญโญ Formulates composed-policy alignment in chatbots for the first time, introducing four empirical interaction patterns and contract-based evaluation.
  • Experimental Thoroughness: โญโญโญโญโญ Combines a 1,000-case industrial traffic audit, 30 diverse enterprise worlds, 9 contemporary LLMs, systematic ablations, and rigorous human gold audits.
  • Writing Quality: โญโญโญโญโญ Highly organized narrative, rigorous formal definitions, and clear analytical taxonomies.
  • Value: โญโญโญโญโญ Offers indispensable tools and evaluation protocols for deploying safe, compliant LLM chatbots in regulated enterprise domains.