SCOPEBENCH: Evaluating Cultural Norm Scope Awareness in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/HAOJUN-LIANG/ScopeBench
Area: Multimodal VLM
Keywords: Cultural Norm Scope, LLM Safety, Scope Inflation and Deflation, Multi-Task Benchmark, Cross-Cultural Alignment
TL;DR¶
This paper introduces SCOPEBENCH, a comprehensive benchmark operationalizing cultural norm scope as a first-class safety dimension across 3,323 norms from 49 countries and 9 cultural clusters, revealing that prompt formatting inverts dominant error types and scope classification accuracy decouples from behavioral safety in 21 frontier LLMs.
Background & Motivation¶
Safety evaluation frameworks for large language models have predominantly treated harmfulness as culturally invariant, benchmarking whether systems refuse universally dangerous queries without accounting for normative applicability. However, deployed AI systems face real-world interactions where the binding nature of a social standard varies drastically across boundaries: a practice considered standard etiquette in one culture may be perceived as offensive or irrelevant elsewhere, whereas foundational human rights and non-derogable protections bind all humanity. Mischaracterizing this boundary produces severe failures in both directionsโeither imposing parochial, community-specific conventions upon outsiders as mandatory global rules, or relativizing core universal protections into local customs.
Concurrently, research on cross-cultural natural language processing has focused on factual cultural recall and stylistic contextual adaptation, but these benchmarks treat norm scope as an unexamined constant. Models are evaluated on their ability to retrieve regional trivia or adapt conversational tone, yet existing setups take for granted that every cataloged norm is uniquely tied to its geographic origin without questioning whom it actually binds. Consequently, prior frameworks lack the diagnostic power to determine whether a model's safety decisions arise from principled scope recognition or format-induced heuristic guessing, leaving both cultural chauvinism (universalizing local customs) and destructive moral relativism (localizing universal human rights) entirely unmeasured.
To resolve this blind spot, cultural scope must be formalized and operationalized as an essential alignment and safety axis, disentangling stable normative knowledge from prompt-presentation artifacts and evaluating downstream enforcement in realistic conversations. Core idea: establish SCOPEBENCH, a dual-anchored benchmark of 3,323 norms spanning 49 countries across nine cultural clusters with unified formalizations of Scope Inflation, Scope Deflation, and Scope Displacement errors across classification, joint/independent attribution, and three-stage behavioral safety tasks.
Method¶
Overall Architecture¶
The SCOPEBENCH evaluation framework spans dataset curation, formal error modeling, and a four-task empirical pipeline tracing model capabilities from abstract metalinguistic knowledge to interactive behavioral enforcement. Candidate norms are gathered through a dual-stream design: universal norms are drawn from international human rights instruments and ethics corpora, while culture-specific norms are elicited via native in-country surveys across 49 countries and existing regional corpora. Each item undergoes a rigorous 3-annotator majority validation threshold (\(\ge 2/3\)) and dual-level deduplication, producing 975 UNIVERSAL and 2,348 SPECIFIC norms. Models are evaluated on standalone binary scope classification (Task 1), joint multiple-choice cultural attribution (Task 2a), per-option independent binary attribution (Task 2b), and three-stage simulated user behavioral safety interaction (Task 3).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Dual-Stream Norm Sources<br/>Universal (UDHR/Ethics) + Specific (49 Countries/9 Clusters)"] --> B["Dual-Anchored Validation and Filtering<br/>975 Universal Norms + 2348 Specific Norms"]
B --> C["Unified Tripartite Scope Error Taxonomy<br/>Scope Inflation / Deflation / Displacement"]
C --> D["Task 1: Isolated Norm Scope Classification<br/>Predict UNIVERSAL vs. SPECIFIC without metadata"]
C --> E["Task 2a/2b: Joint & Independent Cultural Attribution<br/>Quantify option anchoring & Cluster Proximity Gap (CPG)"]
C --> F["Task 3: Three-Stage Behavioral Safety Evaluation<br/>Adversarial generation โ Model response โ Automated judgment"]
D --> G["Structural Safety Failure Analysis<br/>Format error inversion & Knowledge-behavior decoupling"]
E --> G
F --> G
Key Designs¶
1. Dual-Anchored Validation and Filtering: Disentangling Universal Protections from Culture-Specific Practices Prior cultural datasets treat items uniformly without normative stratification, generating widespread ambiguity between fundamental protections and local etiquette. SCOPEBENCH addresses this by establishing two disjoint streams grounded in external normative authorities: the UNIVERSAL stream anchors 975 norms in codified international agreements (such as the Universal Declaration of Human Rights, Geneva and Helsinki medical declarations) and cross-cultural safety benchmarks (ETHICS-CM, HarmBench, Social Chemistry 101), representing obligations that must not be waived in any cultural context. The SPECIFIC stream collects 2,348 community-bounded practices across 49 countries in 9 cultural clusters via structured surveys of native cultural residents with lived in-country experience, supplemented by corpora like SAFEWORLD. Each item is vetted by three independent annotators with relevant background, retaining candidates only under a \(\ge 2/3\) majority agreement (discarding 1,022 ambiguous items, a 23.5% filter rate). Two-level deduplication within and across streams eliminates semantic overlap, yielding an internally consistent ground-truth pool with human validation inter-rater agreement of Cohen's \(\kappa = 0.78\).
2. Unified Tripartite Scope Error Taxonomy: Aligning Failure Definitions Across Diverse Formats To isolate genuine normative misconceptions from prompting format artifacts, the framework operationalizes three structural scope error types identically across all evaluation settings: - Scope Inflation (SI): treating a culture-specific norm as universally binding, thereby illegitimately imposing parochial customs on outsiders; - Scope Deflation (SD): treating a universal protection as culture-specific, thereby relativizing non-derogable ethical requirements; - Scope Displacement (SP): acknowledging that a norm is culture-specific but attributing it to an incorrect cultural community. In attribution tasks (Task 2a/2b), metrics are mathematically formulated as proportion of errors. For Task 2a, denoting specific norms as \(S\) and universal norms as \(U\), with explicit universal option \(E\): $$ \text{SIR}{2a} = \frac{|{n \in S : \hat{y}n = E}|}{|S|}, \quad \text{SDR}} = \frac{|{n \in U : \hat{yn \neq E}|}{|U|} $$ To quantify cultural geographic coherence, displacement errors are decomposed into same-cluster hard distractors (\(h_n\)) versus cross-cluster easy distractors (\(e_n\)), defining the Cluster Proximity Gap (CPG): $$ \text{CPG}} = \text{SPR $$ A positive CPG confirms that displacement errors concentrate near cultural boundaries rather than occurring uniformly at random.}^{\text{hard}} - \text{SPR}_{2a}^{\text{easy}
3. Comparative Attribution Formats and Three-Stage Behavioral Interaction: Isolating Anchoring and Knowledge-Behavior Gaps To reveal how presentation framing affects model judgments and whether metalinguistic knowledge governs conversational behavior, the framework executes two attribution paradigms and a simulated multi-stage interaction: - Joint Multiple-Choice (Task 2a) vs. Independent Binary (Task 2b): Task 2a presents all candidate countries together with explicit universal option E and abstention option F. Task 2b decomposes attribution into four independent Yes/No queries per candidate country in isolation. This eliminates option E's visual anchoring, exposing latent overgeneralization tendencies. - Three-Stage Behavioral Safety Protocol (Task 3): In Stage 1, 5,671 subtly incorrect, plausibly phrased scope-violating statements are generated across SD, SI, and SP paradigms. In Stage 2, evaluated models assess the statement's cultural accuracy under zero-shot instructions without any hints of error. In Stage 3, an automated LLM judge classifies responses into Appropriate (AR), Evasive (ER), or Harmful (HR), directly capturing downstream safety vulnerability.
Benchmark & Evaluation Protocol¶
The benchmark evaluates 21 frontier LLMs spanning seven major families (OpenAI, Anthropic, Google, Meta, Qwen, DeepSeek, Mistral), incorporating proprietary flagships and open-weight architectures. All models are evaluated zero-shot under deterministic decoding where supported. Task 3 employs an automated evaluation protocol calibrated against human expert classifications (achieving Cohen's \(\kappa = 0.72\) on 200 stratified validation pairs) and verified for cross-provider stability across 10,500 samples using secondary judges Claude Opus 4.7 and Grok-3 (Spearman rank correlation \(\rho \ge 0.94\)).
Key Experimental Results¶
Main Results: Scope Classification vs. Behavioral Safety (Tasks 1 and 3)¶
In Task 1 standalone classification, models exhibit low inflation (\(\le 7.6\%\)) but severe deflation (up to \(70.0\%\)). However, when tested in Task 3 behavioral interaction, near-perfect Task 1 classifiers fail to maintain safety, registering harmful-response rates (\(\text{HR}_{\text{total}}\)) exceeding \(40\%\).
| Model | Model Type | Task 1 Acc (%) | Task 1 SDR (%) | Task 1 SIR (%) | Task 3 \(\text{HR}_{\text{SD}}\) (%) | Task 3 \(\text{HR}_{\text{SP}}\) (%) | Task 3 \(\text{HR}_{\text{SI}}\) (%) | Task 3 \(\text{HR}_{\text{total}}\) (%) | Task 3 \(\text{AR}_{\text{total}}\) (%) |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.4 | Proprietary | 93.3 | 12.8 | 4.2 | 52.0 | 52.7 | 4.2 | 32.5 | 67.5 |
| GPT-4.1 | Proprietary | 77.1 | 70.0 | 3.4 | 65.2 | 56.9 | 8.6 | 38.3 | 61.6 |
| o3-mini | Proprietary | 92.1 | 13.0 | 5.8 | 49.0 | 44.1 | 5.7 | 29.0 | 70.9 |
| Claude Opus 4.7 | Proprietary | 89.3 | 23.2 | 5.5 | 47.6 | 51.4 | 5.1 | 31.6 | 67.5 |
| Claude Sonnet 4.6 | Proprietary | 91.2 | 11.6 | 7.6 | 67.6 | 56.4 | 6.2 | 37.5 | 62.3 |
| Gemini 2.5 Flash | Proprietary | 92.9 | 16.6 | 3.2 | 72.7 | 58.0 | 7.1 | 39.5 | 60.3 |
| Llama-4-Maverick | Open-weight | 97.7 | 0.5 | 3.1 | 74.2 | 59.9 | 10.2 | 41.8 | 58.2 |
| Llama-4-Scout | Open-weight | 97.9 | 0.2 | 2.9 | 71.3 | 60.0 | 10.7 | 41.5 | 58.4 |
| Llama-3.1-70B | Open-weight | 98.1 | 0.4 | 2.6 | 74.3 | 60.5 | 10.9 | 42.4 | 57.6 |
| Qwen3-235B | Open-weight | 85.4 | 46.9 | 1.1 | 77.7 | 63.8 | 11.3 | 44.5 | 55.5 |
| DeepSeek-V3 | Open-weight | 93.8 | 3.3 | 7.4 | 86.5 | 70.9 | 16.9 | 51.2 | 48.8 |
| Mistral Large 3 | Proprietary | 97.3 | 0.3 | 3.7 | 73.3 | 58.7 | 10.0 | 41.0 | 58.9 |
| Dolphin-R1-24B | Open-weight | 97.7 | 2.1 | 2.4 | 73.3 | 60.4 | 11.0 | 42.2 | 57.8 |
(Claude Haiku 4.5 is excluded from cohort rank bounds due to a 19.8% evasion rate driven by systematic role refusal).
Ablation & Analysis: Prompt Formatting Inverts Error Dominance (Task 2a vs. Task 2b)¶
Evaluating models across joint (Task 2a) versus independent (Task 2b) attribution reveals an inversion of the dominant error mode: joint presentation suppresses Scope Inflation while inducing Scope Deflation, whereas independent evaluation unmasks latent Scope Inflation exceeding 30%.
| Model | 2a SIR (%) | 2b SIR (%) | 2a SDR (%) | 2b SDR (%) | 2a CPG (%) | 2b CPG (%) | Framing Sensitivity Profile |
|---|---|---|---|---|---|---|---|
| GPT-5.4 | 5.1 | 10.1 | 7.7 | 0.1 | -0.5 | +4.5 | CPG transitions to strictly positive under independent queries |
| Llama-4-Maverick | 1.8 | 33.0 | 18.4 | 1.4 | -0.2 | +3.2 | Removing option E causes an 18-fold surge in Scope Inflation |
| Llama-4-Scout | 1.8 | 32.3 | 18.7 | 1.8 | -0.2 | +2.6 | Severe latent overgeneralization unmasked without joint choices |
| Llama-3.1-70B | 1.9 | 32.2 | 18.1 | 1.6 | +0.3 | +3.9 | Consistent positive CPG confirms cultural boundary sensitivity |
| Qwen3-235B | 5.8 | 8.6 | 4.9 | 0.2 | -1.2 | +2.9 | Removes option choice interference; reveals geographic clustering |
| Dolphin-R1-24B | 1.7 | 29.9 | 22.1 | 0.9 | -1.7 | +4.0 | Bidirectional shift: SDR drops from 22.1% to 0.9%, SIR spikes |
| Mistral Large 3 | 1.8 | 29.3 | 15.0 | 1.2 | +0.2 | +3.9 | Strong bidirectional anchoring across option modes |
Key Findings¶
- Format-Induced Inversion of Dominant Errors: In standalone classification (Task 1), Scope Deflation dramatically exceeds Scope Inflation for all models (e.g., GPT-4.1 achieves SDR of 70.0% vs. SIR of 3.4%). In Task 2a, joint presentation with an explicit universal option E keeps SIR below 2.2% while driving SDR up to 22.1%. In Task 2b, querying each country independently causes SIR to spike to 33.0% while suppressing SDR to under 1.85%. The prompt structure fundamentally shifts model error profiles.
- Empirical Decoupling Between Knowledge and Interactive Safety: Task 1 classification accuracy is statistically uncorrelated with Task 3 harmful-response rates (Pearson \(r = 0.09, p = 0.69\)). Six Llama and Mistral models achieving \(\ge 97\%\) accuracy in Task 1 still produce harmful responses to 40.3%โ42.4% of scope-violating statements. Conversely, GPT-4.1, the weakest Task 1 classifier (77.1% accuracy, 70% SDR), achieves an \(\text{HR}_{\text{total}}\) of 38.3%โoutperforming all near-ceiling Task 1 models.
- Scope Deflation as the Primary Behavioral Hazard: Across 19 of 21 models, the vulnerability ordering follows \(\text{HR}_{\text{SD}} > \text{HR}_{\text{SP}} > \text{HR}_{\text{SI}}\). Models reliably reject localized norms falsely generalized to the world (\(\text{HR}_{\text{SI}}\): 1.8%โ17.1%), but readily accept statements that falsely restrict universal human rights to individual cultures (\(\text{HR}_{\text{SD}}\): 44.3%โ86.5%).
Highlights & Insights¶
- Scope as an Independent Alignment Property: Formalizes the boundary of normative bindingness as distinct from refusal and factual recall, providing mathematical definitions for inflation, deflation, and displacement.
- Bidirectional Anchoring Discovered: Demonstrates that the presence of an explicit "universal" option simultaneously protects specific norms from inflation while drawing universal norms into unwarranted localization, cautioning against single-format multiple-choice benchmarks.
- Dismantling Aggregate Refusal Metrics: Analyzes Claude Haiku 4.5's refusal patterns to establish the vital distinction between capability-driven evasion (hedging due to uncertainty) and role-restriction refusal (declining the task entirely), exposing how mechanical refusals distort apparent safety scores.
Limitations & Future Work¶
- English-Only Presentation: Evaluating non-Western cultural norms exclusively in English creates translational nuances and may bias evaluations against culturally embedded concepts that lack direct English equivalents.
- Single Generator in Task 3: All 5,671 scope-violating statements originate from GPT-5.5; diversifying adversarial generation across model architectures remains an important step.
- Multilingual and Gradient Boundaries: Future work should extend SCOPEBENCH to native languages and investigate soft or gradient normative boundaries for traditions that blend universal ethics with regional customs.
Related Work & Insights¶
- vs. Standard LLM Safety Benchmarks (HarmBench / SORRY-Bench / AIR-Bench): Standard benchmarks assume harm is global and homogeneous, measuring refusal on toxic prompts. SCOPEBENCH introduces the normative scope dimension, evaluating whether models know whom a norm is meant to bind.
- vs. Cultural Adaptability Benchmarks (SAFEWORLD / CulturalBench / NormAd): Existing cultural benchmarks tie all items to individual source countries without universal/specific distinction. SCOPEBENCH introduces dual-stream grounding and isolates proximity biases through a hard/easy distractor design.
Rating¶
- Novelty: โญโญโญโญโญ Establishes cultural scope as a first-class safety dimension with a mathematically rigorous error taxonomy.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous four-task progression across 21 frontier models from 7 families, substantiated by human baselines and multi-judge audits.
- Writing Quality: โญโญโญโญโญ Exceptionally lucid narrative prose, precise formal definitions, and deeply insightful empirical analysis.
- Value: โญโญโญโญโญ Highly impactful framework for mitigating both cultural chauvinism and moral relativism in globally deployed AI systems.