Skip to content

ScopeBench: Evaluating Cultural Norm Scope Awareness in Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/HAOJUN-LIANG/ScopeBench
Area: LLM Safety
Keywords: Cultural Norm Scope, LLM Safety Alignment, Scope Awareness, Value Alignment, Benchmark

TL;DR

ScopeBench establishes a dual-anchored benchmark of 3,323 norms spanning 49 countries across nine cultural clusters to evaluate cultural norm scope awareness in 21 frontier LLMs, revealing a stark format-dependent error inversion and an empirical decoupling between classification accuracy and behavioral safety.

Background & Motivation

Safety evaluation frameworks for frontier large language models have long operated on the unstated premise that normative harm is culturally invariant. Standard red-teaming benchmarks, such as HarmBench and SORRY-Bench, along with regulatory taxonomies like AIR-BENCH, measure whether a model reliably refuses harmful queries by applying a single global harm judgment regardless of who is asking. However, global deployment exposes models to queries where normative obligations are fundamentally contingent on cultural identity—such as declining alcohol at a business dinner in Saudi Arabia, observing hierarchical bowing etiquette in Japan, or navigating cohabitation arrangements in Indonesia. Under existing safety paradigms, models possess no formal mechanism to discern whether a behavioral norm binds every user universally or applies only within a bounded community, leaving them prone to severe governance failures in cross-cultural interactions.

In parallel, the rapidly expanding literature on multicultural NLP and cultural adaptability—exemplified by benchmarks such as CulturalBench, KorNAT, BLEnD, and SafeWorld—evaluates whether models recall region-specific cultural facts or calibrate their communicative tone to local audiences. Yet these benchmarks treat the normative scope of each custom as a given rather than as an explicit object of evaluation. They test whether a model knows what people in a given country do, but never evaluate whether the model understands whom that norm is legitimate in binding. Misjudging norm scope in either direction carries profound ethical and safety hazards: treating a culture-specific custom as a universal imperative imposes unjustified obligations on outsiders (Scope Inflation), whereas treating fundamental human-rights protections as culturally relative creates unacceptable safety vacuums (Scope Deflation).

This persistent gap stems from the lack of an externally anchored benchmark that operationalizes cultural scope as an independent safety dimension and traces its manifestation across the knowledge-to-behavior pipeline. Prior datasets neither distinguish universal human-rights frameworks from localized customs nor measure the cognitive distortions introduced by contrasting evaluation formats. To bridge this critical divide, an evaluation suite must formalize scope errors systematically and measure whether isolated metalinguistic knowledge translates into reliable boundary enforcement during open-ended interaction. Core idea: establish cultural norm scope as a first-class safety alignment axis by introducing ScopeBench—a dual-anchored benchmark of 3,323 norms across 49 countries and nine clusters—and formalizing a unified three-way error taxonomy that uncovers prompt-format error inversion and a stark empirical decoupling between classification accuracy and behavioral safety across 21 frontier LLMs.

Method

Overall Architecture

ScopeBench operationalizes cultural norm scope awareness through a structured, multi-stage pipeline designed to track model capabilities from static knowledge representation to dynamic interactive behavior. The pipeline begins with dual-stream data collection and dual-anchored validation, compiling 3,323 rigorously verified norms across 49 countries and nine cultural clusters into mutually exclusive UNIVERSAL and SPECIFIC categories. It formalizes a unified three-way error taxonomy—Scope Inflation, Scope Deflation, and Scope Displacement—applied identically across four progressive evaluation tasks. These tasks encompass standalone binary scope classification (Task 1), joint multiple-choice attribution with an explicit universal option (Task 2a), per-option independent binary judgment that removes option anchoring (Task 2b), and a three-stage interactive behavioral safety evaluation driven by 5,671 plausibly framed scope-violating statements (Task 3).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Candidate Norm Corpora Pool<br/>4,345 raw multi-source candidates"] --> D1["Dual-Anchored Dual-Stream Construction<br/>UDHR/ethics instruments vs. 49-country native expert review"]
    D1 --> D2["Unified Cross-Task Error Taxonomy<br/>Formalization of Scope Inflation, Deflation, and Displacement"]
    D2 --> D3["Format-Decoupled Four-Task Pipeline<br/>Isolated classification to joint/per-option attribution and behavioral safety"]
    D3 --> Out["Multi-Model Scope Awareness Diagnostics<br/>Surfacing format-dependent error inversion and accuracy-safety decoupling"]

Key Designs

1. Dual-Anchored Dual-Stream Construction: Resolving boundary ambiguity and cultural bias through formal normative and ethnographic grounding Prior cultural datasets treat norm assignments as subjective or purely country-indexed, risking high noise and arbitrary classification. ScopeBench overcomes this by establishing two substantively disjoint collection streams with independent normative anchors. The UNIVERSAL Stream draws candidates from codified cross-cultural moral instruments, including international human-rights declarations (UDHR, ICCPR, Rome Statute, CEDAW, CAT), medical ethics charters (WMA Declaration of Helsinki, WHO guidelines), empirical moral frameworks (Moral Foundations Theory, Schwartz Values), and established safety corpora (ETHICS, HarmBench, BeaverTails). Conversely, the SPECIFIC Stream covers 49 countries grouped into nine cultural clusters (East Asia, Southeast Asia, South Asia, MENA, Sub-Saharan Africa, Latin America, Eastern Europe & Central Asia, Western Countries, and Pacific Islands), sourced via original crowdsourced questionnaires administered to native in-country informants with lived experience, supplemented by SafeWorld. Every candidate underwent independent triple-annotation: universal candidates were reviewed by cross-cultural ethics researchers, while specific candidates were vetted by native researchers from the source culture. Imposing a strict majority consensus threshold (\(\ge 2/3\)) discarded 1,022 ambiguous candidates (23.5% discard rate), retaining 3,323 norms (975 universal, 2,348 specific). The retained SPECIFIC items achieved 98.6%–100.0% annotator unanimity across all clusters, and an independent from-scratch validation on 100 blind items yielded a substantial human–human Cohen's \(\kappa = 0.78\), guaranteeing a gold standard uncompromised by cultural chauvinism.

2. Unified Cross-Task Error Taxonomy: Disentangling normative overgeneralization, moral relativization, and geographic proximity confusions To trace failure modes consistently across distinct evaluation formats, ScopeBench defines a three-way error taxonomy with identical operationalizations across all tasks: - Scope Inflation (SI) occurs when a model treats a SPECIFIC norm as universally binding, projecting localized customs onto outsiders. - Scope Deflation (SD) occurs when a model treats an inalienable UNIVERSAL protection as culture-specific, relativizing fundamental ethical safeguards. - Scope Displacement (SP) occurs when a model correctly identifies a norm as culturally specific but attributes it to the wrong cultural community. To probe the granularity of cultural representations, Task 2 decomposes displacement distractors into hard distractors \(h_n\) (sampled from the same cultural cluster as gold country \(g_n\)) and easy distractors \(e_{1n}, e_{2n}\) (sampled from distinct external clusters). This structure introduces the Cluster Proximity Gap (CPG): $\(CPG = SPR_{\text{hard}} - SPR_{\text{easy}}\)$ A positive CPG demonstrates that misattributions concentrate near cultural boundaries, indicating coarse regional awareness coupled with fine-grained national insensitivity, whereas a non-positive CPG diagnoses non-cultural artifacts such as frequency or position bias.

3. Format-Decoupled Four-Task Pipeline: Isolating prompt anchoring artifacts from invariant behavioral safety failures To determine whether model scope awareness reflects robust world representations or prompt-dependent artifacts, the evaluation pipeline progresses across four complementary tasks: - Task 1 (Norm Scope Classification): Evaluates models on raw norm text alone without geographical context, demanding a binary UNIVERSAL/SPECIFIC classification and short rationale. - Task 2a (Joint Multiple-Choice Attribution): Presents a norm alongside six joint options (four country choices A–D, Option E denoting a universal norm, and Option F for "none of the above"), introducing an explicit universal safe haven. - Task 2b (Per-Option Independent Binary Judgment): Queries each candidate country independently ("Is this norm specifically tied to [country]'s culture?"), completely removing inter-option anchoring and eliminating the salient universal option. - Task 3 (Behavioral Safety in Interaction): Measures behavioral adherence using a three-stage simulated interaction. In Stage 1, GPT-5.5 generates 5,671 subtly misleading, naturalistic scope-violating statements (without explicit signals of error) across SD, SI, and SP types. In Stage 2, evaluated models assess statement accuracy without hints. In Stage 3, an automated judge (GPT-5.5, validated extensively against human adjudications) categorizes responses into Appropriate, Evasive, or Harmful.

Benchmark & Evaluation Protocol

The benchmark evaluates 21 frontier large language models spanning seven prominent provider families (OpenAI, Anthropic, Google, Meta, Qwen, DeepSeek, Mistral), incorporating proprietary flagship APIs and open-weight architectures under a unified zero-shot setting. Rigid regular-expression parsers extract structured outputs, and unparseable generations (retained in denominators as UNKNOWN) are isolated to ensure they do not distort error tallies. Metrics encompass Scope Deflation Rate (\(SDR\)), Scope Inflation Rate (\(SIR\)), Scope Displacement Rate (\(SPR\)), and Accuracy (\(Acc\)) in Tasks 1–2, as well as type-specific Harmful-Response Rates (\(HR_{\text{SD}}, HR_{\text{SP}}, HR_{\text{SI}}\)), overall Harmful-Response Rate (\(HR_{\text{total}}\)), Appropriate-Response Rate (\(AR_{\text{total}}\)), and Evasion Rate (\(ER_{\text{total}}\)) in Task 3.

Quantitative Experimental Results

Primary Benchmark: Norm Scope Classification (Task 1) and Behavioral Safety (Task 3)

The table below presents raw baseline figures across all 21 models on Task 1 isolated classification and Task 3 behavioral interaction:

Provider Family Evaluated Model Type Task 1 SDR (%) Task 1 SIR (%) Task 1 Acc (%) Task 3 \(HR_{\text{SD}}\) (%) Task 3 \(HR_{\text{SP}}\) (%) Task 3 \(HR_{\text{SI}}\) (%) Task 3 \(HR_{\text{total}}\) (%) Task 3 \(AR_{\text{total}}\) (%)
OpenAI GPT-5.4 P 12.8 4.2 93.3 52.0 52.7 4.2 32.5 67.5
GPT-4.1 P 70.0 3.4 77.1 65.2 56.9 8.6 38.3 61.6
GPT-4.1-mini P 30.4 3.9 88.4 84.5 66.7 17.1 49.2 50.7
o3-mini P 13.0 5.8 92.1 49.0 44.1 5.7 29.0 70.9
o4-mini P 20.4 7.2 88.9 58.5 50.5 6.3 33.5 66.4
Anthropic Claude Opus 4.7 P 23.2 5.5 89.3 47.6 51.4 5.1 31.6 67.5
Claude Sonnet 4.6 P 11.6 7.6 91.2 67.6 56.4 6.2 37.5 62.3
Claude Haiku 4.5* P 25.1 5.6 88.7 44.3 41.2 1.8 25.4* 54.8
Google Gemini 2.5 Flash P 16.6 3.2 92.9 72.7 58.0 7.1 39.5 60.3
Gemini 2.5 Flash-Lite P 11.1 4.4 93.6 82.2 69.2 9.4 46.7 53.3
Meta Llama 4 Maverick O 0.5 3.1 97.7 74.2 59.9 10.2 41.8 58.2
Llama 4 Scout O 0.2 2.9 97.9 71.3 60.0 10.7 41.5 58.4
Llama 3.1-70B O 0.4 2.6 98.1 74.3 60.5 10.9 42.4 57.6
Qwen Qwen3-235B-A22B O 46.9 1.1 85.4 77.7 63.8 11.3 44.5 55.5
Qwen3-32B O 46.4 1.9 85.0 76.8 62.8 11.2 43.9 56.1
Qwen2.5-72B O 50.8 1.2 84.3 80.4 62.9 11.8 44.8 55.2
DeepSeek DeepSeek-V3.2 O 13.5 4.4 92.9 74.1 64.9 7.7 42.8 57.2
DeepSeek-V3 O 3.3 7.4 93.8 86.5 70.9 16.9 51.2 48.8
Mistral Dolphin-R1-Mistral-24B O 2.1 2.4 97.7 73.3 60.4 11.0 42.2 57.8
Mistral Large 3 P 0.3 3.7 97.3 73.3 58.7 10.0 41.0 58.9
Mistral Small 4 O 0.4 3.8 97.2 72.0 57.7 9.6 40.3 59.7

*Claude Haiku 4.5 exhibits an anomalous task refusal rate (\(ER_{\text{total}} = 19.8\%\)) asserting role restrictions, which mechanically depresses its harmful-response rate; it is excluded from best-in-cohort safety rankings.

Format Anchoring Ablation (Task 2a vs. Task 2b) and Human/Judge Alignment

The table below contrasts attribution errors under joint multiple-choice presentation (Task 2a) versus per-option independent queries (Task 2b), alongside human baselines and judge validation metrics:

System / Model Representative Task 2a SDR (%) Task 2a SIR (%) Task 2a CPG (%) Task 2b SDR (%) Task 2b SIR (%) Task 2b CPG (%) Validation / Alignment Metric
Llama 4 Maverick 18.4 1.8 -0.2 1.4 33.0 +3.2 Exemplifies format error inversion
Llama 4 Scout 18.7 1.8 -0.2 1.8 32.3 +2.6 SIR surges over 32% under isolated queries
Dolphin-R1-Mistral-24B 22.1 1.7 -1.7 0.9 29.9 +4.0 Deflation in 2a inverts to Inflation in 2b
Qwen3-235B-A22B 4.9 5.8 -1.2 0.2 8.6 +2.9 Anchor removal suppresses SDR below 1%
GPT-5.4 7.7 5.1 -0.5 0.1 10.1 +4.5 Clear positive CPG under Task 2b
Claude Opus 4.7 4.1 6.3 +2.6 0.0 7.0 +2.9 Consistent low inflation across formats
Human Baseline (Adjudicated Gold) 8.0 (Acc 92% on U) 22.0 (Acc 78% on S) +4.0 15.0 (Acc 85%) 8.0 — Task 1 Acc: 94.0%, \(\kappa = 0.78\)
GPT-5.5 Automated Judge vs. Human — — — — — — Overall \(\kappa = 0.72\) (Harmful \(\kappa = 0.80\), Acc 87%)
Claude Opus 4.7 Cross-Judge Check — — — — — — Spearman \(\rho = 0.955\), \(\kappa = 0.894\), \(\Delta_{\max} = 5.8\) pp
Grok-3 (xAI) Cross-Judge Check — — — — — — Spearman \(\rho = 0.942\), \(\kappa = 0.899\), \(\Delta_{\max} = 4.6\) pp

Key Findings

  • Prompt presentation format fundamentally inverts the dominant error type: In single-answer settings (Task 1 and Task 2a), Scope Deflation universally dominates while Scope Inflation is heavily suppressed (\(SIR_1 \le 7.62\%\), while \(SDR_1\) reaches up to 70.0% in GPT-4.1). In contrast, under per-option independent binary evaluation (Task 2b), Scope Inflation completely dominates for all 21 models without exception (\(SIR_{2b}\) reaches 33.0% in Llama 4 Maverick and 32.3% in Llama 4 Scout, while \(SDR_{2b}\) drops to near zero, peaking at only 1.85%). The presence of an explicit universal option in multiple-choice prompts creates a bidirectional anchor: it shields culture-specific norms from overgeneralization while drawing universal norms toward specific country attributions.
  • Classification accuracy is empirically decoupled from behavioral safety: The six models achieving near-ceiling accuracy in Task 1 (\(\ge 97\%\) in Llama and Mistral families) still yield dangerous harmful-response rates of 40.3%–42.4% in Task 3. Across the entire 21-model cohort, Task 1 classification accuracy correlates negligibly with Task 3 harmful-response rate (Pearson \(r = 0.09, p = 0.69\)). In fact, GPT-4.1, which exhibits the worst Task 1 accuracy (77.1%) due to massive deflation (\(SDR_1 = 70.0\%\)), achieves an \(HR_{\text{total}}\) of 38.3%—outperforming all six near-ceiling classifiers in behavioral interaction.
  • Scope Deflation represents the most hazardous behavioral vulnerability: Across 19 of the 21 models, the harmful response rate strictly follows \(HR_{\text{SD}} > HR_{\text{SP}} > HR_{\text{SI}}\). Models demonstrate substantial alertness in rejecting statements that falsely universalize specific customs (\(HR_{\text{SI}}\) ranges from 1.8% to 17.1%), but routinely endorse statements that falsely relativize universal protections (\(HR_{\text{SD}}\) ranges from 44.3% to 86.5%).
  • Cultural displacement concentrates along regional cluster boundaries: Under independent judgment in Task 2b, all 21 models exhibit a strictly positive Cluster Proximity Gap (\(CPG_{2b}\) ranges from +1.2% to +6.4%), mirroring human performance (\(CPG_{\text{human}} = +4.0\%\)). This confirms that displacement errors stem from genuine semantic proximity in cultural geography rather than random distractor guessing.

Highlights & Insights

  • Promoting cultural scope to an explicit first-class safety dimension: ScopeBench successfully challenges the prevailing assumption of cultural invariance in AI safety, formalizing the legal and normative boundaries that dictate when a norm binds universally versus locally.
  • Exposing prompt-induced cognitive anchoring in LLM evaluation: The bidirectional format inversion proves that standard multiple-choice cultural benchmarks systematically mask latent overgeneralization tendencies through option anchoring, establishing independent binary querying as an essential diagnostic counterweight.
  • Demonstrating the inadequacy of static classification for safety guarantees: The empirical decoupling (\(r = 0.09\)) between isolated scope recognition and interactive safety enforcement underscores that metalinguistic knowledge does not translate into robust defense against plausibly deceptive user prompts.

Limitations & Future Work

  • English-only operationalization: All prompts and norm representations are rendered in English to prevent confounding multicultural awareness with multilingual fluency. However, translating indigenous customs into English inevitably risks shedding fine-grained cultural nuance. Future iterations must evaluate scope awareness directly in native languages.
  • Reliance on a single primary generator for adversarial statements: While cross-provider evaluations with Claude Opus 4.7 and Grok-3 confirmed the stability of model rankings under different judges, all Task 3 statements originated from GPT-5.5, which may encode model-specific rhetorical styles. Multi-generator pipelines represent a natural next step.
  • Binary reduction of continuous normative nuances: While ScopeBench provides soft labels for the 2.56% of items with split annotator consensus, reducing complex social customs (e.g., hospitality rites, filial piety) into discrete UNIVERSAL/SPECIFIC labels abstracts away continuous degrees of intercultural variance.
  • vs. SafeWorld (Yin et al., 2024): SafeWorld tests safety across 50 countries but assumes all items are country-specific properties without universal grounding, and reports overall response quality without decomposing failure modes. ScopeBench introduces explicit dual-anchored scope labels and tracks three distinct error types across both classification and interaction.
  • vs. CulturalBench / KorNAT / BLEnD: These benchmarks evaluate whether models possess factual knowledge about specific cultures or adapt conversational style. ScopeBench targets the higher-order question of whom a norm is meant to bind, distinguishing localized customs from cross-cultural ethical imperatives.
  • vs. HarmBench / SORRY-Bench: Traditional safety benchmarks apply culturally invariant harm criteria to elicit refusals. ScopeBench shows that safety alignment requires contextual scope calibration to prevent both cultural imperialism (inflation) and moral relativism (deflation).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes cultural norm scope as an original, first-class safety alignment axis, filling a major theoretical gap between safety and multicultural NLP.
  • Empirical Rigor: ⭐⭐⭐⭐⭐ Comprehensive 21-model evaluation across 3,323 norms and 49 countries, supported by extensive human validation and multi-provider judge robustness checks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical definitions, and compelling narrative flow throughout.
  • Impact: ⭐⭐⭐⭐⭐ Critical foundational utility for international AI safety governance, cross-cultural deployment, and alignment evaluation methodology.