Not What, But How: A Communicative Audit of LLM Response Framing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/siddheshih/FRANZ
Area: LLM Evaluation / Communicative Framing & Pragmatics
Keywords: Response Framing Audit, Cultural Positioning Bias, Anthropomorphic Cues, Generalizing Language, Gricean Maxims
TL;DR¶
Addressing the limitation of conventional cultural benchmarks that focus exclusively on factual correctness while ignoring framing style, this paper proposes FRANZ, a communicative audit framework spanning four sociolinguistic dimensionsโcultural positioning, generalizing language, anthropomorphic cues, and Gricean maximsโand demonstrates across SQUARE (a 376k-query corpus from 57 subreddits across 7 countries) that insider positioning and affective anthropomorphism are tightly coupled with pronounced cross-model and cross-national divergence.
Background & Motivation¶
As large language models (LLMs) are increasingly deployed to answer subjective, open-ended queries, user perception and behavioral decisions are critically shaped not merely by factual accuracy, but by how responses are framed and communicated. However, existing cultural alignment benchmarksโsuch as BLEnd, CultureBank, and ComPOโpredominantly rely on multiple-choice questions or single-ground-truth factual recall, flattening rich cultural interactions into mere common-sense tests. This "what" over "how" evaluation paradigm completely overlooks the communicative stances and pragmatic styles adopted by models during subjective intercultural exchanges.
In subjective inquiry, Cultural Positioning Bias (CPB) represents a particularly insidious challenge: models often naturally adopt an empathetic "insider" stance toward dominant cultural groups while defaulting to an observational, detached "outsider" stance for others. Worse still, in the absence of an insider perspective, models tend to compensate by deploying broad-stroke generalizing language, inadvertently perpetuating stereotypes. Furthermore, cultural positioning is not an isolated phenomenon; it interacts intimately with anthropomorphic cues used to manufacture rapport, as well as with baseline conversational cooperativeness.
Prior studies have investigated cultural positioning, linguistic stereotyping, anthropomorphism, and Gricean conversational maxims in complete isolation, lacking a unified sociolinguistic audit framework and large-scale, culturally situated subjective datasets. Core idea: shift evaluation from what models encode to how responses are framed by introducing FRANZ, a four-dimensional communicative audit framework spanning cultural positioning, generalizing language, anthropomorphism, and Gricean maxims, evaluated across 376k subjective queries in SQUARE via Generalized Estimating Equations.
Method¶
Overall Architecture¶
The FRANZ evaluation and analysis pipeline operates in two primary phases: large-scale culturally grounded subjective corpus curation (SQUARE) and multi-judge pragmatic feature annotation paired with Generalized Estimating Equations (GEE) statistical modeling.
The input comprises 376,350 subjective queries sourced from 57 country-level subreddits across 7 countries, mapped to 19 culturally grounded categories. Three open-weight instruction-tuned LLMs (Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-14B-Instruct) generate responses in a zero-shot, un-steered setting. FRANZ then deploys four specialized LLM-as-a-judge evaluators to independently assess cultural positioning stance, the presence of generalizing language, 9 distinct anthropomorphic cues, and adherence to 4 Gricean conversational maxims. Finally, population-level GEE regression models untangle individual model framing tendencies and feature co-occurrence probabilities.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["SQUARE Cross-National Subjective Queries<br/>57 Subreddits / 7 Countries / 19 Categories"] --> B["Direct LLM Response Generation<br/>Llama-3.1 / Gemma-3 / Ministral-3"]
B --> C["SQUARE Subjective Query Corpus Construction"]
C --> D["FRANZ Four-Dimensional Multi-Judge Response Audit"]
D --> E["Generalized Estimating Equations Statistical Modeling"]
E --> F["Output: Individual Pragmatic Signatures & Framing Biases"]
Key Designs¶
1. SQUARE Subjective Query Corpus Construction: Grounding Cultural Contexts and Nuanced Semantics from Natural Discourse
Standard cultural benchmarks rely on synthetic or academic multiple-choice questions that fail to mirror authentic user inquiry across diverse cultural contexts. To overcome this limitation, the authors leveraged the Pushshift Reddit archive, starting from 28 seed subreddits matching r/Ask<Country> or r/AskA<Nationality> and manually expanding to 408 subreddits. After filtering out NSFW subreddits, multimodal boards, news feeds, and inactive communities, 57 subreddits across 7 countries (India, China, Philippines, USA, Korea, Russia, Turkey) were retained, yielding 376,350 cleaned posts. To address the coarse granularity of prior taxonomies (e.g., broad catch-all labels like "Governance and Society"), the authors synthesized categories from CLIcK, CaLMQA, and Thompson et al.'s typology into 19 culturally grounded categories (such as Kinship, Clothing & Grooming, Food & Drinks, Speech & Language). Zero-shot categorization using Llama-3.1-8B-it with majority voting over 3 runs achieved 87.6%โ90.8% human-expert agreement (\(\kappa=0.59\)), establishing a robust foundation for controlled cross-model comparisons.
2. FRANZ Four-Dimensional Multi-Judge Response Audit: Grounded Pragmatic Decoupling Across Sociolinguistic Facets
To move beyond surface sentiment analysis and lexicons, FRANZ operationalizes four distinct yet interrelated communicative dimensions, each evaluated by an independent LLM judge with dedicated rubrics: - Cultural Positioning: Grounded in anthropological emics vs. etics, this judge classifies whether a response adopts an insider stance (communicating from within the referenced community using first-person pronouns "I/we/us", shared cultural norms, and tacit understandings) or an outsider stance (observational, third-person "they/them" framing). - Generalizing Language: Building on Beukeboom's linguistic stereotyping framework, this judge assesses whether the text makes overarching group-level claims. It evaluates group identity terms, grammatical structures (bare plurals, quantified adverbs, indefinite singulars), and the attribution of enduring collective characteristics. It is scored by Llama-3.1-70B-Instruct to handle complex syntactic nuances. - Anthropomorphic Cues: Drawing from Ibrahim et al.'s taxonomy, physical embodiment cues (unverifiable in text) are excluded, leaving 9 rigorously defined cues: personal relationships, personal history, explicit relationship status, desires, agency, emotions, validation, empathy, and relatability. - Gricean Maxims: Decomposes cooperative dialogue into Quality (well-supported claims without ungrounded speculation), Quantity (adequate information without omissions or verbosity), Relation (strict topical relevance without drift), and Manner (concise, orderly, unambiguous presentation).
3. Generalized Estimating Equations Statistical Modeling: Controlling Clustering and Isolating Causal Coupling
Because each Reddit post receives responses from all three evaluated LLMs, treating post-response pairs as independent observations would violate fundamental regression assumptions. FRANZ employs Generalized Estimating Equations (GEE) grouping observations by Post ID into clusters of size 3, using a Binomial family with logit link functions. For any cause-effect pair of characteristics \((C, E)\), the marginal probability shift is formulated as:
where \(\Delta\) quantifies how the presence of characteristic \(C\) shifts the likelihood of characteristic \(E\). Wald tests evaluate whether individual model tendencies exceed chance, and false discovery rate (FDR) corrections establish relative cross-model rankings within top-3 categories per country, rigorously distinguishing genuine causal coupling from independent co-occurrence.
Loss & Training¶
The evaluated models represent modern instruction-tuned open-weight architectures (Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-14B-Instruct-2512), hosted on vLLM using bfloat16 precision. Un-steered generation used standard decoding parameters: \(\text{temperature}=0.7\), \(\text{top\_p}=0.95\), and \(\text{max\_tokens}=512\). Across 376k queries, the generation stage yielded \(\sim\)1.1M responses, and the auditing stage executed \(\sim\)16M judge invocations, consuming roughly 7 days of compute across 8 NVIDIA A100 GPUs.
Key Experimental Results¶
Main Results¶
Human validation across 150 stratified sample pairs per task confirmed strong alignment between human experts (A1, A2) and the automated LLM judges. Individual model tendencies were subsequently evaluated across SQUARE's top country-category subsets.
Table 1: Human Validation of FRANZ LLM-as-a-Judge Across Dimensions (150 Samples per Characteristic)
| Response Characteristic | Sub-facet | A1-LLM Accuracy | A2-LLM Accuracy | Inter-Annotator Agreement (A1-A2 PA) |
|---|---|---|---|---|
| Cultural Positioning | Insider | 0.75 | 1.00 | 0.86 |
| Outsider | 0.99 | 0.96 | 0.97 | |
| Generalizing Language | Generalizing | 0.71 | 0.71 | 1.00 |
| Non-Generalizing | 1.00 | 0.98 | 0.99 | |
| Anthropomorphism | Personal History | 1.00 | 1.00 | 1.00 |
| Emotions | 0.98 | 0.98 | 1.00 | |
| Empathy | 0.94 | 0.95 | 0.99 | |
| Validation | 0.83 | 0.87 | 0.98 | |
| Conversational Maxims | Quality | 0.94 | 0.92 | 0.95 |
| Quantity | 0.71 | 0.82 | 0.90 | |
| Manner | 0.71 | 0.84 | 0.89 | |
| Relation | 0.51 | 0.73 | 0.80 |
Table 2: Relative Model Rankings by Characteristic Prevalence across Top Country-Category Subsets (FDR-corrected, \(p < 0.05\))
| Country (Total Queries) | Top Category (Count) | Insider Positioning | Generalizing Language | Emotions | Validation | Empathy |
|---|---|---|---|---|---|---|
| India (194,986) | Social Relations (45,715) | M > L > G | M > G > L | M > G > L | M > G > L | M > G \(\sim\) L |
| Education & Career (34,629) | M > G > L | M > G > L | G > M > L | G > M > L | G > L > M | |
| Health & Wellness (24,961) | M > G \(\sim\) L | M > G > L | M > G > L | G > M > L | G > L > M | |
| China (48,585) | Speech & Language (18,140) | M > G \(\sim\) L | M > G > L | G > M > L | G > M > L | G > M > L |
| Education & Career (8,750) | M > G > L | M > G > L | M > G > L | G > M > L | G > M \(\sim\) L | |
| Social Relations (5,383) | M > L \(\sim\) G | M > G > L | M > G > L | G > M > L | M > G > L | |
| Philippines (38,478) | Social Relations (15,172) | M > L > G | M > G > L | M > G > L | G > M > L | G > M > L |
| USA (36,915) | Social Relations (16,134) | M > L > G | M > G > L | M > G > L | G > M > L | G > M > L |
| Korea (27,464) | Food & Drink (7,287) | M > G > L | L \(\sim\) M > G | M > G > L | G > M > L | M > G > L |
| Russia (20,738) | Social Relations (7,807) | M > L > G | M > G > L | M > G > L | G > M > L | G \(\sim\) L \(\sim\) M |
| Turkey (9,183) | Social Relations (4,097) | M > L > G | M > G > L | M > G \(\sim\) L | G > M > L | M > G > L |
(Note: L = Llama-3.1-8B-it, G = Gemma-3-12b-it, M = Ministral-3-14B-it; > indicates statistically significant difference; \(\sim\) indicates no significant difference)
Ablation & Interaction Analysis¶
The investigation assessed maxim violation rates across countries as well as the bidirectional coupling between communicative features.
Table 3: Percentage of Conversational Maxim Violations across Evaluated LLMs (%)
| Maxim | Country | Llama-3.1-8B Violation (%) | Gemma-3-12b Violation (%) | Ministral-3-14B Violation (%) | Dominant Model Pattern |
|---|---|---|---|---|---|
| Quality | India | 16.8 | 12.3 | 22.7 | Gemma violates least (G > L > M adherence) |
| China | 11.2 | 7.1 | 10.8 | Consistently high quality across models | |
| USA | 12.6 | 11.5 | 15.0 | Gemma achieves lowest violation rate | |
| Quantity | India | 93.8 | 88.9 | 93.0 | Widespread verbosity and over-answering |
| China | 90.8 | 86.6 | 89.2 | Severe failure to satisfy conciseness | |
| USA | 96.1 | 91.5 | 95.1 | Over 90% violation due to length padding | |
| Relation | India | 80.3 | 70.9 | 73.0 | Gemma adheres best to core topic |
| China | 76.3 | 63.9 | 66.1 | Frequent topical expansion / drift | |
| USA | 87.1 | 78.9 | 82.0 | Gemma maintains highest topical relevance | |
| Manner | India | 58.8 | 54.6 | 61.5 | Gemma is most readable and orderly |
| China | 49.3 | 46.0 | 48.6 | Moderate manner violations across models | |
| USA | 69.0 | 58.3 | 70.4 | Gemma exhibits lowest manner violation |
Key Findings¶
- Distinct Model-Level Pragmatic Signatures: Across independent features, Ministral exhibits the strongest inclination toward insider positioning (+11% over Llama and Gemma) while also employing the highest rate of generalizing language (\(M > G > L\) across nearly all categories). Gemma consistently leads in conversational maxim adherence and validating cues. In contrast, Llama appears most constrained, exhibiting lower rates across positioning, generalizations, and affective anthropomorphism.
- Bidirectional Coupling between Positioning and Anthropomorphism: When models adopt an insider stance, the probability of incorporating affective cues (emotions, validation, empathy) increases significantly. Conversely, the presence of anthropomorphic cues heavily inflates the likelihood that a response is judged as insider-positioned. Crucially, effect sizes for Anthropomorphism \(\to\) Positioning (\(\Delta \approx 0.16 \sim 0.30\)) are substantially larger than Positioning \(\to\) Anthropomorphism (\(\Delta \approx 0.02 \sim 0.05\)).
- Orthogonality of Conversational Maxims and Cultural Framing: Adherence to Gricean maxims is largely independent of cultural positioning and generalizing language. A response can be highly cooperative, clear, and structured while remaining culturally detached or relying on blatant cultural generalizations, highlighting the inadequacy of standard instruction-tuning for cultural debiasing.
Highlights & Insights¶
- Shifting the Paradigm from What to How: Moves beyond multiple-choice cultural knowledge retrieval, providing the first systematic quantification of how models frame open-ended subjective responses across cultural communities.
- Unveiling the "Camouflage" Risk of Anthropomorphism: Demonstrates that anthropomorphic warmers (e.g., "I understand how you feel") can artificially inflate perceived cultural insiderness, misleading both automated evaluators and human users into assuming genuine cultural depth.
- Curating High-Confidence Agreement Subsets: Releases filtered subsets where all three models receive unanimous FRANZ judgments, offering reliable diagnostic probes for downstream multi-objective alignment and stereotyping audits.
Limitations & Future Work¶
- Reddit Community Proxy Skew: Sourcing data exclusively from national subreddits captures an online, English-proficient, and demographically skewed subpopulation, which cannot represent the entire cultural spectrum of each nation.
- Surface Feature Confounding in LLM Judges: Both insider positioning and affective anthropomorphism rely heavily on first-person pronouns ("I", "we"), meaning judge models may partially conflate surface grammatical cues with deep communicative stance despite explicit prompt controls.
- Western Pragmatic Bias in Gricean Maxims: The Gricean criteria of directness and conciseness reflect Western communicative norms; indirectness, circumlocution, and conversational hedging that are valued in high-context cultures are systematically penalized as violations of Manner or Quantity.
Related Work & Insights¶
- vs. BLEnd / CultureBank / CaLMQA: Existing benchmarks evaluate factual retrieval against fixed ground truths; FRANZ shifts focus to the stylistic, rhetorical, and pragmatic framing of subjective interactions.
- vs. InsideOut (Wan et al., 2026): While prior work examined cultural positioning bias in isolation within scripted interviews, FRANZ integrates positioning with linguistic generalization, anthropomorphic cues, and pragmatic maxims on organic conversational discourse.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering four-dimensional sociolinguistic audit framework investigating how LLMs frame subjective cultural responses.
- Experimental Thoroughness: โญโญโญโญโญ 376k authentic queries, 57 subreddits, 3 open-weight models, 16M judge evaluations, and rigorous GEE modeling.
- Writing Quality: โญโญโญโญโญ Cohesive theoretical motivation, lucid structural exposition, and exemplary statistical discipline.
- Value: โญโญโญโญโญ Highly actionable diagnostic tool for evaluating cultural alignment, stereotyping risks, and multi-objective alignment trade-offs in LLMs.