Skip to content

Not What, But How: A Communicative Audit of LLM Response Framing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/siddheshih/FRANZ
Area: LLM Evaluation / Communicative Framing & Pragmatics
Keywords: Response Framing Audit, Cultural Positioning Bias, Anthropomorphic Cues, Generalizing Language, Gricean Maxims

TL;DR

Addressing the limitation of conventional cultural benchmarks that focus exclusively on factual correctness while ignoring framing style, this paper proposes FRANZ, a communicative audit framework spanning four sociolinguistic dimensionsโ€”cultural positioning, generalizing language, anthropomorphic cues, and Gricean maximsโ€”and demonstrates across SQUARE (a 376k-query corpus from 57 subreddits across 7 countries) that insider positioning and affective anthropomorphism are tightly coupled with pronounced cross-model and cross-national divergence.

Background & Motivation

As large language models (LLMs) are increasingly deployed to answer subjective, open-ended queries, user perception and behavioral decisions are critically shaped not merely by factual accuracy, but by how responses are framed and communicated. However, existing cultural alignment benchmarksโ€”such as BLEnd, CultureBank, and ComPOโ€”predominantly rely on multiple-choice questions or single-ground-truth factual recall, flattening rich cultural interactions into mere common-sense tests. This "what" over "how" evaluation paradigm completely overlooks the communicative stances and pragmatic styles adopted by models during subjective intercultural exchanges.

In subjective inquiry, Cultural Positioning Bias (CPB) represents a particularly insidious challenge: models often naturally adopt an empathetic "insider" stance toward dominant cultural groups while defaulting to an observational, detached "outsider" stance for others. Worse still, in the absence of an insider perspective, models tend to compensate by deploying broad-stroke generalizing language, inadvertently perpetuating stereotypes. Furthermore, cultural positioning is not an isolated phenomenon; it interacts intimately with anthropomorphic cues used to manufacture rapport, as well as with baseline conversational cooperativeness.

Prior studies have investigated cultural positioning, linguistic stereotyping, anthropomorphism, and Gricean conversational maxims in complete isolation, lacking a unified sociolinguistic audit framework and large-scale, culturally situated subjective datasets. Core idea: shift evaluation from what models encode to how responses are framed by introducing FRANZ, a four-dimensional communicative audit framework spanning cultural positioning, generalizing language, anthropomorphism, and Gricean maxims, evaluated across 376k subjective queries in SQUARE via Generalized Estimating Equations.

Method

Overall Architecture

The FRANZ evaluation and analysis pipeline operates in two primary phases: large-scale culturally grounded subjective corpus curation (SQUARE) and multi-judge pragmatic feature annotation paired with Generalized Estimating Equations (GEE) statistical modeling.

The input comprises 376,350 subjective queries sourced from 57 country-level subreddits across 7 countries, mapped to 19 culturally grounded categories. Three open-weight instruction-tuned LLMs (Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-14B-Instruct) generate responses in a zero-shot, un-steered setting. FRANZ then deploys four specialized LLM-as-a-judge evaluators to independently assess cultural positioning stance, the presence of generalizing language, 9 distinct anthropomorphic cues, and adherence to 4 Gricean conversational maxims. Finally, population-level GEE regression models untangle individual model framing tendencies and feature co-occurrence probabilities.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["SQUARE Cross-National Subjective Queries<br/>57 Subreddits / 7 Countries / 19 Categories"] --> B["Direct LLM Response Generation<br/>Llama-3.1 / Gemma-3 / Ministral-3"]
    B --> C["SQUARE Subjective Query Corpus Construction"]
    C --> D["FRANZ Four-Dimensional Multi-Judge Response Audit"]
    D --> E["Generalized Estimating Equations Statistical Modeling"]
    E --> F["Output: Individual Pragmatic Signatures & Framing Biases"]

Key Designs

1. SQUARE Subjective Query Corpus Construction: Grounding Cultural Contexts and Nuanced Semantics from Natural Discourse

Standard cultural benchmarks rely on synthetic or academic multiple-choice questions that fail to mirror authentic user inquiry across diverse cultural contexts. To overcome this limitation, the authors leveraged the Pushshift Reddit archive, starting from 28 seed subreddits matching r/Ask<Country> or r/AskA<Nationality> and manually expanding to 408 subreddits. After filtering out NSFW subreddits, multimodal boards, news feeds, and inactive communities, 57 subreddits across 7 countries (India, China, Philippines, USA, Korea, Russia, Turkey) were retained, yielding 376,350 cleaned posts. To address the coarse granularity of prior taxonomies (e.g., broad catch-all labels like "Governance and Society"), the authors synthesized categories from CLIcK, CaLMQA, and Thompson et al.'s typology into 19 culturally grounded categories (such as Kinship, Clothing & Grooming, Food & Drinks, Speech & Language). Zero-shot categorization using Llama-3.1-8B-it with majority voting over 3 runs achieved 87.6%โ€“90.8% human-expert agreement (\(\kappa=0.59\)), establishing a robust foundation for controlled cross-model comparisons.

2. FRANZ Four-Dimensional Multi-Judge Response Audit: Grounded Pragmatic Decoupling Across Sociolinguistic Facets

To move beyond surface sentiment analysis and lexicons, FRANZ operationalizes four distinct yet interrelated communicative dimensions, each evaluated by an independent LLM judge with dedicated rubrics: - Cultural Positioning: Grounded in anthropological emics vs. etics, this judge classifies whether a response adopts an insider stance (communicating from within the referenced community using first-person pronouns "I/we/us", shared cultural norms, and tacit understandings) or an outsider stance (observational, third-person "they/them" framing). - Generalizing Language: Building on Beukeboom's linguistic stereotyping framework, this judge assesses whether the text makes overarching group-level claims. It evaluates group identity terms, grammatical structures (bare plurals, quantified adverbs, indefinite singulars), and the attribution of enduring collective characteristics. It is scored by Llama-3.1-70B-Instruct to handle complex syntactic nuances. - Anthropomorphic Cues: Drawing from Ibrahim et al.'s taxonomy, physical embodiment cues (unverifiable in text) are excluded, leaving 9 rigorously defined cues: personal relationships, personal history, explicit relationship status, desires, agency, emotions, validation, empathy, and relatability. - Gricean Maxims: Decomposes cooperative dialogue into Quality (well-supported claims without ungrounded speculation), Quantity (adequate information without omissions or verbosity), Relation (strict topical relevance without drift), and Manner (concise, orderly, unambiguous presentation).

3. Generalized Estimating Equations Statistical Modeling: Controlling Clustering and Isolating Causal Coupling

Because each Reddit post receives responses from all three evaluated LLMs, treating post-response pairs as independent observations would violate fundamental regression assumptions. FRANZ employs Generalized Estimating Equations (GEE) grouping observations by Post ID into clusters of size 3, using a Binomial family with logit link functions. For any cause-effect pair of characteristics \((C, E)\), the marginal probability shift is formulated as:

\[\Delta = P(E=1 \mid C=1) - P(E=1 \mid C=0)\]

where \(\Delta\) quantifies how the presence of characteristic \(C\) shifts the likelihood of characteristic \(E\). Wald tests evaluate whether individual model tendencies exceed chance, and false discovery rate (FDR) corrections establish relative cross-model rankings within top-3 categories per country, rigorously distinguishing genuine causal coupling from independent co-occurrence.

Loss & Training

The evaluated models represent modern instruction-tuned open-weight architectures (Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-14B-Instruct-2512), hosted on vLLM using bfloat16 precision. Un-steered generation used standard decoding parameters: \(\text{temperature}=0.7\), \(\text{top\_p}=0.95\), and \(\text{max\_tokens}=512\). Across 376k queries, the generation stage yielded \(\sim\)1.1M responses, and the auditing stage executed \(\sim\)16M judge invocations, consuming roughly 7 days of compute across 8 NVIDIA A100 GPUs.

Key Experimental Results

Main Results

Human validation across 150 stratified sample pairs per task confirmed strong alignment between human experts (A1, A2) and the automated LLM judges. Individual model tendencies were subsequently evaluated across SQUARE's top country-category subsets.

Table 1: Human Validation of FRANZ LLM-as-a-Judge Across Dimensions (150 Samples per Characteristic)

Response Characteristic Sub-facet A1-LLM Accuracy A2-LLM Accuracy Inter-Annotator Agreement (A1-A2 PA)
Cultural Positioning Insider 0.75 1.00 0.86
Outsider 0.99 0.96 0.97
Generalizing Language Generalizing 0.71 0.71 1.00
Non-Generalizing 1.00 0.98 0.99
Anthropomorphism Personal History 1.00 1.00 1.00
Emotions 0.98 0.98 1.00
Empathy 0.94 0.95 0.99
Validation 0.83 0.87 0.98
Conversational Maxims Quality 0.94 0.92 0.95
Quantity 0.71 0.82 0.90
Manner 0.71 0.84 0.89
Relation 0.51 0.73 0.80

Table 2: Relative Model Rankings by Characteristic Prevalence across Top Country-Category Subsets (FDR-corrected, \(p < 0.05\))

Country (Total Queries) Top Category (Count) Insider Positioning Generalizing Language Emotions Validation Empathy
India (194,986) Social Relations (45,715) M > L > G M > G > L M > G > L M > G > L M > G \(\sim\) L
Education & Career (34,629) M > G > L M > G > L G > M > L G > M > L G > L > M
Health & Wellness (24,961) M > G \(\sim\) L M > G > L M > G > L G > M > L G > L > M
China (48,585) Speech & Language (18,140) M > G \(\sim\) L M > G > L G > M > L G > M > L G > M > L
Education & Career (8,750) M > G > L M > G > L M > G > L G > M > L G > M \(\sim\) L
Social Relations (5,383) M > L \(\sim\) G M > G > L M > G > L G > M > L M > G > L
Philippines (38,478) Social Relations (15,172) M > L > G M > G > L M > G > L G > M > L G > M > L
USA (36,915) Social Relations (16,134) M > L > G M > G > L M > G > L G > M > L G > M > L
Korea (27,464) Food & Drink (7,287) M > G > L L \(\sim\) M > G M > G > L G > M > L M > G > L
Russia (20,738) Social Relations (7,807) M > L > G M > G > L M > G > L G > M > L G \(\sim\) L \(\sim\) M
Turkey (9,183) Social Relations (4,097) M > L > G M > G > L M > G \(\sim\) L G > M > L M > G > L

(Note: L = Llama-3.1-8B-it, G = Gemma-3-12b-it, M = Ministral-3-14B-it; > indicates statistically significant difference; \(\sim\) indicates no significant difference)

Ablation & Interaction Analysis

The investigation assessed maxim violation rates across countries as well as the bidirectional coupling between communicative features.

Table 3: Percentage of Conversational Maxim Violations across Evaluated LLMs (%)

Maxim Country Llama-3.1-8B Violation (%) Gemma-3-12b Violation (%) Ministral-3-14B Violation (%) Dominant Model Pattern
Quality India 16.8 12.3 22.7 Gemma violates least (G > L > M adherence)
China 11.2 7.1 10.8 Consistently high quality across models
USA 12.6 11.5 15.0 Gemma achieves lowest violation rate
Quantity India 93.8 88.9 93.0 Widespread verbosity and over-answering
China 90.8 86.6 89.2 Severe failure to satisfy conciseness
USA 96.1 91.5 95.1 Over 90% violation due to length padding
Relation India 80.3 70.9 73.0 Gemma adheres best to core topic
China 76.3 63.9 66.1 Frequent topical expansion / drift
USA 87.1 78.9 82.0 Gemma maintains highest topical relevance
Manner India 58.8 54.6 61.5 Gemma is most readable and orderly
China 49.3 46.0 48.6 Moderate manner violations across models
USA 69.0 58.3 70.4 Gemma exhibits lowest manner violation

Key Findings

  • Distinct Model-Level Pragmatic Signatures: Across independent features, Ministral exhibits the strongest inclination toward insider positioning (+11% over Llama and Gemma) while also employing the highest rate of generalizing language (\(M > G > L\) across nearly all categories). Gemma consistently leads in conversational maxim adherence and validating cues. In contrast, Llama appears most constrained, exhibiting lower rates across positioning, generalizations, and affective anthropomorphism.
  • Bidirectional Coupling between Positioning and Anthropomorphism: When models adopt an insider stance, the probability of incorporating affective cues (emotions, validation, empathy) increases significantly. Conversely, the presence of anthropomorphic cues heavily inflates the likelihood that a response is judged as insider-positioned. Crucially, effect sizes for Anthropomorphism \(\to\) Positioning (\(\Delta \approx 0.16 \sim 0.30\)) are substantially larger than Positioning \(\to\) Anthropomorphism (\(\Delta \approx 0.02 \sim 0.05\)).
  • Orthogonality of Conversational Maxims and Cultural Framing: Adherence to Gricean maxims is largely independent of cultural positioning and generalizing language. A response can be highly cooperative, clear, and structured while remaining culturally detached or relying on blatant cultural generalizations, highlighting the inadequacy of standard instruction-tuning for cultural debiasing.

Highlights & Insights

  • Shifting the Paradigm from What to How: Moves beyond multiple-choice cultural knowledge retrieval, providing the first systematic quantification of how models frame open-ended subjective responses across cultural communities.
  • Unveiling the "Camouflage" Risk of Anthropomorphism: Demonstrates that anthropomorphic warmers (e.g., "I understand how you feel") can artificially inflate perceived cultural insiderness, misleading both automated evaluators and human users into assuming genuine cultural depth.
  • Curating High-Confidence Agreement Subsets: Releases filtered subsets where all three models receive unanimous FRANZ judgments, offering reliable diagnostic probes for downstream multi-objective alignment and stereotyping audits.

Limitations & Future Work

  • Reddit Community Proxy Skew: Sourcing data exclusively from national subreddits captures an online, English-proficient, and demographically skewed subpopulation, which cannot represent the entire cultural spectrum of each nation.
  • Surface Feature Confounding in LLM Judges: Both insider positioning and affective anthropomorphism rely heavily on first-person pronouns ("I", "we"), meaning judge models may partially conflate surface grammatical cues with deep communicative stance despite explicit prompt controls.
  • Western Pragmatic Bias in Gricean Maxims: The Gricean criteria of directness and conciseness reflect Western communicative norms; indirectness, circumlocution, and conversational hedging that are valued in high-context cultures are systematically penalized as violations of Manner or Quantity.
  • vs. BLEnd / CultureBank / CaLMQA: Existing benchmarks evaluate factual retrieval against fixed ground truths; FRANZ shifts focus to the stylistic, rhetorical, and pragmatic framing of subjective interactions.
  • vs. InsideOut (Wan et al., 2026): While prior work examined cultural positioning bias in isolation within scripted interviews, FRANZ integrates positioning with linguistic generalization, anthropomorphic cues, and pragmatic maxims on organic conversational discourse.

Rating

  • Novelty: โญโญโญโญโญ Pioneering four-dimensional sociolinguistic audit framework investigating how LLMs frame subjective cultural responses.
  • Experimental Thoroughness: โญโญโญโญโญ 376k authentic queries, 57 subreddits, 3 open-weight models, 16M judge evaluations, and rigorous GEE modeling.
  • Writing Quality: โญโญโญโญโญ Cohesive theoretical motivation, lucid structural exposition, and exemplary statistical discipline.
  • Value: โญโญโญโญโญ Highly actionable diagnostic tool for evaluating cultural alignment, stereotyping risks, and multi-objective alignment trade-offs in LLMs.