Skip to content

Not What, But How: A Communicative Audit of LLM Response Framing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/siddheshih/FRANZ
Area: Model Evaluation
Keywords: Communicative Framing Audit, Cultural Positioning, Anthropomorphic Cues, Generalizing Language, Gricean Conversational Maxims

TL;DR

Introduces FRANZ, an automated communicative audit framework, and SQUARE, a 376k-sample subjective query corpus from Reddit, revealing that LLM framing exhibits persistent model-level signatures and a strong bidirectional coupling between insider cultural positioning and affective anthropomorphism.

Background & Motivation

As large language models increasingly function as everyday conversational partners, an immense fraction of user inquiries does not seek verifiable factual propositions with a singular objective truth. Instead, users routinely bring forward culturally situated, value-laden, and deeply subjective questions concerning social etiquette, interpersonal relationships, lifestyle norms, and moral choices. However, existing evaluations of cultural competence in NLP have remained overwhelmingly fixated on factual correctness and cloze-style benchmarks, testing whether models retrieve specific regional trivia while completely overlooking how responses are communicatively framed. The stance, tone, and rhetorical framing adopted by an LLM fundamentally mediate user trust, emotional engagement, and the accidental propagation of essentialist stereotypes.

The fundamental tension hindering progress in this domain stems from the multifaceted and interdependent nature of communicative stance. Prior investigations into response characteristicsโ€”such as cultural positioning bias or anthropomorphismโ€”have examined these behavioral dimensions strictly in isolation. In natural human discourse, however, attempting to convey warmth or empathy without an authentic, locally grounded community stance frequently causes language models to resort to broad-stroke, sweeping generalizations that reduce complex lived realities into essentialist clichรฉs. Furthermore, existing cultural benchmarks rely on synthetic multiple-choice questions that entirely miss the stylistic pragmatics, contextual nuance, and informal register of spontaneous user inquiry across diverse geographical regions.

To overcome these barriers, this work shifts the diagnostic paradigm from assessing what LLMs know to systematically interrogating how LLM outputs are framed when answering culturally grounded subjective queries. The authors source open-ended discussions from international Reddit communities and formulate an integrated evaluation battery spanning linguistic, pragmatic, and sociolinguistic axes. Core idea: develop FRANZ, an automated multidimensional communicative audit framework alongside the 376k-sample SQUARE corpus, leveraging LLM-as-a-judge classifiers and Generalized Estimating Equations (GEE) to characterize cultural positioning, anthropomorphism, generalizing language, and conversational maxims across open-weight models.

Method

Overall Architecture

The FRANZ (FRAmework for respoNse characteriZation) diagnostic pipeline integrates large-scale subjective corpus curation with a four-pronged LLM-as-a-judge automated annotation suite and population-averaged statistical modeling. Unfiltered, real-world user questions collected across country-specific Reddit communities form the SQUARE dataset. Candidate open-weight LLMs generate zero-shot responses to these queries, which are then passed to specialized automated judges calibrated against extensive human annotations. The judges score each output across cultural positioning (insider vs. outsider stance), nine distinct non-embodied anthropomorphic cues, the presence of generalizing language, and adherence to four Gricean conversational maxims. Finally, Generalized Estimating Equations (GEE) model both independent model-level behavioral rates and cross-characteristic co-occurrence shifts while explicitly controlling for within-query correlation across model outputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real-world Subjective Queries<br/>SQUARE Corpus (376K samples)"] --> B["Open-weight LLM Generations<br/>(Llama-3.1 / Gemma-3 / Ministral)"]
    B --> C["SQUARE Corpus Curation<br/>57 Country Subreddits & 19 Categories"]
    C --> D["FRANZ Multi-dimensional Auditing<br/>Positioning / Generalizing / Cues / Maxims"]
    D --> E["GEE Statistical Modeling<br/>Cluster-robust Repeated Measures & Shift ฮ”"]
    E --> F["Empirical Diagnostic Insights<br/>Model Signatures & Coupling Analysis"]

Key Designs

1. SQUARE Corpus Curation: Grounding subjective cultural inquiry in authentic social communities

To resolve the limitation that existing benchmarks rely on synthetic, cloze-style multiple-choice questions unreflective of real communicative needs, the authors curate the SQUARE (Subjective QUeries Across REddit) dataset from the public Pushshift Reddit crawl. Starting with seeds of r/Ask<Country> and r/AskA<Nationality>, the collection is manually expanded across 408 geographic subreddits and rigorously filtered to eliminate NSFW content, automated news bots, image-heavy feeds, and inactive threads. The resulting 57 subreddits are mapped to 7 target countries: India, China, the Philippines, the United States, South Korea, Russia, and Turkey. Drawing upon cultural taxonomies and lexical typology, a zero-shot Llama-3.1-8B-it classifier, refined via human validation and majority voting, categorizes each query into one of 19 culturally grounded categories (such as social relations, food and drink, kinship, and historical events), yielding 376,350 natural subjective query samples.

2. FRANZ Multi-dimensional Auditing: A unified sociolinguistic and pragmatic evaluation battery

Rather than examining isolated rhetorical markers, FRANZ establishes four operationalized evaluation channels grounded in linguistic and anthropological theory: 1. Cultural Positioning: Anchored in the anthropological distinction between emics and etics, this judge classifies whether a model adopts an "Insider" perspective (signaling shared membership, local cultural codes, and native lived experience) or a detached "Outsider" stance; 2. Generalizing Language: Building on Beukeboom's syntactic framework, this component identifies essentialist claims attributing enduring traits or behavioral norms to entire demographic groups through bare plurals, indefinite singulars, or quantified adverbs (audited using Llama-3.1-70B-it to ensure grammatical fidelity); 3. Anthropomorphic Cues: Explicitly pruning physically embodied interactions that cannot be reliably verified from text alone (e.g., eating or physical contact), FRANZ audits nine non-embodied cues spanning personal relationships, personal history, explicit relationship status, desires, agency, emotions, validation, empathy, and relatability; 4. Gricean Maxims: Decomposes cooperative dialogue efficiency along Grice's four normative conversational maxims: Quality (grounded vs. unverified assertions), Quantity (adequately informative without excessive redundancy), Relation (topically coherent without thematic drift), and Manner (perspicuous, structured, and intelligible).

3. GEE Statistical Modeling: Cluster-robust inference for cross-characteristic co-occurrence

Because three LLMs generate responses to the identical set of input queries, their outputs cannot be treated as statistically independent observations. Standard chi-square tests or ordinary logistic regression would inflate type-I error. The authors therefore adopt Generalized Estimating Equations (GEE) with a Binomial error distribution and logit link function, clustering repeated observations by Reddit post ID to yield consistent population-averaged estimates. To quantify whether the presence of a causal characteristic \(C\) shifts the probability of an effect characteristic \(E\), they define the risk difference metric \(\Delta\):

\[\Delta = P(E=1 \mid C=1) - P(E=1 \mid C=0)\]

Fitting stratified GEEs across country-category slices and applying false discovery rate (FDR) corrected Wald tests enables the authors to determine both within-model prevalence rankings and statistically rigorous cross-characteristic coupling dynamics.

Benchmark & Evaluation Protocol

The empirical evaluation examines three prominent open-weight models in the 8โ€“14B parameter range: Llama-3.1-8B-it, Gemma-3-12B-it, and Ministral-3-14B-it. Generations are elicited under a neutral instruction prompt without explicit persona steering to capture default behavior. Llama-3.1-8B-it serves as the judge for cultural positioning, anthropomorphism, and Gricean maxims, while Llama-3.1-70B-it assesses generalizing language. To establish the validity of the automated evaluators, two independent human annotators annotated 150 stratified query-response pairs per characteristic, totaling 4,500 human validation judgments.

Key Experimental Results

Main Results: Human Validation of the Automated Evaluators

FRANZ automated judges demonstrate high agreement with human annotator judgments across all audited framing characteristics:

Response Characteristic Sub-category A1-LLM Accuracy (ACC) A2-LLM Accuracy (ACC) Inter-annotator Positive Agreement (PA)
Cultural Positioning Insider Stance 0.75 1.00 0.86
Outsider Stance 0.99 0.96 0.97
Generalizing Language Generalizing Present 0.71 0.71 1.00
Non-Generalizing 1.00 0.98 0.99
Anthropomorphism Emotions 0.98 0.98 1.00
Validation 0.83 0.87 0.98
Empathy 0.94 0.95 0.99
Desires 0.96 0.97 1.00
Agency 0.95 0.96 0.99
Conversational Maxims Quality 0.94 0.92 0.95
Quantity 0.71 0.82 0.90
Relation 0.51 0.73 0.80
Manner 0.71 0.84 0.89

Gricean Maxim Violation Rates Across Countries

Violations of Gricean conversational maxims (reported as percentages where lower values indicate superior adherence) reveal clear structural divergence among models across representative countries:

Country Evaluated Model Quality Violation (%) Quantity Violation (%) Relation Violation (%) Manner Violation (%)
India Llama-3.1-8B-it 16.8 93.8 80.3 58.8
Gemma-3-12B-it 12.3 88.9 70.9 54.6
Ministral-3-14B-it 22.7 93.0 73.0 61.5
China Llama-3.1-8B-it 11.2 90.8 76.3 49.3
Gemma-3-12B-it 7.1 86.6 63.9 46.0
Ministral-3-14B-it 10.8 89.2 66.1 48.6
USA Llama-3.1-8B-it 12.6 96.1 87.1 69.0
Gemma-3-12B-it 11.5 91.5 78.9 58.3
Ministral-3-14B-it 15.0 95.1 82.0 70.4
Korea Llama-3.1-8B-it 11.4 93.9 78.6 54.0
Gemma-3-12B-it 8.8 89.1 65.1 48.0
Ministral-3-14B-it 17.1 93.0 67.4 54.5

Key Findings

  • Distinct and Persistent Model Signatures: Across virtually every examined country and thematic category, models exhibit consistent behavioral rankings. Ministral-3-14B-it displays the strongest propensity toward insider cultural positioning (+11% over same-size models) as well as generalizing language; Gemma-3-12B-it achieves the highest communicative discipline with minimal maxim violations; and Llama-3.1-8B-it remains the most stylistically conservative and constrained.
  • Bidirectional Coupling Between Insider Positioning and Affective Anthropomorphism: When an LLM adopts an insider stance, the likelihood of incorporating empathetic, validating, or emotional phrasing surges significantly (\(\Delta > 0\)). In reverse, the inclusion of affective anthropomorphic markers strongly increases the odds of a response being perceived as culturally insider-positioned.
  • Orthogonality of Generalizing Language and Maxim Adherence: The frequency of making broad-stroke generalizations about social groups is statistically independent of cultural positioning and conversational maxims. Adhering closely to cooperative communicative norms does not prevent a model from emitting essentialist generalizations.

Highlights & Insights

  • Refocusing Evaluation on Communicative Pragmatics: Provides a much-needed methodology for auditing open-ended language model behavior beyond narrow factual accuracy benchmarks, targeting how linguistic stance shapes social impact.
  • Ecologically Valid Corpus Curation: Leverages spontaneous community discourse across 57 national subreddits, capturing genuine subjective queries that artificial multiple-choice benchmarks fail to represent.
  • Critical Caveat for Alignment Research: Demonstrates that optimizing models for polite conversation or maxim adherence does not organically eliminate cultural stereotyping or essentialist overgeneralization, underscoring the necessity of multi-objective alignment constraints.

Limitations & Future Work

  • Sampling Bias in English Reddit Communities: Sourcing from English-language subreddits introduces a demographic skew toward English-proficient, highly educated, or expatriate populations rather than native non-English speakers.
  • Potential Judge Confounding via Politeness Formulas: Evaluating LLMs using other LLMs may conflate conventional conversational fillers (e.g., "I'd be glad to help") with substantive anthropomorphic validation or empathy.
  • Need for Multilingual Expansion: Extending FRANZ to audit indigenous languages and native non-Latin script communities remains an essential avenue for future research.
  • vs. Cultural Positioning Bias (Wan et al., 2026): While Wan et al. audited positioning as an isolated variable within structured interviews, FRANZ situates positioning alongside anthropomorphism, generalization, and maxims in natural subjective queries.
  • vs. Factual Cultural Benchmarks (CultureBank, BLEnD): Traditional benchmarks evaluate whether models possess factual knowledge against static ground-truth answers; FRANZ analyzes how models formulate communicative stances when queries have no ground truth.

Rating

  • Novelty: โญโญโญโญโญ Pioneering multi-faceted communicative audit framework moving evaluation from factual content to rhetorical framing.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous scale analyzing over 1.1M responses across 376k queries with GEE statistical modeling and human validation.
  • Writing Quality: โญโญโญโญโญ Precise conceptual definitions, lucid argumentation, and comprehensive empirical presentation.
  • Value: โญโญโญโญโญ Essential diagnostic methodology for detecting cultural stance divergence, anthropomorphic manipulation, and stereotyping in LLMs.