Skip to content

CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/amazingljy1206/CommunityBench
Area: LLM Reasoning
Keywords: Community-Level Alignment, LLM Alignment, Preference Distribution Prediction, Pluralistic Alignment, Social Simulation

TL;DR

This paper proposes community-level alignment as a scalable middle ground between universal and individual alignment, introducing CommunityBench—a benchmark spanning 6,919 communities and 12,149 instances across four tasks grounded in Common Identity and Common Bond theory—evaluating 17 foundation models to reveal critical failures in distributional reasoning and demonstrating that community signals transfer effectively to individual social simulation.

Background & Motivation

Large language model (LLM) alignment methodologies are designed to ensure that model outputs reflect human intentions, normative principles, and societal values, playing a foundational role in guaranteeing safety, trustworthiness, and social acceptability across open-ended and high-stakes deployment environments. Nevertheless, current alignment strategies predominantly operate at two divergent extremes of granularity. The prevailing paradigm enforces a one-size-fits-all approach, steering models toward monolithic gold standards such as general helpfulness, honesty, and harmlessness; however, this formulation inherently assumes a flat, universal value consensus that systematically silences and marginalizes minority cultures, sub-demographic perspectives, and specialized community norms. Conversely, individual-level alignment seeks to calibrate responses to the idiosyncrasies of every distinct user, but it inevitably encounters severe practical obstacles: real-world interactions for individual users are acutely sparse, while continuously training, fine-tuning, or retrieving personal histories across billions of users incurs prohibitive engineering and computational expenses.

In human society, individuals do not interact as isolated atoms, nor do they adhere to an undifferentiated global hivemind; rather, they naturally self-organize into social clusters and communities characterized by high intra-group normative cohesion and shared behavioral expectations. This intrinsic organizational structure suggests that community-level alignment can serve as an ideal "middle ground" for pluralistic AI alignment. By leveraging community dynamics, system developers can aggregate noisy individual behaviors into robust, culturally grounded supervisory signals without expending immense resources on per-user customization, all while preserving the rich diversity of pluralistic viewpoints. Despite the theoretical appeal of this middle-ground paradigm, whether state-of-the-art foundation models possess the capacity to internalize, navigate, and represent group-specific norms remains fundamentally unexamined, largely because prior benchmarks evaluate either coarse demographic buckets or isolated modal choices rather than structured community behaviors.

Bridging this gap requires an evaluation framework that systematically captures collective identity, internal disagreement, characteristic rhetorical discourse, and observable behavioral traces across thousands of authentic social groups. The core motivation of this paper is to operationalize classical social psychology—specifically Common Identity and Common Bond (CICB) theory—into a rigorous computational benchmark powered by public online community interactions. Core idea: Treating subreddits as observable online community proxies, this paper formalizes four group facets—shared identity, within-group heterogeneity, characteristic discourse, and behavioral traces—into the 12,149-instance CommunityBench across 6,919 communities, evaluating 17 foundation models on vote-smoothed preference distributions and demonstrating that community-conditioned representations effectively transfer to downstream individual attitude modeling.

Method

Overall Architecture

CommunityBench treats Reddit subreddits as structured proxies for online communities, executing a four-stage pipeline: corpus collection and deduplication, dual-level community representation profiling, query-option instance construction, and Bayesian opinion distribution estimation. Grounded in Common Identity and Common Bond (CICB) theory, macro-level community profiles summarize shared worldviews from top-50 upvoted posts, while micro-level conversation profiles capture interaction dynamics and stance diversity from top-20 comments per thread. Candidate options are curated via Maximal Marginal Relevance (MMR) and stratified sampling across sentiment and popularity buckets to eliminate lexical and length heuristics. Continuous ground-truth preference distributions are then estimated from net vote scores via Beta-smoothed pseudo-counts, powering four diagnostic tasks: Preference Identification (PI), Preference Distribution Prediction (DP), Community-Consistent Generation (CG), and Community Identification (CI).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Raw Reddit Corpus<br/>(1,000+ subreddits, deduplication & bot removal)"] --> B["Stage 1: Dual-Level Community Profiling<br/>Common Identity (Top-50 posts) + Common Bond (Top-20 comments)"]
    B --> C["Stage 2: Contrasting Candidate Construction<br/>MMR filtering + Sentiment & popularity stratification"]
    C --> D["Stage 3: Bayesian Opinion Distribution Estimation<br/>Beta prior smoothing + Pseudo-count vote recovery"]
    D --> E["Stage 4: Four Community Alignment Tasks<br/>PI / DP / CG / CI"]
    E --> F["Downstream Transfer<br/>SocioBench individual attitude simulation"]

Key Designs

1. Dual-Level Community Profiling: Grounding Group Dynamics in CICB Theory

Conventional demographic steering often relies on broad categorical labels (e.g., nationality or age group) that fail to capture the nuanced communicative styles, unspoken conventions, and local consensus that organically evolve within real communities. Guided by Common Identity and Common Bond (CICB) theory, the benchmark constructs representations along two complementary axes. At the subreddit level (Common Identity), the top-50 upvoted posts are aggregated to extract recurring group identities, shared ethical principles, and collective worldviews, supporting concise summary profiles. At the single-thread level (Common Bond), the top-20 comments are preserved to reflect localized conversational dynamics, vernacular, humor, and immediate interpersonal bonding. In empirical evaluation, CommunityBench offers three distinct settings—fine-grained (raw conversational history), summary (synthesized persona), and coarse (lightweight metadata)—enabling models to condition directly on rich conversational context without prior exposure to the specific community.

2. Contrasting Candidate Construction: Eliminating Lexical and Popularity Shortcuts via MMR

When candidate options exhibit minimal semantic divergence or when one option drastically dominates in length or grammatical polish, models easily exploit superficial shortcuts rather than genuinely inferring community-specific values. To produce informative and challenging candidate sets, the pipeline computes emotional polarity using DistilBERT and extracts dense semantic embeddings using Sentence-BERT. Maximal Marginal Relevance (MMR) is subsequently applied to select candidate comments that remain strictly relevant to the post request while maximizing mutual contrast. The final options are assembled via a stratified greedy strategy across sentiment buckets and popularity quantiles, discarding candidate sets with pairwise cosine similarity exceeding 0.8, insufficient sentiment diversity, or identical vote scores. This setup forces models to reason over substantive ideological alignment rather than superficial text artifacts.

3. Bayesian Opinion Distribution Estimation: Recovering Pluralistic Disagreement from Net Votes

Online communities are rarely unanimous; internal debate and dissenting perspectives are central features of group culture that alignment benchmarks must preserve. Because Reddit exposes only the net score \(s_i = U_i - D_i\) rather than raw upvote and downvote counts, naive normalization would improperly truncate or distort negatively scored options. The authors introduce a Bayesian pseudo-count reconstruction mechanism. Using the thread-level upvote ratio \(r\) and scaling factor \(b = |2r - 1|\), the total interaction volume is estimated as \(N_i = |s_i| / b\). For non-negative scores, votes are apportioned as \((r N_i, (1 - r) N_i)\); for negative scores, they are mapped to \(((1 - r) N_i, r N_i)\), preserving \(U_i - D_i = s_i\) while allocating small non-zero support mass to downvoted candidates. A Beta prior \(\text{Beta}(\alpha_0, \beta_0)\) anchored to the post approval rate (\(\alpha_0 = r k_0, \beta_0 = (1 - r) k_0\), default strength \(k_0 = 10\)) smoothly regularizes sparse observations into normalized posterior probabilities:

\[d_i = \frac{\alpha_0 + U_i}{\sum_{j=1}^n (\alpha_0 + U_j)}\]

This formulation maintains a benchmark-wide mean preference entropy of 1.54, faithfully reflecting pluralistic debate and establishing a robust target distribution for distributional evaluation.

4. Four Community Alignment Tasks: Comprehensive Diagnostic Loop from Modal Choice to Provenance Recovery

To evaluate the multi-faceted nature of group-conditioned behavior, CommunityBench formalizes four distinct evaluation tasks: - Preference Identification (PI): Conditioned on a community profile \(p\), a query \(q\), and candidate options \(O\), the model selects the single option \(\hat{o}\) matching the community's modal endorsement, testing its grasp of shared identity; - Preference Distribution Prediction (DP): The model predicts the complete probability simplex \(\hat{d} \in \Delta^{n-1}\) across all options, evaluated via Kendall's \(\tau\) (ranking order), Top-1 accuracy, and Jensen–Shannon Divergence (JSD) to assess fidelity to internal group heterogeneity; - Community-Consistent Generation (CG): The model produces open-ended responses matching the community's characteristic discourse, judged by an LLM-as-a-judge panel (GPT-4o, Grok-4-Fast, Gemini-2.5-Flash) and ranked via BTL-Elo; - Community Identification (CI): Serving as the diagnostic inverse of alignment, the model observes a query, candidate responses, and an empirical preference distribution to infer which community produced the pattern, testing its recognition of behavioral traces.

Benchmark & Evaluation Protocol

CommunityBench comprises 12,149 instances across 6,919 communities selected via \(k\)-center greedy diversity sampling over hashed \(n\)-gram representations (roughly 3,037 instances per task), spanning December 2020 through September 2025 with an average of 4.0 options per query. Evaluations cover 17 foundation models across Qwen2.5/3, Llama3.1/3.3, GLM-4, InternLM3, Mistral, DeepSeek-V3/R1, GPT-4o, and Grok-4. Except for DeepSeek-R1 (granted an 8,192-token reasoning budget for PI, DP, and CI), all models use greedy decoding (temperature=0) with a 1,024-token cap under vLLM. In the downstream transfer study, Qwen2.5-7B-Instruct is fine-tuned with LoRA (\(r=16, \alpha=32\)) on 15,790 multi-task CommunityBench instances and evaluated zero-shot across 10 societal domains in SocioBench to test generalizability to individual attitude simulation.

Key Experimental Results

Main Results

The evaluation across 17 foundation models under the fine-grained conversational profile setting reveals distinct capability ceilings between modal selection and continuous distribution modeling.

Model PI (Acc ↑) DP (JSD ↓) DP (Kendall's \(\tau\) ↑) DP (Top-1 Acc ↑) CI (Acc ↑) CG (BTL-Elo ↑)
Qwen2.5-7B-Instruct 0.3526 0.1600 0.0790 0.3029 0.5866 -262.68
Qwen2.5-14B-Instruct 0.3596 0.1402 0.0817 0.2960 0.6853 -154.87
Qwen2.5-72B-Instruct 0.3675 0.1355 0.1365 0.3336 0.6896 -149.18
Qwen3-8B 0.3573 0.1222 0.0407 0.1297 0.3914 87.77
Qwen3-14B 0.3813 0.1216 0.1128 0.2522 0.4852 272.39
Qwen3-32B 0.3698 0.1172 0.1610 0.3312 0.6086 237.80
Llama3.1-8B-Instruct 0.2963 0.2457 0.0445 0.2605 0.4737 -166.28
Llama3.1-70B-Instruct 0.3424 0.1871 0.0828 0.3039 0.7702 -54.90
Llama3.3-70B-Instruct 0.3434 0.1825 0.1100 0.3224 0.7380 -56.64
InternLM3-8B-Instruct 0.3023 0.1308 0.0294 0.2124 0.4924 -509.44
Mistral-7B-Instruct-v0.3 0.3135 0.1491 0.0409 0.2591 0.4855 -300.56
GLM-4-9B-0414 0.3312 0.1724 0.0559 0.2838 0.5125 176.85
GLM-4-32B-0414 0.3645 0.1434 0.1127 0.2894 0.6718 233.49
DeepSeek-V3-0324 0.4034 0.1358 0.1471 0.3345 0.8232 206.22
DeepSeek-R1-0528 0.4120 0.1360 0.1695 0.3184 0.8209 812.30
GPT-4o 0.3862 0.1309 0.1413 0.3230 0.8430 256.28
Grok-4 0.3734 0.1136 0.1676 0.3454 0.8223 478.67

Ablation Study

To verify that models rely on meaningful community context rather than lexical heuristics or majority shortcuts, the authors evaluated heuristic baselines, lexical matching, and profile ablations.

Model / Baseline PI (Acc ↑) DP (JSD ↓) CI (Acc ↑) Note
Random Baseline 0.2549 – 0.2459 Uniform random choice over candidates
Majority Voting 0.2601 – 0.3585 Training set empirical mode
Uniform Distribution – 0.1626 – Uniform probability allocation over options
TF-IDF + SVM 0.2908 – 0.3673 Classical lexical frequency baseline
Qwen2.5-72B (Full Profile) 0.3675 0.1355 0.6896 Default fine-grained conversational profile
Qwen2.5-72B (w/o Profile) 0.3230 0.1626 0.5478 Dropping profile drops PI by 4.45% and CI by 14.18%
Qwen3-32B (w/o Profile) 0.3157 0.1523 0.4916 Substantial drop across all evaluated tasks
Llama3.1-70B (w/o Profile) 0.3133 0.1608 0.5513 Severe performance degradation without profile
Llama3.3-70B (w/o Profile) 0.3230 0.1549 0.5290 Confirms reliance on structured community context

In the downstream SocioBench individual attitude simulation transfer study (Qwen2.5-7B backbone), the Community-Aligned Model outperformed both the one-size-fits-all and individual-level prompting baselines across 8 out of 10 social domains, increasing average accuracy from 0.267 (One-size) and 0.283 (Individual) to 0.295, with the largest improvements observed in Environment (0.277 vs 0.241), Family (0.283 vs 0.255), and Health (0.284 vs 0.237).

Key Findings

  • Distributional reasoning is the primary failure mode: While frontier models achieve moderate success in identifying modal preferences (PI up to 0.4120), their rank-order consistency on preference distributions remains deficient (Kendall's \(\tau \le 0.1695\)). Models tend to output overly smoothed, uniform-like probabilities that fail to capture the actual intensity and ordering of dissenting opinions. In contrast, human annotators attain \(\tau\) values of 0.505–0.615 and JSDs of 0.041–0.052, highlighting immense headroom for future distributional modeling.
  • Steep accuracy degradation in long-tail communities: Model accuracy correlates strongly with subreddit subscriber counts. Models perform reliably on mainstream, highly populated subreddits encountered frequently during pretraining, but fail markedly on niche and long-tail communities, proving that current architectures struggle to rapidly infer specialized cultural norms from static context windows alone.
  • Richer context monotonically boosts alignment fidelity: Comparing coarse metadata, LLM-generated summary personas, and fine-grained raw conversational logs demonstrates that fine-grained conversational traces deliver the highest alignment accuracy and distributional correlation, confirming that authentic interpersonal interactions carry vital stylistic and normative grounding.

Highlights & Insights

  • Theoretical-to-Empirical Translation: Successfully grounds abstract social psychology (CICB theory) into four concrete, mathematically formal machine learning tasks, providing an actionable and scalable framework for pluralistic alignment research.
  • Bayesian Pseudo-Count Vote Inversion: Overcomes platform data constraints by deriving a principled pseudo-count formulation paired with a Beta prior, turning coarse net voting scores into continuous, noise-resilient preference distributions without manual re-annotation.
  • Reconciling Individual and Universal Granularity: Demonstrates that community-level supervision serves as a powerful inductive bias for downstream individual modeling, outperforming direct individual-level prompt retrieval while avoiding the extreme cost of personalized model fine-tuning.

Limitations & Future Work

  • Platform and Demographic Representativeness: Because all communities are sourced from Reddit, the corpus reflects inherent platform biases—predominantly English-speaking, younger, and tech-oriented demographics—limiting direct extrapolation to census-representative societal values. Cross-platform validation (e.g., StackExchange, multilingual forums) remains essential.
  • Confounding Variables in Online Engagement: Public vote signals are inevitably influenced by thread exposure time, algorithmic placement, moderation interventions, and vote fuzzing, meaning vote distributions approximate behavioral endorsement rather than purely uncorrupted normative judgments.
  • Norm Drift and Hierarchical Governance: The benchmark represents a static temporal snapshot (median date: August 2025) and cannot measure longitudinal norm drift; furthermore, future research must address hierarchical arbitration mechanisms to determine when models should prioritize universal safety invariants over local community conventions or individual desires.
  • vs PRISM / GlobalQA / OpinionQA: Prior pluralistic alignment benchmarks typically rely on broad, static demographic buckets (e.g., country of origin, race) or multiple-choice survey items; CommunityBench grounds alignment in 6,919 naturalistic online social communities and introduces explicit continuous distribution modeling.
  • vs ComPO / Com-Align: While recent community-focused datasets explore open-ended generation or pairwise ranking in restricted domains, CommunityBench establishes a unified four-task diagnostic loop spanning modal identification, distribution prediction, generation, and reverse community attribution, paired with downstream individual simulation transfer.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first unified community-level alignment benchmark with rigorous sociological grounding and continuous distribution modeling.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 17 foundation models, rigorous human double-blind audits, judge self-bias analyses, and downstream SFT transfer.
  • Writing Quality: ⭐⭐⭐⭐⭐ Impeccably structured, clear mathematical definitions, and deeply cohesive narrative prose throughout.
  • Value: ⭐⭐⭐⭐⭐ Offers a transformative and scalable paradigm that bridges the long-standing gap between monolithic one-size-fits-all models and expensive individual customization.