VASTU: Language Models Struggle to Recognize Online Community Values¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/scuba-illinois/VASTU
Area: Social Computing
Keywords: community values, preference modeling, content curation, parametric adaptation, prompting bottleneck
TL;DR¶
This paper formalizes community-conditioned preference prediction and introduces the VASTU benchmark comprising 75,000 comments across 15 Reddit communities, finding that parametric adaptation (AUROC 0.71–0.74) substantially outperforms frontier prompting (≤0.62) where vanilla LLMs discard over 80% of locally valued content due to rigid, generic quality heuristics.
Background & Motivation¶
Online communities cultivate distinct discourse norms and subcultures over time, meaning what their members collectively celebrate often diverges sharply from platform-wide, sanitized notions of "high quality." However, natural language processing and content moderation research has long concentrated on punitive tasks, such as filtering hate speech, toxicity, and misinformation. In contrast, computational modeling for positive content curation—such as surfacing constructive commentary, routing relevant exemplars to newcomers, and personalizing feed ranking—remains severely understudied. Developing models that reliably recognize locally valued discourse in its natural conversational context is an indispensable prerequisite for building community-aware reward models, constructing weak preference pairs, and enabling pluralistic AI governance.
The central tension in recognizing community preferences is that community-valued content is fundamentally distinct from generic helpfulness, eloquence, or non-toxicity. A response can be objectively well-argued in the abstract yet penalized in context for being pedantic or redundant; conversely, in gaming and humor subreddits, concise insider references, witty memes, or casual profanity are passionately rewarded for relatability, whereas an academic forum would discard them as spam. Prior inquiries have either characterized community norms through descriptive qualitative taxonomies or attempted monolithic platform-wide alignment, leaving an open empirical question: can state-of-the-art language models accurately detect whether a community will approve of naturally occurring human-written content?
This paper addresses this gap by utilizing Reddit's behavioral upvoting patterns as an aggregated signal of community approval, evaluating whether models can predict local preferences given conversational context and community identity. Core idea: formalize community approval detection as community-conditioned preference prediction, construct the 75k-sample VASTU benchmark spanning 15 diverse subreddits across 5 domains, and systematically benchmark feature-based models, supervised encoders, fine-tuned SLMs, and frontier prompted LLMs to expose their capability boundaries and failure modes.
Method¶
Overall Architecture¶
The VASTU framework consists of two core components: a rigorous community preference data curation pipeline and a multi-paradigm evaluation testbed operating under standardized evaluation protocols. Given a community \(c\), a target reply comment \(x\), and optional context \(z\) (parent post title/body or direct parent comment), the models predict whether \(x\) belongs to the upper tier of community approval (top 5% of upvotes).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Raw Reddit Discourse Corpus<br/>15 Subreddits across 5 Domains / 75k Comments"] --> B["Community Quantile Stratified Sampling<br/>HIGH(top 5%) vs LOW(bottom 30%)"]
B --> C["Feature Extraction & Context Variants<br/>Embeddings + Prosociality + LIWC + Context"]
C --> D["Dual-Track Model Evaluation Suite<br/>Supervised Adaptation vs Frozen LLM Prompting"]
D --> E["Systematic Error Profiling & Taxonomy<br/>FPR-FNR Trade-offs & 6 Thematic Failure Modes"]
Key Designs¶
1. Community Quantile Stratified Sampling: Establishing Extreme Separability
To extract robust behavioral preference signals from noisy online discourse, the authors select 15 topically diverse Reddit communities categorized into 5 domains: Gaming (e.g., r/Games, r/2007scape), Science (e.g., r/science, r/askscience), Q&A, Advice, and Politics. From a 9-month archive (May 2016 to February 2017), the pipeline systematically filters out deleted or removed comments, banned accounts, moderator-distinguished posts, deterministic bot accounts, and ultra-short comments (\(\le 5\) tokens). Rather than predicting continuous, noisy raw upvote counts, approval is operationalized via community-specific score quantiles: comments above the 95th percentile (\(q=0.95\)) are labeled HIGH, while comments below the 70th percentile (\(q=0.70\)) are labeled LOW. The dataset samples 2,500 balanced comments per class per subreddit (capped at 5,000 per community), yielding 75,000 balanced instances. This deliberate threshold gap prioritizes conceptual separability, ensuring that HIGH instances represent the unambiguous upper tier of community-celebrated contributions.
2. Multi-Dimensional Feature Fusion: Coupling Dense Semantics with Psycholinguistic Signals
To probe whether explicit linguistic indicators complement deep contextual representations, the authors extract five feature groups: dense semantic embeddings via SentenceBERT (all-mpnet-base-v2), six prosociality dimensions (politeness, agreement, support, donation, laughter, gratitude), surface-level stylometrics (VADER sentiment, Flesch reading ease, interrogative ratio), Detoxify toxicity scores, and psycholinguistic markers from the LIWC 2015 dictionary. In the supervised encoder family (BERT-base and ModernBERT), the dense transformer representations of the text and the engineered linguistic feature vectors are integrated in parallel using learnable adaptive fusion weights before feeding into a two-layer classification head. This architecture allows the model to leverage general semantic representations while explicitly tuning into community-specific vocabulary frequencies and tone profiles.
3. Progressive Contextual Injection and Prompting Strategy Matrix: Decoupling Knowledge Acquisition
To dissect how conversational context and background knowledge influence detection, the benchmark establishes three context conditions: comment-only, +community (prepending subreddit name and description), and +community+parent (incorporating the direct parent comment). Across frontier prompted models (GPT-4o-mini, GPT-4o, and reasoning model GPT-5-mini), four prompting paradigms are evaluated: standard Zero-Shot, Chain-of-Thought (CoT) prompting for step-by-step normative evaluation, balanced Few-Shot prompting (\(k=3\)) sampled from the same subreddit across distinct threads, and Value-Augmented Prompting (VAP) providing Schwartz's Basic Human Values framework (10 universal human values) as an analytical lens. This matrix enables a rigorous comparison between parametric domain adaptation and inference-time prompting heuristics.
4. Dual-Track Quantitative & Qualitative Error Diagnosis: Exposing the Auditor Bias
To understand why prompted LLMs underperform smaller supervised baselines, the authors conduct error profiling across False Positive Rates (FPR) and False Negative Rates (FNR). The resulting trade-off curves reveal that vanilla LLMs exhibit extreme conservatism, acting as rigid gatekeepers rather than curators. Research personnel subsequently conducted an inductive thematic analysis on 150 misclassified instances (10 per community) under the comment-only CoT setting, developing a 6-factor qualitative failure taxonomy: T1 Wrong community norm, T2 "More effort is better" bias, T3 Missing thread context, T4 Evidence bar mismatch, T5 Humor/sarcasm misinterpretation, and T6 Profanity penalty. This diagnosis illustrates how standard safety tuning (RLHF) induces an over-sanitized helpfulness bias that systematically misjudges informal community discourse.
Key Experimental Results¶
Main Results¶
Under a community-stratified 80/20 train/test split, models across four distinct families were evaluated on AUROC, F1 score, and Brier calibration score (for continuous probability outputs).
| Model Family | Model | Training / Inference Setup | Scope / Strategy | AUROC ↑ | F1 ↑ | Brier ↓ |
|---|---|---|---|---|---|---|
| Baselines | 1-Toxicity | Inverted Detoxify score | Zero-shot heuristic | 0.45 ± 0.05 | 0.41 ± 0.06 | 0.81 ± 0.02 |
| Baselines | DialogRPT | Reddit score-pretrained ranking model | Cross-domain zero-shot | 0.59 ± 0.04 | 0.55 ± 0.04 | 0.17 ± 0.06 |
| Feature-based | Logistic Regression | Full feature set + L2 regularization | Global (pooled) | 0.65 | 0.61 | 0.23 |
| Feature-based | Logistic Regression | Full feature set + L2 regularization | Local (per-community) | 0.69 ± 0.04 | 0.64 ± 0.03 | 0.22 ± 0.01 |
| Feature-based | XGBoost | Gradient boosted decision trees | Global (pooled) | 0.66 | 0.61 | 0.23 |
| Feature-based | XGBoost | Gradient boosted decision trees | Local (per-community) | 0.67 ± 0.04 | 0.63 ± 0.02 | 0.23 ± 0.01 |
| Supervised Encoders | BERT-base | Text + Feature fusion head | Global (pooled) | 0.70 | 0.67 | 0.22 |
| Supervised Encoders | BERT-base | Text + Feature fusion head | Local (per-community) | 0.71 ± 0.05 | 0.68 ± 0.03 | 0.23 ± 0.02 |
| Supervised Encoders | ModernBERT | Text + Feature fusion head | Global (pooled) | 0.73 | 0.70 | 0.21 |
| Supervised Encoders | ModernBERT | Text + Feature fusion head | Local (per-community) | 0.74 ± 0.04 | 0.72 ± 0.03 | 0.21 ± 0.02 |
| Prompted LLMs | GPT-4o-mini | Frozen in-context inference | Zero-Shot | 0.56 | 0.38 | – |
| Prompted LLMs | GPT-4o | Frozen in-context inference | Zero-Shot | 0.58 | 0.41 | – |
| Prompted LLMs | GPT-4o | Frozen in-context inference | Few-Shot (k=3) | 0.60 | 0.42 | – |
| Prompted LLMs | GPT-4o | Frozen in-context inference | Value-Augmented | 0.60 | 0.48 | – |
| Reasoning LRM | GPT-5-mini | Extended test-time reasoning | Zero-Shot | 0.53 | 0.20 | – |
| Reasoning LRM | GPT-5-mini | Extended test-time reasoning | CoT | 0.55 | 0.37 | – |
| Fine-tuned SLMs | Gemma-3-4B-IT | 4-bit LoRA (1 epoch) | Supervised fine-tuning | 0.65 | 0.42 | – |
| Fine-tuned SLMs | Qwen3-4B-Instruct | 4-bit LoRA (1 epoch) | Supervised fine-tuning | 0.64 | 0.43 | – |
Ablation Study¶
The table below tracks the progressive performance shift (AUROC and F1) across input context variants:
| Model | comment-only AUROC | comment-only F1 | +community AUROC | +community F1 | +community+parent AUROC | +community+parent F1 | Parent Context AUROC Gain |
|---|---|---|---|---|---|---|---|
| GPT-4o-mini | 0.56 | 0.38 | 0.57 | 0.37 | 0.59 | 0.41 | +0.02 |
| GPT-4o | 0.58 | 0.41 | 0.61 | 0.40 | 0.62 | 0.44 | +0.01 |
| GPT-5-mini | 0.53 | 0.20 | 0.54 | 0.24 | 0.55 | 0.21 | +0.01 |
| Gemma-3-4B-IT (LoRA) | 0.65 | 0.42 | 0.66 | 0.43 | 0.71 | 0.49 | +0.05 |
| Qwen3-4B-Instruct (LoRA) | 0.64 | 0.43 | 0.65 | 0.41 | 0.70 | 0.48 | +0.05 |
Key Findings¶
- Parametric adaptation decisively beats prompting: Open-weight SLMs with only 4B parameters (Gemma-3-4B and Qwen3-4B reaching 0.64–0.71 AUROC) consistently outshine frontier prompted models orders of magnitude larger, such as GPT-4o (≤0.62 AUROC), while ModernBERT Local achieves the top benchmark mark of 0.74 AUROC. Community-specific norms represent empirical knowledge that must be learned from supervision rather than prompted into emergence.
- Reasoning fails on implicit social norms: Extended reasoning proves counterproductive. GPT-5-mini demonstrates the weakest zero-shot performance across all models (0.53 AUROC, 0.20 F1), and adding Chain-of-Thought to GPT-4o degrades AUROC from 0.58 to 0.56. Reasoning models tend to over-analyze conversational text using generic moral or academic rubrics, blinding them to casual cultural affinity.
- Conversational grounding is the primary inference lever: Injecting direct parent comment context yields substantial AUROC gains (+0.05 to +0.06 in SLMs, e.g., Gemma-3-4B moving from 0.66 to 0.71), producing roughly \(6\times\) the benefit of prepending static community descriptions. Many comments derive their entire value from conversational relevance and humor timing, which are invisible in isolation.
- Opposing directional drivers across subreddits: SHAP analysis on XGBoost proves that linguistic features operate with high domain heterogeneity. Prosociality strongly predicts community value in Advice subreddits (+0.121 mean SHAP) but exhibits negative coefficients in Gaming (-0.056) and Science (-0.048). Similarly, first-person self-focus is highly rewarded in personal advice (+0.098) but heavily penalized in objective science discourse (-0.070).
Highlights & Insights¶
- From punitive moderation to positive curation: The paper formalizes a crucial transition in NLP for social media, moving beyond harm detection toward understanding what communities actively reward and celebrate.
- Exposing the false-negative gatekeeper syndrome: Prompted LLMs discard over 80% of community-approved content under vanilla settings because their safety-tuned alignment equates "value" with sanitized, formal, and lengthy text, penalizing punchy, relatable discourse.
- Practical viability of decentralized SLMs: Demonstrating that 4B parameter models tuned on community data beat massive commercial LLMs provides strong empirical backing for modular, low-latency, privacy-preserving curation architectures deployed via local adapters (e.g., LoRA routing).
Limitations & Future Work¶
- Proxy limitations of Reddit scores: Upvotes serve as an established behavioral proxy for collective approval but remain vulnerable to non-linguistic confounders such as submission timing, thread depth, and bandwagon effects.
- Temporal and platform bounds: Datasets originate from Reddit during 2016–2017, capturing the demographic and stylistic realities of that specific period, which may not translate identically to contemporary platforms like Discord or TikTok.
- Future trajectories: Promising extensions include formulating the curation problem as end-to-end listwise ranking, integrating structural posting metadata, and developing dynamic mixture-of-experts (MoMoE) architectures with community-specialized routing.
Related Work & Insights¶
- vs DialogRPT (Gao et al., 2020): While DialogRPT is pretrained on Reddit feedback, its platform-wide ranking objective ignores localized community variations, achieving only 0.59 AUROC on VASTU and underscoring the necessity of community conditioning.
- vs CommunityBench (Lin & Wei, 2026): CommunityBench benchmarks model generation against community personas, whereas VASTU tackles the foundational detection and reward modeling task of recognizing valued human discourse in situ.
- vs ValueScope (Park et al., 2024): ValueScope maps social interaction norms onto high-level value taxonomies, while VASTU evaluates direct computational discriminators across model architectures for downstream curation systems.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First systematic formulation and benchmark of community-conditioned preference prediction on human discourse.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 model paradigms, 15 subreddits, 75k comments, complete with ablations, SHAP stability, and qualitative failure coding.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid narrative flow, impeccable empirical framing, and highly insightful error taxonomy.
- Value: ⭐⭐⭐⭐⭐ Provides foundational empirical evidence and benchmarks for community-aware reward modeling, feed reranking, and pluralistic alignment.