Skip to content

Psychologically Potent, Computationally Invisible: LLMs Generate Social-Comparison-Eliciting Posts They Fail to Detect

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zhao4hua4/XHS-SCoRE
Area: Social Computing
Keywords: Social comparison, Xiaohongshu text, generation-detection dissociation, reader-grounded evaluation, computational invisibility

TL;DR

This paper introduces XHS-SCoRE, a reader-grounded benchmark evaluating social comparison elicitation from Xiaohongshu posts, discovering that while LLMs successfully generate posts with strong psychological potency that shift human standing perceptions and affective states, prompted classifiers reliably fail to detect the same relational cues due to severe neutralization and directional skew.

Background & Motivation

Social media operates as an omnipresent arena for social comparison, where individuals continually assess their own standing by evaluating the lifestyle, achievements, appearance, and consumption displayed by others. Such comparisons frequently trigger envy, anxiety, depression, and general negative affect. On lifestyle-oriented platforms like Xiaohongshu (RedNote), comparison cues rarely appear as overt, explicit comparatives; instead, they are subtly woven into everyday narrative framing, reported dialogue, and culturally specific expressions of aspiration or distress. This poses a fundamental challenge for natural language processing: can computational models recover the reader-perceived direction of social comparison elicited by naturalistic posts from an authentic first-person perspective? The issue is particularly pressing as generative AI models are increasingly deployed to mass-produce persuasive, affect-calibrated platform content at scale.

Traditional NLP approaches to psychological and social constructs predominantly treat labels as static, author-centric, or text-inherent properties, ignoring the fact that social comparison is fundamentally a relational appraisal dependent on the reader's background and social position. While large language models are widely adopted as zero-shot or few-shot classifiers for social media auditing and computational social science, their reliability remains fragile when detecting pragmatic, culturally situated reader experiences. This creates a severe governance risk: generative models can easily synthesize highly potent, comparison-inducing content targeting specific demographics, yet prompted detection pipelines may systematically overlook or mischaracterize these exact cues in natural feeds, creating dangerous blind spots in content moderation and digital well-being auditing.

To address this generation–detection gap, this paper introduces a reader-grounded elicitation task conditioned on a defined demographic profile, establishing a balanced benchmark across upward comparison (UPWARD), downward comparison (DOWNWARD), and no clear comparison (NEUTRAL), and empirically validating the psychological potency of generated stimuli through controlled human experiments. Core idea: By pairing the reader-grounded benchmark XHS-SCoRE with a controlled behavioral validation experiment, this work exposes a fundamental "generation–detection dissociation" where LLMs can generate psychologically potent social-comparison posts yet systematically fail to recover the same relational cues under prompted classification, formalizing the mechanisms of neutralization collapse and directional skew.

Method

Overall Architecture

The methodology integrates an end-to-end framework spanning reader-grounded natural corpus collection and reliability calibration, multi-tiered classifier benchmarking, corpus-constrained controllable generation, and a randomized controlled psychological experiment. First, natural Xiaohongshu text posts are gathered via a first-person immediate-response protocol to form the class-balanced XHS-SCoRE benchmark, calibrated against human re-rating stability and ambiguity baselines. Second, state-of-the-art prompted LLMs and supervised Chinese encoder baselines are evaluated on test splits, followed by an in-depth error topology analysis. Third, lexical and semantic frame markers extracted from corpus keyness analysis are translated into structural prompt constraints for controllable stimulus synthesis. Finally, a between-subjects human pilot study using jsPsych measures shifts in perceived standing and comparison-related affect, empirically grounding the psychological potency of the generated content against detector failures.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Natural Xiaohongshu Text-Only Posts"] --> B["First-Person Immediate-Response Collection Protocol"]
    B --> C["XHS-SCoRE Benchmark & Reliability Calibration<br/>UP / NEU / DOWN Balanced 4,632 Posts Each"]
    C --> D["Dual-Track Classifier Evaluation & Error Topology"]
    D -->|Corpus Keyness & Semantic Frame Analysis| E["Corpus-Constrained Target-Directed Text Generation"]
    E --> F["Randomized Controlled Human Behavioral Pilot<br/>Standing Appraisal, Assimilative Emotion & Affect Shifts"]
    F --> G["Confirmation of Generation Potency vs. Detection Dissociation"]

Key Designs

1. First-Person Immediate-Response Collection Protocol: Grounding Relational Appraisal Over Author Intent

Standard annotation frameworks often assume a population-invariant semantic consensus or focus on author intent. However, social comparison represents a relational inference made by the reader regarding relative social standing, heavily influenced by the reader's socio-cultural background. To capture this construct, the authors established a single-collector immediate-response protocol: 67 active Xiaohongshu users (female undergraduate students aged 18–24 from Hong Kong universities with mainland Chinese backgrounds, constraining demographic variance) collected 10 posts per category (UPWARD, DOWNWARD, NEUTRAL) daily over 7 consecutive days on their personal devices, yielding a balanced dataset of 13,916 posts (2,452,665 Chinese characters). Inclusion strictly enforced text-only self-containment, discarding multimodal posts dependent on images or video. Calibration on 210 randomized posts with two independent demographic-matched raters revealed 63.2%–68.4% raw in-group agreement and ~82% delayed (24h+) test-retest stability. Crucially, human raters re-labeled 20.4% of UP and 17.0% of DOWN posts as NEU under text-only browsing, establishing an essential human ambiguity baseline for diagnosing automated model collapse.

2. Dual-Track Classifier Evaluation & Error Topology Analysis: Uncovering Neutralization and Directional Skew

To contrast prompt-based reasoning with in-domain representations, the authors deployed a dual-track evaluation: four prompted LLMs across parameter and capability tiers (frontier closed-source GPT-5, open-source MoE Qwen3-235B-A22B-Instruct, lightweight MoE Qwen3-30B-A3B-Instruct, and cost-efficient GPT-4.1-nano) were evaluated in a first-person zero-shot JSON-output format, alongside three fine-tuned Chinese pre-trained encoders (Chinese-BERT-wwm-ext, Chinese-RoBERTa-wwm-ext, Chinese-MacBERT-base). Diagnostic prompt variants (Persona-primed, Few-shot, and Cue-explicit) probed robustness across prompting regimes. The error analysis identified two pervasive structural failures in prompted LLMs: first, massive "neutralization," where comparison-eliciting texts are overwhelmingly collapsed into NEUTRAL (e.g., GPT-5 predicted 60.1% of instances as NEU; Qwen3-30B predicted 65.9%), drastically exceeding human ambiguity baselines; second, severe directional asymmetry, where DOWNWARD comparison cues (hardship narratives, low agency, parental conflict) are disproportionately washed out compared to UPWARD cues, with Qwen3-235B uniquely displaying an excessive upward-skew profile (predicting 42.5% UP with 0.670 recall, but only 0.282 DOWN recall).

3. Corpus-Constrained Controllable Generation & Human Behavioral Validation: Decoupling Potency from Detectability

To rigorously test whether models can realize constructs they cannot reliably classify, the authors conducted Wmatrix 7 keyness and log-likelihood analysis on the corpus, isolating distinct linguistic signatures: UPWARD posts feature aspirational mobility, luxury travel/dining, and extreme evaluative adjectives, whereas DOWNWARD posts heavily concentrate on family disputes, parental control verbs (scolding, coercing), victim-positioned passive constructions ("被"), and dense negations. These rules were compiled into structural generation constraints for GPT-4.1-nano to synthesize realistic, 170–200 character Xiaohongshu posts. Subsequently, a between-subjects human pilot (N=29 demographic-matched users: UP N=10, DOWN N=9, NEU N=10) in jsPsych presented 7 randomized condition-specific posts, measuring perceived relative standing, self-other similarity, domain-specific comparison emotions, and PANAS affect scales. The empirical behavioral data confirmed that generated posts strongly shifted perceived standing and induced expected emotional profiles, providing definitive psychological evidence of generative potency despite detector fragility.

Loss & Training

Supervised encoder baselines (Chinese-BERT-wwm-ext, Chinese-RoBERTa-wwm-ext, Chinese-MacBERT-base, each containing 102,269,955 trainable parameters) were fine-tuned on the training split using cross-entropy loss augmented with label smoothing (0.10–0.15). Training was conducted in FP32 on a single 24GB RTX 4090 D GPU with batch size 16 and gradient accumulation steps 2 (effective batch size 32), capped at a maximum sequence length of 512 tokens. Learning rates were tuned to \(2.0 \times 10^{-5}\) (BERT), \(2.5 \times 10^{-5}\) (RoBERTa), and \(3.0 \times 10^{-5}\) (MacBERT) using linear warmup ratios (0.15–0.20) and weight decay of 0.01 over 15 epochs, checkpointed by validation Macro-F1.

Key Experimental Results

Main Results

On the 2,783 balanced test posts of XHS-SCoRE, prompted zero-shot LLM classifiers were benchmarked against supervised Chinese encoder models.

Model Model Type Accuracy (Acc) Macro-F1 Recall UP Recall NEU Recall DOWN Predicted (as) NEU
GPT-5 Frontier Closed LLM 0.521 0.518 0.410 0.752 0.402 0.601
Qwen3-235B-A22B Open Large MoE 0.491 0.480 0.670 0.522 0.282 0.425
GPT-4.1-nano Lightweight Commercial LLM 0.469 0.469 0.379 0.630 0.397 0.558
Qwen3-30B-A3B Lightweight Open MoE 0.430 0.400 0.364 0.748 0.179 0.659
CN-BERT-wwm Supervised Encoder SFT 0.670 0.671 0.666 0.636 0.708 0.360
CN-RoBERTa-wwm Supervised Encoder SFT 0.680 0.679 0.695 0.585 0.759 0.307
CN-MacBERT-base Supervised Encoder SFT 0.665 0.665 0.633 0.631 0.730 0.349

Ablation Study

Diagnostic evaluation of alternative prompting strategies (Zero-Shot ZS, Persona-Primed PP, Few-Shot FS, Cue-Explicit CE) alongside human ambiguity baselines and supervised references, tracking comparison-to-neutral collapse (\(UP \to NEU\) and \(DOWN \to NEU\)).

Model / Baseline Prompt Condition Macro-F1 Total Pred. NEU (%) \(UP \to NEU\) (%) \(DOWN \to NEU\) (%)
Human Re-rating Baseline Text-only Human Check — — 20.4 17.0
CN-RoBERTa-wwm Supervised Encoder SFT 67.9 30.7 18.6 15.1
GPT-5 Zero-Shot (ZS) 51.8 60.1 53.3 51.8
GPT-5 Persona-Primed (PP) 51.6 65.9 62.2 51.9
GPT-5 Few-Shot (FS) 50.2 59.4 47.8 56.2
GPT-5 Cue-Explicit (CE) 55.1 33.4 26.0 27.8
GPT-4.1-nano Zero-Shot (ZS) 46.9 55.8 54.2 50.4
GPT-4.1-nano Persona-Primed (PP) 38.0 77.4 79.9 65.8
GPT-4.1-nano Few-Shot (FS) 38.1 69.8 62.0 72.0
GPT-4.1-nano Cue-Explicit (CE) 44.9 68.2 68.6 57.8
Qwen3-235B Zero-Shot (ZS) 48.0 42.5 29.6 45.8
Qwen3-235B Persona-Primed (PP) 46.0 66.7 50.6 68.0
Qwen3-235B Few-Shot (FS) 51.4 54.5 44.8 49.0
Qwen3-235B Cue-Explicit (CE) 54.8 30.8 24.6 24.0
Qwen3-30B Zero-Shot (ZS) 40.0 65.9 62.1 60.7
Qwen3-30B Persona-Primed (PP) 47.8 49.1 40.6 46.1
Qwen3-30B Few-Shot (FS) 46.9 39.7 31.2 36.5
Qwen3-30B Cue-Explicit (CE) 53.7 28.0 25.7 19.5

Key Findings

  • Statistically Significant Generation Potency: The human behavioral pilot revealed that generated stimuli elicited strong standing manipulation effects (adjusted \(R^2 = 0.570, p < 0.001\)), with DOWN assimilative emotions surging in the DOWN condition (\(M=6.33, SD=1.41\)) compared to other conditions (\(M=1.85, SD=1.90\); \(t(27) = 6.312, p < 0.001, d = 2.534\)). Undesirable comparison emotions followed a strict monotonic progression (\(DOWN > NEU > UP\); \(F(2, 26) = 14.26, p < 0.001, \eta^2_p = 0.52\)), proving the psychological reality of generated texts.
  • Pervasive Neutralization Collapse: Prompted models routinely default to NEUTRAL on active comparison posts. Zero-shot GPT-5 collapsed 53.3% of UP and 51.8% of DOWN posts into NEU, while Qwen3-30B collapsed over 60% of both, far surpassing the human ambiguity baseline (20.4% UP, 17.0% DOWN). This failure erases socially consequential relational signals at the detector output.
  • Acute Blind Spot for Downward Comparison (DOWN): Downward comparison cues (grounded in conflict and constrained agency) are particularly inaccessible to prompted LLMs. Zero-shot DOWN recall dropped as low as 0.179 on Qwen3-30B and 0.282 on Qwen3-235B, compared to 0.759 on supervised RoBERTa.
  • Diagnostic Scaffolding via Cue-Explicit Prompts: Incorporating an explicit linguistic cue inventory significantly mitigated neutralization, elevating GPT-5 Macro-F1 to 55.1% and reducing predicted NEU from 60.1% to 33.4%. This demonstrates that the relational signal is textually present and learnable, but remains structurally invisible under routine prompt formulations.

Highlights & Insights

  • Introduction of the "Reader-Grounded Elicitation" Paradigm: Moves NLP beyond rigid, author-centric or text-intrinsic sentiment labeling to formalize how socially situated readers perceive relative standing from implicit platform discourse.
  • Empirical Demonstration of Generation–Detection Dissociation: Couples computational text classification with an experimental behavioral validation loop, proving that high fluency and psychological efficacy in LLM generation do not imply reliable detection of the same construct.
  • Critical Implications for Platform Content Governance: Demonstrates that off-the-shelf prompted LLM auditors risk dangerous false-negative rates, systematically classifying psychologically harmful comparison posts as "neutral" content.

Limitations & Future Work

  • Text-Only Scope on a Multimodal Platform: Xiaohongshu posts heavily rely on image curation, layout, and visual aesthetic cues that may amplify or modify comparison elicitation; future benchmarks should incorporate multimodal features.
  • Demographic Homogeneity: Evaluators and participants were recruited from a specific demographic cohort (female mainland-origin students in Hong Kong) to minimize group variance; cross-cultural, cross-age, and gender-diverse generalizability requires broader empirical auditing.
  • Pilot Scale of Behavioral Validation: While statistically robust (\(N=29\)) for construct validation, longitudinal tracking of real-world exposure dynamics is necessary to quantify long-term psychological impacts.
  • vs. Classical Text-Based Psychological Measurement (e.g., LIWC, Sap et al., 2020): Prior work primarily extracts author-level psycholinguistic proxies or demographic markers, whereas this work pioneers reader-conditioned relational standing appraisal.
  • vs. LLMs in Computational Social Science (Ziems et al., 2024): Extends beyond reporting aggregate classification accuracy by exposing qualitative failure topologies (neutralization and directional skew) and contrasting generative potency with classification fragility.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First to formalize the reader-grounded social comparison elicitation task and demonstrate generation-detection dissociation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Includes a 13.9k post benchmark, dual-track modeling across 7 architectures, comprehensive error topologies, and a controlled human behavioral validation pilot.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous psycholinguistic operationalization, and precise conceptual exposition.
  • Value: ⭐⭐⭐⭐⭐ Highly impactful for automated platform governance, algorithmic auditing, and digital mental health research.