RedNote-Vibe: A Lifestyle Social Media Dataset for Detecting AI-Generated Text and Analyzing Its Impact¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ydli-ai/RedNote-Vibe
Area: LLM Agent
Keywords: AIGT detection, social media benchmark, psycholinguistic features, user engagement, cross-model generalization
TL;DR¶
The authors introduce RedNote-Vibe, the first longitudinal Chinese lifestyle social media dataset spanning 2020 to 2025 across 148k posts and 17 LLMs, accompanied by the PsychoLinguistic AIGT Detection Framework (PLAD) that extracts 31 theory-grounded cognitive features to achieve superior generalization to unseen models and uncover how strategic human-AI collaboration drives peak engagement.
Background & Motivation¶
The rapid proliferation of Large Language Models (LLMs) has fundamentally transformed digital content creation, leading to an unprecedented influx of AI-Generated Text (AIGT) across public discourse. Consequently, developing robust detection systems has become a crucial research frontier. However, existing benchmarks and detection pipelines predominantly center on formal, knowledge-driven domains such as academic literature, news corpora, and informational discussion forums like Reddit and Quora. In these structured settings, text value hinges on factual precision and logical consistency, where machine artifacts often manifest in distinct token frequencies or syntactic anomalies. In sharp contrast, lifestyle-sharing and social-commerce platforms like RedNote (Xiaohongshu)โserving hundreds of millions of monthly active usersโprioritize personal narratives, subjective emotions, and consumer recommendations. Here, content success is measured not by encyclopedic accuracy, but by audience resonance reflected in likes, comments, and collections. The emergence of LLMs enables automated generation of sensationalized or fabricated personal stories, directly threatening the foundational authenticity and mutual trust of community ecosystems.
This emerging landscape poses two critical hurdles that prior research fails to address. First, existing datasets are largely static snapshots devoid of temporal progression and lack fine-grained engagement metadata, preventing researchers from tracking how generative text naturally evolves alongside shifting platform dynamics. Second, as LLM generation aligns closer to colloquial human patterns, superficial linguistic artifacts such as n-gram frequencies and syntactic parse trees are rapidly neutralized. Furthermore, conventional black-box fine-tuned classifiers (e.g., RoBERTa and BERT variants) often suffer from severe performance collapse when confronted with unobserved model families or novel reasoning paradigms, while offering zero human-interpretable insight into why a given post is classified as synthetic.
These dual deficiencies reveal a fundamental bottleneck: surface-level statistical patterns are inherently fragile against modern prompting and continuous model iteration. Instead, reliable and interpretable detection requires probing the deep cognitive and psycholinguistic mechanisms underlying human language productionโsuch as autobiographical emotional grounding, narrative perspective coordination, and conversational stanceโwhich statistical sampling from autoregressive LLMs fails to replicate. Core idea: construct RedNote-Vibe, a comprehensive five-year lifestyle social media dataset with complete engagement metrics and 17 LLM variants, and introduce the PsychoLinguistic AIGT Detection Framework (PLAD) that leverages 31 theory-grounded cognitive features to deliver robust, interpretable detection and quantify user engagement dynamics.
Method¶
Overall Architecture¶
The RedNote-Vibe benchmark and the PLAD framework form an integrated end-to-end analytical pipeline. At the data layer, 148,000 lifestyle notes were retrieved across ten core domains (Career, Wellness, Travel, Health, Food, Pets, Education, Sports, Fashion, and Relationships) spanning October 2020 through July 2025, with text filtered above 100 Chinese characters and enriched with timestamp and interaction metrics (likes, comments, collections). The corpus is partitioned into: (1) a pre-LLM human-authored ground-truth set published prior to November 2022; (2) an AI-generated set containing parallel variants synthesized by 17 leading LLMs across 6 major providers using the pre-LLM notes as seeds; and (3) an unlabeled exploration set of 90,000 post-LLM real-world posts.
At the methodology layer, PLAD departs from black-box embedding classification by projecting raw text into a 31-dimensional psycholinguistic feature space categorized into four cognitive dimensions. The feature extraction pipeline couples direct statistical computations with standardized proxy-LLM scoring guided by Chain-of-Thought (CoT) reasoning over expert-designed rubrics. Finally, interpretable gradient-boosted decision trees (CatBoost and XGBoost) perform multi-granular detection tasks while enabling game-theoretic SHAP feature attribution to explain content-level discrimination and author-level engagement drivers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-domain Lifestyle Post Input<br/>Text content + Timestamps + Engagement metrics"] --> B["Dataset Construction & Variant Alignment<br/>Pre-LLM human seeds / 17 LLM pool / Post-LLM set"]
B --> C["Psycholinguistic Feature Space Quantification<br/>4 cognitive dimensions spanning 31 features"]
C --> D["Proxy-LLM Semantic Scoring & Statistical Extraction<br/>Standardized JSON schema + CoT expert rubrics"]
D --> E["Tree-based Hierarchical Classification & Attribution<br/>Binary AIGT / 6-way provider / 17-way model"]
Key Designs¶
1. Dataset Construction & Variant Alignment: unpolluted seed-driven generation under lifestyle stylistic constraints
Conventional AIGT collection pipelines rely on unconstrained direct prompting, which generates artificial texts that differ drastically from authentic social media posts in length, layout, and colloquial formatting. To simulate realistic adversarial conditions, RedNote-Vibe strictly curates human seed posts published before November 2022, entirely eliminating synthetic contamination from the training base. For each seed note and its domain, a structured prompt forces the LLM pool (spanning DeepSeek-R1, DeepSeek-V3, GPT-4o, Claude-3.7-sonnet, Gemini-2.5-flash, Qwen-2.5-32B, GLM-4, etc.) to: (1) maintain thematic and domain alignment without verbatim copying; (2) authentically mimic lifestyle writing conventions, including colloquial syntax, expressive punctuation, and minor disfluencies typical of social platforms; (3) preserve natural length distributions; and (4) output strictly in a JSON schema containing title and content fields. This rigorous alignment guarantees that the resulting synthetic variants represent challenging, human-indistinguishable adversarial benchmarks.
2. Psycholinguistic Feature Space Quantification: grounding detection in 4 cognitive dimensions and 31 features
To counteract the inevitable decay of surface-level classifiers, PLAD establishes a 31-dimensional feature vector rooted in established psychological and psycholinguistic theories: - Emotional & Social Grounding: assesses affective depth and relational presence, quantifying Emotional Intensity (Gross's emotion regulation theory), Personal Emotional Grounding (Conway's autobiographical memory model), Sensory Detail Richness, Social Connectedness, Empathetic Engagement, Interactive/Dialogic Stance (Bakhtin's dialogism), and Unique Emoji Ratio / Emoji Density; - Cognitive Architecture: captures perspective-taking and organizational logic, including Perspectival Complexity (Baker-Brown's integrative complexity), Narrative Structure Flexibility (Labov & Waletzky narrative analysis), Dialectical Argumentation Strength (Toulmin argument model), Self-Correction (Hyland's metadiscourse), Axiological Coherence (Graham's moral foundations theory), Temporal Orientation/Integration (Reichenbach's tense logic), and sentence/word length statistics; - Lexical Identity & Stylistic Signature: measures idiolect uniqueness and natural irregularities, encompassing Lexical-Stylistic Personalization, Prosodic Rhythm Consistency (Hallidayan functional grammar), Imperfection (Clark & Fox Tree's discourse disfluency theory), Rhetorical Sophistication, Punctuation / Number Ratios, Type-Token Ratio (TTR), Word Frequency Entropy, and Word Burstiness; - Cohesion & Textual Flow: evaluates textual progression and repetition patterns, measuring Lexical Cohesion, Inter-Sentential Semantic Similarity, Immediate Repetition Density, Phrasal Repetition Frequency, and Sentence Burstiness. Pairwise correlation analysis across all 465 feature pairs shows an average absolute Pearson correlation of only 0.1884 (with 80% having \(|r| < 0.3\)), confirming that the features provide mutually orthogonal, complementary signals.
3. Proxy-LLM Semantic Scoring & Statistical Extraction: chain-of-thought guided psychometric measurement
Traditional psycholinguistic software (such as LIWC) relies on static dictionary keyword matching, which fails to evaluate discourse context, subtle metaphors, and pragmatic tone. PLAD resolves this via a hybrid extraction pipeline: statistical indices (e.g., TTR, punctuation ratios, emoji metrics) are computed deterministically, while complex semantic constructs are scored using an advanced proxy LLM (qwen-turbo) via dynamic prompt assembly. Each dimension is governed by a formal JSON schema specifying: (1) theoretical construct definitions; (2) anchored 0.0-to-1.0 scoring rubrics with concrete behavioral markers; (3) key textual indicators; and (4) human-validated few-shot exemplars with detailed Chain-of-Thought reasoning steps. Benchmark validation confirms that proxy-LLM scores achieve high Spearman correlations (\(r = 0.720 - 0.785\)) against expert human annotations on concreteness, sentiment, and formality, drastically surpassing dictionary-based LIWC (\(r = 0.429 - 0.570\)).
4. Tree-based Hierarchical Classification & Attribution: interpretable boosting models with SHAP engagement modeling
The extracted 31-dimensional feature representation \(f(x) \in \mathbb{R}^{31}\) is passed to gradient-boosted decision trees (CatBoost and XGBoost) trained via multi-class cross-entropy minimization: $$ \mathcal{L}{\text{CE}} = - \sum y_i \log(p_i) $$ Decision trees provide superior tabular inductive bias over shallow neural nets on numerical cognitive scores while eliminating black-box opacity. More importantly, this formulation integrates seamlessly with Shapley Additive exPlanations (SHAP) from cooperative game theory. SHAP computes exact, locally consistent feature attribution values, enabling the framework to explain why specific models exhibit distinct stylistic footprints and uncover the psycholinguistic drivers governing audience likes, comments, and bookmarking behaviors.
Key Experimental Results¶
Main Results¶
The authors benchmark three hierarchical tasks derived from RedNote-Vibe: binary AIGT detection (Human vs. AI), 6-way AI provider identification (OpenAI, Google, Anthropic, DeepSeek, Alibaba, Zhipu), and fine-grained 17-way model attribution. With an 8:1:1 train/validation/test split, macro-averaged precision, recall, and accuracy are reported below (extracted directly from Table 4 of the original paper):
| Category | Method | 17-way Model Acc. (%) | 17-way Prec. (%) | 6-way Provider Acc. (%) | 6-way Prec. (%) | Binary AIGT Acc. (%) | Binary Prec. (%) | Binary Recall (%) |
|---|---|---|---|---|---|---|---|---|
| Statistical Baselines | StyloAI (2024) | 23.20 | 21.68 | 41.21 | 37.33 | 77.90 | 75.23 | 72.55 |
| Statistical Baselines | Ullah et al. (2024) | 26.01 | 23.83 | 46.41 | 41.05 | 78.59 | 75.91 | 73.83 |
| Statistical Baselines | Binoculars (2024) | 24.05 | 21.13 | 43.72 | 40.48 | 77.89 | 72.07 | 74.17 |
| Pretrained Transformers | FT. BERT (2019) | 33.76 | 32.97 | 50.47 | 44.38 | 88.24 | 85.91 | 88.29 |
| Pretrained Transformers | FT. ALBERT (2019) | 31.65 | 32.26 | 44.34 | 41.98 | 84.58 | 83.59 | 79.63 |
| Pretrained Transformers | FT. RoBERTa (2019) | 32.21 | 35.20 | 47.64 | 38.22 | 89.52 | 87.85 | 88.88 |
| Specialized Pipelines | Sniffer (2023) | 30.04 | 30.63 | 45.65 | 44.39 | 84.45 | 82.55 | 81.30 |
| Specialized Pipelines | POGER (2024) | 31.88 | 31.28 | 47.16 | 45.41 | 84.89 | 79.17 | 82.03 |
| Specialized Pipelines | LLM-Idiosyncrasies (2025a) | 32.31 | 32.91 | 49.96 | 49.81 | 89.07 | 88.09 | 90.15 |
| Proposed Framework | PLAD-XGBoost | 34.04 | 32.11 | 53.30 | 50.73 | 87.79 | 86.45 | 85.31 |
| Proposed Framework | PLAD-CatBoost | 36.94 | 35.87 | 51.89 | 50.06 | 89.62 | 88.70 | 87.28 |
In fine-grained 17-way model identification, PLAD-CatBoost outperforms the strongest transformer baseline (FT. BERT) by 3.18% in accuracy and 2.90% in precision. In binary detection, PLAD achieves the top overall accuracy (89.62%) and precision (88.70%).
Ablation Study & Generalization to Unseen Models¶
To assess the structural necessity of each cognitive pillar, dimension-level ablations were conducted using CatBoost (reported as Macro-average F1 (%) in Table 6):
| Configuration | 17-way Model F1 (%) | 6-way Provider F1 (%) | Binary AIGT F1 (%) | Notes |
|---|---|---|---|---|
| CatBoost (Full 31 Features) | 36.16 | 48.66 | 87.98 | Complete psycholinguistic representation |
| w/o Emotional Dimension | 27.68 (-8.48) | 43.22 (-5.44) | 86.32 (-1.66) | Severe drop in fine-grained model attribution |
| w/o Cognitive Dimension | 31.40 (-4.76) | 43.56 (-5.10) | 86.44 (-1.54) | Loss of structural and argumentative cues |
| w/o Lexical & Stylistic Dimension | 26.81 (-9.35) | 38.66 (-10.00) | 85.00 (-2.98) | Largest performance degradation across all tasks |
| w/o Cohesion Dimension | 32.29 (-3.87) | 44.76 (-3.90) | 86.57 (-1.41) | Degradation in discourse progression tracking |
To evaluate zero-shot resilience against newly deployed models, the latest GPT-o3 and Gemini-2.5 were withheld from training data while keeping older models from OpenAI and Google in the 6-way provider task (Table 5):
| Method | Seen Accuracy (%) | GPT-o3 Unseen Recall (%) | Gemini-2.5 Unseen Recall (%) |
|---|---|---|---|
| BERT-base | 42.42 / 45.23 | 25.35 | 52.31 |
| ALBERT-base | 42.73 / 42.61 | 28.31 | 50.00 |
| LLM-Idiosyncrasies | 44.34 / 41.71 | 22.54 | 53.85 |
| PLAD (Ours) | 46.46 / 43.72 | 56.34 (+31.0% vs. BERT) | 58.46 (+4.6% vs. LLM-Idio) |
PLAD detects unseen reasoning-heavy architectures like GPT-o3 with 56.34% recall, compared to only 22.54%โ28.31% for transformer baselines, confirming that psycholinguistic signatures generalize robustly across model generations.
Key Findings¶
- Synthetic text exhibits unnatural rhythmic smoothness and suppressed human imperfection: Feature importance analysis identifies Prosodic Rhythm Consistency, Type-Token Ratio (TTR), and Imperfection as the top three predictors. LLMs produce overly regular prosody and maintain artificially elevated lexical diversity, while strictly eliminating natural hesitation markers, spoken false starts, and conversational self-corrections (\(p < 0.001\)), creating a detectable "over-polished" footprint.
- Audience cognitive investment dictates the human-AI engagement gap: In low-barrier interactions (likes), AI posts perform reasonably well due to attractive formatting. However, in deeply personal experiential domains (Travel, Career, Relationships), human-authored posts systematically dominate in median engagement. Furthermore, AI content distributions are tightly clustered around the mean and rarely produce breakout viral posts. Crucially, the gap narrows for high-investment interactions (comments), indicating distinct interaction dynamics.
- AI-augmented authors outperform both purists and full AI-adopters: Author-level tracking across 829 creators (\(\ge 4\) posts) categorizes users into Traditional (68.7%), AI-reliant (27.7%), and AI-augmented (3.5%). Creators who strategically combine generative AI draft generation with human editorial refinement achieve substantially higher likes, collections, and comments than either pure human or fully automated pipelines.
Highlights & Insights¶
- Operationalizing psychological theory into rigorous computational detection: Rather than chasing ephemeral statistical artifacts, PLAD translates established cognitive paradigmsโsuch as Conway's autobiographical memory and Halliday's functional grammarโinto structured evaluation rubrics, creating the first interpretable, theory-grounded AIGT framework for social platforms.
- Exposing the inflated accuracy of academic benchmarks: Prior benchmarks reported \(>98\%\) detection rates on naive, unconstrained LLM outputs. By imposing colloquial, seed-aligned prompting that mimics lifestyle sharing, RedNote-Vibe reveals that realistic in-the-wild detection accuracy drops to ~89%, providing an authentic stress-test for commercial defense systems.
- Disentangling the linguistic determinants of social media engagement: Through game-theoretic SHAP modeling, the paper shows that passive likes are driven by surface-level punctuation and word frequency entropy; collections depend on axiological consistency and logical coherence; and active comments require strong lexical personalization, empathetic engagement, and dialogic stance.
Limitations & Future Work¶
- Single-platform and language specificity: The empirical dataset is harvested exclusively from Chinese-language RedNote. While the 31 psycholinguistic constructs are conceptually language-agnostic, cross-lingual and cross-platform transferability to ecosystems like Instagram or Reddit remains to be empirically verified.
- Computational overhead of proxy-LLM scoring: Relying on an auxiliary LLM (qwen-turbo) to extract nuanced semantic scores introduces latency and API cost compared to single-pass RoBERTa classifiers, posing challenges for real-time, high-throughput content moderation pipelines.
- Absence of multimodal visual signals: RedNote is fundamentally an image-and-video-centric platform where cover aesthetics, visual typography, and cross-modal harmony strongly influence post virality and synthetic authenticity; extending PLAD into vision-language multimodal analysis represents an essential next step.
Related Work & Insights¶
- vs. SM-D (Sun et al., 2025b): SM-D explores informational discussion threads on Reddit and Quora; RedNote-Vibe provides the first dedicated Chinese lifestyle social media dataset, tracking 5 years of longitudinal content evolution and user interaction metrics.
- vs. Sniffer & LLM-Idiosyncrasies (Li et al., 2023; Sun et al., 2025a): While previous source attribution frameworks rely on hidden activation embeddings that overfit to specific model architectures, PLAD's cognitive feature space maintains strong zero-shot transferability when encountering next-generation reasoning models like GPT-o3.
- vs. Binoculars & DetectGPT (Hans et al., 2024; Mitchell et al., 2023): Zero-shot perplexity curvature methods degrade sharply when LLM outputs are constrained to emulate conversational imperfections; PLAD bypasses perplexity thresholds by measuring deep discourse coherence and affective regulation.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering lifestyle social media AIGT dataset coupled with a highly innovative 31-feature psycholinguistic detection paradigm.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 17 LLMs, 5-year longitudinal analysis, zero-shot unseen model testing, and detailed SHAP attribution.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative progression, rigorous psycholinguistic theoretical grounding, and elegant empirical visualizations.
- Value: โญโญโญโญโญ Sets a new benchmark for social AIGT detection while offering invaluable strategic insights into creator behavior and algorithmic platform governance.