Skip to content

towards detecting ai assisted responses in online surveys

content_hash: b9b93fab3c141473 ---# Towards Detecting AI-Assisted Responses in Online Surveys

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mike-qz-wang/ASURRE
Area: Social Computing and Social Media
Keywords: Online Surveys, Machine-Generated Text Detection, Agentic Simulation, Behavioural Cues, Zero-Shot Aggregation

TL;DR

Addressing the failure of existing machine-generated text (MGT) detectors against persona-grounded agentic survey completion (AUROC ~0.61), this paper introduces ASURRE, a benchmark spanning three real-world surveys and three AI usage strategies, and proposes SPABD, a training-free detector leveraging four respondent-level behavioural cues anchored on just five verified human responses that boosts average agentic AUROC to 0.75.

Background & Motivation

Online surveys serve as a foundational instrument across the social sciences for measuring human attitudes, behaviours, and complex social phenomena. However, the pervasive availability of generative large language models (LLMs) poses an existential threat to the validity of survey-based empirical research: motivated by desires to minimize cognitive effort, maximize financial compensation on crowdsourcing platforms, or manipulate measurement outcomes, participants increasingly rely on generative tools to craft or refine open-ended responses. Existing machine-generated text (MGT) detectors are almost exclusively tailored for isolated, long-form continuous text such as Wikipedia articles or news essays, relying heavily on token-level log-probability statistics, curvature metrics, or domain-specific neural classifiers. When deployed against the short, fragmented, and disparate open-text responses typical of multi-field surveys, these conventional per-question detectors degrade severely.

The challenge is exacerbated by a qualitative shift in how AI is utilized in survey environments. Moving beyond naive single-prompt generation, participants can leverage persona-grounded agents that maintain contextual state across an entire survey instrument. By conditioning on a synthesized persona portrait, such agents generate plausible, internally consistent, and naturalistic answers that effectively strip away superficial AI stylistic artifacts. At the individual question level, these outputs are virtually indistinguishable from human writing, pushing the discriminative ability of existing SOTA text detectors to near-chance levels. Yet, while human survey responses exhibit rich cognitive heterogeneity, idiosyncratic length variances, and selective non-response behaviours, LLMs inevitably manifest systematic regularities and cross-field homogenization dictated by their training distributions.

This paper tackles the challenge from a novel perspective: rather than examining isolated response strings, it treats a respondent's complete set of answers as an integrated behavioural trace. Core idea: shift survey AI detection from isolated single-answer text scoring to respondent-level behavioural auditing, deriving four unsupervised behavioural cues that exploit LLM regularities and aggregating them via one-tail robust statistical anchoring on a handful of verified human references.

Method

Overall Architecture

The paper formalizes AI survey completion detection by modeling each respondent \(r\) through their submitted answer set \(R_r = \{(q_k, a_{r,k})\}_{k=1}^{K_r}\), where \(q_k\) represents the prompt text and \(a_{r,k}\) is the corresponding free-text response. In contrast to conventional per-question detectors \(f(q_k, a_{r,k}) \in [0, 1]\), the proposed framework constructs a respondent-level detector \(f(R_r) \in [0, 1]\). To systematically evaluate this setting, the authors introduce the ASURRE benchmark spanning three distinct usage strategies (Revision, Full Generation, and Agentic Completion) and formulate SPABD (Survey-Pool Anchored Behavioural Detection), a training-free aggregator built upon four distinct behavioural cues.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input respondent answer set R_r = {(q_k, a_k)}"] --> B["ASURRE Three-Tier Usage Modeling<br/>Distinguishing Revision, Full Generation, and Agentic Fill"]
    B --> C["Respondent-Level Behavioural Cue Profiling<br/>Extracting minimum length, length CV, QA similarity, answered fields"]
    C --> D["One-Tail Robust Aggregation via SPABD<br/>Median/MAD normalization over K verified humans with max-z selection"]
    D --> E["Output respondent-level AI anomaly Score(r)"]

Key Designs

1. ASURRE Three-Tier Usage Modeling: Characterizing the progression from superficial polishing to agentic completion

Addressing the absence of standardized evaluation corpora for AI-assisted surveys, the authors create ASURRE. The benchmark pairs authentic pre-2020 human responses across three diverse surveys (phd2019, springer2017, osmi_mh)—guaranteeing zero historical AI contamination—with synthetic responses generated under three realistic usage strategies: - LLM Revision: The respondent writes the original text, and an LLM is tasked purely with stylistic polishing while preserving semantic intent, specific facts, and personal stances; - Full Generation: The respondent delegates authoring entirely to an LLM given only the question prompt, producing plausible but ungrounded answers; - Agentic Completion: A sophisticated two-stage agentic simulation powered by Claude. Stage A synthesizes an 8-dimensional persona portrait (affect, engagement level, English proficiency, grammar error level, survey seriousness, writing style, recurring themes, and empirical length habits) from a real respondent's profile. Stage B initializes a fresh agent session that answers the survey question-by-question, maintaining sequential memory, deciding whether to answer or skip (signaled by [SKIP]), and executing a post-generation humanization pass to suppress stereotypical LLM markers. Four prompting modes are evaluated: Natural, Controlled (enforcing heuristic skip rules), Length-restrictive, and Combined.

2. Respondent-Level Behavioural Cue Profiling: Exploiting structural regularities across multi-field responses

Rather than concatenating fields into a single long string, the framework extracts four respondent-level behavioural cues \(c(r)\) using word counting and a lightweight 80M-parameter sentence encoder (all-MiniLM-L6-v2): - Shortest Answer Length \(min\_len(r) = \min_i \ell_i\): While humans frequently offer terse one- or two-word shorthand (e.g., median 2 words in phd2019), LLMs display an intrinsic floor effect by generating complete syntactic clauses (AI median \(\ge 6\) words; expected AI direction \(\uparrow\)); - Length Coefficient of Variation \(length\_cv(r) = \text{std}_i(\ell_i) / \text{mean}_i(\ell_i)\): Humans allocate effort unevenly across prompts (CV ~0.67), whereas LLM generation remains rigidly uniform across fields even when persona-prompted (CV drops to 0.33; expected AI direction \(\downarrow\)); - Mean Question-Answer Semantic Similarity \(mean\_qa_sim(r) = \frac{1}{K_r} \sum_{i=1}^{K_r} \phi(q_i) \cdot \phi(a_i)\): Humans often digress or recount tangential anecdotes, whereas LLMs remain strictly on-topic relative to the prompt text (expected AI direction \(\uparrow\)); - Answered Field Count \(n\_answered(r) = K_r\): Humans routinely skip non-mandatory or fatigue-inducing fields, whereas default conversational models exhibit an over-compliance bias, completing significantly more fields than their human counterparts (e.g., in osmi_mh, humans median 2 vs. AI median 10; expected AI direction \(\uparrow\)).

3. One-Tail Robust Aggregation via SPABD: Guarding against targeted evasion through directional extreme-value anchoring

Individual behavioural cues can be neutralized by targeted prompting (e.g., a skip policy attenuates \(n\_answered\), and length constraints compress \(min\_len\)). To build resilience without requiring expensive per-survey supervised training, SPABD employs an unsupervised, reference-anchored one-tail aggregation. Using a tiny reference set of \(K\) verified human respondents from the target survey pool (default \(K=5\)), the detector estimates the robust location \(\mu_c = \text{median}_{r \in \mathcal{R}} c(r)\) and scale \(\sigma_c = \text{median}_{r \in \mathcal{R}} |c(r) - \mu_c|\) for each cue \(c\). Given the known directional prior \(\text{sgn}(c) \in \{+1, -1\}\), test respondent \(r\) is scored by: $$ \text{Score}(r) = \max_c \left( \text{sgn}(c) \cdot \frac{c(r) - \mu_c}{\sigma_c} \right) $$ By taking the maximum normalized deviation (Max-z), the scoring mechanism ensures that if an adversary manipulates one or two dimensions (e.g., artificially varying word counts), any remaining unperturbed behavioural dimension exhibiting typical LLM regularity will trigger detection.

Key Experimental Results

Main Results

Evaluation across the ASURRE benchmark benchmarks six existing zero-shot and supervised detectors: a prompt-based classifier (Zero-shot), token probability and curvature estimators (DetectGPT, Fast-DetectGPT, Log-Likelihood, Binoculars), and an adversarial classifier (RADAR).

Dataset Evaluation Regime Ours (SPABD) Best Existing Baseline Gain (AUROC) Top Baseline Method
phd2019 Agentic Fill (Mean) 0.753 0.613 +0.140 Binoculars
springer2017 Agentic Fill (Mean) 0.687 0.613 +0.074 Log-Likelihood
osmi_mh Agentic Fill (Mean) 0.813 0.591 +0.222 DetectGPT
All Surveys Agentic Fill (Mean) 0.751 0.606 +0.145 Mixed

Under stringent false positive rate (FPR) constraints essential for operational deployment, SPABD demonstrates a substantial performance advantage: at FPR=5%, SPABD achieves a true positive rate (TPR@5%) of 27.7% across all 12 agentic settings, compared to 12.0% for the best existing baseline. At FPR=1%, SPABD catches 15.5% of AI submissions versus just 4.5% for existing detectors—more than a threefold improvement.

Ablation Study

Extensive ablations analyze reference pool scaling \(K\), aggregation operators, open-ended field counts, and cross-LLM generator robustness.

Configuration / Variable Description Mean AUROC Remarks
Full model (SPABD, K=5) Max-z one-tail, 4 cues 0.751 Standard setting, 50-seed std = 0.09
Scaling K=10 10 human reference samples 0.776 Seed std drops to 0.06
Scaling K=100 100 human reference samples 0.792 Seed std narrows to 0.02
Aggregator: Sum-z / Mean-z Linear unweighted aggregation 0.776 Slightly higher in clean setups; vulnerable to evasion
Aggregator: Trimmed-sum-z Excludes maximum absolute z 0.689 Discards decisive anomaly signals
Supervised Upper Bound Logistic regression (5 human + 5 AI) 0.856 Requires explicit AI labels
Fields Retained 50% First 50% open-text fields kept 0.655 Outperforms full-survey best baseline (0.61)
Fields Retained 25% First 25% open-text fields kept 0.612 Degrades toward single-field MGT baseline

Key Findings

  • Usage strategy dictates detection viability: Full Generation represents the easiest regime for token-level detectors, where Fast-DetectGPT and Log-Likelihood achieve 0.79–0.93 AUROC. Revision exhibits intermediate difficulty (0.55–0.74). Crucially, Agentic Completion collapses all existing text-level detectors to near-random performance (0.50–0.65), while SPABD restores practical discriminability.
  • Robustness of Max-z against adversarial blind attacks: Against a blind attacker prompted to artificially vary answer lengths and skip random fields, Mean-z performance drops from 0.899 to 0.722, whereas Max-z retains 0.839 by exploiting surviving cues (such as \(n\_answered\), which stays at 1.000).
  • Invariance across generator backbones: When evaluating agentic completions produced by gpt-oss-120b and Gemini-3-Flash on OSMI, existing baselines fluctuate erratically and frequently drop below chance (AUROC < 0.30), whereas SPABD maintains consistent detection AUROC between 0.886 and 0.909 across all three backbones.

Highlights & Insights

  • Conceptual Paradigm Shift: The work identifies a fundamental mismatch in applying document-level MGT detectors to survey contexts, demonstrating that while LLM agents can effectively masquerade as humans at the micro-text level, their macro-level survey engagement patterns reveal irreducible behavioural constraints.
  • Lightweight and Non-Invasive: SPABD requires no model retraining, fine-tuning, or repeated prompt perturbations (unlike DetectGPT); it operates efficiently using an 80M sentence encoder and just 5 historical human responses, making it directly deployable as an audit plugin for online survey platforms.

Limitations & Future Work

  • Dependency on Multi-Field Instruments: SPABD's discriminative power relies on having multiple open-ended fields per respondent. In short instruments with only a single open-ended field, cross-answer metrics like length CV and coverage become undefined, and performance reverts to standard short-text MGT detection.
  • Sensitivity to In-Survey Anchor Distribution: Behavioural baselines do not transfer across different survey populations. Utilizing reference responses from an external survey drops average AUROC from 0.751 to 0.646. Practitioners must supply a small, authentic pilot sample from the target instrument.
  • Multilingual Validation: The ASURRE benchmark primarily evaluates English surveys. While demographic analyses indicate no systematic bias against non-native English speakers, behavioural cue dynamics across non-English survey populations warrant further empirical investigation.
  • vs Per-Question Statistical Detectors (DetectGPT, Fast-DetectGPT, Binoculars): These methods analyze token log-probabilities within single texts, which degrade on brief responses and fail against persona-adapted phrasing; SPABD bypasses token-level evasion by aggregating behavioural statistics across a respondent's complete submission.
  • vs Supervised Neural Detectors (RADAR): Adversarially trained classifiers like RADAR suffer severe out-of-domain transfer penalties on survey text (yielding 0.28 AUROC on phd2019); SPABD avoids training mismatch entirely through unsupervised, non-parametric statistical calibration.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering study exposing the threat of persona-grounded agentic survey falsification and proposing respondent-level behavioural auditing.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 3 real-world survey datasets, multiple frontier LLMs, and 4 distinct prompting modes.
  • Writing Quality: ⭐⭐⭐⭐⭐ Precise mathematical and experimental formulations with compelling narrative progression.
  • Value: ⭐⭐⭐⭐⭐ Highly actionable, training-free methodology offering immediate practical defenses for survey integrity in computational social science.