Skip to content

Personalization over Generalization: History-Aware Sentence-Level AI Detection in Collaborative Writing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/adirizq/phas-id
Area: LLM Reasoning
Keywords: Sentence-level AI Detection, Human-AI Collaborative Writing, History-Aware Personalization, Dual Cross-Attention Fusion, GhostFill

TL;DR

Addressing the limitation that conventional sentence-level AI detectors enforce a generic one-size-fits-all boundary that ignores individual writing styles, PHAS-ID grounds predictions in an author's labeled historical writing via dual macro-micro cross-attention and introduces GhostFill to faithfully synthesize collaborative interleaving dynamics.

Background & Motivation

The proliferation of Large Language Models (LLMs) has amplified concerns regarding academic dishonesty and synthetic misinformation. In modern human-AI collaborative writing workflows, documents are rarely purely machine-generated or purely human-authored; instead, they consist of interwoven human text and machine-generated suggestions. Traditional document-level detectors fail completely on these mixed-authorship compositions. Consequently, sentence-level AI detection (SL-AID)—identifying whether each individual sentence was written by a human, generated by an AI, or jointly revised—has emerged as a foundational capability for auditing collaborative documents. However, existing detectors (such as DeBERTaV3 fine-tuning, token-probability tracking via SeqXGPT, and coherence-augmented classifiers like GL-CLiC) yield poor performance on realistic benchmarks like CoAuthor, where the previous state-of-the-art achieves only a 61.72 Macro F1.

The fundamental breakdown of existing methodologies stems from treating all human authors identically. In real-world collaborative authoring, human writers exhibit profound stylistic variance in how they interact with AI tools: some heavily edit and synthesize suggestions into their personal voice, while others accept raw model completions verbatim. As theoretically characterized in authorship verification research, the boundary between human-written and machine-generated content is inherently author-specific rather than universal. When a generic detector lacks access to an author's baseline style, it frequently conflates an author's idiosyncratic phrasing with AI-generated text or misattributes polished machine sentences to the human.

Compounding this architectural blind spot, existing synthetic benchmarks (such as SeqXGPT-Bench and HACo-Det) exhibit synthetic artifacts that fail to capture real collaborative dynamics. Whereas actual collaboration in CoAuthor is characterized by rapid, stochastic interleaving (an average AI run length of just 1.4 sentences and transition variance \(\sigma^2 = 6.6\)), synthetic corpora generate long, contiguous blocks of single-class text. This structural mismatch creates an artificially simplified training distribution. This paper directly addresses both bottlenecks by conditioning sentence-level classification on author-specific historical context and benchmarking against realistic style-conditioned data. Core idea: formulate sentence-level AI detection as an author-personalized verification task via PHAS-ID, fusing macro-level class prototypes and micro-level sentence associative memory from an author's labeled history, and introducing GhostFill to simulate realistic style-grounded collaborative interleaving.

Method

Overall Architecture

PHAS-ID leverages \(K\) labeled support documents \(S = \{S_1, \dots, S_K\}\) written by the same author as the target document \(T\) to ground sentence predictions in historical stylistic behavior. At inference time, StyleDistance first retrieves the top-\(K\) stylistically relevant support documents from the author's archive. Both target and support documents are parsed into sentence sequences and encoded using a shared DeBERTaV3 backbone and an intra-document self-attention module to capture paragraph-level discourse context. Next, a Dual Cross-Attention Fusion module conditions the contextualized target representations on the author's history across two granularities: a macro-level label representation capturing broad class prototypes, and a micro-level sentence memory performing soft associative retrieval of historical sentence-label pairs. Finally, residual fusion with layer normalization feeds a classification head that predicts per-sentence labels (Human, AI, or Human-AI).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input target document and K labeled author support documents"] --> B["Intra-Document Self-Attention<br/>Shared DeBERTaV3 and self-attention encode sentence context"]
    B --> C["Dual Cross-Attention Fusion<br/>Macro class prototypes and micro sentence associative memory"]
    C --> D["Residual Normalization & Prediction Head<br/>Fused representations classify sentences into 3 classes"]
    D --> E["GhostFill Benchmark Generation<br/>Style-aware realistic collaborative interleaving"]

Key Designs

1. Intra-Document Self-Attention: Bridging Local Sentences to Global Discourse

Because initial sentence embeddings produced by the backbone language model treat each sentence in isolation, they fail to capture intra-document argumentative flow and narrative coherence. Given target document sentences \(T=(t_1, \dots, t_N)\) and support sentences \(S_k=(s_1^k, \dots, s_m^k)\), PHAS-ID extracts the [CLS] token hidden states from a shared DeBERTaV3 backbone \(E_\theta\), yielding raw sentence matrices \(H_T \in \mathbb{R}^{N \times d}\) and \(H_S^k \in \mathbb{R}^{m \times d}\). To enable sentences within the same document to contextualize one another, a shared multi-head intra-document self-attention layer \(\text{SA}_\phi\) is applied: $$ \tilde{H}T = \text{SA}\phi(Q=H_T, K=H_T, V=H_T), \quad \tilde{H}S^k = \text{SA}\phi(Q=H_S^k, K=H_S^k, V=H_S^k) $$ Standard attention masking prevents attention to padded sentences. The resulting contextualized support documents are then flattened across document and sequence dimensions into an unbroken cross-document memory bank \(\bar{H}_S \in \mathbb{R}^{Km \times d}\).

2. Dual Cross-Attention Fusion: Multiscale Grounding via Prototypes and Associative Memory

Target sentence representations alone lack author-specific stylistic anchors. PHAS-ID injects personalization through two parallel multi-head cross-attention mechanisms: At the macro level, support sentences corresponding to class \(c \in \{1, \dots, C\}\) are mean-pooled and transformed through class-specific MLPs \(f_c\) into a macro prototype matrix \(H_{\text{cls}}^L \in \mathbb{R}^{C \times d}\). At the micro level, these prototypes are broadcast back across all individual support sentences to construct an aligned sentence-label matrix \(H_{\text{sent}}^L \in \mathbb{R}^{Km \times d}\). The dual cross-attentions operate in parallel: $$ Z_{\text{cls}} = \text{CA}{\text{cls}}^{\psi_c}(Q=\tilde{H}T, K=H^L, V=H}^L) $$ $$ Z_{\text{sent}} = \text{CA}_{\text{sent}}^{\psi_s}(Q=\tilde{H}_T, K=\bar{H}}S, V=H^L) $$ Here, }\(Z_{\text{cls}}\) aligns the target sentence with high-level stylistic class profiles, while \(Z_{\text{sent}}\) uses historical support sentences as query keys and their corresponding ground-truth label representations as associative values. This enables the model to resolve ambiguous sentences by softly retrieving how the specific author previously formulated and edited analogous expressions.

3. Residual Normalization & Prediction Head: Stabilizing Noisy Stylistic Boundaries

To prevent author context from overpowering immediate textual content while ensuring robust gradient propagation, the prediction head merges the feature representations via residual addition: $$ \hat{H}T = \text{LayerNorm}(\tilde{H}T + Z + Z}) $$ A classification multi-layer perceptron }\(\text{MLP}_\omega\) maps \(\hat{H}_T\) to predicted sentence labels \(\hat{Y} = \{\hat{y}_1, \dots, \hat{y}_N\}\). The network is trained end-to-end using weighted Cross-Entropy loss regularized with label smoothing (\(\epsilon = 0.05\)). This smoothing softens decision boundaries against subtle stylistic drift and annotation noise in historical metadata.

4. GhostFill Style-Aware Synthesis Pipeline: Replicating Empirical Interleaving Statistics

To overcome the lack of modern, challenging test sets, GhostFill constructs high-fidelity evaluation data. Given target author \(a\) from the Blog Authorship Corpus, StyleDistance pairs \(a\) with a stylistically closest author \(b\) from the collaborative CoAuthor dataset. GhostFill extracts author \(b\)'s empirical transition probability \(\lambda\) and mean AI run length \(\bar{r}\), and stochastically masks sentence spans in \(a\)'s human text according to these parameters. Using an Author Writing Sheet that extracts persona attributes across Plot, Creativity, Development, and Language Use, frontier models (GPT-5.4, Sonnet 4.6, DeepSeekV4) infill the masked spans, followed by candidate selection that retains genuine authorial style while mirroring real-world collaborative dynamics.

Loss & Training

The framework builds on DeBERTaV3-base (183M parameters), totaling 199M trainable parameters. Training utilizes the AdamW optimizer with decoupled learning rates: \(\eta_{\text{plm}} = 1 \times 10^{-5}\) for the PLM backbone and \(\eta_{\text{mlp}} = 5 \times 10^{-5}\) for newly initialized attention and classification modules. Training follows a linear warmup over the initial 10% steps followed by cosine learning rate decay, using early stopping with a patience of 3 epochs monitored on validation Macro F1. On a single NVIDIA RTX 4090 GPU, training completes in approximately 3 minutes per epoch per 1,000 target documents.

Key Experimental Results

Main Results

Evaluation on the real-world CoAuthor benchmark under Seen Authors and Unseen Authors settings demonstrates that PHAS-ID significantly outperforms all generic detectors and few-shot LLM baselines across precision (P), recall (R), and F1 scores.

Setting & Method Human F1 AI P AI R AI F1 H-AI F1 Macro F1
Seen Authors Setting
SeqXGPT 81.49 58.60 11.97 19.88 13.16 38.18
PLM Fine-tune (DeBERTaV3) 82.46 58.68 41.41 48.55 43.51 58.17
SimLLM 80.02 45.00 52.61 48.51 42.49 57.01
PTD Framework 81.05 50.57 12.73 20.33 40.89 47.42
GL-CLiC (Previous SOTA) 82.26 54.08 49.10 51.47 51.44 61.72
P-LLM (Few-shot GPT-4o) 74.10 30.23 36.37 33.02 44.20 50.44
P-LLM (Few-shot Llama 4) 66.96 30.01 39.41 34.07 36.50 45.85
PHAS-ID (Ours) 81.95 50.44 64.12* 56.33* 55.03* 64.44*
Unseen Authors Setting
SeqXGPT 79.67 83.16 8.01 14.62 7.03 33.77
PLM Fine-tune 78.28 41.38 64.30 50.36 42.37 57.00
SimLLM 77.68 44.55 61.36 51.62 36.92 55.41
PTD Framework 78.46 39.39 3.96 7.19 26.99 37.55
GL-CLiC 81.90 60.85 37.83 46.65 13.90 47.48
P-LLM (Few-shot GPT-4o) 76.56 25.39 31.10 27.96 30.39 44.97
PHAS-ID (Ours) 86.11* 52.39 57.92 54.86* 48.33* 63.10*

Note: Asterisk (*) indicates statistically significant improvement over the second-best method (\(p < 0.05\)).

Ablation Study

Ablation experiments verify that performance gains stem from genuine author personalization rather than generic context expansion, alongside sensitivity analysis on the number of support documents \(K\).

Support Configuration Human F1 AI F1 Human-AI F1 Macro F1 Note
No Supp (\(K=0\)) 80.37 51.30 46.92 59.53 Unconditioned intra-document model
Global Supp 80.46 50.59 48.29 59.78 Random documents from other authors
Topic Supp 79.67 51.16 48.92 59.92 Same topic, mismatched authors
Human Only 82.41 53.67 53.70 63.26 Only human-authored history from target author
Full PHAS-ID (\(K=5\)) 81.95 56.33* 55.03* 64.44* Full author-matched labeled history

Scaling of Support Documents \(K\): - \(K=0\): Macro F1 is 59.53. - \(K=1\): Jumps sharply to 63.17 (+3.64 points), immediately outperforming the prior SOTA (GL-CLiC at 61.72). - \(K=2\): Rises to 64.20, achieving most of the personalization gains. - \(K=5\): Peaks at 64.44 Macro F1, with AI F1 steadily scaling from 51.30 to 56.33.

Key Findings

  • Personalization Is the Decisive Factor: Providing mismatched support documents from the general dataset pool (Global Supp, 59.78) or from the same topic by different writers (Topic Supp, 59.92) collapses performance back to the zero-support baseline (59.53). This confirms that improvements arise from author-specific stylistic grounding rather than extra context length or class balance signals.
  • Superior Generalization to Unseen Authors: While GL-CLiC overfits to training authors and experiences a massive drop of 14.24 Macro F1 on unseen authors, PHAS-ID drops by only 1.34 points (from 64.44 to 63.10), outpacing the next best baseline by 6.10 points.
  • Robustness in Cross-Generator Scenarios: When trained on CoAuthor (GPT-3.5) and evaluated on GhostFill (generated by frontier models GPT-5.4, Sonnet 4.6, DeepSeekV4), PHAS-ID achieves 55.28 Macro F1, delivering a 13.84 point absolute gain in AI F1 over standard PLM fine-tuning and surpassing GL-CLiC's 44.87 Macro F1.
  • Sandwich Errors Dominate Boundary Failures: Error analysis reveals that 28.89% of all misclassifications—and 55.58% of all AI-class errors—are "sandwich" errors, where an isolated AI sentence flanked by human sentences is over-smoothed into a human prediction by the intra-document attention layer.

Highlights & Insights

  • Reframing Detection via Authorship Verification: Instead of hunting for universal statistical artifacts across frontier LLMs, the paper grounds detection in author deviation, recognizing that the human-AI boundary is inherently personalized.
  • Decoupled Dual-Scale Cross-Attention: The dual mechanism provides macro-level category anchoring while using micro-level historical sentences as keys and ground-truth labels as values, creating an efficient soft associative memory with only 16M additional parameters.
  • Practicality of Human-Only History: The finding that purely human-written history (Human Only) achieves 63.26 Macro F1 provides an immediate deployment path, eliminating the need for sentence-level annotation in real-world collaborative platforms.

Limitations & Future Work

  • Dependency on Author Historical Data: In cold-start or anonymous settings lacking author metadata (such as HACo-Det), PHAS-ID falls back to random context sampling and underperforms unconstrained fine-tuned models.
  • Stylistic Drift Over Time: The model assumes a static stylistic profile, leaving its vulnerability to longitudinal writing drift and evolving vocabulary unaddressed.
  • Susceptibility to Context Over-Smoothing: Intra-document self-attention over-smooths single-sentence insertions, resulting in sandwich errors at transition boundaries that could be resolved with CRF decoders or local attention windows.
  • vs GL-CLiC (Adi et al., 2025): GL-CLiC relies on global coherence and lexical metrics within a static model, which overfits to seen authors and suffers a 14.24-point collapse on unseen authors; PHAS-ID achieves stable generalization via dynamic reference conditioning.
  • vs SeqXGPT (Wang et al., 2023): SeqXGPT uses token log-probabilities from proxy open LLMs, achieving an AI recall of only 11.97% on collaborative benchmarks; PHAS-ID achieves 64.12% AI recall through stylistic grounding.
  • vs HACo-Det (Su et al., 2025): HACo-Det generates uniform contiguous blocks that underestimate detection difficulty; GhostFill matches empirical transition probabilities and run lengths, establishing a realistic benchmark.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering personalization and authorship verification principles for sentence-level AI detection.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage of seen/unseen authors, cross-domain, cross-generator, and multi-source context ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Fluid prose narrative, clear mathematical formalization, and transparent error characterization.
  • Value: ⭐⭐⭐⭐⭐ Delivers critical practical insights and datasets for auditing academic integrity in human-AI collaborative authoring.