When LLMs Say No: Diagnosing and Mitigating Conservative Bias in Authorship Verification¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/galactixx/when-llms-say-no
Area: LLM Reasoning
Keywords: Authorship Verification, Conservative Bias, Content Conflation, Prompt Interventions, Stylometry
TL;DR¶
Addressing the severe conservative bias in zero-shot LLM authorship verification where models disproportionately predict "different author" (NO) due to confusing topical divergence with authorial differences, this paper introduces lightweight framing and contrastive reweighting prompt interventions that increase same-author accuracy by up to 13.1 percentage points and macro-F1 by 3.1 points without fine-tuning.
Background & Motivation¶
Authorship verification (AV) asks whether two distinct texts \((T_1, T_2)\) were authored by the same individual. The task serves as a foundational building block for forensic linguistics, dispute resolution, academic integrity audits, and digital forensics. Historically, AV systems have relied on classical stylometric methodologies that track low-level stylistic markers such as character/word \(n\)-grams, function word frequencies, punctuation preferences, and syntactic parsing distributions. With the recent proliferation of instruction-tuned large language models, zero-shot prompting frameworks like PromptAV and Prompt-CAVE have emerged as attractive alternatives. These zero-shot approaches eliminate the necessity for expensive labeled training corpora while naturally generating human-readable reasoning rationales alongside their binary classifications or confidence scores.
However, zero-shot prompting suffers from a critical, directional failure mode: models exhibit an extreme conservative bias, systematically over-predicting different authorship (NO) on true same-author pairs. The core tension arises because an author's invariant stylistic signature is fundamentally entangled with topical subject matter. In realistic cross-domain and cross-topic verification, an author frequently writes about diverse subjects, adopts different emotional registers, and employs distinct genre conventions across texts. Standard instruction-tuned LLMs confuse these surface semantic variations with genuine differences in authorship. In 65% of the evaluated baseline model-dataset experimental cells, models achieve over 0.80 accuracy on different-author (NO) pairs while dropping below 0.50 on same-author (YES) pairs. Consequently, standard aggregate accuracy and macro-F1 metrics obscure a catastrophic class-directional blind spot.
This paper tackles this issue by empirically diagnosing the underlying causal mechanism of conservative bias and developing minimal, non-invasive prompt-level mitigations. By conducting rigorous qualitative examinations of generated reasoning chains and quantitatively modeling false-negative errors against semantic distance measured via sentence embeddings, the authors demonstrate that topical divergence systematically drives LLMs to reject true same-author pairs. Core idea: inject lightweight prompt interventions that explicitly instruct the model to disregard topical themes and prioritize invariant surface-form habits (Framing), while contrastively separating low-level stylometric evidence from content-sensitive cues and placing higher decision weight on stylometry (Contrastive Reweighting).
Method¶
Overall Architecture¶
The proposed diagnostic and mitigation framework directly intervenes at the prompt interface of zero-shot LLM authorship verification systems. Given an unlabeled text pair \((T_1, T_2)\), existing baseline prompts (such as Chain-of-Thought, PromptAV, and the structured JSON-based Prompt-CAVE) query the model on broad stylistic dimensions including punctuation, special characters, capitalization, idioms, tone, and sentence structure. Without explicit guidance regarding topic-style disentanglement, models naturally succumb to content conflation, allowing semantic divergence to overwhelm underlying stylometric regularities.
To rectify this without parameter updates, the paper introduces two modular prompt-level interventions: Task Framing and Contrastive Reweighting. These instructions can be applied individually or combined synergistically as a unified intervention. In the restructured reasoning trajectory, the model first parses the task through the framing instruction, actively filtering out topical distractions and focusing cognitive resources on invariant surface-form habits across topics. Next, under contrastive reweighting, the model explicitly segregates extracted features into low-level stylometric markers versus content-sensitive signals, assigning higher decision priority to the former. Finally, the model outputs either an calibrated confidence score or a structured JSON object containing feature judgments and the final YES/NO decision.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Text Pair (T1, T2)"] --> B["Stage 1: Task Framing<br/>Discount topic content; focus on invariant surface-form habits"]
B --> C["Stage 2: Contrastive Reweighting<br/>Separate low-level stylometry from content; prioritize stylometry"]
C --> D["Stage 3: Multi-Variable Stylometric Reasoning<br/>PromptAV step-by-step scoring / Prompt-CAVE JSON feature evaluation"]
D --> E["Calibrated Decision Output: Confidence Score or YES/NO"]
Key Designs¶
1. Task Framing: Disregarding topical themes to isolate invariant surface-form habits This intervention addresses the vulnerability of LLMs to semantic distractors, where differences in characters, fictional settings, or topical subject matter are falsely treated as evidence of distinct authors. Standard prompts instruct models to look at broad linguistic variables, but models instinctively prioritize high-level semantics. The Task Framing intervention explicitly injects negative constraints and an anchor directive into the user prompt:
Do not focus on what the texts are about. Focus on recurring surface-form habits that could remain stable across topics.
2. Contrastive Reweighting: Explicitly separating low-level stylometry from content and prioritizing stylometric evidence This intervention tackles the aggregation bottleneck where models fail to weigh subtle stylometric consistency against glaring semantic differences. In standard reasoning chains, models often note that punctuation and syntax are identical, yet conclude with NO because the emotional tone or subject matter differs. Contrastive Reweighting forces a two-tier evidential hierarchy:
When comparing the texts, explicitly separate low-level stylometric evidence from content-sensitive evidence, and give more weight to low-level stylometric evidence.
3. Semantic-Distance Diagnostic Mechanism: Quantifying the content conflation hypothesis
To validate that conservative errors stem from content divergence rather than random hallucination, the authors develop a quantitative diagnostic protocol. For each text pair, the first 384 tokens of each document are encoded using the pretrained sentence embedding model all-mpnet-base-v2. Semantic distance \(d(T_1, T_2)\) is computed as the cosine distance between the embeddings. A pooled logistic regression with cluster-robust standard errors (clustered by text pair) models false-negative errors on gold-YES pairs:
$\(P(\hat{y} = \text{NO} \mid y = \text{YES}, d(T_1, T_2)) = \sigma\left(\alpha + \beta \cdot d(T_1, T_2) + \gamma_{\text{dataset}} + \delta_{\text{model}}\right)\)$
The empirical results reveal that the odds ratio per standard-deviation increase in semantic distance is \(1.89\times\) for PromptAV (\(p = 2.34 \times 10^{-62}\)) and \(2.51\times\) for Prompt-CAVE (\(p = 6.30 \times 10^{-82}\)). Furthermore, binning gold-YES pairs into semantic distance quintiles (Q1 to Q5) demonstrates that false-negative rates rise monotonically with semantic distance, establishing the empirical imperative for prompt reframing.
Benchmark & Evaluation Protocol¶
The benchmark evaluates 4 diverse public authorship verification datasets encompassing 4,000 balanced text pairs: 1. Blog-Auth: 1,000 pairs of personal blog posts characterized by informal tone and conversational syntax (500 YES / 500 NO); 2. Fanfic: 1,000 pairs from fan fiction covering varied fictional worlds, characters, and stylistic registers (500 YES / 500 NO); 3. IMDb62: 1,000 pairs of movie reviews with moderate formality across distinct films (503 YES / 497 NO); 4. CrossNews: 1,000 pairs sampled from the Article–Article subset of the CrossNews benchmark covering professional journalism (500 YES / 500 NO).
Experiments are conducted across 5 open-weight instruction-tuned LLMs from 3 model families: Gemma 3 12B-IT, Gemma 3 27B-IT, Mistral-Small-3.2-24B-Instruct, Qwen3-14B, and Qwen3-32B. Inference is served via vLLM on a single NVIDIA H100 GPU using greedy decoding (temperature = 0) with a 4,096-token context budget and non-thinking mode for Qwen3. No system prompts or labeled training examples are used, preserving a strict zero-shot setting.
Key Experimental Results¶
Main Results¶
The baseline evaluation reveals a severe class-directional asymmetry across models and datasets, where models act as conservative verifiers that over-reject same-author pairs.
| Model | Baseline Method | Blog-Auth (Acc / YESA / NOA) | Fanfic (Acc / YESA / NOA) | IMDb62 (Acc / YESA / NOA) | CrossNews (Acc / YESA / NOA) |
|---|---|---|---|---|---|
| Gemma 3 12B | CoT | 0.59 / 0.28 / 0.91 | 0.53 / 0.09 / 0.97 | 0.69 / 0.48 / 0.90 | 0.73 / 0.47 / 0.98 |
| Gemma 3 12B | PromptAV | 0.54 / 0.13 / 0.95 | 0.51 / 0.04 / 0.98 | 0.61 / 0.25 / 0.98 | 0.65 / 0.35 / 0.96 |
| Gemma 3 12B | Prompt-CAVE | 0.53 / 0.08 / 0.98 | 0.50 / 0.00 / 1.00 | 0.54 / 0.09 / 0.99 | 0.61 / 0.25 / 0.98 |
| Gemma 3 27B | PromptAV | 0.58 / 0.29 / 0.88 | 0.57 / 0.24 / 0.89 | 0.67 / 0.47 / 0.87 | 0.72 / 0.62 / 0.81 |
| Gemma 3 27B | Prompt-CAVE | 0.62 / 0.36 / 0.88 | 0.54 / 0.12 / 0.96 | 0.72 / 0.47 / 0.96 | 0.78 / 0.66 / 0.91 |
| Mistral 24B | PromptAV | 0.64 / 0.49 / 0.78 | 0.60 / 0.52 / 0.68 | 0.69 / 0.63 / 0.75 | 0.77 / 0.70 / 0.83 |
| Mistral 24B | Prompt-CAVE | 0.59 / 0.39 / 0.79 | 0.57 / 0.61 / 0.54 | 0.64 / 0.63 / 0.64 | 0.77 / 0.82 / 0.73 |
| Qwen3-14B | PromptAV | 0.56 / 0.18 / 0.94 | 0.52 / 0.13 / 0.91 | 0.60 / 0.25 / 0.96 | 0.63 / 0.30 / 0.97 |
| Qwen3-14B | Prompt-CAVE | 0.47 / 0.07 / 0.86 | 0.39 / 0.05 / 0.72 | 0.46 / 0.07 / 0.86 | 0.51 / 0.12 / 0.90 |
| Qwen3-32B | PromptAV | 0.60 / 0.35 / 0.86 | 0.58 / 0.46 / 0.69 | 0.69 / 0.57 / 0.81 | 0.75 / 0.74 / 0.76 |
| Qwen3-32B | Prompt-CAVE | 0.59 / 0.37 / 0.82 | 0.59 / 0.57 / 0.61 | 0.71 / 0.69 / 0.74 | 0.81 / 0.79 / 0.83 |
Across all 60 baseline model-dataset cells, 65% (39/60) exhibit NO accuracy \(> 0.80\) while YES accuracy falls \(< 0.50\). For instance, Gemma 3 12B with Prompt-CAVE on Fanfic achieves 1.00 NO accuracy but 0.00 YES accuracy, completely failing to recognize true authors when fictional settings differ.
Ablation Study¶
The following table reports the performance of each prompt intervention variant averaged across all five models, illustrating the individual and combined impact of Framing (+FR) and Contrastive Reweighting (+CO).
| Base Family | Variant | Blog-Auth (Acc / F1 / YESA / NOA) | Fanfic (Acc / F1 / YESA / NOA) | IMDb62 (Acc / F1 / YESA / NOA) | CrossNews (Acc / F1 / YESA / NOA) | Global Metrics (Mean \(\Delta\) YESA / Win Rate) |
|---|---|---|---|---|---|---|
| PromptAV | BASE | 0.58 / 0.54 / 0.29 / 0.88 | 0.55 / 0.50 / 0.28 / 0.83 | 0.65 / 0.63 / 0.43 / 0.87 | 0.70 / 0.69 / 0.54 / 0.87 | Baseline |
| +FR (Framing) | 0.59 / 0.56 / 0.35 / 0.84 | 0.56 / 0.54 / 0.41 / 0.71 | 0.66 / 0.65 / 0.52 / 0.80 | 0.72 / 0.72 / 0.64 / 0.81 | +9.3 pt (19/20 improved) | |
| +CO (Contrastive) | 0.58 / 0.53 / 0.28 / 0.88 | 0.55 / 0.51 / 0.36 / 0.74 | 0.65 / 0.63 / 0.46 / 0.85 | 0.70 / 0.69 / 0.56 / 0.84 | +2.8 pt (13/20 improved) | |
| +BOTH (FR + CO) | 0.58 / 0.55 / 0.34 / 0.82 | 0.55 / 0.55 / 0.50 / 0.60 | 0.66 / 0.65 / 0.56 / 0.76 | 0.73 / 0.72 / 0.68 / 0.77 | +13.1 pt (18/20 improved) | |
| Prompt-CAVE | BASE | 0.56 / 0.52 / 0.25 / 0.87 | 0.52 / 0.47 / 0.27 / 0.77 | 0.61 / 0.59 / 0.39 / 0.84 | 0.70 / 0.69 / 0.53 / 0.87 | Baseline |
| +FR (Framing) | 0.58 / 0.56 / 0.34 / 0.82 | 0.53 / 0.51 / 0.37 / 0.68 | 0.64 / 0.65 / 0.52 / 0.76 | 0.72 / 0.74 / 0.66 / 0.78 | +11.2 pt (19/20 improved) | |
| +CO (Contrastive) | 0.54 / 0.49 / 0.19 / 0.89 | 0.51 / 0.44 / 0.19 / 0.82 | 0.59 / 0.56 / 0.30 / 0.89 | 0.69 / 0.68 / 0.49 / 0.88 | -6.7 pt (2/20 improved) | |
| +BOTH (FR + CO) | 0.56 / 0.54 / 0.27 / 0.85 | 0.52 / 0.50 / 0.34 / 0.69 | 0.63 / 0.63 / 0.42 / 0.83 | 0.70 / 0.72 / 0.60 / 0.80 | +4.8 pt (14/20 improved) |
Statistical significance confirmed via Holm-adjusted one-sided McNemar tests and paired permutation tests indicates: - For PromptAV, +BOTH achieves a +13.1 percentage point average increase in YES accuracy (18/20 settings improved, 13/20 significant at \(p < 0.05\)) while lifting macro-F1 by 3.1 points. - For Prompt-CAVE, +FR alone yields the greatest gain, improving YES accuracy in 19/20 settings (+11.2 points average, 16/20 significant at \(p < 0.05\)) and increasing macro-F1 by 4.8 points (15/20 settings improved). In contrast, +CO hurts Prompt-CAVE (-6.7 points YES accuracy), highlighting that rigid JSON output generation interacts poorly with complex contrastive instructions.
Key Findings¶
- Error Concentration Across Semantic Quintiles: False-negative rates increase monotonically from Q1 (semantically closest) to Q5 (semantically farthest). On Blog-Auth and Fanfic, the false-negative rate in Q5 exceeds Q1 by 20–40 percentage points.
- Uniform Error Reduction Without Slope Flattening: Effective interventions shift the false-negative curves downward roughly equally across all quintiles (Q1–Q5) rather than flattening the slope. This indicates that while prompt reframing mitigates the conservative threshold, underlying semantic distance remains an influential cognitive attractor.
- Threshold Tuning Does Not Resolve Rationalization Bias: A manual audit of 600 near-threshold (\(0.3 \le s \le 0.5\)) gold-YES outputs shows that 92.7% (556/600) contained explicit rationales arguing for different authorship. Post-hoc threshold shifting simply forces label conversion without fixing the flawed reasoning, and the optimal threshold \(\tau^*\) varies dramatically across domains (0.324 on Blog-Auth vs. 0.464 on CrossNews).
Highlights & Insights¶
- Diagnosing the Class-Directional Blind Spot: Pinpoints that seemingly moderate aggregate F1 scores in zero-shot LLM benchmarks conceal extreme directional failure modes where models act as aggressive rejection filters.
- High-Impact, Zero-Cost Prompt Reframing: Demonstrates that two sentences of targeted natural language instructions can shift model error profiles by 13.1 percentage points without fine-tuning, latency overhead, or specialized architectures.
- Format-Intervention Interaction: Uncovers a vital insight for prompt engineering: unconstrained generation (PromptAV) benefits most from multi-level contrastive decomposition (+BOTH), whereas tightly constrained JSON generation (Prompt-CAVE) functions best with top-level negative framing alone (+FR).
Limitations & Future Work¶
- Trade-off with Different-Author Discrimination: Mitigating false negatives causes a moderate reduction in NO accuracy (dropping 5–15 points depending on the dataset). In high-stakes forensic or legal settings where false positives carry severe consequences, this trade-off must be carefully balanced.
- Evaluation Limited to Open-Weight Models: The study evaluates five instruction-tuned models up to 32B parameters; future work should examine whether frontier reasoning models (e.g., OpenAI o1/o3, DeepSeek-R1) or closed-source commercial APIs exhibit identical content conflation dynamics.
- Approximation of Content Divergence: Semantic distance is approximated using sentence embeddings over the initial 384 tokens, which may miss narrative discrepancies occurring later in long documents.
Related Work & Insights¶
- vs PromptAV (Hung et al., EMNLP 2023): PromptAV established the baseline of instructing LLMs to reason over stylistic variables. This work identifies its vulnerability to content conflation across topics and provides plug-and-play framing instructions that resolve its false-negative skew.
- vs Prompt-CAVE (Ramnath et al., NAACL 2025): Prompt-CAVE introduced structured JSON feature explanations. This paper reveals that its rigid format exacerbates conservative bias on smaller models (reducing YES accuracy to near zero) and establishes that standalone Framing (+FR) provides the optimal fix.
- vs One-Shot Style Transfer (Miralles-González et al., 2025 / OSST): OSST utilizes token log-probabilities to disentangle style from content. In contrast, this paper operates entirely in black-box zero-shot prompt space without requiring access to log-likelihoods, ensuring superior practical applicability.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Exposes a critical directional failure mode in zero-shot AV and establishes the content conflation hypothesis]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans 5 models, 4 datasets, 20 experimental settings, robust statistical hypothesis tests, and manual audits]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, coherent narrative progression, and well-structured empirical tables and figures]
- Value: ⭐⭐⭐⭐ [Offers actionable prompt engineering guidelines for zero-shot text classification, forensics, and evaluation systems]