Skip to content

TraceMIA: Interpretable Membership Inference Attack via Lexical Patterns in Reasoning Traces

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Safety
Keywords: Membership Inference Attack, Large Reasoning Models, Lexical Patterns, Introspective Reasoning, Interpretable Auditing

TL;DR

TraceMIA uncovers that large reasoning models exhibit a prominent lexical dichotomy during introspective deliberationโ€”unseen texts elicit a "guessing-like" style while training members elicit a "remember-like" styleโ€”and leverages chi-square filtering on reasoning traces from a small reference set to achieve high-precision, interpretable black-box membership auditing with human-readable textual evidence.

Background & Motivation

Large language models and large reasoning models (LRMs) increasingly rely on web-scale training corpora that inevitably ingest copyrighted literature, proprietary code, and protected academic publications, raising urgent intellectual property and compliance concerns. Membership inference attacks (MIAs) have emerged as essential auditing tools to verify whether specific datasets were ingested during pre-training. Traditional MIA methods primarily operate on token-level probabilities, evaluating perplexity thresholds, lowest-\(k\)% token log-likelihoods (Min-K%), or neighborhood perturbation margins. However, with the transition from standard pre-trained foundation models to reasoning models post-trained with extensive reinforcement learning, severe distribution shifts render fragile token-level statistical features ineffective, causing detection accuracy across prior MIA benchmarks to collapse dramatically.

To adapt to reasoning models under black-box settings, recent pioneering attempts like BlackSpectrum have started extracting membership signals from chain-of-thought traces, but they project reasoning trajectories into opaque, high-dimensional latent semantic spaces using dense embedding encoders. While this latent projection recovers some signal, it completely strips away audit interpretability. In real-world copyright litigation and compliance investigations, content publishers and legal authorities require human-readable, transparent factual evidence that can withstand legal scrutiny; opaque similarity scores or latent distance metrics fail to provide actionable, verifiable proof of unauthorized data consumption.

Addressing this critical gap between detection capability and evidential transparency, the authors conducted an empirical exploration of how reasoning models articulate self-reflection. They discovered that an LRM's internal memory state is explicitly leaked through its macroscopic lexical choices and tone: when prompted to reflect introspectively on whether a passage was seen during training, non-member traces consistently default to epistemic hedging and speculative vocabulary (e.g., maybe, perhaps, possibly, cutoff), whereas member traces naturally exhibit assertive provenance-tracing and recollection terms (e.g., remember, source, recognize, wording). The core idea of this paper is to bypass opaque latent vector encodings and directly exploit the lexical dichotomy between speculative guessing and memory recollection in introspective reasoning traces, applying statistical chi-square filtering on a small reference set to build discriminative cue lexicons that yield both high detection accuracy and transparent, human-readable evidence.

Method

Overall Architecture

TraceMIA operates under a realistic black-box auditing setting where the auditor can only query the target reasoning model and observe its generated text, including the exposed <think>...</think> reasoning trace, without access to model weights, token-level logits, or internal layer activations. The auditing pipeline proceeds through three tightly coupled phases: Phase 1 elicits introspective reasoning traces from the target model for a small labeled reference set; Phase 2 applies frequency pruning and Yates-corrected chi-square contingency testing to construct positive (member-cue) and negative (non-member-cue) lexicons; Phase 3 queries any unseen target passage, extracts its reasoning trace, and computes a length-normalized cue frequency difference score alongside concrete lexical highlights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query Passage & Labeled Reference Set"] --> B["Introspective Reasoning Elicitation<br/>Introspective prompt triggers deep reflection in &lt;think&gt; tags"]
    B --> C["Data-Driven Lexicon Construction<br/>Frequency filtering + Yates-corrected chi-square test yields W+ / W-"]
    C --> D["Normalized Frequency Scoring<br/>Length-normalized cue count difference Slex"]
    D --> E["Membership Decision & Transparent Textual Evidence"]

Key Designs

1. Introspective Reasoning Elicitation: Activating latent memory traces via structured self-reflection

Standard black-box attacks typically evaluate final continuation completions or prompt the model for direct binary classification, overlooking the rich deliberative dynamics unfolding within reasoning traces. Relying solely on final self-reports (such as a binary JSON answer) leaves audits vulnerable to calibration biases and truncation errors when long chains of thought exhaust context windows. TraceMIA designs a targeted introspective prompt \(\mathcal{P}(x)\) that explicitly prompts the model to reflect on four provenance indicators: where the passage might originate, how familiar its writing style feels, whether specific phrases or facts are recognized, and what corpus it might belong to. With thinking mode enabled (enable_thinking=True), the model emits an extensive reasoning trajectory \(\tau_x\) within <think>...</think> tags. The concluding JSON schema is maintained solely as a structural termination trigger, while the downstream scoring pipeline operates exclusively on the preprocessed word sequence \(V(\tau_x)\) within the trace.

2. Data-Driven Lexicon Construction: Isolating stylistic signals without manual stop-word curation

Hand-crafted lexicons of certainty and uncertainty cues fail to generalize across diverse text genres such as scientific preprints and creative fiction due to substantial cross-domain vocabulary shifts. TraceMIA establishes a data-driven lexicon extraction protocol requiring only an inexpensive reference set \(\mathcal{D}_{\text{ref}} = \mathcal{D}_m \cup \mathcal{D}_n\) (\(K=25\) samples per class). For each candidate word \(w\), the algorithm tallies document frequencies \(a, b, c, d\) across member and non-member traces to form a \(2 \times 2\) contingency table. Words with overall document frequency \(df(w) = a+c < f_{\min}\) (\(f_{\min}=3\)) are pruned to suppress small-sample noise. Next, a chi-square independence test with Yates' continuity correction is evaluated. Using significance threshold \(p_{\text{thresh}} = 0.05\) and class-conditional document frequencies \(\pi_m(w) = a/K\) and \(\pi_n(w) = c/K\), surviving words are split into two complementary lexicons:

\[W^+ = \{w \mid df(w) \ge f_{\min}, p_{\chi^2}(w) < p_{\text{thresh}}, \pi_m(w) > \pi_n(w)\}\]
\[W^- = \{w \mid df(w) \ge f_{\min}, p_{\chi^2}(w) < p_{\text{thresh}}, \pi_n(w) > \pi_m(w)\}\]

This statistical procedure naturally filters out high-frequency function words (e.g., the, is, that), because they appear with statistically balanced document frequencies across both classes and fail the significance test, entirely removing the need for manual stop-word engineering.

3. Normalized Frequency Scoring: Cancelling trace-length variance for verifiable auditing

During inference on an arbitrary query passage \(x\), the model executes a single forward pass to generate trace \(\tau_x\). Because reasoning lengths vary widely depending on passage complexity, raw cue counts would systematically bias scores toward verbose generations. TraceMIA treats the trace vocabulary as a multiset \(V(\tau_x)\) and normalizes the net difference between member-cue and non-member-cue occurrences by the total trace length:

\[S_{\text{lex}}(x) = \frac{|V(\tau_x) \cap W^+| - |V(\tau_x) \cap W^-|}{\max(|V(\tau_x)|, 1)} \times 100\]

A high positive \(S_{\text{lex}}(x)\) indicates that the trace is heavily dominated by recollection and verification vocabulary characteristic of training members, while a negative score indicates a predominance of modal hedges and hypothetical framing characteristic of unseen inputs. Because the score directly maps to exact lexical matches, an auditor can present the precise set of triggered cue words as verifiable textual proof.

Loss & Training

TraceMIA is completely training-free and non-parametric during test time. The underlying reasoning models remain entirely frozen with no gradient computations, LoRA adapters, or parameter updates. Traces are elicited following standard reasoning configurations (\(T = 0.6\), Top-\(p = 0.95\), max length 1024 tokens). Lexicon induction requires only a one-time offline chi-square computation that executes in seconds.

Key Experimental Results

Main Results

The framework is evaluated across three benchmark datasetsโ€”WikiMIA-25, ArxivReasoning (computer science and machine learning abstracts), and BookReasoning (literary texts)โ€”at input lengths of 32, 64, and 128 tokens on Qwen3-8B, DeepSeek-R1-Distill-Qwen-14B (DS-14B), and QwQ-32B. Performance is measured by Area Under the ROC Curve (AUC-ROC) and True Positive Rate at 5% False Positive Rate (TPR@5%FPR).

Dataset Length Model Metric TraceMIA (lexicon) BlackSpectrum (Latent SOTA) SaMIA (Baseline) CDD (Baseline)
WikiMIA-25 32 Qwen3-8B AUC / TPR 0.9546 / 0.7000 0.5650 / 0.0630 0.5859 / 0.1585 0.5013 / 0.0525
WikiMIA-25 32 QwQ-32B AUC / TPR 0.9873 / 0.9741 0.7230 / 0.1440 0.6211 / 0.1077 0.5020 / 0.0538
WikiMIA-25 64 Qwen3-8B AUC / TPR 0.9722 / 0.8508 0.5800 / 0.0650 0.5168 / 0.0743 0.4994 / 0.0489
WikiMIA-25 128 QwQ-32B AUC / TPR 0.9914 / 0.9929 0.7550 / 0.1780 0.6167 / 0.1395 0.4953 / 0.0409
ArxivReasoning 32 Qwen3-8B AUC / TPR 0.7739 / 0.2744 0.5790 / 0.0680 0.6611 / 0.1877 0.4972 / 0.0448
ArxivReasoning 64 QwQ-32B AUC / TPR 0.9200 / 0.6619 0.5610 / 0.0540 0.6489 / 0.1483 0.4941 / 0.0386
ArxivReasoning 128 Qwen3-8B AUC / TPR 0.8675 / 0.4925 0.5680 / 0.1020 0.6639 / 0.2021 0.4873 / 0.0252
BookReasoning 32 DS-14B AUC / TPR 0.7596 / 0.3323 0.5170 / 0.0520 0.5447 / 0.1167 0.4998 / 0.0496
BookReasoning 64 QwQ-32B AUC / TPR 0.8562 / 0.5332 0.5810 / 0.0470 0.5980 / 0.1262 0.5109 / 0.0710
BookReasoning 128 DS-14B AUC / TPR 0.8318 / 0.5232 0.4520 / 0.0410 0.7747 / 0.4121 0.4998 / 0.0496

Ablation Study

Systematic controlled ablations were conducted to isolate the source of discriminative signal, testing prompt framing, lexicon origin, and comparing trace scoring against direct binary model self-reports.

Ablation Setting Benchmark Scenario AUC TPR@5%FPR Lexicon Size |W+| / |W-| Key Takeaway / Observation
Introspective Prompt + Lexicon (TraceMIA) WikiMIA-25 (Qwen3-8B, L=32) 0.9546 0.7264 29.0 / 13.8 Full model: introspective framing triggers statistically robust lexical divergence
Introspective Prompt + Fixed Lexicon WikiMIA-25 (Qwen3-8B, L=32) 0.9276 0.6292 Hand-crafted (26 / 26) Competitive on encyclopedic text, but degrades severely on literary domains
Neutral Summarization + Lexicon Learning WikiMIA-25 (Qwen3-8B, L=32) 0.6002 0.1319 4.6 / 2.0 Generic CoT suppresses memory reflection; discriminative cues drop by 80%, AUC drops 0.3544
Neutral Summarization + Fixed Lexicon WikiMIA-25 (Qwen3-8B, L=32) 0.5192 0.1030 Hand-crafted (26 / 26) Performance collapses near random chance, proving signal stems from introspective task
Input Text Lexicon (Input-Lexicon) WikiMIA-25 (Qwen3-8B, L=64) 0.5811 0.1176 1 / 2 Applying chi-square to raw input yields only 1-2 cues, proving signal is in the trace
Reasoning Trace Lexicon (Trace-Lexicon) WikiMIA-25 (Qwen3-8B, L=64) 0.9722 0.8671 20 / 20 Substantial gains of +0.3911 AUC and +0.7495 TPR over input surface text
Binary Self-Report (Self-Report) WikiMIA-25 (QwQ-32B, L=32) 0.8840 (PosRate 0.415) โ€” Direct JSON judgment suffers from conservative bias and 19% parse failures
TraceMIA (lexicon) WikiMIA-25 (QwQ-32B, L=32) 0.9873 0.9741 31.6 / 12.8 Outperforms binary self-report by +0.103 AUC and smoothly scores truncated traces

Key Findings

  • Superiority over Latent Embeddings: TraceMIA outperforms BlackSpectrumโ€”the only prior trace-based methodโ€”by 0.14 to 0.39 AUC across all configurations. Projecting reasoning traces into dense continuous embeddings discards subtle, localized stylistic markers that discrete lexical filtering preserves.
  • Introspective Elicitation is Indispensable: Prompt ablation demonstrates that swapping introspective prompts with neutral summarization causes AUC to plunge from 0.9546 to 0.6002, with learned discriminative words dropping from 29.0 to 4.6. This verifies that lexical divergence is not an artifact of chain-of-thought generation per se, but an explicit manifestation of introspective memory retrieval.
  • Cross-Model Convergence on Non-Member Cues: Jaccard similarity analysis of lexicons across Qwen3-8B, DS-14B, and QwQ-32B shows that negative cue lexicons (\(W^-\)) exhibit a high top-10 overlap of 0.538 (sharing terms like future, hypothetical, cutoff, fictional). Across different post-training procedures, reasoning models share a universal epistemic hedging dialect when confronting unseen texts.
  • Sensitivity to Verbatim Memorization: Performing controlled character perturbations (10% to 30%) on member passages while preserving topics results in a monotonic decay of \(S_{\text{lex}}\). This confirms that TraceMIA specifically tracks exact textual memorization rather than broad topical familiarity.

Highlights & Insights

  • Cognitive Stylistic Perspective on Model Safety: The paper reframes membership inference from a rigid probability-density estimation problem into an interpretable cognitive linguistics inquiry, demonstrating that an LLM's certainty and hesitation are transparently encoded in its choice of words.
  • Auditing with Actionable Legal Evidence: By discarding opaque latent similarity metrics in favor of explicit lexical matching, TraceMIA provides human-auditable evidence chains, pinpointing the exact occurrences of recollection words that substantiate copyright infringement.
  • Exceptional Sample Efficiency and Truncation Resilience: Requiring only 25 reference samples per class, TraceMIA delivers superior data efficiency. Furthermore, when verbose reasoning cuts off before emitting the terminating JSON tag, TraceMIA reliably extracts membership signals from the partial trace where binary self-reports fail completely.

Limitations & Future Work

  • Prerequisite of Exposed Reasoning Traces: The method is strictly contingent on accessing the model's <think> trace; closed-source commercial APIs that censor or summarize internal reasoning remain out of reach.
  • Vulnerability to Intentional Style Defense: Because lexical markers are explicit in the generated text, model vendors could conceivably suppress hedging tokens or apply style-homogenization post-processing to evade detection.
  • Domain Dispersion in Creative Texts: In unconstrained literary domains like BookReasoning, lexical diversity reduces the density of common cue words, requiring larger reference budgets (\(K \ge 100\)) to match encyclopedic performance.
  • vs Token-Level Probability MIAs (Min-K%, Min-K%++, Gap-K%): Token-level methods require white-box logit access and fail under post-training distribution shifts in LRMs; TraceMIA operates strictly on black-box generated text and capitalizes on reasoning dynamics.
  • vs Continuation-Sampling Black-Box MIAs (SaMIA, CDD, SimMIA): Sampling-based approaches rely on expensive multi-temperature generation and edit-distance calculations over final outputs; TraceMIA extracts high-fidelity signals from a single introspective forward pass.
  • vs Latent Trace Encoders (BlackSpectrum): While BlackSpectrum maps traces to continuous latent spaces, obscuring interpretability and losing fine-grained cues, TraceMIA demonstrates that discrete, human-readable lexical patterns achieve superior accuracy alongside full audit transparency.

Rating

  • Novelty: โญโญโญโญโญ Pioneering discovery of the lexical dichotomy in introspective reasoning traces.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 5 models, 3 distinct corpora, multiple input lengths, and extensive ablations.
  • Writing Quality: โญโญโญโญโญ Exceptionally structured presentation, sound theoretical formulation, and lucid empirical analysis.
  • Value: โญโญโญโญโญ Delivers an interpretable, practical, and highly accurate tool for black-box copyright auditing in the reasoning model era.