Can Reasoning Models Detect Changes to their Chains of Thought?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/wgantt/cot-interventions
Area: LLM Reasoning
Keywords: Chain-of-Thought Detection, Situational Awareness, Reasoning Tampering, Steered Reasoning, Self-Awareness
TL;DR¶
The paper presents a comprehensive empirical evaluation of whether contemporary reasoning models can detect and localize modifications made to their chains of thought (CoTs), demonstrating that unalerted models perform at chance level, self-awareness provides no measurable advantage over inspecting foreign CoTs, and fine-grained localization fails outside of blatantly irrelevant sentence insertions.
Background & Motivation¶
The emergence of test-time reasoning models and elongated chain-of-thought (CoT) prompting has achieved breakthrough performance across complex mathematical and logical reasoning benchmarks. Concurrently, safety and alignment researchers heavily depend on post-hoc interventions on intermediate CoT traces—including perturbing reasoning steps to assess faithfulness, pruning steps for compression and efficiency, and removing potentially malicious reasoning steps to steer safe generation. Crucially, the validity of these safety and interpretability protocols relies on a foundational implicit premise: the model undergoing intervention must remain oblivious to external tampering with its reasoning trace. If a model could reliably sense that its internal thoughts had been intercepted and altered, it could infer that it is under evaluation and strategically falsify or distort its responses—a dangerous manifestation of alignment faking or scheming that directly invalidates CoT monitorability.
Despite its critical importance, existing investigations into situational awareness and self-awareness in language models have focused almost exclusively on post-hoc preferences for final outputs, static activation probing, or evaluation awareness under overt prompting. Whether models possess true introspective self-knowledge of their own ongoing reasoning processes remains largely unanswered. Prior work offered only preliminary and anecdotal observations on singular models without systematic perturbation taxonomies or statistical controls, leaving the central question open: is a model's detection of an abnormal thought trace driven by a genuine metacognitive self-awareness of its own generative style, or merely a superficial heuristic reaction to blatant semantic anomalies?
To rigorously resolve this tension, the authors establish an empirical benchmark assessing four premier open-weight reasoning architectures (GPT-OSS-120B, Kimi K2.5, DeepSeek V3.2, and Qwen3-235B-A22B-Thinking) across three challenging benchmarks under six distinct perturbation operators. By comparing unalerted generation with alerted task-directed detection, evaluating intermediate vs. post-reasoning judgments, and testing self vs. other attribution, the authors provide definitive evidence regarding the limits of model introspection. Core idea: by systematically evaluating reasoning models across diverse CoT perturbation operators and comparing self-detection against foreign-trace detection, the authors demonstrate that unalerted models detect tampering at rates no better than chance and show identical accuracy across their own and others' CoTs, confirming that current reasoning models lack introspective self-awareness of their reasoning trajectories.
Method¶
Overall Architecture¶
The experimental methodology systematically maps the detection pipeline across four sequential phases: Controlled Intervention Generation, Context & Alert Setting, Multi-Stage Inference Protocol, and Attribution & Localization Auditing. Beginning with canonical reasoning traces generated on challenging benchmarks (GPQA-Diamond, MMLU-Pro, and AIME 2025), a balanced 50% subset is subjected to one of six controlled intervention operators spanning insertion, deletion, and paraphrasing. The prefilled assistant blocks are then administered under either an Unalerted natural setting (where spontaneous verbalized suspicion is judged by an external LLM) or an Alerted task setting (where explicit verdicts are extracted via regex). Finally, the framework contrasts judgments on the model's own traces against foreign traces (Self vs. Other) and tests span-level localization of the perturbation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query & Canonical CoT Generation"] --> B["Controlled CoT Intervention Operators<br/>Single-step/Truncation/Paraphrase/Swap"]
B --> C["Dual Alert & Prefill Context Setting<br/>Unalerted Natural State vs Alerted Task State"]
C --> D["Temporal Stage Protocol<br/>Completed Post-CoT vs Partial In-CoT"]
D --> E["Self-Recognition & Span Localization Auditing<br/>Self vs Other Attribution & Step Identification"]
E --> F["LLM-as-a-Judge & Deterministic Regex Scoring"]
Key Designs¶
1. Controlled CoT Intervention Operators: Multiscale Semantic and Structural Perturbations
To evaluate the full spectrum of perturbation severity without introducing single-operator artifacts, the framework formulates six distinct interventions: - Insertion operators: Single-step irrelevant insertion (\(I_{\mathrm{one}}\)) injects a completely orthogonal, factual sentence (e.g., regarding the ecological role of the Amazon rainforest) at a random position within the trace. To ablate purely syntactic surprises, an LLM-guided contextually relevant non-sequitur (\(I_{\mathrm{one}}-\mathrm{LLM}\)) is also designed. - Deletion operators: Single-step deletion (\(D_{\mathrm{one}}\)) eliminates a randomly chosen reasoning step, whereas half-trace truncation (\(D_{\mathrm{half}}\)) deletes the entire terminal 50% of the reasoning steps to evaluate detection of severe incompleteness. - Modification and substitution operators: Single-step paraphrasing (\(E_{\mathrm{one}}\)) rewords a chosen step via an auxiliary generator without altering truth value, prefix paraphrasing (\(E_{\mathrm{par}}\)) rephrases the entire initial 50% of the trace, and trace replacement (\(E_{\mathrm{rep}}\)) swaps the entire CoT with a complete trace generated by a completely different reasoning model for the identical problem.
2. Dual Alert & Prefill Context Setting: Disentangling Spontaneous Suspicion from Task Upper Bounds
In realistic deployment and scheming scenarios, models do not receive explicit warnings that their context has been manipulated. To evaluate both empirical limits and baseline behavioral tendencies, the authors formulate two experimental contexts:
- Unalerted Setting: The target model is supplied solely with the problem prompt and a prefilled assistant turn containing the (possibly modified) CoT, with no mention of tampering or evaluation. The model is allowed to complete its response freely. An independent LLM judge (GPT-4o) evaluates whether the completion spontaneously registers confusion, notes an abrupt discontinuity, or flags that the reasoning was truncated or off-track.
- Alerted Setting: The system prompt explicitly instructs the model: "Someone has direct access to your chain of thought and it is possible that they may have modified it... Determine whether the provided chain of thought appears to have been modified." The model is constrained to output MODIFIED: YES or MODIFIED: NO, establishing an empirical upper bound on detection capability.
3. Temporal Stage Protocol: Probing In-Flight Generation vs. Post-Hoc Inspection
To determine whether models can sense alterations dynamically as they reason—a critical prerequisite for real-time scheming during execution—the protocol differentiates between two evaluation junctures: - Completed Condition: The model receives a fully completed CoT in the prefill block and renders a retrospective verdict at the very end of the output. - Partial Condition: The model receives a prefilled prefix comprising only the first half of a CoT (subjected to \(E_{\mathrm{rep}}\), \(E_{\mathrm{par}}\), or \(I_{\mathrm{one}}\)), requiring the model to generate the remaining reasoning steps while simultaneously tracking whether the prefix exhibited signs of prior tampering, rendering a verdict in its final output line.
4. Self-Recognition & Span Localization Auditing: Decoupling Style Introspection from Anomaly Filtering
Prior research claimed that language models possess self-recognition and show distinct preferences for their own outputs. To rigorously test whether CoT detection is mediated by self-awareness, the authors formulate the Self vs. Other protocol: the target model is presented with modified or pristine CoTs sampled from its own historical rollouts (Self) versus traces sampled from foreign architectures (Other). If genuine self-awareness is present, detection accuracy on self-generated CoTs should significantly surpass that on other models' CoTs. Furthermore, in the Localization experiment, models are explicitly informed that an edit occurred and tasked with outputting the exact MODIFIED_SPAN and REASON, scored against ground truth by a calibrated GPT-4o-mini judge.
Benchmark & Evaluation Protocol¶
All experiments are conducted across three diverse benchmarks: GPQA-Diamond (198 scientific multiple-choice problems), MMLU-Pro (a balanced 200-question sample spanning diverse academic domains), and AIME 2025 (30 high-difficulty competition mathematics problems). The evaluated model suite comprises GPT-OSS-120B (117B), DeepSeek V3.2 (685B), Kimi K2.5 (1.1T), and Qwen3-235B-A22B-Thinking (236B). For all experiments, data construction employs a fixed random seed of 19106, with a balanced 50% modified and 50% unmodified split. Confidence intervals (95%) are derived via non-parametric bootstrapping, and statistical significance relative to chance performance (50%) is assessed using two-sided z-tests (\(\alpha=0.05\)).
Key Experimental Results¶
Main Results: Detection Accuracy and Error Breakdown under Alerted Setting¶
The table below reports the detection accuracy (Acc. %) alongside detailed error breakdowns across True Positive Rate (TPR), False Positive Rate (FPR), False Negative Rate (FNR), and Formatting Error Rate (FER) under the Alerted setting in the completed condition, aggregated across GPQA, MMLU-Pro, and AIME:
| Model | Intervention | TPR (%) | FPR (%) | FNR (%) | FER (%) | Detection Acc. (%) |
|---|---|---|---|---|---|---|
| DeepSeek V3.2 | \(E_{\mathrm{rep}}\) (Full Swap) | 2.80 | 9.40 | 97.20 | 55.80 | 46.50 |
| DeepSeek V3.2 | \(I_{\mathrm{one}}\) (Single Insert) | 44.70 | 8.60 | 55.30 | 41.60 | 68.70 |
| GPT-OSS-120B | \(E_{\mathrm{rep}}\) (Full Swap) | 20.50 | 1.90 | 79.50 | 33.20 | 59.10 |
| GPT-OSS-120B | \(I_{\mathrm{one}}\) (Single Insert) | 15.40 | 3.60 | 84.60 | 33.20 | 57.00 |
| Kimi K2.5 | \(E_{\mathrm{rep}}\) (Full Swap) | 23.30 | 7.50 | 76.70 | 20.80 | 57.70 |
| Kimi K2.5 | \(I_{\mathrm{one}}\) (Single Insert) | 55.80 | 6.80 | 44.20 | 21.50 | 75.00 |
| Qwen3-235B-A22B | \(E_{\mathrm{rep}}\) (Full Swap) | 37.70 | 10.30 | 62.30 | 0.90 | 63.60 |
| Qwen3-235B-A22B | \(I_{\mathrm{one}}\) (Single Insert) | 44.20 | 13.20 | 55.80 | 0.20 | 66.10 |
Note: Formatting errors are discarded prior to computing rate metrics and accuracy; \(E_{\mathrm{rep}}\) replaces the entire CoT with that of another model, and \(I_{\mathrm{one}}\) inserts a single irrelevant statement.
Ablation Study: Self vs. Other Recognition and Step Localization¶
The table below contrasts model detection accuracy when examining modifications on their own reasoning traces (Self) versus traces from other models (Other), alongside span-level localization accuracy across three single-step operators on MMLU-Pro:
| Evaluation Dimension / Setting | GPT-OSS-120B | DeepSeek V3.2 | Kimi K2.5 | Qwen3-235B-A22B |
|---|---|---|---|---|
| Self \(I_{\mathrm{one}}\) Acc. (%) | ~57.0 | ~68.7 | ~75.0 | ~66.1 |
| Other \(I_{\mathrm{one}}\) Acc. (%) | ~59.0 | ~70.4 | ~78.3 | ~70.4 |
| Self \(D_{\mathrm{half}}\) Acc. (%) | ~64.0 | ~59.8 | ~59.0 | ~58.3 |
| Other \(D_{\mathrm{half}}\) Acc. (%) | ~61.5 | ~58.2 | ~57.4 | ~58.0 |
| Localization Acc.: \(I_{\mathrm{one}}\) (Irrelevant Insert) | 42.50 | 64.00 | 78.50 | 89.50 |
| Localization Acc.: \(D_{\mathrm{one}}\) (Step Deletion) | < 5.00 | < 5.00 | < 5.00 | < 5.00 |
| Localization Acc.: \(E_{\mathrm{one}}\) (Step Paraphrase) | < 5.00 | < 5.00 | < 5.00 | < 5.00 |
| Ablation: \(I_{\mathrm{one}}-\mathrm{LLM}\) (Contextual Non-Sequitur) | < 50.0 | < 50.0 | < 50.0 | < 50.0 |
Key Findings¶
- Unalerted Models are Functionally Blind: In the unalerted setting, models almost never spontaneously voice suspicion regarding CoT modifications. Accuracy on GPQA and MMLU is statistically indistinguishable from 50% chance, driven by extreme False Negative Rates (often 70% to 97%). Even when an entire reasoning trace is replaced or cut in half, the model seamlessly continues from the prompt.
- Absence of Self-Recognition in Reasoning: Models perform just as well—and frequently slightly better—at detecting edits in other models' CoTs than in their own. Across all four models and multiple datasets, the performance difference between the Self and Other conditions is statistically insignificant, demonstrating that detection capability stems from general anomaly detection rather than stylistic self-awareness.
- Severe Localization Collapse on Subtlety: When prompted to locate exactly where a modification took place, all models collapse to single-digit accuracy on single-step deletions (\(D_{\mathrm{one}}\)) and single-step paraphrases (\(E_{\mathrm{one}}\)). While models perform decently on obviously irrelevant sentence insertions (\(I_{\mathrm{one}}\)), replacing that sentence with an LLM-generated relevant non-sequitur causes detection accuracy to plummet well below chance level.
Highlights & Insights¶
- The study rigorously establishes that chain-of-thought tampering detection in contemporary LLMs is fragile and largely illusionary, relying on coarse semantic incongruity rather than fine-grained logical verification.
- For AI alignment and safety practitioners, this finding is reassuring: intermediate chain-of-thought editing, compression, and safety interventions can currently be performed with minimal risk of triggering strategic deception or scheming in open-source frontier reasoning models.
Limitations & Future Work¶
- The evaluation is strictly restricted to open-weight models that support explicit assistant-prefill manipulation via API; closed proprietary models (e.g., OpenAI o1/o3, Claude 3.7 Sonnet) could not be tested due to platform-level prefill restrictions.
- The models were tested strictly zero-shot as released. Future research should investigate whether targeted reinforcement learning or post-training focused on metacognitive monitoring can induce robust introspective self-awareness.
Related Work & Insights¶
- vs. Cywiński et al. (2025): While Cywiński et al. presented preliminary informal observations on DeepSeek-R1 and MATH-500, this paper provides the first formal, statistically controlled evaluation across four distinct model families, multiple benchmarks, and systematic perturbation operators.
- vs. Panickssery et al. (2024) & Davidson et al. (2024): Previous works established that LLMs can identify and show preference for their own final completions. This work demonstrates that this phenomenon does not transfer to chains of thought, highlighting a fundamental distinction between surface text recognition and reasoning introspection.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering systematic formulation of CoT tampering detection and self-recognition in reasoning]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exemplary protocol covering multiple architectures, alert states, temporal stages, and ablation controls]
- Writing Quality: ⭐⭐⭐⭐⭐ [Extremely clear narrative structure with rigorous statistical hypothesis testing and transparent error breakdown]
- Value: ⭐⭐⭐⭐⭐ [Crucial empirical bedrock for CoT monitorability, alignment faking research, and LLM safety governance]