Agreement Is Not Truth: Characterizing and Controlling Consistent-but-Wrong Reasoning in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mjdddol/agreement-is-not-truth
Area: LLM Reasoning
Keywords: Self-Consistency, Consistent-but-Wrong Reasoning, Risk-Controlled Prediction, Verifier Calibration, Test-Time Compute Scaling
TL;DR¶
The paper proves a theoretical lower bound demonstrating that self-consistency answer histograms alone cannot distinguish modal-correct from modal-wrong reasoning instances, and introduces RISC (Risk-Controlled Self-Consistency)—a framework that fuses verifier signals with majority ratios and calibrates acceptance thresholds via Learn-then-Test to guarantee finite-sample risk bounds while cutting API spend by ~9x.
Background & Motivation¶
In large language model (LLM) multi-step reasoning, self-consistency (SC) and its budget-adaptive variants are standard techniques for improving task accuracy, while modern test-time compute scaling frequently treats high sampling agreement as an operational proxy for confidence and correctness. However, equating consensus across independently sampled chains of thought with ground truth rests on an unverified premise. In practice, models repeatedly arrive at identical erroneous answers across independent paths due to shared knowledge blind spots, multi-step retrieval bottlenecks, or misleading lexical heuristics. Even when 90% of sampled trajectories agree on a single answer, existing certificates guarantee only that the returned answer matches the asymptotic majority, completely silent on whether that asymptotic majority is factually true.
This failure stems from a fundamental theoretical barrier in selective classification: decision rules that observe only the empirical output answer distribution \(\mu\) suffer from intrinsic indistinguishability. In the population limit, a question whose modal answer is correct and an unrelated question whose modal answer is wrong can induce the exact same answer distribution \(\mu\) under the same generator. Because an SC-only rule scores both instances identically, it cannot separate them; consequently, within any family of instances sharing the same distribution, no SC-only selective rule can drive conditional error below the class's inherent wrong-mode share. Empirically, consistent-but-wrong (CBW) errors are pervasive, governed by task structure rather than generator choice, and persist even across frontier reasoning models trained with reinforcement learning.
This paper breaks the informational bottleneck of generator-internal answer histograms by incorporating an orthogonal external verification signal into a calibrated risk-control pipeline. Core idea: prove that self-consistency distributions alone cannot guarantee correctness and establish a wrong-mode error floor, then formulate Risk-Controlled Self-Consistency (RISC) to combine majority ratios with external verifier scores under Learn-then-Test calibration, certifying finite-sample risk targets and abstaining when no valid threshold exists.
Method¶
To overcome the inability of self-consistency to certify correctness, RISC builds an end-to-end framework spanning theoretical error characterization, rationale-conditioned external verification, non-parametric score combination, and distribution-free finite-sample risk control. The generator samples \(K\) independent reasoning paths, aggregates the modal answer, extracts a representative rationale supporting that consensus, queries an external verifier for orthogonal validation, and decides whether to accept the majority answer or abstain under an admissibility-checked statistical certificate.
Overall Architecture¶
The RISC pipeline consists of four sequential stages: majority consensus extraction, rationale-conditioned verification, joint risk scoring, and Learn-then-Test risk calibration. A query problem is first processed by the generator to sample multiple chains of thought and derive the majority ratio. The first supporting chain is then audited by an external verifier to produce an acceptance rate. The two signals are fused into a composite score, which is checked against a calibrated threshold to output the majority prediction or trigger abstention.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query Question q"] --> B["Majority Consensus Extraction<br/>Sample K chains & compute majority ratio ρ"]
B --> C["Chain-of-Thought Verification<br/>Verifier audits first majority chain for yes-rate sv"]
C --> D["Joint Risk Score Computation<br/>Unweighted average s = 1/2(ρ + sv)"]
D --> E["Learn-then-Test Calibration & Admission<br/>Check UCB(τ*) ≤ α, accept if valid else abstain"]
E -->|Accept| F["Output Majority Prediction ŷ"]
E -->|Abstain| G["Trigger Safe Abstention"]
Key Designs¶
1. Majority Consensus Extraction: Answer Histogram Aggregation and Indistinguishability Bounds
Addressing the ungrounded reliance on consensus as proof of truth, this design samples \(K=16\) independent reasoning trajectories from generator \(g\) for question \(q\), extracts standardized answers, and computes the majority ratio \(\rho(q) = \frac{1}{K_{\text{valid}}} \sum_{k=1}^K \mathbb{I}[y_k = \hat{y}(q)]\). Beyond providing a baseline consensus signal, this component grounds Theorem 1 and Corollary 1: for any SC-only certifier \(C\) depending solely on the answer distribution \(\mu\), the conditional selective error within an indistinguishable sub-family \(T_\mu\) sharing distribution \(\mu\) is bounded below by:
$\(\beta_\mu(C) = \Pr[\text{mode}(\mu) \neq y(q) \mid C \text{ accepts}, q \in T_\mu] \ge \frac{p_2}{p_1 + p_2}\)$
where \(p_2\) denotes the proportion of modal-wrong instances in that sub-family. Empirical verification shows that 98.3% of high-agreement instances (\(\rho \ge 0.9\)) reside in mixed classes containing both correct and incorrect modes, demonstrating that sweeping consensus thresholds \(\rho\) cannot surpass this data-inherent error floor without external signals.
2. Chain-of-Thought Verification: Orthogonal Rationale Auditing via Independent Verifiers
Targeting the limitation that generator token probabilities and self-reflection fail to inject orthogonal evidence, this module queries an independent external verifier to evaluate the substantive reasoning behind the consensus. The pipeline extracts the representative chain \(r^*\)—defined as the first sample in sampling order whose extracted answer matches the majority prediction \(\hat{y}(q)\)—and submits the question, rationale, and candidate answer to the verifier for \(K_v=4\) binary queries. The verifier inspects the full step-by-step reasoning and emits a visible yes or no, yielding the empirical verifier score \(s_v(q) \in \{0, 0.25, 0.5, 0.75, 1.0\}\). Ablation studies confirm that stripping the reasoning rationale degrades the area under the risk-coverage curve (AURC from 9.6% to 14.4%), validating that the visible chain-of-thought is the indispensable carrier of factual and logical consistency.
3. Joint Risk Score Computation: Non-Parametric Monotone Signal Fusion
Addressing potential overfitting and boundary collapse in multi-source calibration, this design systematically explores product, minimum, logistic regression, and isotonic generalized additive models (GAM) before establishing the unweighted arithmetic mean as the optimal score function:
$\(s(q) = \frac{1}{2}(\rho(q) + s_v(q))\)$
This design preserves sensitivity across the full spectrum. When a generator exhibits unyielding consensus (\(\rho=1.0\)) on an erroneous trajectory, a low verifier score \(s_v\) penalizes the aggregate score \(s(q)\), breaking the dense tie cluster at \(\rho=1.0\) that blunts consensus-only ranking. Meanwhile, unlike multiplication or minimum pooling, the arithmetic mean prevents a single noisy verifier query from collapsing the score, achieving an AURC of 7.76% and outperforming agreement alone (10.18%) and verification alone (10.65%).
4. Learn-then-Test Calibration & Admission: Distribution-Free Finite-Sample Risk Guarantees
Resolving the arbitrary nature of heuristic acceptance cutoffs, this component applies Learn-then-Test (LTT) over a held-out calibration split \(\mathcal{D}_{\text{cal}}\) under exchangeability. For each candidate threshold \(\tau\) in the score grid \(\mathcal{T} = \{s(q) : q \in \mathcal{D}_{\text{cal}}\}\), the system computes the Clopper-Pearson upper confidence bound \(\text{UCB}(\tau)\) on selective error at confidence level \(1 - \delta / |\mathcal{T}|\) using a Bonferroni correction. It selects the least restrictive valid threshold:
$\(\tau^* = \min \{ \tau \in \mathcal{T} : \text{UCB}(\tau) \le \alpha \}\)$
If no candidate threshold satisfies \(\text{UCB}(\tau) \le \alpha\), the admissibility check rejects all candidates and RISC safely abstains on all test queries. This mechanism provides rigorous mathematical protection against under-calibrated risk, turning selective self-consistency into an auditable statistical protocol.
Loss & Training¶
RISC is a training-free test-time inference and risk-control framework. Generators and verifiers operate under fixed decoding parameters without gradient updates. Open-weight generators (Qwen2.5-7B, Qwen2.5-14B, Llama-3.1-8B, Mistral-7B-Instruct) are served locally via vLLM on a single NVIDIA RTX A5000 (24GB VRAM) using bfloat16 or NF4 quantization at \(T=0.7, p=0.95\). Proprietary endpoints (GPT-5.4-mini) operate at API defaults (\(T=1.0\)). Verification runs \(K_v=4\) queries with binary output extraction. Offline calibration uses 50/50 splits repeated over \(R=10\) random seeds with target risk bound \(\delta=0.10\).
Key Experimental Results¶
Main Results¶
The framework was evaluated across 8,222 valid instances spanning five datasets (GSM8K, StrategyQA, MATH500, ARC-Challenge, CommonsenseQA) and five generators, alongside evaluations on frontier reasoning models.
The table below reports the consistent-but-wrong rate ([email protected]) and conditional high-agreement error (Err@\(\rho \ge 0.9\)) across reasoning-trained generators on the two polar benchmark tasks:
| Generator Model | GSM8K [email protected] (%) | GSM8K Err@\(\rho \ge 0.9\) (%) | StrategyQA [email protected] (%) | StrategyQA Err@\(\rho \ge 0.9\) (%) | Description & Profile |
|---|---|---|---|---|---|
| Five-Generator Cohort Mean | 2.1 | ~2.0 | 20.8 | 15.0–29.7 | Baseline instruction-tuned cohort mean |
| DeepSeek-R1-Distill-Qwen-7B | 4.0 | 3.1 | 20.0 | 27.1 | Compact distilled reasoning model |
| DeepSeek-V4-Pro (Thinking) | 1.7 | 1.4 | 11.3 | 8.5 | Frontier long-reasoning model; preserves task gap |
| Claude Sonnet 4.6 (Extended Thinking) | 1.3 | 1.4 | 11.5 | 11.9 | Frontier model; retains >10% error at 90% consensus |
At an operational coverage of 0.70, the per-task selective error comparison between standard self-consistency (SC-only) and RISC is summarized below:
| Dataset | SC-Only Selective Error (%) | RISC Selective Error (%) | Error Delta \(\Delta\) (pp) | Analytical Context |
|---|---|---|---|---|
| GSM8K | 4.0 | 2.1 | -1.9 | Formal arithmetic; verification eliminates residual tails |
| MATH500 | 21.6 | 19.3 | -2.2 | Multi-step symbolic math; verifier filters spurious paths |
| ARC-Challenge | 5.9 | 4.7 | -1.3 | Science multiple choice; external audit rectifies distractors |
| StrategyQA | 23.9 | 25.0 | +1.1 | Implicit multi-hop retrieval; verifier gain saturated |
| CommonsenseQA | 9.7 | 12.0 | +2.3 | Commonsense priors; verifier noise causes minor penalty |
Ablation Study¶
The following table presents the ablation across score formulations, auxiliary distribution statistics, and score combinations evaluated on the \(N=6,154\) sub-cohort:
| Scoring Formulation / Baseline | AURC (%) [95% CI] | Coverage at \(\alpha=0.10\) (%) | Core Ablation Takeaways |
|---|---|---|---|
| \(\rho\) Alone (SC-Only) | 10.18 [9.21, 11.12] | 67.90 [63.00, 72.47] | Pure agreement; bounded by wrong-mode lower bound |
| Negative Entropy \(-\text{Entropy}\) | 10.09 [9.11, 11.03] | 67.57 [62.59, 70.85] | Distributional dispersion; indistinguishable from \(\rho\) |
| Margin (Top vs. Runner-Up) | 9.88 [8.90, 10.82] | 69.20 [64.61, 73.84] | Distance to competitor; slight improvement, no paradigm shift |
| Verifier Score \(s_v\) Alone | 10.65 [9.66, 11.66] | 67.50 [62.40, 71.80] | Isolated external audit; prone to point-estimate noise |
| \(\frac{1}{2}(\rho + s_v)\) (RISC Default) | 7.76 [6.95, 8.54] | 72.38 [68.64, 76.21] | Dual orthogonal fusion; lowers AURC by >2.4pp |
| Logistic Regression Combiner | 7.94 [7.13, 8.78] | 73.60 [69.10, 77.40] | Parametric fitting; within CI of unweighted mean |
| Isotonic GAM Combiner | 7.91 [7.11, 8.76] | 73.40 [69.00, 77.20] | Non-parametric monotone fit; fails to outperform mean |
Key Findings¶
- Task Identity Dominates Consistent-but-Wrong Variance: A two-way ANOVA reveals that dataset identity accounts for nearly all variance in [email protected] (\(\eta^2_{\text{task}} = 0.91, F(4, 16) = 50.7\)), whereas generator identity contributes negligibly (\(\eta^2_{\text{gen}} = 0.02\)). Clustered bootstrap yields \(\Pr[\eta^2_{\text{task}} > 5\eta^2_{\text{gen}}] = 1.0\), demonstrating that high-agreement failure is a topological feature of the reasoning task rather than an idiosyncratic generator glitch.
- Reasoning RL Attenuates but Does Not Eradicate CBW: While frontier reasoning models (DeepSeek-V4-Pro, Claude Sonnet 4.6) achieve higher base accuracy, their conditional error on StrategyQA at \(\rho \ge 0.9\) remains high at 8.5% and 11.9% (compared to 1.4% on GSM8K), confirming that scaling test-time search without orthogonal verification cannot close the truth-consensus gap.
- Selective Risk Reductions Concentrate at Conservative Coverage: Pooled across all instances, RISC achieves an error reduction of 3.01pp (\([-3.75, -2.29]\)) at 0.50 coverage, 0.87pp at 0.71 coverage, and 0.63pp at 0.91 coverage, effectively screening out high-confidence false assertions where acceptance errors are most costly.
- Local Generation with Lightweight Verification Outperforms Full API Scaling: At a 0.90 selective coverage target, pairing a local Qwen2.5-7B generator with GPT-5.4-mini verification achieves an metered API cost of \(1.17 per 1,000 questions—an ~8.9x reduction compared to full GPT-5.4-mini majority voting (\)10.47 per 1,000 questions)—while showing no statistically detectable error difference on GSM8K, StrategyQA, ARC, and CSQA.
Highlights & Insights¶
- Formal Invalidation of Agreement as Truth: Establishes a rigorous indistinguishability theorem proving that decision rules restricted to the self-consistency answer distribution cannot separate correct from incorrect consensus in the worst case, defining a formal wrong-mode error floor.
- Statistical Risk Guarantees for LLM Consensus: Integrates Learn-then-Test and Clopper-Pearson bounds into LLM test-time aggregation, enabling selective prediction that satisfies a pre-specified risk target \(\alpha\) with finite-sample statistical validity.
- Asymmetric Verification Efficiency: Leverages the fact that rationale verification is input-heavy but requires only a single-token binary output, allowing deployments to run generative sampling on affordable local hardware and reserve minimal external API calls for verification.
Limitations & Future Work¶
- Vulnerability to Task Shift and Exchangeability Breakdown: The LTT validity guarantee relies on exchangeability between calibration and deployment. Direct cross-task transfer results in severe violation rates of 31% to 63%, requiring per-task recalibration under distribution shift.
- Base Generator Accuracy Ceiling: RISC operates strictly as an acceptance and abstention layer; it does not rewrite or repair reasoning chains. When a generator completely misses the correct solution across all \(K\) trajectories, RISC can only control risk by abstaining, unable to expand coverage.
- Dependence on Transparent Thought Rationales: Auditing requires access to explicit chain-of-thought text. If reasoning models migrate entirely to hidden latent representations, external verification cannot inspect the intermediate steps, diminishing its auditing efficacy.
Related Work & Insights¶
- vs Adaptive Consistency (Aggarwal et al., 2023): Adaptive Consistency establishes stopping bounds to identify the asymptotic majority answer early, certifying consensus rather than correctness; this paper proves that the majority itself is frequently wrong, substituting external verification for consensus-only stopping.
- vs Confidence-Weighted Voting (Taubenfeld et al., 2025 / CISC): CISC weights chains by internal token probabilities to adjust point predictions; this work shows that internal confidence fails to resolve the 65.2% tie density at \(\rho=1.0\), whereas external verifiers introduce genuinely orthogonal information that cuts AURC by over 2.4pp.
- vs Self-Refine / Reflexion (Madaan et al., 2023; Shinn et al., 2023): Self-refinement prompts models to critique and revise their own outputs, which often fails when the model is trapped in a consistent misconception; RISC bypasses cyclic self-delusion by keeping candidates fixed and introducing an external auditor for calibrated risk control.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the first theoretical indistinguishability proof for self-consistency, providing a rigorous statistical risk-control framework for LLM reasoning consensus.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 5 datasets, 5 generators, 3 frontier reasoning architectures, and 6 verifiers, complete with two-way ANOVA and inter-rater reliability audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Impeccable narrative flow, mathematically precise proofs, and exceptionally transparent documentation of limitations, empirical costs, and transfer boundaries.
- Value: ⭐⭐⭐⭐⭐ Delivers high foundational value for dependable LLM deployment, risk-controlled decision-making, and cost-effective hybrid local-cloud inference pipelines.