Skip to content

DualStake: Dual-Path Confidence Calibration in Deep Research Agent

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/FloXXXt/DualStake
Area: LLM Agent
Keywords: Deep Research Agent, Confidence Calibration, Reinforcement Learning, Evidence Confidence, Stake Reward

TL;DR

Addressing the severe overconfidence of Deep Research agents during multi-step retrieval-augmented reasoning, this paper uncovers that post-retrieval Evidence Confidence (E-Conf) carries stronger uncertainty signals than post-generation Answer Confidence (A-Conf) and largely shapes it, motivating DualStakeโ€”a dual-path RL calibration framework using margin-clipped stake rewards that drastically cuts calibration error without sacrificing task accuracy.

Background & Motivation

Deep Research agents, which operate through multi-round retrieval, evidence aggregation, and decision-oriented generation, have emerged as a dominant paradigm for solving complex knowledge-intensive tasks. In real-world deployment and human-AI collaborative workflows, a reliable agent must not only generate accurate answers but also faithfully communicate its internal epistemic boundaries through well-calibrated confidence scores. Such calibrated uncertainty is crucial for downstream risk-sensitive decisions, reliable abstention, and building user trust. However, current Deep Research pipelines optimized with long-horizon reinforcement learning (such as GRPO) suffer from severe overconfidence, where expressed confidence values completely detach from the empirical likelihood of answer correctness.

Almost all existing confidence calibration frameworks for large language models treat inference as a single-step black box, eliciting verbalized confidence solely after final answer generation. This static post-answer setup fails fundamentally in agentic Deep Research workflows, where an agent's epistemic state evolves dynamically across multiple rounds of acquiring, filtering, and synthesizing external documents. Relying exclusively on end-of-sequence confidence cannot capture this evolving uncertainty trajectory, nor can it identify at which intermediate reasoning step false certainty creeps into the agent's internal state.

To investigate this evolving uncertainty, the authors augment the standard Search-R1 pipeline by introducing step-level confidence queries after each external retrieval step, defining the confidence elicited after the final retrieval step as Evidence Confidence (E-Conf), and comparing it directly with the standard Answer Confidence (A-Conf). Through extensive linear probing and activation patching across Transformer layers, they discover two surprising mechanistic phenomena: E-Conf representations naturally preserve a vastly stronger correctness-discriminative signal than A-Conf, and A-Conf is causally driven and shaped by the preceding E-Conf rather than independently reflecting final answer correctness. Core idea: jointly supervise post-retrieval Evidence Confidence and post-generation Answer Confidence within a dual-path reinforcement learning framework using margin-clipped, symmetric stake rewards (DualStake), aligning expressed confidence with empirical correctness while preventing extreme value divergence from corrupting task accuracy.

Method

Overall Architecture

DualStake builds upon the multi-step Deep Research interaction protocol instantiated by Search-R1. Given an input query, the agent autonomously issues search queries <search>query</search> or returns a final response. Whenever external evidence is fetched from the retrieval engine (an E5 dense encoder querying a Wikipedia index) and returned in <information>...</information> blocks, the agent is prompted to output its immediate step confidence within <confidence>X</confidence>. The confidence produced after the final retrieval step is designated as Evidence Confidence (\(c_{\text{evi}}\)), whereas the confidence verbalized after emitting <answer>...</answer> is designated as Answer Confidence (\(c_{\text{ans}}\)).

To align both confidence markers with ground-truth answer correctness without triggering degenerate policy updates, DualStake incorporates a dual-path stake-based calibration reward into the Group Relative Policy Optimization (GRPO) framework. The overall reward combines base correctness, formatting compliance, and two symmetric margin-clipped stake rewards dynamically controlled via linear warm-up scheduling.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query & Multi-Round Retrieval<br/>Input + Search Interaction"] --> B["Step Confidence Augmented Pipeline<br/>Interleaved E-Conf & Final A-Conf Elicitation"]
    B --> C["Margin-Clipped Stake Calibration Reward<br/>Symmetric Reward/Penalty Scaling with Bound Clamping"]
    C --> D["Dual-Path Joint Supervision & Warmup<br/>Synchronized Gradient Flow via E-Conf & A-Conf Paths"]
    D --> E["Calibrated Decision & Selective Abstention<br/>High Accuracy with Low ECE and Minimized AURC"]

Key Designs

1. Step Confidence Augmented Pipeline: Capturing Epistemic Dynamics and Representational Divergence Standard verbalized calibration elicits confidence only at the very end of the sequence, ignoring the epistemic updates occurring during intermediate retrieval steps. DualStake addresses this gap by prompting the agent to emit an explicit confidence integer between 1 and 9 within <confidence>X</confidence> immediately after receiving retrieved passages. Mechanistic analysis reveals that uncalibrated models already exhibit significantly lower Expected Calibration Error on E-Conf than on A-Conf across diverse benchmarks. Furthermore, layer-wise linear probing from Layer 0 to 28 demonstrates that hidden states at the E-Conf token position achieve consistently higher AUROC in predicting answer correctness than those at the A-Conf token position. Activation patching experiments further confirm this causal asymmetry: patching the final E-Conf token position with the mean hidden state from correct samples shifts the expected A-Conf digit upward by up to \(\Delta = +0.363\) (peaking at Layer 12), whereas patching the final answer token yields only an insignificant shift (\(\Delta = +0.073\)). This proves that A-Conf is strongly determined by E-Conf, underscoring the necessity of capturing and supervising confidence at the evidence stage.

2. Margin-Clipped Stake Calibration Reward: Preventing Extremes and Enforcing Symmetric Accountability To penalize unwarranted overconfidence while rewarding justified self-assurance, the calibration objective draws from strictly proper scoring rules through a betting analogy. Treating the normalized confidence score \(c \in [0, 1]\) as the model's stake on its reasoning, a correct answer (\(q \in [0, 1]\) evaluated via token-level F1) yields a positive reward proportional to \(c\), whereas an incorrect answer incurs an identical penalty proportional to \(c\). Directly optimizing the raw stake objective \((2q - 1) \cdot c\), however, leads policy gradients to push confidence values toward 0.0 or 1.0 to maximize reward or escape penalties, destabilizing training and degrading question-answering accuracy. To enforce stable optimization, DualStake introduces margin clipping:

\[R_s = (2q - 1) \cdot \min\Big(\max(c, m^-), m^+\Big)\]

where \(m^+\) caps the maximum gain from high confidence on correct answers (set to 0.9), and \(m^-\) prevents the agent from evading penalties by outputting near-zero confidence on incorrect answers (set to 0.1). This clipping bounds the effective reward gradient range and forces the model to learn nuanced, reliable intermediate probability scores.

3. Dual-Path Joint Supervision & Warmup Scheduling: Coordinated Alignment and Smooth Convergence Solely supervising Evidence Confidence fails to propagate adequate calibration to the end-user facing Answer Confidence, leaving A-Conf ECE elevated at 0.463. Conversely, supervising only Answer Confidence directly distorts generation logits and harms final accuracy (dropping from 0.375 to 0.367). DualStake unifies both supervisory signals into an end-to-end composite reward:

\[R_{\text{DualStake}} = R_{\text{corr}} + R_{\text{form}} + \alpha(t) \cdot R_{\text{evi}}^s + \beta(t) \cdot R_{\text{ans}}^s\]

where \(R_{\text{corr}} = q\) preserves core task fidelity, and \(R_{\text{form}} = 0.1\) enforces the structural presence of well-formed confidence tags. The two margin-clipped stake terms \(R_{\text{evi}}^s\) and \(R_{\text{ans}}^s\) supervise E-Conf and A-Conf independently. To protect early exploration while the agent learns basic search and answer structures, calibration coefficients \(\alpha(t)\) and \(\beta(t)\) follow a linear warm-up schedule: holding at 0 for initial training steps \(t_{\text{start}}\), then scaling linearly to peak weights \(\alpha = 0.25\) and \(\beta = 0.25\). This balanced allocation achieves optimal Pareto trade-offs across calibration error, discrimination, and task correctness.

Key Experimental Results

Main Results

Experiments are conducted on 8 challenging open-domain QA benchmarks: NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, and SimpleQA, using Qwen2.5-7B as the primary backbone. Comparisons include inference-only baselines, post-hoc methods (Temperature Scaling, Sequence Probability), and contemporary RL calibration approaches (MSCR, RLCR). Reported metrics include Exact Match accuracy (ACC), Expected Calibration Error (ECE, lower is better), Area Under the ROC Curve (AUC, higher is better), and Brier Score (BS, lower is better).

Method Type ACCโ†‘ ECEโ†“ AUCโ†‘ Brier Score (BS)โ†“
Qwen2.5-7B (Base) Inference Only 0.137 0.633 0.581 0.555
GRPO (Vanilla) Post-hoc Base 0.370 0.518 0.552 0.497
+ Temperature Scaling Post-hoc Calib. 0.370 0.151 0.538 0.249
+ Sequence Probability Post-hoc Calib. 0.370 0.322 0.511 0.335
MSCR (Xuan et al., 2026b) Training-time 0.335 0.300 0.647 0.299
RLCR (Damani et al., 2025) Training-time 0.367 0.188 0.682 0.258
DualStake (Ours) Training-time 0.375 0.178 0.712 0.220

Note: DualStake attains the highest AUC (0.712) and best Brier Score (0.220) while slashing ECE from 0.518 to 0.178 relative to standard GRPO. Crucially, DualStake achieves an accuracy of 0.375, outperforming all other training-time calibration methods without suffering the accuracy degradation seen in MSCR (0.335).

On Qwen2.5-7B-Instruct, DualStake maintains an ACC of 0.367 while achieving ECE 0.160, AUC 0.742, and BS 0.213 (compared to GRPO's ECE 0.480, AUC 0.644, BS 0.443). On the compact Qwen3-4B model, DualStake attains ACC 0.359, ECE 0.182, AUC 0.726, and BS 0.237, demonstrating solid cross-architecture robustness.

Ablation Study

The table below details the ablation on margin clipping and the allocation of supervision weights between E-Conf (\(\alpha\)) and A-Conf (\(\beta\)) across the 8 benchmarks using Qwen2.5-7B (with fixed total calibration weight \(\alpha + \beta = 0.5\)):

Configuration (\(\alpha, \beta\)) Margin Clipping Avg. ACCโ†‘ Avg. ECEโ†“ Avg. AUCโ†‘ Avg. BSโ†“ Observations
E-Conf Only (0.5, 0.0) w/o Margin 0.370 0.528 0.504 0.512 Unbounded stake leads to severe training instability
E-Conf Only (0.5, 0.0) w/ Margin 0.376 0.463 0.702 0.425 E-Conf supervision alone leaves A-Conf ECE elevated
A-Conf Biased (0.1, 0.4) w/ Margin 0.369 0.169 0.688 0.227 Skewing weight toward A-Conf drops AUC and task ACC
A-Conf Only (0.0, 0.5) w/o Margin 0.336 0.221 0.693 0.247 Unclipped A-Conf optimization severely harms accuracy
A-Conf Only (0.0, 0.5) w/ Margin 0.367 0.172 0.700 0.223 Slight accuracy drop compared to balanced configuration
Balanced Dual-Path (0.25, 0.25) w/ Margin (Default) 0.375 0.178 0.712 0.220 Best overall trade-off: highest AUC and lowest Brier Score

Key Findings

  • Margin clipping is essential for RL stability: Without margin clipping, the agent attempts to game the calibration objective by pushing confidence to extreme values (1 or 9), causing erratic reward trajectories during training. Adding the \([0.1, 0.9]\) clipping bounds stabilizes policy gradients and improves average accuracy by up to 0.039.
  • Single-path supervision is fundamentally insufficient: Supervising only E-Conf leaves A-Conf miscalibrated (ECE 0.463), whereas supervising only A-Conf leads to gradient conflict with answer generation, depressing accuracy to 0.367. The balanced dual-path configuration (\(\alpha = \beta = 0.25\)) achieves Pareto-optimal calibration and accuracy.
  • Substantial gains in selective prediction and abstention: When ranking examples by expressed A-Conf, DualStake reduces the area under the riskโ€“coverage curve (AURC) from 0.600 to 0.516 across four multi-hop QA datasets (a 13.5% relative error reduction). At a 20% coverage budget, selective accuracy increases from 53.5% to 65.9% on NQ and from 29.9% to 54.2% on SimpleQA.

Highlights & Insights

  • Unraveling the causal direction of uncertainty in agents: By combining layer-wise probing with activation patching, the paper presents compelling mechanistic evidence that external evidence intake serves as the primary uncertainty anchor in multi-step agents, establishing that end-of-sequence verbalized confidence is largely an echo of earlier evidence appraisal.
  • Clean stake design with bounded betting dynamics: Formulating confidence calibration as symmetric stake rewards with hard margin clipping elegantly resolves the long-standing tension between exploration, overconfidence penalties, and extreme score exploitation in RLHF/RLAIF workflows.
  • Seamless plug-and-play adaptability: The dual-path calibration formulation requires no architectural alterations or complex auxiliary heads, making it directly portable to any tool-use, multi-turn RAG, or long-horizon reasoning system.

Limitations & Future Work

  • Compute constraints and model scale coverage: The empirical investigation focuses primarily on 3B to 7B parameter models; validating dual-path dynamics on 70B+ scales and massive proprietary Deep Research systems remains an open direction.
  • Static step elicitation and uniform margins: The current pipeline elicits confidence after every retrieval step and uses globally static margin thresholds (0.1 and 0.9); developing adaptive elicitation schedules conditioned on reasoning depth or problem ambiguity would be a valuable next step.
  • vs Search-R1 (Jin et al., 2025): While Search-R1 pioneered autonomous retrieval and reasoning via RL, it optimizes solely for answer correctness and exacerbates overconfidence; DualStake augments the Search-R1 pipeline with step-level confidence queries and dual-path RL supervision.
  • vs RLCR (Damani et al., 2025) & MSCR (Xuan et al., 2026b): Prior RL-based calibration works focus strictly on post-answer confidence, failing to harness intermediate epistemic shifts. DualStake's joint supervision of E-Conf and A-Conf yields superior discrimination (AUC 0.712 vs 0.682/0.647) while better preserving task accuracy.

Rating

  • Novelty: โญโญโญโญโญ Pioneering mechanistic and algorithmic exploration of evolving step-level confidence in Deep Research agents.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation spanning 8 benchmarks, 3 model families, linear probing, activation patching, and selective prediction.
  • Writing Quality: โญโญโญโญโญ Rigorous conceptual framing, clear narrative progression from mechanistic analysis to reward formulation, and pristine presentation.
  • Value: โญโญโญโญโญ Highly impactful for advancing trustworthy AI, agentic abstention, and risk-aware deployment in autonomous research systems.