Skip to content

Teaching LLMs to Abstain via Fine-Grained Semantic Confidence Reward

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Notbesidemoon/FiSCoRe
Area: Hallucination Detection
Keywords: hallucination mitigation, selective abstention, semantic confidence, GRPO, knowledge boundary

TL;DR

Addressing the ambiguous knowledge boundary caused by coarse-grained global signals in LLM abstention alignment, this paper proposes FISCORE, an RL framework that derives sample-specific semantic confidence rewards from clustering cluster sizes via GRPO, and introduces the monotonically sound F1rel reliability metric.

Background & Motivation

Large language models (LLMs) have achieved remarkable success across diverse generative tasks, yet their propensity to hallucinateโ€”fabricating plausible but false information when confronted with questions beyond their knowledge scopeโ€”remains a severe threat to their deployment in high-stakes domains. To alleviate hallucinations, selective abstention fine-tuning has emerged as a primary alignment strategy to teach models to answer when knowledgeable and abstain otherwise. Supervised approaches typically partition training data into known and unknown queries based on correctness, or employ uncertainty measures like semantic entropy to substitute unknown responses with rejection phrases.

However, existing abstention methods overwhelmingly depend on coarse-grained, global signalsโ€”such as a single binary correctness label or an aggregated group-level entropy scalarโ€”to guide the model's decision-making. These approaches ignore the fine-grained, sample-specific confidence variations among distinct candidate responses under the same prompt. Consequently, the learned knowledge boundary remains blunt and imprecise: the model frequently either over-abstains on queries it could answer correctly (damaging helpfulness) or recklessly outputs hallucinations on low-confidence inputs (eroding truthfulness).

The core tension lies in the fact that aggregating multi-sample outputs into a single global uncertainty metric discards the nuanced confidence distribution across candidate trajectories, leaving individual rollouts without fine-grained supervisory signals. This paper's angle of attack is that generative semantic consensus among sampled responses naturally reflects an answer's intrinsic confidence, which can be directly decoupled at the individual sample level. Core idea: convert the cardinality of semantic equivalence clusters from sampled rollouts into fine-grained per-sample confidence rewards, train LLMs under Group Relative Policy Optimization (GRPO) to verbalize high confidence for consensus answers and express uncertainty for low-consensus ones to trigger post-hoc abstention, and evaluate alignment with a monotonically sound reliability metric \(F1_{\text{rel}}\).

Method

Overall Architecture

FISCORE (Fine-grained Semantic Confidence Reward) is a reinforcement learning framework designed to cultivate calibrated abstention behavior. Given an input question, the model generates a group of structured rollouts containing both an answer and a verbalized confidence tag. Next, an external bidirectional Natural Language Inference (NLI) model groups the candidate answers into semantically equivalent clusters. The size of each semantic cluster serves as a proxy for the sample's intrinsic confidence. Finally, this sample-level confidence reward is combined with auxiliary ground-truth accuracy and format rewards to optimize the policy network via Group Relative Policy Optimization (GRPO).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Question x"] --> B["GRPO Policy Rollout: G Samples<br/>Generate answer and confidence tags"]
    B --> C["Semantic Clustering & Sample Confidence<br/>DeBERTa Bidirectional Entailment"]
    C --> D["Fine-Grained Confidence Reward Rc<br/>Reward sure for large clusters, unsure for small"]
    B --> E["Auxiliary Rewards Computation<br/>Ground-Truth Accuracy Ra & Format Rf"]
    D --> F["Advantage Normalization & Policy Update<br/>GRPO Objective Parameter Optimization"]
    E --> F
    F -->|Converged Policy| G["Inference: Single Greedy Decoding Pass<br/>Execute post-hoc abstention when unsure"]

Key Designs

1. Fine-Grained Semantic Confidence Reward: Overcoming Global Entropy Aggregation

Prior uncertainty-based alignment techniques (such as SE-Tuning or GRPO-SE) collapse the semantic distribution of \(G\) rollouts into a single scalar semantic entropy \(H(x)\), thereby broadcasting an identical abstention label across the entire group and blinding the optimizer to per-sample confidence divergence. FISCORE overcomes this by leveraging a pretrained NLI model (DeBERTa-v2-xlarge-mnli) to evaluate bidirectional entailment between generated answers, greedily assigning semantically equivalent outputs into clusters \(C_i\). Because generative consensus directly mirrors the model's epistemic confidence, large clusters correspond to high-confidence generations, while small or singleton clusters signify speculative deviations. Setting the abstention threshold to \(\tau = \lceil G / 2 \rceil\), a target confidence label \(\hat{U}_i = \text{sure}\) is assigned if cluster cardinality \(|C_i| \ge \tau\), and \(\hat{U}_i = \text{unsure}\) if \(|C_i| < \tau\). The confidence reward for the \(i\)-th rollout is defined as:

\[R_c = \mathbb{I}[U_i = \hat{U}_i]\]

This sample-level mechanism provides granular supervisory feedback, teaching the policy to preserve majority consensus answers even under generally challenging prompts while discarding fragile outliers.

2. Auxiliary Accuracy and Structured Format Rewards: Suppressing Consensus Hacking

Relying exclusively on semantic consensus introduces a vulnerability to reward hacking, where the policy collapses into consistently generating an identical, high-consensus but fabricated false answer to artificially inflate cluster size and maximize \(R_c\). To prevent this mode collapse, FISCORE integrates an auxiliary accuracy reward \(R_a\), assigning 1 if the extracted answer is semantically equivalent to the ground-truth solution and 0 otherwise. Furthermore, to enable robust rule-based parsing of outputs, a format reward \(R_f\) incentivizes the presence of paired <answer> and <confidence> tags (0.125 per tag) and awards an additional 0.5 when the answer precedes the confidence tag. The composite reward is formulated as:

\[R_{\text{total}} = w_c R_c + w_a R_a + w_f R_f\]

with hyperparameters set to \(w_c = 1, w_a = 4, w_f = 2\). Crucially, a hard gating constraint is imposed: \(R_c\) and \(R_a\) are added only when the format is fully valid (\(R_f = 1\)), ensuring that the NLI clustering pipeline operates on clean, uncorrupted strings.

3. GRPO Optimization and Zero-Overhead Post-Hoc Inference: Group Relative Updates and Efficient Serving

The policy is trained using Group Relative Policy Optimization (GRPO), eliminating the need for a separate value/critic model. For each question \(x\), GRPO samples a cohort of \(G\) rollouts \(Y = \{y_1, \dots, y_G\}\) from the reference policy, normalizes advantages \(\hat{A}_{i,t}\) across the group via reward mean and standard deviation, and optimizes the policy parameters \(\theta\) under clipping ratio \(\epsilon\) and KL penalty \(\beta\):

\[\mathcal{J}_{\text{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min \left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{i,t} \right) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})\]

During downstream deployment, no multi-sample generation or clustering is performed. The model executes a single greedy decoding pass to output the answer and verbalized confidence. If <confidence> decodes to unsure, the system triggers post-hoc abstention, restricting the multi-sample clustering overhead entirely to offline training.

4. Monotonically Sound Reliability Metric \(F1_{\text{rel}}\): Remedying Negative Incentive Flaws

Evaluating abstention methods requires balancing helpfulness (\(F1_{\text{ans}}\), accuracy preservation on known questions) and truthfulness (\(F1_{\text{abs}}\), rejection rate on unknown questions). The authors demonstrate that the conventional Reliability Score (RS) suffers from a critical theoretical pathology: its partial derivative with respect to incorrect guesses on unanswerable questions (\(\frac{\partial \text{RS}}{\partial N_4}\)) can become strictly positive, perversely rewarding reckless hallucination over abstention. To rectify this, the paper defines \(F1_{\text{rel}}\) as the harmonic mean of \(F1_{\text{ans}}\) and \(F1_{\text{abs}}\):

\[F1_{\text{rel}} = \frac{2 \cdot F1_{\text{ans}} \cdot F1_{\text{abs}}}{F1_{\text{ans}} + F1_{\text{abs}}} = \frac{4 N_1 N_5}{4 N_1 N_5 + 2 N_2 N_5 + N_1 N_3 + N_1 N_4 + N_3 N_5 + N_4 N_5}\]

Mathematical derivation confirms that \(\frac{\partial F1_{\text{rel}}}{\partial N_4} \le 0\) holds universally, guaranteeing that any increase in catastrophic hallucinations strictly penalizes the model's reliability score.

Key Experimental Results

Main Results

The framework is trained on the relation prediction dataset Pararel (in-domain, ID) and comprehensively evaluated on the Pararel validation split alongside three out-of-distribution (OOD) knowledge-intensive QA benchmarks (TriviaQA, Natural Questions, and SciQ). Baselines are tested across Llama3-8B-Instruct and Qwen2.5-7B-Instruct backbones.

Model & Benchmark Metric FISCORE (Ours) GRPO-SE (RL Baseline) R-Tuning (SFT Baseline) ICL-IDK (Prompting)
Llama3-8B / Pararel (ID) \(F1_{\text{rel}}\) / Acc 74.3 / 59.4 73.9 / 35.1 79.0 / 40.5 63.2 / 31.7
Llama3-8B / TriviaQA (OOD) \(F1_{\text{rel}}\) / Acc 70.8 / 61.7 65.0 / 47.3 49.5 / 26.3 44.5 / 55.3
Llama3-8B / NQ (OOD) \(F1_{\text{rel}}\) / Acc 61.3 / 21.7 49.5 / 13.0 43.5 / 10.8 53.0 / 18.1
Llama3-8B / SciQ (OOD) \(F1_{\text{rel}}\) / Acc 62.3 / 52.4 57.0 / 46.4 39.5 / 14.6 54.5 / 56.4
Qwen2.5-7B / Pararel (ID) \(F1_{\text{rel}}\) / Acc 76.6 / 51.9 79.0 / 31.7 82.7 / 31.8 64.1 / 27.4
Qwen2.5-7B / TriviaQA (OOD) \(F1_{\text{rel}}\) / Acc 81.1 / 51.3 72.0 / 41.6 54.4 / 15.1 56.6 / 44.9
Qwen2.5-7B / NQ (OOD) \(F1_{\text{rel}}\) / Acc 66.8 / 20.4 55.3 / 14.2 38.7 / 4.5 43.3 / 19.4
Qwen2.5-7B / SciQ (OOD) \(F1_{\text{rel}}\) / Acc 64.6 / 58.0 61.0 / 50.7 48.3 / 19.9 24.5 / 60.5

Ablation Study

Ablations on reward components and abstention threshold \(\tau\) conducted on TriviaQA with Qwen2.5-7B-Instruct.

Config \(F1_{\text{ans}}\) \(F1_{\text{abs}}\) \(F1_{\text{rel}}\) Acc Note
FISCORE Full Model (\(\tau=5, w_a=4\)) 83.3 79.1 81.1 51.3 Balanced full model
w/o Format Reward (\(R_f\)) 71.5 43.6 54.2 (-26.9) 51.5 Tag parsing degradation biases clustering
w/o Confidence Reward (\(R_c\)) 69.5 2.1 4.1 (-77.0) 56.1 Model loses awareness of abstention
w/ Continuous Confidence Reward 83.0 82.9 82.9 (+1.8) 49.1 Fine-grained score allows flexible thresholds
Threshold \(\tau = 3\) 79.3 65.6 71.8 54.7 Overly eager answering, low abstention rate
Threshold \(\tau = 7\) 74.7 81.2 77.8 36.5 Over-conservative abstention impairs helpfulness
Accuracy Weight \(w_a = 0\) 0.0 98.2 0.0 2.3 Reward hacking leads to complete abstention collapse

Key Findings

  • Robust OOD Generalization Over SFT: SFT baselines (like R-Tuning) excel on ID training data (\(F1_{\text{rel}}\) of 79.0/82.7) but collapse catastrophically on OOD benchmarks like TriviaQA (plummeting to 49.5/54.4). FISCORE leverages reinforcement learning to maintain stable OOD performance at 70.8 and 81.1, demonstrating genuine generalizability.
  • Granular Signals Outperform Global Entropy: FISCORE and GRPO-SE share the identical RL pipeline and backbone; the sole difference is per-sample cluster labeling versus group-level entropy. FISCORE consistently beats GRPO-SE by 5.3 to 11.5 points in \(F1_{\text{rel}}\) across all OOD datasets, validating the importance of granular supervision.
  • Divergence Between Raw Knowledge and Metacognition: Llama3-8B exhibits higher raw QA accuracy than Qwen2.5-7B, yet its post-alignment \(F1_{\text{rel}}\) reliability is lower across multiple benchmarks. This highlights that a model's extensive knowledge store does not inherently guarantee self-awareness of its own boundaries.

Highlights & Insights

  • Cluster Cardinality as Sample-Specific Confidence Proxy: By translating multi-rollout NLI cluster sizes into discrete per-sample rewards, the framework elegantly bypasses the information bottleneck inherent in global entropy scalars.
  • Mathematical Correction of Benchmark Metrics: Proving the perverse positive derivative of Reliability Score (RS) and introducing the monotonic \(F1_{\text{rel}}\) establishes a theoretically sound evaluation standard for hallucination mitigation.
  • Decoupled Training Complexity and Lightweight Serving: While multi-sample rollouts and NLI clustering incur computational overhead during offline GRPO training, online serving requires only single-pass greedy decoding to trigger instant post-hoc abstention.

Limitations & Future Work

  • Static Threshold Setting: The abstention threshold \(\tau\) must be predetermined prior to training, preventing dynamic post-deployment adjustment across tasks with varying hallucination tolerance.
  • Potential Cognitive Bias in Autoregressive Confidence: Because the model generates the answer tokens prior to emitting the confidence tag, the answer prefix may induce confirmation bias during self-reflection.
  • Scale and Pretraining Domain Constraints: Experiments are constrained to 7B/8B model sizes and relation-extraction training sets; validation on larger models (e.g., 70B+) and diverse reasoning tasks remains an open direction.
  • vs R-Tuning / SE-Tuning: Supervised fine-tuning forces uncertain queries to memorize hardcoded refusal phrases ("I don't know"), which severely overfits and collapses on out-of-domain distributions. FISCORE trains self-reflective confidence via RL, retaining helpfulness alongside robustness.
  • vs GRPO-SE: GRPO-SE assigns a uniform group-level label using semantic entropy, ignoring variance among sampled responses; FISCORE rewards rollouts based on their specific cluster membership, providing richer learning gradients.

Rating

  • Novelty: โญโญโญโญโ˜† Converting cluster consensus into sample-level RL rewards paired with the monotonic \(F1_{\text{rel}}\) derivation is both technically sound and impactful.
  • Experimental Thoroughness: โญโญโญโญโ˜† Extensive testing across two major LLMs, four QA benchmarks, detailed ablations, and rigorous metric derivations.
  • Writing Quality: โญโญโญโญโญ Well-structured, mathematically rigorous, with clear motivational narratives and convincing counter-examples.
  • Value: โญโญโญโญโญ Establishes a highly effective, deployable paradigm for selective abstention and hallucination mitigation in language models.