MIRAGE: Reliability Assessment via Grounded Evidence for Table Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Table Reasoning, Reliability Assessment, Selective Prediction, Abstention Calibration, Uncertainty Estimation
TL;DR¶
Addressing the critical failure mode where large language models produce stable but structurally ungrounded tabular answers, this paper introduces MIRAGE, a structure-grounded reliability framework that estimates residual structural uncertainty across sampled execution traces and employs quantile-based calibration to achieve low-risk, interpretable selective table reasoning.
Background & Motivation¶
Table question answering and complex structured reasoning represent a pivotal milestone for deploying large language models into enterprise analytics, financial auditing, and scientific computing. In conventional language generation tasks, uncertainty estimation and selective prediction frameworks predominantly treat model reliability as a mathematical property of the predicted answer distribution. High token probability, sampled trajectory consensus (such as Self-Consistency), and low semantic entropy across paraphrased generations are widely trusted as universal hallmarks indicating that a model prediction is robust and can be safely returned to users. Under this paradigm, high agreement among independent stochastic generations is conventionally expected to protect against hallucinations and reasoning errors.
However, table reasoning exposes a fundamentally distinct and pervasive structural failure mode that answer-distribution metrics fundamentally fail to capture. Unlike free-form open-domain text generation, tabular reasoning requires resolving implicit multi-hop structural obligations: linking natural language mentions to specific table rows and ambiguous entities, binding question targets to the correct schema columns, applying intended symbolic aggregation functions, and coordinating joins across multi-table dependencies. Under subtle table ambiguities—such as colliding column headers (e.g., "Sales" versus "Revenue"), identical entity names in different geographic regions, or missing join tables—models regularly exhibit systematic inductive biases. Consequently, independent reasoning traces repeatedly arrive at the exact same hallucinated answer with high mutual agreement and inflated self-reported confidence. In these circumstances, high answer stability is merely a mirage that conceals profound structural grounding failures.
To establish trustworthy table reasoning systems, reliability assessment must look beyond superficial answer consistency and explicitly audit whether the intermediate reasoning process has satisfied the underlying structural constraints necessary for an answer to be trusted. We refer to the remaining hazard that bypasses answer distributions as "residual structural uncertainty": the risk that an answer is incorrectly grounded, improperly computed, or structurally unsupported by available tabular evidence. Core idea: This paper introduces MIRAGE (Model Inference Reliability Assessment via Grounded Evidence), a structure-grounded reliability framework that extracts multi-dimensional grounding, computation, and consensus signals from test-time execution traces, predicts answer correctness with a lightweight estimator, and applies quantile-based abstention calibration to produce risk-controlled answer-or-abstain decisions accompanied by lightweight diagnostic reliability certificates.
Method¶
Overall Architecture¶
MIRAGE operates as an external, non-invasive reliability verification layer positioned downstream of any base table reasoning model. Given a natural language query \(q\), table context \(T\), and a candidate answer generated by a base reasoner, MIRAGE does not require fine-tuning the underlying model parameters or generating secondary chain-of-thought rationales. Instead, it systematically audits observable test-time behaviors across sampled or perturbed reasoning traces to quantify residual structural uncertainty. The operational pipeline consists of five consecutive phases: perturbed trace sampling, multi-dimensional reliability signal extraction, lightweight reliability estimation, quantile-based abstention calibration, and test-time selective decision-making accompanied by diagnostic reliability certificates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Query q + Table Context T"] --> B["Perturbed Trace Sampling<br/>Decode N stochastic & schema-perturbed traces"]
B --> C["Reliability Signal Extraction<br/>Outcome consistency / Grounding / Computation signals"]
C --> D["Reliability Estimator<br/>Two-layer MLP predicts correctness probability p"]
D --> E["Quantile-Based Abstention Calibration<br/>Evaluate nonconformity s(x) on D_cal to set q_alpha"]
E -->|s(x) <= q_alpha| F["Safe Acceptance: Output Answer + Reliability Certificate"]
E -->|s(x) > q_alpha| G["Active Abstention: Output Abstain + Structural Failure Diagnosis"]
Key Designs¶
1. Perturbed Trace Sampling: Single greedy executions conceal structural fragility -> Stochastic and schema-perturbed decoding collects trace sets -> Exposes latent binding vulnerabilities
A single greedy execution trace rarely reveals whether a model's column binding or entity resolution is stable or accidental. To make structural vulnerabilities observable at inference time, MIRAGE prompts the base reasoner under stochastic decoding and controlled table serialization perturbations (such as column re-orderings and alternate serialization formats). This produces a structured trace collection \(\mathcal{Z}(q, T) = \{(z_i, y_i, E_i, m_i)\}_{i=1}^N\), where each trace \(z_i\) provides not only an executed candidate answer \(y_i\), but also an identified evidence region binary mask \(E_i\) and execution metadata \(m_i\) detailing chosen schema columns, resolved entity candidates, and symbolic operation types. By evaluating how the model behaves across perturbed schema presentations, MIRAGE effectively unmasks brittle bindings that appear stable under default prompt serializations.
2. Reliability Signal Extraction: Answer distributions miss structural failures -> Multi-dimensional extraction of grounding, computation, and consensus signals -> Fully captures residual structural uncertainty
Rather than collapsing trajectory information into raw answer frequencies, MIRAGE computes a compact, interpretable feature vector \(\mathbf{x} \in \mathbb{R}^d\) spanning three structural dimensions from observable trace behaviors without requiring gold annotations or hidden model activations: - Outcome Consistency: Quantifies answer dispersion by calculating execution entropy \(H_{\text{exec}} = -\sum_y P(y) \log P(y)\) and top-2 answer vote margin \(A_{\text{margin}}\). - Grounding Reliability: Evaluates schema binding volatility across \(K\) table serialization variations via \(V_{\text{schema}} = 1 - \frac{1}{K}\sum_{k=1}^K \mathbb{I}(\hat{y}(q, \widetilde{T}_k) = \hat{y}(q, T))\), alongside header collision \(C_{\text{header}}\) and schema ambiguity \(D_{\text{ambig}}\). For entity lookups, it tracks top candidate margin \(E_{\text{gap}}\) and resolution status \(E_{\text{status}}\). For supporting cell masks, it computes pairwise Intersection-over-Union (IoU) evidence stability: $\(S_{\text{evid}} = \frac{2}{N(N-1)}\sum_{i < j} \frac{|m_i \cap m_j|}{|m_i \cup m_j|}\)$ - Computation & Dependency Reliability: Detects operation type conflicts \(C_{\text{agg}}\) (e.g., sum vs. average), operation target divergence \(D_{\text{agg}}\), multi-table join conflicts \(C_{\text{join}}\), and explicit mentions of missing cross-table evidence \(M_{\text{missing}}\).
3. Reliability Estimator: Complex multi-signal interactions resist manual thresholding -> Shallow MLP maps trace features to correctness probabilities -> Provides smooth, robust reliability scoring
Because structural signals interact non-linearly (for instance, high schema ambiguity combined with low entity margin indicates acute failure, whereas isolated minor entropy may be benign), MIRAGE trains a lightweight two-layer Multi-Layer Perceptron (MLP) \(f_\phi(\mathbf{x}) \to \hat{p}\) to estimate the posterior correctness probability \(\Pr(\text{answer is correct} \mid \mathbf{x})\). By remaining intentionally shallow, the estimator avoids the compute overhead and hallucinations associated with second-stage LLM judges, requiring under two GPU hours to train on an A100. Furthermore, the estimator automatically formats the extracted structural evidence into a modular reliability certificate \(c = \{c_{\text{outcome}}, c_{\text{grounding}}, c_{\text{computation}}\}\), indicating whether schema, entity, operation, and dependency obligations are stable, ambiguous, or missing.
4. Quantile-Based Abstention Calibration: Fixed confidence thresholds fail under shifting data distributions -> Nonconformity calibration on held-out splits -> Delivers rigorous risk-controlled selective prediction
To avoid arbitrary heuristic thresholding that leads to either excessive false rejections or catastrophic errors, MIRAGE leverages conformal prediction principles to calibrate abstention thresholds. Defining the nonconformity score as \(s(\mathbf{x}) = 1 - \hat{p}\), the system evaluates scores across a held-out calibration set \(\mathcal{D}_{\text{cal}}\) drawn from exchangeable distributions. Given a user-defined significance level \(\alpha \in (0, 1)\), it identifies the empirical quantile threshold: $\(\hat{q}_\alpha = \text{Quantile}_{1-\alpha}\left(\{s(\mathbf{x}_j)\}_{(\mathbf{x}_j, y_j) \in \mathcal{D}_{\text{cal}}}\right)\)$ At inference time, MIRAGE accepts and emits candidate answer \(\hat{y}\) if and only if \(s(\mathbf{x}) \le \hat{q}_\alpha\); otherwise, it triggers a calibrated abstention decision. This formulation grounds selective decisions in a statistically rigorous data-driven operating point, prioritizing low selective risk without discarding reliable answers.
Benchmark & Evaluation Protocol¶
To isolate and stress-test failure regimes where answer-distribution confidence is deceptive, the paper constructs a controlled diagnostic benchmark comprising 2,000 instances balanced across four structural failure families: Schema Ambiguity, Entity Lookup Ambiguity, Aggregation Conflict, and Missing Cross-Table Evidence. The benchmark is balanced across answerable and abstain decisions (250 examples per family-label cell). Latent table schemas, database queries, and ground-truth answerability states are generated deterministically via Python, with Qwen3-8B utilized strictly for surface lexicalization to prevent taxonomy-to-label shortcut learning.
The evaluation enforces a strict 5-fold group-stratified rotation protocol (1,200 training / 400 calibration / 400 test per fold), ensuring that instances sharing latent schemas or generative templates remain in disjoint folds. The default backbone is Qwen3-8B with \(N_{\text{trace}} = 7\) sampled traces, accompanied by cross-backbone transfer on Llama-3.1-8B-Instruct and real-world tabular evaluation on WikiTableQuestions and FinQA. Performance is measured using global discrimination metrics (AUROC, ECE, Selective Accuracy) and selective risk at a fixed 50% coverage (Risk@50%, representing the error rate among accepted answers).
Key Experimental Results¶
Main Results¶
Under the controlled diagnostic stress test, MIRAGE was evaluated against seven prominent uncertainty estimation, semantic entropy, conformal, and hallucination detection baselines across five group-disjoint rotations. All baselines were re-fitted or re-calibrated on identical splits.
| Method | AUROC ↑ | ECE ↓ | Sel. Acc ↑ | Risk@50% ↓ |
|---|---|---|---|---|
| MaxProb | 0.38 ± 0.03 | 0.30 ± 0.04 | 0.50 ± 0.02 | 0.50 ± 0.03 |
| Predictive Entropy | 0.38 ± 0.04 | 0.30 ± 0.04 | 0.50 ± 0.02 | 0.50 ± 0.04 |
| Temperature-Scaled MaxProb | 0.38 ± 0.04 | 0.30 ± 0.03 | 0.50 ± 0.02 | 0.50 ± 0.03 |
| Self-Consistency | 0.55 ± 0.04 | 0.19 ± 0.03 | 0.54 ± 0.04 | 0.46 ± 0.03 |
| Self-Assessment P(True) | 0.57 ± 0.03 | 0.23 ± 0.04 | 0.52 ± 0.03 | 0.45 ± 0.03 |
| Verbalized Confidence | 0.62 ± 0.04 | 0.16 ± 0.03 | 0.51 ± 0.04 | 0.43 ± 0.03 |
| Semantic Entropy | 0.58 ± 0.03 | 0.33 ± 0.04 | 0.51 ± 0.04 | 0.43 ± 0.03 |
| Evidential Semantic Entropy | 0.55 ± 0.05 | 0.20 ± 0.03 | 0.54 ± 0.04 | 0.46 ± 0.04 |
| Selective Conformal Uncertainty (SConU) | 0.50 ± 0.04 | 0.33 ± 0.05 | 0.52 ± 0.03 | 0.48 ± 0.04 |
| Hallucination Detection (SelfCheckGPT) | 0.63 ± 0.03 | 0.17 ± 0.03 | 0.52 ± 0.03 | 0.43 ± 0.03 |
| MIRAGE (Ours) | 0.94 ± 0.02 | 0.08 ± 0.02 | 0.90 ± 0.03 | 0.16 ± 0.02 |
Ablation Study¶
To quantify the marginal value of each structural signal category, the authors conducted feature ablations by iteratively removing signal groups from the full MIRAGE interface under identical evaluation folds.
| Variant | AUROC ↑ | ECE ↓ | Risk@50% ↓ | Sel. Acc ↑ | Coverage |
|---|---|---|---|---|---|
| MIRAGE Full | 0.94 | 0.08 | 0.15 | 0.91 | 0.49 |
| w/o schema signals | 0.79 (-16.0%) | 0.14 (+75.0%) | 0.31 (+106.7%) | 0.78 (-14.3%) | 0.65 (+32.7%) |
| w/o entity signals | 0.87 (-7.4%) | 0.11 (+37.5%) | 0.22 (+46.7%) | 0.84 (-7.7%) | 0.58 (+18.4%) |
| w/o operation signals | 0.90 (-4.3%) | 0.10 (+25.0%) | 0.19 (+26.7%) | 0.87 (-4.4%) | 0.55 (+12.2%) |
| w/o dependency signals | 0.93 (-1.1%) | 0.09 (+12.5%) | 0.16 (+6.7%) | 0.90 (-1.1%) | 0.48 (-2.0%) |
| w/o evidence stability | 0.93 (-1.1%) | 0.08 (±0.0%) | 0.15 (±0.0%) | 0.90 (-1.1%) | 0.49 (±0.0%) |
| answer-distribution only | 0.55 (-41.5%) | 0.19 (+137.5%) | 0.46 (+206.7%) | 0.54 (-40.7%) | 0.91 (+85.7%) |
Key Findings¶
- Distributional metrics collapse when table failures are structural: Standard uncertainty estimators such as Self-Consistency, Semantic Entropy, and MaxProb achieve near-random AUROCs (0.38–0.58) and unacceptable risk levels (Risk@50% of 0.43–0.50). Empirical trace analysis shows that in three of the four diagnostic families, the agreement gap between answerable and unanswerable queries is under 0.06, confirming that consensus reflects model inductive bias rather than structural truth.
- Schema and entity grounding represent the primary pillars of tabular reliability: Ablating schema-grounding signals degrades AUROC by 16.0% and more than doubles Risk@50% (from 0.15 to 0.31), while ablating entity signals increases risk by 46.7%. In contrast, relying strictly on output distributions causes AUROC to collapse to 0.55 and risk to surge to 0.46.
- Selective risk reduction transfers to real-world tables: On the heterogeneous WikiTableQuestions benchmark, MIRAGE achieves comparable global AUROC to SelfCheckGPT (0.61 vs. 0.63) but drastically reduces selective risk at 50% coverage from 0.42 to 0.34. On FinQA, MIRAGE attains 0.69 AUROC and 0.32 Risk@50% (outperforming SelfCheckGPT's 0.65 and 0.38), validating that structural trace signals successfully isolate high-reliability answer subsets in natural environments.
Highlights & Insights¶
- Unveiling the "Confidence Mirage" in structured reasoning: The paper provides a formal, empirical demonstration that high sampling consensus and low semantic entropy can actively mislead selective prediction systems when reasoning failures originate from unresolved schema bindings or entity ambiguities.
- Lightweight, non-invasive architecture: MIRAGE introduces zero changes to base LLM parameters and requires no multi-agent debate or compute-intensive re-generation loops, learning an effective calibration head in under 2 GPU hours on a single A100.
- Modular diagnostic reliability certificates: Rather than acting as a black-box rejector, MIRAGE produces an interpretable reliability certificate decomposed into schema, entity, operation, and evidence statuses, providing clear operational provenance for enterprise decision support.
Limitations & Future Work¶
- Taxonomy transfer gaps: When evaluated on diagnostic failure families completely held out from both training and calibration, transfer performance exhibits degradation (AUROC drops to 0.72 for schema ambiguity and 0.53 for entity lookup), indicating that certain trace features retain domain-specific cues.
- Dependence on observable structured traces: Full MIRAGE relies on accessible intermediate execution signals (such as highlighted cell masks and bound column names). In pure black-box settings that output only raw text answers without elicitation, the system gracefully falls back to outcome-level signals with diminished discriminative power.
- Transition from selective abstention to autonomous repair: Current implementations terminate at binary answer-or-abstain decisions. A natural future evolution is leveraging diagnostic certificate feedback to trigger targeted schema re-ranking or entity disambiguation to autonomously recover from detected structural failures.
Related Work & Insights¶
- vs. Self-Consistency (Wang et al., 2023): Self-Consistency assumes that diverse reasoning chains converging on a single output indicate correctness; this paper demonstrates that in structured table QA, identical hallucinations frequently emerge from colliding entities, a flaw MIRAGE overcomes via grounding signal verification.
- vs. Semantic Entropy (Kuhn et al., 2023; Farquhar et al., 2024): Semantic Entropy measures dispersion over semantic meanings in open text; for tabular answers that are inherently succinct entities or numbers, semantic clustering cannot discern whether underlying SQL-like operations were valid, which MIRAGE resolves through operation conflict metrics.
- vs. Conformal Prediction Baselines (SConU / ConU, Wang et al., 2024, 2025): While generic conformal methods calibrate confidence over raw text outputs, MIRAGE pioneers the adaptation of quantile calibration directly to multi-dimensional residual structural uncertainty in relational data.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the concept of residual structural uncertainty and exposes the failure of distribution-based uncertainty in table reasoning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous 2,000-instance diagnostic benchmark with 5-fold group rotations, dual LLM backbones, lexicalization audits, and real-world transfer evaluations.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid mathematical formulations, compelling narrative structure, disciplined prose free of superficial bullet filler, and highly informative visual flows.
- Value: ⭐⭐⭐⭐⭐ Delivers an actionable, low-overhead blueprint for deploying safe and auditable table reasoning in high-stakes domain applications.