Skip to content

Measuring Source-Resolution Distortion in LLM Output Arbitration: A Source-Bounded Evaluation Framework

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/Ning311/A2R-Benchmark
Area: Information Extraction
Keywords: Arbitration Compression, Source-Resolution Distortion, Decision Ledger, LLM-as-a-Judge, Auditable Arbitration

TL;DR

Addressing the pervasive loss and misattribution of critical source structures when LLMs arbitrate among multiple candidate answers, this paper formalizes Source-Resolution Distortion (SRD), introduces a source-bounded benchmark, and proposes A2Rโ€”a framework that builds a sparse decision ledger before deterministic rendering to substantially cut conflict erasure and boost provenance recoverability.

Background & Motivation

In modern multi-agent systems and multi-LLM ensembles, large language models are increasingly deployed as judges or arbiters to synthesize multiple heterogeneous candidate outputs into a unified, coherent response. In high-stakes domains such as regulatory compliance, legal reasoning, and healthcare technology assessment, individual candidate models often capture complementary or conditionally conflicting perspectives. For instance, in evaluating whether a clinical AI product can enter an international market, one agent may conclude that market, procurement, and technical workstreams can proceed in parallel, while another identifies that regulatory classification serves as a strict prerequisite. A standard LLM arbiter typically synthesizes these responses into a smooth, fluent compromiseโ€”such as advising to "conduct all investigations in parallel while continuing to confirm the regulatory classification throughout the process."

The central tension of this paradigm is that the synthesized answer appears completely fluent and plausible on the surface, yet the underlying compression irreversibly damages the task-relevant structural logic of the source outputs. In information theory terms, arbitration compression folds distinct signals into an aliased composite, converting a hard procedural prerequisite into a generic process reminder. Existing research on LLM-as-a-Judge predominantly examines evaluator robustness, position bias, correlation with human holistic scores, or scalar score aggregation methods. Similarly, conventional RAG and citation evaluations solely verify whether individual output claims are factually supported by an external source. However, an arbitrated output claim can remain fully source-supported while an essential conflict between candidate models is erased, or a critical caveat is detached from the specific conclusion it was meant to qualify.

To bridge this fundamental gap, the authors draw an analogy to digital signal processing, where anti-aliasing filters protect distinguishable spectral components before irreversible downsampling. The paper formalizes a source-bounded evaluation framework where reference source structures are fixed prior to arbitration, enabling precise measurement of structural preservation and erroneous mapping in the final synthesis. Core idea: formalize multi-candidate output arbitration as a lossy compression process causing source-resolution distortion (SRD), and propose A2R (Auditable Arbitration for Resolution), which extracts query-conditioned source units into a sparse decision ledger before constraining deterministic rendering, effectively preventing structural aliasing.

Method

Overall Architecture

The framework establishes a formal definition of Source-Resolution Distortion (SRD) and introduces the A2R arbitration pipeline. The evaluation protocol focuses on pairwise inputs \(x=(q, A, B)\) (with extensions to \(k=3\) candidate setups). Prior to running any arbitration system, domain experts establish the source-side gold reference structure \(\mathcal{S}_q(x) = (\mathcal{U}_A, \mathcal{U}_B, \mathcal{E}_q, \mathcal{P}_q, \mathcal{Q}_q)\), specifying query-relevant atomic source units, query-conditioned logical relations, expected claim provenance, and requisite caveat attachments. The fidelity of any arbitration output is evaluated using six discrete checklist metrics.

To mitigate SRD, the A2R architecture shifts the processing paradigm from black-box holistic generation to a staged "resolve, then aggregate" pipeline: an extractor \(E\) first identifies atomic units \(\hat{\mathcal{U}}\) from the candidate answers; a decision module \(L\) evaluates local logical relations under query \(q\) to produce a structured, sparse decision ledger \(\hat{L}\); and finally, a deterministic renderer \(G\) serializes only authorized claims into the final answer \(\hat{y}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query q and Candidate Answers A, B"] --> B["Source Unit Extractor E<br/>Structured atomic unit extraction"]
    B --> C["Decision Module L<br/>Sparse decision ledger construction"]
    C --> D["Deterministic Constrained Renderer G<br/>Templated claim assembly & provenance inheritance"]
    D --> E["Final Output ลท and Auditable Ledger L"]

Key Designs

1. Source Unit Extractor E: isolating query-relevant atomic claims and constraints from raw candidate text
To prevent early semantic aliasing caused by processing unstructured candidate answers in bulk, extractor \(E\) maps candidates \(A\) and \(B\) into an explicit set of discrete source units \(\hat{\mathcal{U}} = \{u_i\}\). Each unit is represented as a five-tuple \(u_i = (\text{id}_i, \text{source}_i, \text{text}_i, \text{type}_i, \text{flags}_i)\), where \(\text{source}_i \in \{A, B\}\) retains candidate provenance, \(\text{text}_i\) preserves the exact text anchor, and \(\text{type}_i\) with \(\text{flags}_i\) annotate functional roles (such as factual claims, recommendations, caveats, or constraints). By selecting only query-conditioned units and tagging their scoping constraints early on, the extractor provides a high-fidelity discrete representation for subsequent relational reasoning, achieving an empirical extraction recall of approximately 0.91.

2. Decision Module L: constructing a sparse decision ledger of local relations and operational licenses
To prevent the model from smoothing over genuine logical contradictions or misplacing qualifications during holistic generation, decision module \(L\) evaluates small subsets of source units to build a sparse decision ledger \(\hat{L} = \{\ell_m\}\). Each record is defined by a six-tuple: $\(\ell_m = (d_m, I_m, r_m, o_m, c_m, p_m)\)$ where \(d_m\) is a decision ID; \(I_m \subseteq \hat{\mathcal{U}}\) contains the units involved in the local judgment; \(r_m\) denotes the query-conditioned semantic relation (such as equivalent, additive, conflict, or query_invalid); \(o_m\) specifies the resolution operation (merge, condition_sources, drop, or report_unresolved); \(c_m\) specifies the resulting claim authorized for the final output; and \(p_m\) logs the provenance tag (\([A]\), \([B]\), \([A+B]\), \([query]\), \([resolution]\), or \([new]\)). If a caveat qualifies a conclusion, \(I_m\) binds both units together so that \(c_m\) expresses their joint scope within the same statement, permanently preserving caveat attachment edges before generation occurs.

3. Deterministic Constrained Renderer G: eliminating provenance loss and caveat drift caused by free rewriting
To eliminate the severe structural degradation that occurs when an LLM is given free rein to write an aggregated summary, renderer \(G\) employs deterministic templates to assemble the claims authorized in ledger \(\hat{L}\): $\(\hat{y} = G(\hat{L})\)$ The renderer strictly follows the ordering of ledger records, merging adjacent claims on the same topic and inserting necessary grammatical connectives without introducing new propositional content. Provenance brackets, decision IDs, and caveat attachments are directly inherited from the ledger. In cases of preference ambiguity where neither criteria nor evidence suffices to resolve a conflict, the operation report_unresolved deterministically outputs both incompatible alternatives alongside the missing decision criterion. In the final output package, every substantive claim is linked via licensed_by to its underlying decision record \(d_m\), guaranteeing end-to-end auditability and structural recoverability.

Benchmark & Evaluation Protocol

To rigorously quantify SRD, the authors construct an evaluation suite comprising 520 pairwise cases (320 stratified controlled cases and 200 natural open-ended cases) and a 200-case \(k=3\) extension. Six discrete metrics evaluate output packages: - Conflict Erasure Rate (CER): fraction of gold conflicts that the system fails to resolve or condition, \(CER = \frac{n_{\text{eras}}}{n_{\text{res}} + n_{\text{cond}} + n_{\text{eras}}}\); - False Conflict Rate (FCR): fraction of complementary/equivalent items incorrectly reported as conflicting; - Unsupported Addition Rate (UAR): fraction of outputs that introduce unauthorized trap claims; - Harmonic Provenance (Prov-H): harmonic mean of provenance exposure \(P_{\text{cov}} = \frac{n_{\text{ec}} + n_{\text{ew}}}{n_{\text{app}}}\) and provenance accuracy \(P_{\text{acc}} = \frac{n_{\text{ec}}}{n_{\text{ec}} + n_{\text{ew}}}\), computed as \(\text{Prov-H} = \frac{2 P_{\text{cov}} P_{\text{acc}}}{P_{\text{cov}} + P_{\text{acc}}}\); - Caveat Preservation Rate (CPR): fraction of required caveats preserved without scope distortion; - Relation Fidelity (RF*): exposure rate and Macro-F1 of query-conditioned pairwise relation classification.

Key Experimental Results

Main Results

On the 520 pairwise benchmark cases, A2R-Decision is evaluated against seven baseline paradigms (using OpenAI gpt-5.4-2026-03-05 as the primary resolver backend, with relative cost normalized to Checklist Prompt at 1.00).

System CER โ†“ FCR โ†“ UAR โ†“ Prov-H โ†‘ CPR โ†‘ Rel. Cost
Free Judge 0.34 0.05 0.22 0.08 0.58 0.31
Source-Citing Prompt 0.30 0.09 0.18 0.36 0.60 0.46
Checklist Prompt 0.23 0.10 0.15 0.45 0.68 1.00
Source-Tagged Synthesizer 0.22 0.12 0.16 0.52 0.66 1.17
Post-hoc Attribution 0.24 0.09 0.17 0.55 0.64 1.54
CoT-based Synthesizer 0.24 0.06 0.16 0.31 0.64 1.38
LLM-Blender 0.27 0.04 0.16 0.18 0.63 1.22
A2R-Heuristic 0.19 0.08 0.17 0.60 0.66 0.67
A2R-Decision (Ours) 0.16 0.07 0.14 0.67 0.72 1.44

On 50 held-out validation cases, blind reconstruction and decision-utility evaluations confirm the practical benefits of A2R-Decision: reconstruction accuracy reaches 0.79 (vs. 0.58 for Checklist and 0.29 for Free Judge), decision accuracy reaches 0.74, reason coverage reaches 0.71, risk coverage reaches 0.66, and human completion time decreases from 218s to 128s.

Ablation Study

To isolate the individual contributions of the structured decision ledger and the deterministic renderer, the authors evaluate configurations swapping each component:

System Configuration Decision Structure / Renderer CER โ†“ Prov-H โ†‘ Rel. Cost
Checklist Prompt Compact checklist / LLM free final 0.23 0.45 1.00
Checklist-Templated Compact checklist / Deterministic template 0.21 0.53 0.95
A2R-LLM-Final A2R decision ledger / LLM free final 0.18 0.58 1.84
A2R-Decision A2R decision ledger / Deterministic template 0.16 0.67 1.44

On the 40 preference-ambiguous cases, integrating the report_unresolved operation increases correct conditional or unresolved behavior from 52.5% (21/40) to 65.0% (26/40), and decreases incorrectly forced answers from 40.0% (16/40) to 27.5% (11/40).

Key Findings

  • Answer preference decouples from source-resolution fidelity: In blind preference evaluations (Win+Tie/2) on stripped text, A2R-Decision achieves 53% against Checklist Prompt, 51% against Source-Tagged Synth, and 51% against Post-hoc Attribution. However, while free-form baselines (Free Judge, CoT-based Synth, LLM-Blender) produce fluent prose, their Prov-H collapses to 0.08โ€“0.31 and CER remains high (0.24โ€“0.34). This reveals that standard fluency and preference metrics fail to penalize severe structural information loss.
  • Dual protection from decision ledger and constrained rendering: Replacing free-form rewriting with a template (Checklist-Templated) increases Prov-H from 0.45 to 0.53, while replacing the checklist with an A2R ledger under free generation (A2R-LLM-Final) raises Prov-H to 0.58. Combining both in A2R-Decision delivers a synergistic leap to 0.67 Prov-H and 0.16 CER, proving that structural preservation requires constraining both intermediate reasoning states and final text generation.
  • Query-conditioned relation induction: In the 150-group query-contrast experiment, a query-aware classifier reaches 0.73 Macro-F1, outperforming query-blind (0.54) and contrast-query (0.45) variants. This demonstrates that logical relations between source claims are dynamically governed by the query rather than static surface text similarity.

Highlights & Insights

  • Cross-disciplinary anti-aliasing formulation: Translating the digital signal processing principle of anti-aliasing filtering prior to downsampling into NLP arbitration provides a rigorous, principled mental model for tackling multi-source compression artifacts.
  • Separation of exposure from accuracy in provenance: The formulation of Prov-H as the harmonic mean between provenance exposure (\(P_{\text{cov}}\)) and provenance correctness (\(P_{\text{acc}}\)) cleanly penalizes both ungrounded omissions and explicit hallucinations without distorting evaluation baselines.
  • High-efficiency deployment via heuristic extraction: The A2R-Heuristic variant achieves Prov-H of 0.60 and CER of 0.19 at just 0.67x the computational cost of Checklist Prompt, offering an attractive, lightweight option for high-throughput operational pipelines.

Limitations & Future Work

  • Source-bounded scope vs. open-world truth: SRD strictly evaluates whether the arbitration process faithfully preserves the structural elements provided by the input candidates; it does not verify whether the candidate statements themselves are empirically true. If all candidates supply an identical factual error, A2R will preserve it as a grounded source consensus.
  • Upstream extraction bottleneck: If extractor \(E\) misses a critical source unit (current recall is ~0.91), that omitted unit cannot be recovered by the ledger or renderer. While coverage reviews can mitigate this, they incur additional inference calls.
  • Scalability to large multi-agent ensembles: The primary framework is grounded in pairwise arbitration. Extending to \(k=3\) lowers Prov-H to 0.55 and raises CER to 0.22; scaling further to large-scale, heterogeneous agent swarms with complex multi-source compositional graphs remains an open challenge.
  • vs LLM-as-a-Judge and Evaluator Benchmarks (CheckEval, ContextualJudgeBench): While existing evaluator studies examine evaluator bias, position consistency, and prompt robustness, this paper is the first to directly model and quantify the structural distortion occurring inside the synthesized judgment output itself.
  • vs Fine-Grained RAG Citation (CiteEval, FActScore): Conventional citation evaluations check whether isolated claims are factually supported by passages. SRD specifically targets relational structure across multiple candidate outputsโ€”detecting whether genuine conflicts are silently erased or prerequisite scopes are shifted.
  • vs Multi-Candidate Ensembling (LLM-Blender, CoT Synthesizer): Standard ensemble methods fuse candidate outputs using end-to-end sequence-to-sequence or ranking models, inadvertently smoothing over essential nuance; A2R enforces explicit relational records via a ledger, making multi-model aggregation auditable and structurally faithful.

Rating

  • Novelty: โญโญโญโญโญ Formulates source-resolution distortion through an anti-aliasing lens and formalizes source-bounded arbitration compression.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 520 pairwise cases, \(k=3\) checks, blind reconstruction, decision utility, and cross-backend ablations.
  • Writing Quality: โญโญโญโญโญ Impeccable mathematical formulations, clear terminology, and compelling problem-driven narrative prose.
  • Value: โญโญโญโญโญ Essential benchmark and architectural blueprint for multi-agent arbitration, legal/medical report aggregation, and trustworthy LLM-as-a-Judge systems.