Skip to content

QUORUM: Quality-Optimized Routing Using Multiple annotators

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/amazon-science/QUORUM
Area: LLM Reasoning
Keywords: Data Annotation Routing, Human-LLM Collaboration, Contextual Bandit, Bayesian Linear Regression, Multi-Annotator Aggregation

TL;DR

QUORUM introduces an uncertainty-free, budget-aware data annotation routing framework that extracts lightweight linguistic and semantic features to estimate instance difficulty, adaptively routes samples across humans and heterogeneous LLMs via online Bayesian regression, and triggers disagreement-guided re-annotation and consensus aggregation to boost quality while lowering costs.

Background & Motivation

As natural language processing models scale, collecting vast quantities of high-quality supervision remains the premier bottleneck in modern machine learning. While modern Large Language Models (LLMs) offer an unprecedented opportunity to substitute expensive human labor with fast and cost-effective automated labeling, their empirical reliability is fundamentally instance-dependent. For routine, syntactically clean inputs, LLMs frequently achieve parity with human annotators; however, on examples characterized by subtle semantic nuances, complex multi-hop reasoning, specialized domain knowledge, or cultural context, LLMs regularly succumb to hallucinations or flawed reasoning. Consequently, the core operational challenge in real-world data pipelines is no longer deciding strictly between humans or LLMs, but rather learning how to dynamically and cost-effectively route each instance across a heterogeneous pool of annotators under strict budgetary constraints.

Existing collaborative annotation routing approaches predominantly hinge on model-derived uncertainty or confidence estimates to determine when human expertise must be escalated. However, this paradigm introduces severe practical and computational bottlenecks in production pipelines. Commercial closed-source LLMs seldom provide well-calibrated logits or token probabilities, and relying on self-verbalized confidence scores introduces systematic biases while necessitating an extra forward inference step (the confidence estimation phase, CT) for every candidate instance before any routing decision can take place. Furthermore, alternative strategies that rely on supervised classifiers or retrieval augmentation (such as ARAIDA or SANT) incur substantial retraining overhead and cannot naturally generalize to open-ended generative tasks like abstractive text summarization where discrete label confidences do not exist.

To break free from this pathological dependency on model-generated uncertainty signals, this paper approaches instance difficulty directly from intrinsic, deterministic textual and semantic complexity. By bypassing model inference during difficulty appraisal and formalizing multi-annotator allocation as an online contextual bandit equipped with budget preservation and iterative consensus refinement, the authors propose a principled and universally applicable paradigm. Core idea: eliminate model-uncertainty inference overhead by estimating instance difficulty via lightweight lexical, syntactic, and semantic isolation features, dynamically learning annotator quality posteriors via Bayesian linear regression, and iteratively acquiring consensus annotations guided by posterior disagreement under budget constraints.

Method

Overall Architecture

The objective of QUORUM is to maximize cumulative annotation quality over a dataset \(\mathcal{D} = \{d_t\}_{t=1}^T\) subject to a global budget constraint \(B\). The router manages an action arm set \(\mathcal{A} = \{1, \dots, K, H\}\) comprising \(K\) heterogeneous LLM annotators (e.g., Claude Sonnet 4.5, Nova Pro v1.0, Qwen3-32B) and a human expert \(H\), with each arm associated with a known cost \(c_a > 0\). The overall architecture comprises four sequential phases: feature-space difficulty extraction and proactive calibration, budget-aware contextual bandit routing, disagreement-driven iterative re-annotation, and task-specific consensus aggregation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Dataset & Global Budget<br/>Input samples D with budget constraint B"] --> B["Feature-Space Difficulty & Active Calibration<br/>Lexical/syntactic features + semantic distance + Mahalanobis sampling"]
    B --> C["Dynamic Contextual Bandit Routing & Budget Guard<br/>Features + budget state + progress vector with decaying exploration"]
    C --> D["Disagreement-Driven Iterative Refinement & Consensus Aggregation<br/>Posterior uncertainty + label variance priority, weighted vote/centroid"]
    D --> E["High-Quality Final Annotations<br/>Accurate classification labels or consensus summaries"]

Key Designs

1. Feature-Space Difficulty Estimation & Active Calibration: Eliminating LLM Forward Overhead and Warming Up Posteriors Conventional routing frameworks require running costly LLM forward calls to elicit confidence scores or probability distributions prior to routing. QUORUM replaces this dependency with a deterministic feature representation \(\phi(d_t) \in \mathbb{R}^m\) consisting of two fast components: linguistic features (average word length, type-token ratio, average sentence length, normalized Flesch reading ease score, noun ratio, and verb ratio) that measure vocabulary sophistication, information density, and syntactic complexity; and embedding-based semantic isolation features computed via a lightweight multilingual encoder (Stella 1.5B) by taking the cosine distances to the \(k=5\) nearest neighbors in embedding space, where higher distances capture atypical or out-of-distribution instances. Prior to online routing, QUORUM dedicates roughly 20% of the affordable human budget (\(N_{\text{cal}} = \lfloor 0.20 \cdot \lfloor B / c_{\text{human}} \rfloor \rfloor\)) to an active calibration stage: samples exhibiting maximum Mahalanobis uncertainty \(u_t = \sqrt{(\phi(d_t) - \mu)^\top \Sigma^{-1} (\phi(d_t) - \mu)}\) are queried for human labels \(y_t\), and the resulting correctness indicators \(r_{a,t} = \mathbf{1}(\hat{y}_t^{(a)} = y_t)\) initialize the Gaussian posteriors of each LLM arm without incurring online decision risk.

2. Dynamic Contextual Bandit Routing & Budget Guard: Balancing Adaptive Exploration with Late-Stage Expert Preservation Static or greedy routing frequently exhausts the human budget on early ambiguous samples, starving critical late-stage instances of expert oversight. QUORUM constructs a comprehensive contextual state vector for each instance: \(x_t = [\phi(d_t), b_t, p_t, m_t]\), concatenating normalized instance features \(\phi(d_t)\), remaining budget fraction \(b_t\), dataset progression ratio \(p_t = t/T\), and a binary query mask \(m_t\) recording which annotators have already labeled sample \(t\). Each arm maintains posterior parameters \(\theta_a \sim \mathcal{N}(\mu_a, \Sigma_a)\) updated online via Bayesian linear regression. Decision-making incorporates a decaying exploration schedule \(\epsilon_t = \frac{\epsilon_0}{1 + \gamma t}\): with probability \(\epsilon_t\), an LLM is sampled uniformly at random; otherwise, the router exploits expected quality \(q_a = \mu_a^\top x_t\). If the highest predicted quality falls below a preset escalation threshold \(\tau_H\) (set to 0.5), the sample is escalated to human annotator \(H\). Crucially, the total human budget is partitioned into working and reserved components (\(B = B_{\text{working}} + B_{\text{reserved}}\)), structurally preventing early exhaustion of expert supervision before complex edge cases appear in later stages.

3. Disagreement-Driven Iterative Refinement & Consensus Aggregation: Breaking Single-Pass Bottlenecks via Multi-Annotator Synthesis Rather than freezing predictions after a single routing pass, QUORUM utilizes any remaining budget to execute iterative multi-annotator refinement on ambiguous instances. Samples are prioritized using a composite priority metric combining posterior epistemic uncertainty, empirical label disagreement, and lack of human verification: $\(P_t = D_t + \Delta_t + \mathbf{1}(\text{no human}), \quad D_t = \frac{1}{K}\sum_{a=1}^K x_t^\top \Sigma_a x_t, \quad \Delta_t = 1 - \frac{\max_y \text{count}(y)}{|Y_t|}\)$ Instances with the highest priority \(P_t\) iteratively receive additional annotations (up to a ceiling of 3 annotators per sample). Finally, task-specific consensus aggregation resolves conflicting annotations: for discrete classification, weighted majority voting is performed with a dominant human weight (\(w_H = 10, w_{\text{LLM}} = 1\)); for open-ended abstractive summarization where exact label matching fails, QUORUM computes pairwise cosine similarities across all candidate summary embeddings, rewards semantic agreement, and selects the candidate closest to the geometric embedding centroid as the final consensus output.

Benchmark & Evaluation Protocol

Theoretical Guarantees: Concentration Bounds & Asymptotic Zero Regret The paper establishes two foundational theoretical propositions underpinning QUORUM: - Proposition 3.1 (Posterior Concentration): Under bounded context \(\|x_t\|_2 \le L\) and bounded true parameters \(\|\theta_a^*\|_2 \le S\) with Gaussian noise \(r_{a,t} = x_t^\top \theta_a^* + \eta_t, \eta_t \sim \mathcal{N}(0, \sigma^2)\), when exploration samples feasible annotators infinitely often such that \(\lambda_{\min}(\Sigma_{a,t}^{-1}) \to \infty\), the quadratic form vanishes (\(x_t^\top \Sigma_{a,t} x_t \le \frac{L^2}{\lambda_{\min}(\Sigma_{a,t}^{-1})} \to 0\)), guaranteeing that quality estimates concentrate around true parameters with probability at least \(1-\delta\). - Proposition 3.2 (Vanishing Average Regret): Assuming a strictly positive margin \(\Delta_{\min} > 0\) separating the context-dependent oracle annotator \(a_t^*\) from sub-optimal arms, the greedy choice under decaying exploration \(\epsilon_t = \epsilon_0 / (1+\gamma t)\) selects the oracle annotator for all exploitation rounds beyond a finite threshold \(T_0\). Consequently, cumulative regret satisfies \(\lim_{T \to \infty} \frac{R(T)}{T} = 0\), proving asymptotic convergence to oracle routing.

Experimental Settings and Pricing Schedule Evaluations are conducted under two realistic regimes: AUDITOR STYLE (fixed human annotation budget varying from 10% to 70%, deciding between human and the cheapest LLM) and DOLLARS (fixed monetary budget with heterogeneous model pricing based on AWS Bedrock May 2026 rates: Claude Sonnet 4.5 at $0.05, Nova Pro v1.0 at $0.03, Qwen3-32B at $0.01, and human at $0.105 per annotation). Evaluation metrics include Accuracy, ROUGE-1, cumulative cost, and Machine Cumulative Accuracy (MCA): $\(\text{MCA} = \frac{\sum_{i \in \mathcal{H}} \mathbf{1}[\hat{y}_i \neq y_i]}{|\mathcal{H}|}\)$ where higher MCA signifies that expensive human interventions were allocated precisely to instances where standalone LLMs would have failed.

Key Experimental Results

Main Results

The following table summarizes performance and cumulative costs under the AUDITOR STYLE setting across 10% and 70% human annotation budgets (values in parentheses denote standalone average LLM baselines):

Dataset Samples Human Budget QUORUM (Ours) HyPAC CoAnnotating CDI SANT ARAIDA
AG's News (LLM: 0.834) 7,600 10% Acc / Cost
70% Acc / Cost
0.800 / \(144**<br/>**0.936** / **\)555 0.794 / $152
0.903 / $608
0.800 / $152
0.922 / $608
0.800 / $152
0.921 / $608
0.798 / $152
0.896 / $608
0.787 / $144
0.886 / $608
SST2 (LLM: 0.944) 870 10% Acc / Cost
70% Acc / Cost
0.968 / \(16**<br/>**0.990** / **\)64 0.966 / $17
0.969 / $70
0.948 / $17
0.978 / $70
0.941 / $17
0.985 / $70
0.940 / $17
0.975 / $70
0.939 / $17
0.949 / $70
MMLU-Redux (LLM: 0.633) 3,000 10% Acc / Cost
70% Acc / Cost
0.778 / \(57**<br/>**0.954** / **\)219 0.764 / $60
0.936 / $240
0.772 / $60
0.939 / $240
0.759 / $60
0.923 / $240
0.754 / $60
0.899 / $240
0.749 / $60
0.826 / $240
PubMed (LLM: 0.701) 30,000 10% Acc / Cost
70% Acc / Cost
0.683 / \(562**<br/>**0.905** / **\)2159 0.678 / $591
0.891 / $2366
0.678 / $591
0.890 / $2366
0.677 / $591
0.892 / $2366
0.673 / $591
0.714 / $2366
0.666 / $562
0.762 / $2366
CNN/DailyMail (LLM: 0.363) 11,490 10% R-1 / Cost
70% R-1 / Cost
0.425 / \(10,455**<br/>**0.816** / **\)72,501 0.423 / $11,604
0.806 / $80,544
0.423 / $11,604
0.810 / $80,544
0.423 / $11,604
0.807 / $80,544
— / —
— / —
— / —
— / —

Under the DOLLARS setting with strict monetary caps, QUORUM achieves even wider margins on the Pareto frontier. On PubMed at 70% budget, QUORUM attains an accuracy of 0.787 (MCA 0.242), outperforming HyPAC (0.673) by +11.4 percentage points. On CNN/DailyMail summarization at 70% budget, QUORUM reaches 0.638 ROUGE-1, substantially ahead of CoAnnotating (0.613) and HyPAC (0.407).

Ablation Study & In-Depth Analysis

Table A: Feature-Based Difficulty vs. Model Uncertainty Routing (Accuracy)

Budget Fraction Dataset LLM Uncertainty Features Random Features QUORUM Linguistic + Semantic Features (Ours)
10% AG's News 0.800 ± 0.002 0.697 ± 0.009 0.800 ± 0.001
10% SST2 0.948 ± 0.001 0.901 ± 0.012 0.968 ± 0.001
10% Global-MMLU (es) 0.741 ± 0.001 0.518 ± 0.012 0.749 ± 0.001
70% AG's News 0.932 ± 0.002 0.712 ± 0.007 0.936 ± 0.002
70% SST2 0.978 ± 0.002 0.921 ± 0.010 0.990 ± 0.001
70% Global-MMLU (es) 0.910 ± 0.001 0.692 ± 0.009 0.912 ± 0.001

Table B: Downstream Impact of Max Annotations per Instance (DOLLARS Setting)

Max Annotations Allowed AG's News (Accuracy) PubMed (Accuracy) MMLU-Redux (Accuracy) CNN/DailyMail (ROUGE-1)
1 (Single-pass decision) 0.852 0.740 0.721 0.387
2 (Up to 2 annotations) 0.873 0.756 0.754 0.398
3 (QUORUM Default) 0.891 0.791 0.766 0.410

Table C: Routing Preprocessing Overhead (Seconds, Scalability Comparison)

Method AG's News (7.6k) PubMed (30k) Global-MMLU (14.3k) Preprocessing Bottleneck Description
Random 0.007s 0.035s 0.014s Zero inference overhead
CoAnnotating 0.006s + CT 0.038s + CT 0.014s + CT Requires extra sampling for confidence estimation (CT ~2 hours)
CDI 0.503s + CT 0.547s + CT 0.661s + CT Requires prompt-based confidence verbalization (CT ~2 hours)
HyPAC 0.046s + CT 0.150s + CT 0.075s + CT Relies on PAC bounds over uncertainty estimates
SANT 18.052s + TT 161.014s + TT 40.853s + TT Model retraining & active learning triage (TT ~20 mins)
QUORUM (Ours) 0.494s + ET 1.467s + ET 0.593s + ET Lightweight embedding & text features only (ET ~4 mins)

Key Findings

  • Deterministic Features Rival and Surpass Model Confidence: Table A highlights that structured linguistic complexity and semantic distance features systematically match or outperform computationally heavy LLM verbalized confidence (e.g., SST2 10% budget yields 0.968 vs. 0.948). Meanwhile, Table C reveals that QUORUM bypasses multi-hour LLM prompting stages (CT) and classifier retraining (TT), slashing preprocessing down to roughly 4 minutes of local embedding extraction.
  • Redundancy Rescues Ambiguous and Generative Tasks: Table B confirms that increasing allowed annotations from 1 to 3 yields consistent monotonic gains, with CNN/DailyMail ROUGE-1 climbing from 0.387 to 0.410. Disagreement-driven multi-annotation provides critical error-correction where single-pass models fall prey to hallucinations.
  • Robustness Under Real-World Human Label Noise: On the SentiMP-En dataset featuring three real human annotators with subjective disagreement and intrinsic noise rates (8% to 17%), QUORUM achieves top accuracies (0.7271 to 0.7500) with a modest budget of 200 human labels, significantly surpassing CDI (0.6854 to 0.7292) and standalone LLMs (0.6040).
  • Seamless Cross-Lingual Portability: Without language-specific prompt tuning or calibrated confidence scores, QUORUM transfers directly to Spanish and Japanese benchmarks (Global-MMLU and XLSum), outperforming HyPAC and CoAnnotating while preserving low costs.

Highlights & Insights

  • Decoupling Instance Difficulty from Model Execution: Formulating task complexity via text-intrinsic lexical density, reading ease, and embedding space topological sparsity enables an elegant separation between "how difficult an example is" and "querying the model itself," eliminating vendor lock-in and token consumption during routing.
  • Contextual State with Explicit Budget Reserves: Dynamically feeding remaining budget fraction \(b_t\) and dataset scan progression \(p_t\) directly into the contextual bandit context \(x_t\), coupled with an explicit human reserve budget \(B_{\text{reserved}}\), provides robust mathematical insurance against premature budget exhaustion on early samples.
  • Dual-Mode Consensus Aggregation: Seamlessly bridging weighted majority voting for discrete classification with pairwise embedding similarity centroid selection for text summarization allows one unified architecture to govern both closed- and open-ended NLP data pipelines.

Limitations & Future Work

  • Authors' Stated Limitations: While validated across Spanish and Japanese, the experimental evaluation remains centered on major language benchmarks. Further investigation is necessary to test generalization across low-resource dialects and specialized, highly jargon-dense vertical domains (such as dense legal contracts or complex pathology reports).
  • Identified Weaknesses: The semantic isolation metric is bounded by the representational fidelity of the underlying text encoder (Stella 1.5B); severe distribution shifts could bias neighbor distance vectors. Furthermore, selecting the centroid summary in open-ended generation optimizes for median consensus but may systematically discard creative, high-variance candidate outputs.
  • Future Directions: Integrating reinforcement learning with human feedback (RLHF) directly into the bandit policy to support non-stationary stream routing, and replacing naive embedding centroid selection with lightweight LLM-as-a-judge critique mechanisms during summary aggregation.
  • vs. CoAnnotating (Li et al., 2023) & CDI (Gligorić et al., 2025): Both methods rely heavily on self-verbalized confidence or output entropy, necessitating supplementary inference passes and calibrated probabilities. QUORUM operates completely confidence-free, slashing preprocessing latency while achieving higher accuracy under tight budgets.
  • vs. HyPAC (Zeng et al., 2026): While HyPAC introduces PAC-style error guarantees via importance sampling, its partitioning mechanism remains anchored to model uncertainty. QUORUM establishes Bayesian posterior concentration and asymptotic zero regret proofs while incorporating dynamic multi-annotator consensus.
  • vs. ARAIDA (Huang et al., 2024b) & SANT (Huang et al., 2024a): These frameworks demand supervised classifier fitting and active learning triage (TT overhead), rendering them computationally intensive and inapplicable to open-ended summarization tasks. QUORUM handles both discrete labels and free-form text summarization out of the box.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Decouples routing from model-derived uncertainty using deterministic linguistic-semantic features and online Bayesian bandits.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Spans classification, QA, and summarization across multiple budget tiers, languages, and real-world noisy annotator setups.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear theoretical proofs, and well-structured empirical ablations.
  • Value: ⭐⭐⭐⭐⭐ Directly addresses the critical economic and reliability trade-offs of modern enterprise LLM data annotation pipelines.