QUORUM: QUality-Optimized Routing Using Multiple annotators¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/amazon-science/QUORUM
Area: Multimodal VLM
Keywords: Data Annotation Routing, Multi-Annotator Consensus, Contextual Bandits, Budget-Aware Optimization, Feature-Based Difficulty Estimation
TL;DR¶
QUORUM discards the expensive model-derived uncertainty estimation paradigm, employing lightweight linguistic and embedding-based distance features with Gaussian posterior updates to dynamically allocate heterogeneous annotators and trigger multi-annotator consensus under strict budget constraints.
Background & Motivation¶
Data annotation remains a central bottleneck in modern natural language processing and multimodal foundation model alignment, where acquiring high-quality supervision at scale demands prohibitive human expenditure. While Large Language Models (LLMs) present a cost-effective, high-throughput alternative, their reliability is inherently instance-dependent: automated models solve straightforward, prototypical examples with remarkable precision, but stumble when confronted with nuanced pragmatic cues, subtle reasoning, rare domain vocabulary, or intricate cultural context. In production settings, the operational dilemma has evolved beyond a binary choice between pure human labeling and automated inference, turning instead into an active resource-allocation challenge: how to dynamically route instances across heterogeneous annotators of varying capabilities and costs under a fixed total budget.
Existing collaborative annotation frameworks predominantly rely on model-derived uncertainty or verbalized confidence scores to triage uncertain instances to human experts. However, this established paradigm suffers from two fundamental limitations in real-world deployment: first, it assumes access to well-calibrated confidence scores or internal logits, which are rarely furnished by proprietary commercial LLM APIs and frequently distorted by severe overconfidence; second, estimating uncertainty necessitates running at least one full LLM forward pass (or multiple stochastic rollouts) before a routing decision can be made, incurring non-negligible computational overhead, latency, and monetary expenditure that undermine the economic viability of adaptive routing in high-volume pipelines.
This paper tackles the challenge by circumventing reliance on model uncertainty altogether, proposing to predict instance difficulty directly from intrinsic data features and system budget states within a multi-armed bandit framework. Core idea: construct a lightweight context from surface linguistic metrics and semantic neighborhood distances, maintain Gaussian posterior estimates over annotator competence via Bayesian linear regression, and dynamically orchestrate multi-annotator refinement and consensus aggregation to achieve zero-inference-overhead budget-aware routing.
Method¶
Overall Architecture¶
QUORUM formalizes data annotation allocation across \(T\) samples under a budget constraint \(B\) as a sequential contextual bandit process. The candidate arm pool \(\mathcal{A} = \{1, \dots, K, H\}\) encompasses \(K\) heterogeneous LLM annotators of varying costs alongside human experts \(H\). Rather than querying candidate LLMs to evaluate prediction confidence upfront, QUORUM extracts deterministic linguistic statistics and semantic neighborhood distances from the raw text, concatenating them with remaining budget fractions, global dataset progression, and prior query masks into a rich context vector \(x_t\). The pipeline begins with an upfront calibration phase using a small slice of human budget to initialize annotator posteriors. It then proceeds to an online routing phase where candidates are chosen via decaying \(\epsilon\)-greedy exploration and posterior quality estimates. Finally, difficult or disputed instances are adaptively assigned additional annotations and resolved through task-specific consensus aggregation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Dataset and Annotation Budget Constraint"] --> B["Feature-Space Uncertainty & Initial Calibration<br/>Extract linguistic/semantic signals and anchor priors"]
B --> C["Gaussian Posterior Quality Modeling & Decaying Routing<br/>Estimate expected reward and escalate below threshold"]
C --> D["Budget-Aware Adaptive Refinement<br/>Trigger extra annotations via posterior variance and disagreement"]
D --> E["Task-Specific Consensus Aggregation<br/>Weighted majority voting / semantic centroid pooling"]
E --> F["High-Quality Final Aggregated Annotations"]
Key Designs¶
1. Feature-Space Uncertainty & Initial Calibration: Bypassing Model Inference to Establish Grounded Competence Priors
To eliminate the computational and financial burden of generating model predictions merely to assess routing eligibility, QUORUM shifts difficulty estimation to raw input features \(\phi(d_t)\). This representation combines six scalar linguistic metrics—average word length, type-token ratio, average sentence length, normalized Flesch reading ease score, noun density, and verb density—with a semantic distance vector measuring the cosine distance to the \(k\)-nearest neighbors extracted via a dense multilingual text embedder (Stella-1.5B). The linguistic markers capture syntactic convolution and vocabulary sophistication, while the embedding distance directly reflects out-of-distribution atypicality. Before live routing commences, approximately 20% of the affordable human budget is allocated to an active calibration phase: instances maximizing feature-space Mahalanobis uncertainty \(u_t = \sqrt{(\phi(d_t) - \mu)^\top \Sigma^{-1} (\phi(d_t) - \mu)}\) receive human labels, allowing the system to verify the initial accuracy of each LLM arm and establish empirical Gaussian posteriors without cold-start vulnerability.
2. Gaussian Posterior Quality Modeling & Decaying Routing: Balancing Competence Uncertainty and Cost Constraints
To continuously adapt to heterogeneous annotator performance across evolving text distributions, QUORUM models the quality parameter of each arm \(a \in \mathcal{A}\) as a Gaussian posterior \(\theta_a \sim \mathcal{N}(\mu_a, \Sigma_a)\). Each routing decision is conditioned on the full contextual vector \(x_t = [\phi(d_t), b_t, p_t, m_t]\), where \(b_t\) represents the remaining budget fraction, \(p_t = t/T\) reflects dataset progress, and \(m_t\) is a binary query history mask. Expected quality is evaluated via the inner product \(q_a = \mu_a^\top x_t\). To avoid premature convergence onto low-cost, subpar models during early iterations, QUORUM employs a decaying exploration schedule \(\epsilon_t = \epsilon_0 / (1 + \gamma t)\), selecting an LLM uniformly at random with probability \(\epsilon_t\). In exploitation rounds, if the predicted quality of the best-performing LLM \(q_{a^*}\) fails to meet the human escalation threshold \(\tau_H\) (empirically set to 0.5), the sample is escalated to human supervision. Observed feedback updates \(\mu_a\) and \(\Sigma_a\) analytically via conjugate Bayesian linear regression updates.
3. Budget-Aware Adaptive Refinement: Overcoming Single-Pass Routing with Targeted Multi-Annotator Supervision
Recognizing that ambiguous instances situated near complex decision boundaries cannot be reliably resolved through a single pass, QUORUM introduces an iterative refinement mechanism supported by a partitioned budget \(B = B_{\text{working}} + B_{\text{reserved}}\). The reserved budget prevents greedy exhaustion of human resources on early instances. After the primary pass concludes, remaining funds are directed toward instances ranked by a priority score combining posterior epistemic uncertainty, empirical disagreement, and missing human oversight: $\(P_t = D_t + \Delta_t + \mathbb{I}(\text{no human})\)$ where \(D_t = \frac{1}{K}\sum_{a} x_t^\top \Sigma_a x_t\) measures average posterior variance across candidate models, \(\Delta_t = 1 - \frac{\max_y \text{count}(y)}{|\mathcal{Y}_t|}\) quantifies label disagreement among collected annotations \(\mathcal{Y}_t\), and \(\mathbb{I}(\text{no human})\) introduces a penalty for unverified instances. High-priority instances receive up to three sequential annotations from distinct arms, allowing collective disagreement to trigger decisive human arbitration.
4. Task-Specific Consensus Aggregation: Heterogeneous Reliability-Weighted Voting and Semantic Centroid Selection
Once multi-source annotations are collected, QUORUM synthesizes them into final outputs tailored to the downstream task format. For discrete closed-ended tasks (classification and multiple-choice QA), consensus is resolved via reliability-weighted majority voting, where human judgments are assigned an authoritative weight \(w_H = 10\) against LLM weights \(w_{\text{LLM}} = 1\). For open-ended abstractive summarization where verbatim matching is impossible, QUORUM measures semantic consensus using pairwise cosine similarity across summary embeddings: $\(r_t = \frac{2}{|\mathcal{Y}_t|(|\mathcal{Y}_t|-1)} \sum_{i \neq j} \cos(e_i, e_j)\)$ Instances exceeding a consensus threshold terminate early, and the final predicted summary is chosen as the candidate annotation exhibiting minimum Euclidean distance to the embedding centroid of the entire generated ensemble.
Loss & Training¶
QUORUM updates online posterior parameters without gradient backpropagation, relying on exact Bayesian linear regression recurrences: $\(\Sigma_{a, t+1}^{-1} = \Sigma_{a, t}^{-1} + \frac{1}{\sigma^2} x_t x_t^\top, \quad \mu_{a, t+1} = \Sigma_{a, t+1} \left( \Sigma_{a, t}^{-1} \mu_{a, t} + \frac{r_{a, t}}{\sigma^2} x_t \right)\)$ Priors are initialized uninformatively for LLM arms (\(\mu_a = \mathbf{0}, \Sigma_a = \mathbf{I}\)), while human experts are endowed with an optimistic, low-variance prior (\(\mu_H = 0.3 \cdot \mathbf{1}, \Sigma_H = 0.5\mathbf{I}\)). Default hyperparameters set initial exploration \(\epsilon_0 = 0.3\), decay factor \(\gamma = 0.005\), human escalation threshold \(\tau_H = 0.5\), maximum annotations per instance to 3, and consensus stopping to 0.66. The authors provide theoretical proofs demonstrating that under bounded context assumptions, posterior variance contracts asymptotically and the time-averaged regret vanishes (\(\lim_{T \to \infty} R(T)/T = 0\)), guaranteeing asymptotic convergence toward an oracle routing policy.
Key Experimental Results¶
Main Results¶
QUORUM was extensively evaluated across both AUDITOR STYLE (allocating 10% to 70% human annotation budgets) and DOLLARS settings (fixed monetary budgets under realistic commercial pricing: Claude Sonnet 4.5 at $0.05, Nova Pro at $0.03, Qwen3-32B at $0.01, and humans at $0.105 per item). Benchmarks encompass classification (AG's News, SST2, IMDB, PubMed), multiple-choice reasoning (MMLU-Redux, Global-MMLU), and abstractive summarization (CNN/DailyMail, XLSum).
Table 1: Main performance and cost comparison under the AUDITOR STYLE setting across human budget levels
| Dataset | Metric | Method | 10% Budget (Perf / Cost) | 30% Budget (Perf / Cost) | 50% Budget (Perf / Cost) | 70% Budget (Perf / Cost) |
|---|---|---|---|---|---|---|
| AG's News (LLM: 0.834) | Accuracy ↑ / Cost ($) ↓ | Random | 0.729 / 152 | 0.805 / 304 | 0.794 / 456 | 0.804 / 608 |
| ARAIDA | 0.787 / 144 | 0.834 / 281 | 0.861 / 418 | 0.886 / 555 | ||
| CDI | 0.800 / 152 | 0.841 / 304 | 0.887 / 456 | 0.921 / 608 | ||
| CoAnnotating | 0.800 / 152 | 0.844 / 304 | 0.889 / 456 | 0.922 / 608 | ||
| HyPAC | 0.794 / 152 | 0.840 / 304 | 0.875 / 456 | 0.903 / 608 | ||
| QUORUM (Ours) | 0.800 / 144 | 0.845 / 281 | 0.890 / 418 | 0.936 / 555 | ||
| MMLU-Redux (LLM: 0.633) | Accuracy ↑ / Cost ($) ↓ | Random | 0.706 / 60 | 0.691 / 120 | 0.767 / 180 | 0.901 / 240 |
| CDI | 0.764 / 60 | 0.832 / 120 | 0.889 / 180 | 0.939 / 240 | ||
| CoAnnotating | 0.772 / 60 | 0.836 / 120 | 0.884 / 180 | 0.936 / 240 | ||
| HyPAC | 0.759 / 60 | 0.808 / 120 | 0.865 / 180 | 0.923 / 240 | ||
| QUORUM (Ours) | 0.778 / 57 | 0.815 / 111 | 0.891 / 165 | 0.954 / 219 | ||
| PubMed (LLM: 0.701) | Accuracy ↑ / Cost ($) ↓ | Random | 0.616 / 591 | 0.708 / 1183 | 0.731 / 1775 | 0.752 / 2366 |
| CDI | 0.677 / 591 | 0.749 / 1183 | 0.820 / 1775 | 0.892 / 2366 | ||
| CoAnnotating | 0.678 / 591 | 0.750 / 1183 | 0.821 / 1775 | 0.890 / 2366 | ||
| HyPAC | 0.678 / 591 | 0.749 / 1183 | 0.820 / 1775 | 0.891 / 2366 | ||
| QUORUM (Ours) | 0.683 / 562 | 0.762 / 1094 | 0.835 / 1627 | 0.905 / 2159 | ||
| CNN/DM (LLM: 0.363) | ROUGE-1 ↑ / Cost ($) ↓ | Random | 0.402 / 10455 | 0.521 / 31137 | 0.619 / 51819 | 0.781 / 72501 |
| CDI | 0.423 / 11604 | 0.556 / 34584 | 0.682 / 57564 | 0.807 / 80544 | ||
| CoAnnotating | 0.423 / 11604 | 0.550 / 34584 | 0.677 / 57564 | 0.810 / 80544 | ||
| HyPAC | 0.423 / 11604 | 0.550 / 34584 | 0.677 / 57564 | 0.806 / 80544 | ||
| QUORUM (Ours) | 0.425 / 10455 | 0.560 / 31137 | 0.678 / 51819 | 0.816 / 72501 |
Ablation Study¶
The authors conducted thorough ablations across three core operational dimensions: (1) feature-based difficulty signals versus model uncertainty and random features; (2) downstream performance progression as the multi-annotator limit scales from 1 to 3; and (3) pre-decision computational overhead across routing paradigms.
Table 2: Ablation analysis on feature formulation, multi-annotator depth, and pre-routing latency
| Experiment Dimension | Setting / Dataset | Metric | Empirical Value | Key Observation & Analysis |
|---|---|---|---|---|
| Feature Signal Comparison (10% Budget) | AG's News (LLM-Uncertainty / Random / Ours) | Accuracy | 0.800 / 0.697 / 0.800 | Lightweight static features perfectly match verbalized uncertainty |
| SST2 (LLM-Uncertainty / Random / Ours) | Accuracy | 0.948 / 0.901 / 0.968 | Features surpass confidence-based routing by +2.0% absolute | |
| Global-MMLU es (Uncertainty / Random / Ours) | Accuracy | 0.741 / 0.518 / 0.749 | Random baseline degrades severely, validating structural signal | |
| Multi-Annotator Capacity (#Annotations) | AG's News (#1 / #2 / #3) | Accuracy | 0.852 / 0.873 / 0.891 | Stepped gains demonstrate value of iterative dispute resolution |
| PubMed (#1 / #2 / #3) | Accuracy | 0.740 / 0.756 / 0.791 | High-variance biomedical terms benefit greatly from consensus (+5.1%) | |
| MMLU-Redux (#1 / #2 / #3) | Accuracy | 0.721 / 0.754 / 0.766 | Disagreement-driven arbitration filters out spurious model errors | |
| CNN/DailyMail (#1 / #2 / #3) | ROUGE-1 | 0.387 / 0.398 / 0.410 | Semantic centroid pooling produces sharper, consensus-aligned summaries | |
| Routing Overhead (AG's News, 2k samples) | CDI / CoAnnotating (Confidence CT) | Overhead | 0.503s + CT (~2 hours) | Mandates continuous LLM calls before any routing decision |
| SANT / ARAIDA (Model Training TT) | Overhead | 4.988s + TT (~20 mins) | Expensive offline training limits agility on dynamic data streams | |
| QUORUM (Ours) | Overhead | 0.494s + ET (~4 mins) | 30x faster preprocessing via pure embedding lookup and scalar stats |
Key Findings¶
- Budget state acts as the dominant routing governor: Bandit feature importance analysis reveals that remaining budget ratio \(b_t\) dominates routing weights (mean weight 0.5333), followed by historical query masks \(m_t\) and vocabulary richness. The policy smoothly transitions from exploratory probing to cost-effective automated annotation, strictly reserving expensive human interventions for instances exhibiting high posterior variance and severe disagreement.
- Static features overcome confidence miscalibration: Substituting verbalized uncertainty with deterministic linguistic metrics and embedding distances not only incurs zero inference latency, but also insulates the router from LLM self-reporting bias and overconfidence, outperforming confidence baselines across nearly all budget regimes.
- Robust multilingual and noisy human transfer: In Spanish and Japanese Global-MMLU benchmarks, QUORUM achieves top-tier accuracy, reaching 0.946 on Global-MMLU (ja) at 70% budget (+3.7% over the nearest competitor). Furthermore, on SentiMP-En featuring three distinct, noisy human annotators, QUORUM consistently maintains superior accuracy (0.7500 vs. 0.7271 for CDI), verifying resilience under realistic crowdsourced label noise.
Highlights & Insights¶
- Decoupled routing from candidate model inference: Conventional co-annotation frameworks require running the very models they seek to optimize to estimate confidence; QUORUM breaks this circular dependency, enabling lightning-fast, zero-inference routing decisions that reduce preprocessing time from 2 hours to 4 minutes.
- Unified discrete and continuous consensus paradigm: Seamlessly bridges discrete classification with open-ended generation by pairing reliability-weighted majority voting with semantic centroid extraction in embedding space, establishing a unified formulation for multimodal annotation.
- Reserved budget safeguarding against late-stage starvation: Partitioning available funds into active and reserved tranches prevents greedy early-stage allocation, guaranteeing that challenging edge cases appearing late in the stream still receive expert supervision.
Limitations & Future Work¶
- Author-admitted limitations: The primary benchmarks concentrate on English benchmarks, with multilingual validation confined to Spanish and Japanese. Further validation is required on highly specialized domains such as legal contracts and complex medical imaging workflows.
- Potential structural biases: Surface-level linguistic indicators (e.g., Flesch readability, word lengths, POS ratios) are calibrated for morphologically segmented Indo-European languages; their efficacy diminishes on logographic scripts (such as Japanese and Chinese) or multimodal visual tokens, placing heavier reliance on embedding neighborhood distances.
- Future directions: Integrating visual complexity descriptors (e.g., image entropy, spatial frequency, cross-modal grounding misalignment) would allow QUORUM to directly arbitrate dense vision-language annotation workflows for large vision-language models.
Related Work & Insights¶
- vs CoAnnotating (Li et al., 2023) / CDI (Gligorić et al., 2025): These systems compute entropy or verbalized probabilities via repeated LLM querying, incurring heavy computational overhead; QUORUM utilizes static linguistic and semantic features, achieving superior Pareto cost-efficiency without running model inference.
- vs HyPAC (Zeng et al., 2026): While HyPAC establishes PAC error bounds, it relies on partitioning input space via model uncertainty; QUORUM leverages Bayesian linear regression with online posterior concentration and asymptotic zero-regret guarantees while supporting iterative multi-annotator arbitration.
- vs ARAIDA / SANT (Huang et al., 2024a/b): Prior retrieval-augmented and active learning triage models require supervised offline training; QUORUM maintains an online, closed-form updating procedure that adapts on the fly without offline overhead.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Repositions collaborative data annotation from confidence-based routing to budget-aware contextual bandits driven by intrinsic data features.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across classification, QA, and summarization, supplemented by multilingual evaluations, noise audits, and theoretical proofs.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear narrative structure, and exhaustive empirical reporting.
- Value: ⭐⭐⭐⭐⭐ Highly actionable blueprint for reducing annotation expenditure while maximizing ground-truth quality in industrial AI data pipelines.