Benchmarking Aggregation Queries over Unstructured Text¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/luck-seu/AGGBench
Area: Information Retrieval & RAG
Keywords: Aggregation Queries, Unstructured Text, AGGBench, Retrieval-Augmented Generation, Completeness Evaluation
TL;DR¶
Addressing the severe undercounting flaw of existing RAG and agent systems that prioritize partial evidence rather than exhaustive entity coverage on unstructured corpora, this paper formalizes corpus-bounded entity aggregation querying, constructs AGGBench spanning 498 single- and multi-constraint queries across finance and research domains with verifiable evidence, and evaluates 15 representative systems to uncover fundamental bottlenecks in evidence sparsity and completeness.
Background & Motivation¶
Analytical aggregation queries over large unstructured text collectionsโsuch as counting how many entities across a vast document repository satisfy natural-language constraintsโare vital for financial auditing, legal discovery, biomedical literature synthesis, and investigative reporting. Unlike conventional question answering where locating a single plausible supporting passage suffices to generate an accurate answer ("find one"), aggregation over free text demands exhaustive discovery, deduplication, and counting of every qualifying entity across the entire corpus ("find all"). While structured database methods transform text into relational tables or knowledge graphs prior to executing Text-to-SQL, this requires prohibitively costly schema engineering and inevitably suffers from extraction noise and schema brittleness; semantic-operator pipelines likewise remain confined to predefined execution plans.
In parallel, retrieval-augmented generation (RAG) and autonomous agent frameworks operate directly over raw text but inherit a fundamental relevance-oriented design. Dense retrievers optimize for top-ranked semantic similarity within a fixed top-\(k\) window; when satisfying entity mentions are sparsely scattered or exceed \(k\), crucial evidence chunks are permanently omitted, causing systematic undercounting. Compounding this challenge is the complete lack of completeness-oriented evaluation benchmarks: multi-hop benchmarks such as HotpotQA and 2WikiMultiHopQA contain fewer than 5% aggregation questions and focus on small supporting fact sets, numerical reasoning benchmarks like DROP target local reading arithmetic over individual paragraphs, and broad search datasets such as WideSearch focus on unbounded open-web exploration rather than verifiable, corpus-bounded entity evaluation.
Constructing a rigorous aggregation benchmark over real-world corpora is severely impeded by annotation bottlenecks: reading hundreds of thousands of characters to discover every qualifying entity and grounding each in explicit textual evidence is prohibitively labor-intensive and error-prone for manual annotators. To bridge this divide, this paper establishes a corpus-grounded benchmark construction methodology that combines automated LLM candidate extraction with rigorous human double-blind verification, followed by boolean set algebra to derive multi-constraint queries with exact gold answers. Core idea: formalize unstructured text aggregation as a corpus-bounded entity counting problem, build AGGBench with 498 evidence-grounded queries across finance and research corpora, and systematically benchmark 15 representative RAG and agent systems to expose their severe evidence coverage and undercounting bottlenecks.
Method¶
Overall Architecture¶
The paper formalizes the task of corpus-bounded entity aggregation querying over text: given a text corpus \(\mathcal{C}\) and an aggregation query \(q = (T, \Phi, \Gamma)\), where \(T\) denotes the target entity type, \(\Phi = \{\phi_1, \dots, \phi_m\}\) denotes a set of natural-language boolean constraints, and \(\Gamma\) is an aggregation operator (focused on COUNT, \(\Gamma(\cdot) = |\cdot|\)). A system must identify all entities \(e \in \mathcal{E}(\mathcal{C})\) of type \(T\) for which explicit textual evidence in \(\mathcal{C}\) verifies that \(\Phi(e) = \text{true}\), deduplicate them, and return the total count. The AGGBench construction and evaluation pipeline comprises four tightly coordinated phases: domain-focused corpus curation and chunking, single-constraint query extraction and verbalization, LLM-assisted human evidence grounding with disagreement adjudication, and boolean query composition with multi-dimensional diagnostics.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Domain Corpus Curation<br/>15 SEC 10-K Filings + 44 NLP/AI Papers"] --> B["Single-Constraint Query Generation<br/>Entity type & attribute extraction + verbalization"]
B --> C["Human-in-the-Loop Evidence Grounding<br/>LLM pre-annotation + double-blind validation & adjudication"]
C --> D["Boolean Composition Engine<br/>AND intersection / OR union query derivation"]
D --> E["Multi-Dimensional Diagnostic Evaluation<br/>ACE absolute error + NACE normalized error + recall metrics"]
Key Designs¶
1. Single-Constraint Query Generation: Corpus-Grounded Type and Discriminative Constraint Mining To ensure that queries present realistic aggregation challenges rather than degenerating into isolated "find-one" instances scattered across heterogeneous topics, the pipeline selects domain-focused corpora featuring recurring entity types: 15 SEC Form 10-K annual filings (averaging 393k characters each) for finance and 44 top-tier conference papers from ACL/NeurIPS (averaging 101k characters each) for research. Documents are split into uniform chunks of 1000 characters with 200-character overlaps. The pipeline prompts an LLM to extract entity mentions and candidate types \(m = (n, T, d_i, c_j)\), applies deterministic normalization (lowercasing, whitespace trimming) to merge duplicates, and discards entity types with fewer than 5 canonical entities or appearing in fewer than 2 documents/chunks. Human annotators eliminate vague or overly broad types (e.g., "thing" or "information"). For retained types \(T \in \mathcal{T}\), the pipeline prompts the LLM to extract attribute-value phrases linked to entities as candidate constraints \(\phi\). It filters out constraints occurring fewer than twice or applying to nearly all entities of that type (such as "important" or "large"). Finally, an LLM verbalizes validated pairs into natural-language aggregation questions \(q\) (e.g., pairing type "dataset" with constraint "used for multi-hop question answering" into "How many datasets are used for multi-hop question answering?"), reviewed by humans to discard ambiguous or unnatural phrasings.
2. Human-in-the-Loop Evidence Grounding: Two-Stage Double-Blind Annotation and Adjudication Because discovering every qualifying entity across thousands of chunks poses immense miss rates for human readers alone, the benchmark adopts a high-throughput, high-rigor two-stage annotation protocol. In the first stage, an LLM evaluates all queryโchunk pairs to propose candidate positive entities with document and chunk identifiers, serving strictly as pre-annotations without defining truth labels. In the second stage, 5 trained annotators exhaustively read assigned documents to simultaneously verify whether LLM-proposed candidates are explicitly supported by the text and discover any additional qualifying entities overlooked by the LLM. Quality control utilizes independent dual-annotation across document subsets: whenever annotators disagree, the positive-label annotator presents the exact textual evidence span for cross-examination; if consensus cannot be reached, the sample is flagged as unresolved. Crucially, any query independently flagged as unresolved by two or more annotator pairs is permanently eliminated from the benchmark (15 queries were removed under this rule). On the retained benchmark, positive-label agreement reached 79% after adjudication, with initial raw agreement of 90.23% and Cohen's \(\kappa = 0.496\) across approximately 300 annotation hours, yielding 5,110 validated, evidence-anchored entity annotations over 117 single-constraint queries.
3. Boolean Composition Engine: Algebraic Set Operators for Multi-Constraint Queries Real-world analytical workflows frequently require compound criteria across attributes, yet re-annotating multi-constraint corpora manually would be economically infeasible and introduce labeling drift. The framework introduces a set-algebraic derivation mechanism: for any pair of validated single-constraint queries sharing the same entity type \(q_i = (T, \{\phi_i\})\) and \(q_j = (T, \{\phi_j\})\), compound queries are generated via logical conjunction and disjunction: $\(q_{ij}^{\land} = (T, \{\phi_i \land \phi_j\}), \quad q_{ij}^{\lor} = (T, \{\phi_i \lor \phi_j\})\)$ Their gold answer sets are derived directly through set algebra: \(\mathcal{A}_{q_i} \cap \mathcal{A}_{q_j}\) for conjunction and \(\mathcal{A}_{q_i} \cup \mathcal{A}_{q_j}\) for disjunction, with each entity inheriting verified evidence chunk pointers from the single-constraint stage. Composed queries are verbalized via LLM prompting, and human annotators filter out unnatural phrasings, illogical combinations, or trivial/empty answer sets. This produces 381 multi-constraint queries (51 AND, 330 OR), scaling AGGBench to 498 total queries covering varying degrees of compositional difficulty.
4. Multi-Dimensional Diagnostic Evaluation: Decoupled Counting Error and Dual-Level Recall To diagnose whether a system fails because it missed relevant evidence or because it mishandled entity deduplication and arithmetic aggregation, the evaluation framework establishes decoupled metrics. For predicted count \(\hat{y}\) and gold count \(y\), counting accuracy is measured by Absolute Count Error \(\text{ACE} = |\hat{y} - y|\) and Normalized Absolute Count Error \(\text{NACE} = |\hat{y} - y| / (y + \epsilon)\) (using median values across queries to guard against extreme outliers). To expose the underlying information bottleneck, the framework pairs count error with dual-level recall diagnostics: entity-level recall \(\text{Recall}_{\text{ent}} = |\hat{\mathcal{E}} \cap \mathcal{E}| / |\mathcal{E}|\) measures how completely the system extracted all true satisfying entities, while chunk-level evidence recall \(\text{Recall}_{\text{chunk}}\) quantifies the fraction of annotated gold supporting passages captured by the system's retrieval or reading stages. This decoupling enables researchers to pinpoint whether failures arise from retrieval omission or context-level reasoning breakdowns.
Key Experimental Results¶
Main Results¶
On the standardized chunked corpora (7,611 chunks in Finance, 4,755 chunks in Research) and 498 queries, the paper evaluated 15 representative systems spanning Basic RAG, Structured/Schema-first, and Agentic frameworks, with GPT-4o-mini as the standard backbone.
| Category | Method | Finance NACE(med) โ | Finance ACE(med) โ | Finance Recall(ent) โ | Finance Recall(chk) โ | Research NACE(med) โ | Research ACE(med) โ | Research Recall(ent) โ | Research Recall(chk) โ |
|---|---|---|---|---|---|---|---|---|---|
| Basic RAG | Naive RAG | 0.737 | 9.500 | 0.138 | 0.054 | 0.429 | 3.000 | 0.194 | 0.033 |
| Basic RAG | FLARE | 0.857 | 10.000 | 0.311 | 0.114 | 0.769 | 5.000 | 0.351 | 0.065 |
| Basic RAG | RAT | 0.606 | 7.000 | 0.254 | 0.100 | 0.429 | 3.000 | 0.389 | 0.036 |
| Basic RAG | DeepNote | 0.823 | 10.000 | 0.203 | 0.045 | 0.750 | 5.000 | 0.227 | 0.012 |
| Structured/Schema-first | HippoRAG | 0.750 | 9.500 | 0.144 | 0.057 | 0.500 | 3.000 | 0.220 | 0.042 |
| Structured/Schema-first | LightRAG | 0.667 | 7.000 | 0.170 | 0.058 | 0.364 | 2.000 | 0.223 | 0.029 |
| Structured/Schema-first | PageIndex | 0.800 | 9.500 | 0.143 | โ | 0.750 | 5.000 | 0.080 | โ |
| Structured/Schema-first | Palimpzest | 0.833 | 5.000 | 0.178 | โ | 0.556 | 5.000 | 0.475 | โ |
| Agentic Frameworks | Naive Agent | 0.586 | 6.000 | 0.214 | 0.074 | 0.500 | 3.000 | 0.306 | 0.032 |
| Agentic Frameworks | ReAct | 0.616 | 7.000 | 0.210 | 0.063 | 0.429 | 3.000 | 0.238 | 0.015 |
| Agentic Frameworks | ReadAgent | 0.500 | 7.000 | 0.398 | โ | 0.667 | 4.000 | 0.273 | โ |
| Agentic Frameworks | Multi Agent | 0.594 | 6.000 | 0.220 | 0.076 | 0.500 | 3.000 | 0.290 | 0.026 |
| Agentic Frameworks | Smolagents | 0.768 | 8.000 | 0.186 | 0.069 | 0.538 | 3.000 | 0.260 | 0.049 |
| Agentic Frameworks | Search-o1 | 0.667 | 8.000 | 0.162 | 0.054 | 0.875 | 7.000 | 0.317 | 0.044 |
| Agentic Frameworks | Claude Code | 0.579 | 7.000 | 0.180 | โ | 0.444 | 3.000 | 0.237 | โ |
Ablation Study: Impact of Expanding Retrieval Top-\(k\)¶
Testing whether enlarging retrieval top-\(k\) could trivially alleviate undercounting in RAG systems, the authors varied \(k\) from 5 to 50 on the AGGBench Research domain:
| Method | Retrieval Top-\(k\) | NACE (Mean) โ | NACE (Med) โ | Chunk Recall โ |
|---|---|---|---|---|
| Naive RAG | \(k = 5\) | 0.446 | 0.429 | 0.033 |
| Naive RAG | \(k = 10\) | 0.419 | 0.333 | 0.050 |
| Naive RAG | \(k = 20\) | 0.711 | 0.333 | 0.077 |
| Naive RAG | \(k = 50\) | 1.406 | 0.417 | 0.121 |
| LightRAG | \(k = 5\) | 0.450 | 0.364 | 0.029 |
| LightRAG | \(k = 10\) | 0.418 | 0.375 | 0.044 |
| LightRAG | \(k = 20\) | 0.434 | 0.333 | 0.076 |
| LightRAG | \(k = 50\) | 0.463 | 0.375 | 0.123 |
| DeepNote | \(k = 5\) | 0.621 | 0.667 | 0.015 |
| DeepNote | \(k = 10\) | 0.785 | 0.667 | 0.037 |
| DeepNote | \(k = 20\) | 0.963 | 0.615 | 0.049 |
| DeepNote | \(k = 50\) | 0.920 | 0.556 | 0.094 |
Key Findings¶
- Enlarging retrieval depth hurts count precision: While increasing top-\(k\) from 5 to 50 boosts chunk recall by roughly 0.09 across baselines, Naive RAG's mean NACE degrades sharply from 0.446 to 1.406. Inflating context with unverified distractors overloads LLM deduplication and reasoning, proving that text aggregation cannot be solved by simply scaling retrieval depth.
- Extreme evidence sparsity dominates failure modes: Median query evidence density is only 0.210% overall (0.145% in Finance, 0.294% in Research). Across all evaluated paradigms, chunk recall remains trapped between 0.01 and 0.11, while entity recall rarely exceeds 0.40, revealing persistent and pervasive undercounting.
- Model scale is not an economical remedy: Benchmarking across 5 backbone LLMs (Claude-Haiku-4.5, Gemini-3.5-Flash, DeepSeek-V3.2, Qwen3-32B, and GPT-4o-mini) demonstrated that while stronger models yield marginal recall gains under agentic loops, inference costs skyrocket (e.g., ReadAgent exceeds \(6โ\)7 per domain, and Palimpzest surpasses \(32โ\)38), without resolving evidence omission.
- Disjunctive queries compound omission errors: OR queries exhibit much higher absolute count errors (median ACE of 9โ16 in Finance) compared to AND queries (median ACE of 2โ4). Because disjunction queries have larger true answer sets (average 14.7 entities in Finance), missed candidate passages accumulate linearly into severe undercounts.
Highlights & Insights¶
- First formalization of corpus-bounded text aggregation: Bridges a longstanding gap between structured OLAP querying and free-text QA, defining a clear task setting where completeness, evidence grounding, and deduplication are evaluated concurrently.
- Mathematically grounded benchmark construction: By pairing human-validated single-constraint query primitives with boolean set algebra (intersections for AND, unions for OR), the methodology synthesizes hundreds of rigorous multi-constraint queries without exponential human annotation expense.
- Definitive debunking of naive RAG scaling: Empirically refutes the common assumption that larger context windows and higher top-\(k\) values suffice for exhaustive synthesis, establishing the need for stateful, completeness-oriented exploration algorithms.
Limitations & Future Work¶
- Restricted to COUNT aggregation: The benchmark currently formalizes and evaluates only entity counting (\(\Gamma = \text{COUNT}\)), leaving numerical aggregation operators such as SUM, AVERAGE, MIN/MAX, or GROUP-BY over textual attributes for future work.
- Domain coverage: AGGBench focuses on financial disclosures and AI conference papers. Expanding to legal discovery records, clinical patient histories, and government archives would further test cross-domain robustness.
- Evaluation without dedicated architectures: The study focuses on benchmark construction and diagnosing existing paradigms; designing a dedicated completeness-aware agent architecture remains an open challenge.
Related Work & Insights¶
- vs HotpotQA / 2WikiMultiHopQA / MuSiQue: Standard multi-hop benchmarks focus on constructing reasoning paths across 2โ4 supporting facts to identify a single entity; AGGBench requires exhaustive corpus-wide entity discovery where evidence is scattered across dozens of documents.
- vs DROP: DROP assesses discrete arithmetic reasoning (addition, subtraction, sorting) within a single paragraph; AGGBench evaluates corpus-scale retrieval completeness and cross-document entity deduplication.
- vs WideSearch / DeepWideSearch: Open-web broad information-seeking benchmarks evaluate autonomous browsing over unconstrained, dynamic search engine results without bounded ground truth; AGGBench enforces a closed corpus where every gold entity has verifiable, annotated text spans.
Rating¶
- Novelty: โญโญโญโญโญ Formalizes the neglected problem of corpus-bounded text aggregation and introduces the first dedicated benchmark with evidence grounding.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 15 diverse systems spanning three distinct architectural paradigms across 5 foundation LLMs, supported by detailed diagnostic ablations.
- Writing Quality: โญโญโญโญโญ Clear mathematical formulations, clean benchmark construction flowcharts, and rigorous diagnostic reporting.
- Value: โญโญโญโญโญ Serves as a vital benchmark for advancing enterprise-grade analytical RAG, document intelligence, and completeness-aware agent design.