AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/KYLN24/Agora
Area: LLM Reasoning
Keywords: Agentic Reasoning, Archive-Grounded Benchmark, Cross-Document Reasoning, Workplace Document Intelligence, Deterministic Evaluation
TL;DR¶
AGORA establishes an archive-grounded document reasoning benchmark spanning 8 professional domains, 9,664 authentic workplace documents, and 372M tokens; its 362 deterministic numeric tasks require active file exploration, unit and terminology reconciliation, and multi-hop arithmetic, on which leading frontier models reach only 59.4% accuracy.
Background & Motivation¶
Large language models are rapidly shifting from passive chatbots that answer from internal parametric weights to autonomous agents embedded in complex enterprise workflows. In knowledge-intensive workplace scenarios such as investment due diligence, regulatory compliance audits, and policy research, the primary value of an agent lies in its ability to synthesize scattered evidence and perform rigorous calculations across massive proprietary document archives. However, real-world enterprise repositories are notoriously messy, inconsistently structured, and replete with divergent reporting conventions, conflicting metrics, irregular time intervals, and implicit institutional assumptions. Reasoning in this setting is fundamentally a search-and-verify process under evidence scarcity rather than reading comprehension over a pre-extracted context.
Existing evaluation benchmarks leave critical gaps when measured against these workplace requirements. Traditional multi-hop QA datasets such as HotpotQA and MuSiQue rely almost exclusively on homogeneous Wikipedia paragraphs, failing to mirror the irregular file structures and varied reporting styles of corporate archives. Table and document QA benchmarks such as TAT-QA and FinanceBench typically present pre-selected documents as static reading comprehension tasks without requiring active file system exploration or programmatic computation. Meanwhile, web and desktop agent benchmarks like GAIA, BrowseComp, and OSWorld operate in open-web environments or graphical interfaces, prioritizing interaction mechanics over rigorous, bounded evidence synthesis. The closest prior benchmark, OfficeQA Pro, is confined to a single source of U.S. Treasury bulletins, lacking the cross-domain diversity required to test whether agentic document reasoning capabilities truly generalize.
To bridge these gaps, creating an authentic, cross-domain, and leak-free workplace reasoning benchmark presents three formidable obstacles: synthesizing cross-document tasks that genuinely require non-parametric multi-hop deduction rather than single-file lookup, compiling authentic enterprise archives at scale without exhausting synthesis budgets, and formulating tasks that permit deterministic, judge-free verification. Core idea: Construct AGORA, an archive-grounded benchmark pairing 362 deterministic numeric queries with 9,664 authentic documents (372M tokens) across eight professional domains, built via an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and multi-stage difficulty filtering.
Method¶
Overall Architecture¶
The AGORA benchmark operationalizes archive-grounded reasoning by restricting agents to a bounded local directory of authentic workplace documents inside an isolated sandbox. The construction pipeline follows three sequential phases: heterogeneous document collection and density-based preprocessing, cross-document task synthesis with iterative anti-leakage obfuscation, and multi-stage difficulty filtering with human verification. During evaluation, an agent interacts within an offline Linux sandbox (equipped with 2 vCPUs, 4GB RAM, a 3,600-second timeout, and standard CLI data-science tools) via the mini-swe-agent framework. The agent is provided only a Bash tool to explore /workspace/documents/{{domain}}/, write Python scripts for parsing and reconciliation, and compute the answer, terminating within a 200-turn budget by outputting a single verifiable numeric answer wrapped in <answer>...</answer>.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Domain Document Collection<br/>PDF/MD/Excel/CSV Format Normalization"] --> B["Three-Level Information Density Filtering & Vector DB<br/>Top-100 Seed Chunk Selection"]
B --> C["Cross-Document Multi-Hop Task Synthesis<br/>Semantic Drafting + Bridging Fact Grounding"]
C --> D["Dual Leakage-Preventing Obfuscation<br/>Lexical/Structural Attack Tests & Role Rewriting"]
D --> E["Multi-Stage Difficulty Filtering & De-Parametrization<br/>Closed-Book Filter + 3-Model Consensus Drop + Trajectory Audit"]
E --> F["Isolated Deterministic Sandbox Evaluation<br/>Constrained Bash Tool + Exact Numeric Verification"]
Key Designs¶
1. Three-Level Information Density Filtering & Vector DB: Pinpointing high-density computational anchors across messy archives. Ingesting hundreds of thousands of workplace pages directly into agentic synthesis workflows is computationally prohibitive and yields trivial tasks. AGORA addresses this by converting multi-format files (PDFs converted via dots.ocr into 5-page windows, Markdown chunked into 8,000-token sliding windows with 800-token overlap, and Excel/CSV sheets profiled by column types and statistics) into unified plain text. To identify potent entry points, the pipeline applies a clamped additive heuristic that prioritizes numerical and temporal density: $\(S(c) = \min\left(5 \times 1000 \times \frac{|N(c)|}{|c|}, 30\right) + \min\left(\frac{\text{tokens}(c)}{50}, 20\right) + S_{\text{type}}(c)\)$ where \(|N(c)|\) is the count of numerical tokens, and \(S_{\text{type}}(c)\) awards bonuses for structured tables (+10 for Markdown, +12 for OCR tables) and longitudinal spans (+8 for time columns, bonuses for row/column counts and \(2 \times (\text{year}_{\max} - \text{year}_{\min})\)). Chunks with fewer than 20 tokens or matching table-of-contents patterns are strictly dropped. Chunks scoring \(\ge 10\) are ranked to retain the top-100 seed chunks per domain. Finally, GPT-5.5 generates summaries and keyword tags across chunk, document, and domain levels, embedding them into a dense vector database to empower multi-granularity semantic retrieval during task creation.
2. Cross-Document Multi-Hop Task Synthesis: Enforcing non-trivial multi-file bridging dependencies. Simple question generation frequently lapses into single-file keyword lookups. In AGORA, a Codex agent begins from a designated seed chunk, queries the multi-granularity vector database to retrieve complementary evidence from distinct files, and synthesizes candidate queries under strict constraints prohibiting single-step shortcuts and enforcing minimum hop counts. The synthesis agent produces the natural language query, an executable verification script, and a gold reasoning trajectory. Subsequently, a refinement subagent attempts the question blind to verify solvability, checks alignment across the query stem, evidence chain, and answer, and discards any candidate that fails to span multiple heterogeneous files.
3. Dual Leakage-Preventing Obfuscation: Stripping lexical shortcuts and structural tells. Even when tasks require multi-hop reasoning, models frequently exploit unintended leakage: lexical leakage, where rare query terms retrieve key evidence in 1–2 naive grep calls, and structural leakage, where intermediate entities that should be deduced are explicitly named in the prompt. AGORA introduces an automated leakage attack suite to stress-test candidates. When leakage is detected, entity names and target values are systematically rewritten into business-role descriptions or functional equivalents (e.g., rewriting a named holding company into "the upstream animal-protein group with the largest representation in the highest-ranked stock basket"). The rewrite preserves the underlying multi-file dependency graph, and candidates are re-tested to ensure all retrieval shortcuts are completely neutralized.
4. Multi-Stage Difficulty Filtering & De-Parametrization: Guaranteeing benchmark hardness beyond frontier capabilities. To prevent trivial tasks or answers memorized during pretraining from contaminating the benchmark, AGORA enforces a four-tiered gating protocol. First, each task is presented to DeepSeek-V4-Pro in a closed-book setting; any query solvable from internal parametric knowledge is immediately discarded. Second, surviving tasks are evaluated by a three-model consensus panel (GPT-5.5, DeepSeek-V4-Flash, and DeepSeek-V4-Pro); tasks solved correctly by all three are eliminated as insufficiently challenging. Third, Codex acts as an impartial reviewer launching two parallel solving subagents—one solving blind and one guided by the reference reasoning path—to verify logical validity, unambiguity, and lack of file redundancy. Finally, expert human annotators manually inspect and verify the surviving candidates, producing the curated final set of 362 tasks.
Benchmark & Evaluation Protocol¶
Evaluation runs inside an isolated E2B sandbox with 2 vCPUs, 4GB RAM, and no network access. Agents run under mini-swe-agent, restricted strictly to Bash commands (python3, ripgrep, fd, jq, sed, awk, pandas, openpyxl). Every query is guaranteed to possess a single, deterministic numeric answer (e.g., growth percentages, basis point spreads, dimensionless ratios). The evaluation harness parses the final <answer>...</answer> submission and verifies it against the ground truth with format-normalized tolerance, eliminating the subjectivity, inconsistency, and cost associated with LLM-as-a-judge frameworks.
Key Experimental Results¶
Main Results¶
Eight frontier proprietary and open-weight models were evaluated under their maximum reasoning effort and temperature 1.0. Accuracy (%) across the eight workplace domains and overall performance are summarized below:
| Model | Agri | Arch | Biz | Edu | Fin | Health | Law | Tech | Overall Accuracy |
|---|---|---|---|---|---|---|---|---|---|
| Gemini-3.1-Pro | 66.20% | 68.97% | 72.00% | 59.18% | 41.03% | 64.71% | 51.43% | 49.09% | 59.39% |
| GPT-5.5 | 60.56% | 58.62% | 38.00% | 48.98% | 53.85% | 52.94% | 62.86% | 61.82% | 54.70% |
| GLM-5.1 | 59.15% | 58.62% | 38.00% | 34.69% | 56.41% | 50.00% | 51.43% | 52.73% | 50.00% |
| DeepSeek-V4-Pro | 43.66% | 55.17% | 42.00% | 38.78% | 46.15% | 50.00% | 51.43% | 47.27% | 45.86% |
| DeepSeek-V4-Flash | 45.07% | 41.38% | 34.00% | 46.94% | 38.46% | 44.12% | 31.43% | 36.36% | 40.06% |
| Qwen3.5-35B-A3B | 7.04% | 6.90% | 12.00% | 12.24% | 2.56% | 17.65% | 17.14% | 16.36% | 11.33% |
| Gemini-3.1-Flash-Lite | 4.23% | 6.90% | 2.00% | 10.20% | 5.13% | 5.88% | 2.86% | 12.73% | 6.35% |
| Qwen3.5-9B | 2.82% | 0.00% | 8.00% | 2.04% | 0.00% | 0.00% | 5.71% | 3.64% | 3.04% |
Human Baseline & Retrieval Scope Ablation Study¶
The table below reports human expert performance (conducted on a 40-item subset using VS Code with ripgrep) alongside ablation studies on document retrieval scope and agent scaffolding harnesses across a representative 104-task subset:
| Evaluation Setting / Scope | GPT-5.5 | DeepSeek-V4-Pro | Qwen3.5-35B-A3B | Human Baseline | Key Takeaways |
|---|---|---|---|---|---|
| Human Audit (40 tasks) | 55.0% | 45.0% | 12.5% | 82.5% | Confirms tasks are logically verifiable and solvable by humans |
| Paired domain (Primary) | 54.81% | 46.15% | 11.54% | — | Bounded domain corpus (22M–117M tokens) |
| All domains, foldered | 56.73% | 40.38% | 6.73% | — | All 8 domains mounted; requires top-level routing |
| All documents, flat | 38.46% | 37.50% | 3.85% | — | All 9,664 files in a single flat directory; severe drop (-16.35 pts) |
| Harness: OpenHands | 68.27% | 51.92% | 2.88% | — | Rich scaffolding improves GPT-5.5 by +13.5 pts |
| Harness: OpenClaw | 60.58% | 41.35% | Timeout | — | Divergent harness sensitivity across model architectures |
Key Findings¶
- High benchmark headroom and tier bimodal split: Even the top-performing model, Gemini-3.1-Pro, achieves only 59.39% overall accuracy. The evaluated models divide sharply into a frontier tier (40%–60%) and an underperforming lower tier (3%–11%), separated by an enormous 28.7-point performance chasm. Smaller and lightweight models fail almost completely across domains.
- Cross-domain variance causes leaderboard inversions: Aggregate scores conceal severe domain weaknesses. Gemini-3.1-Pro leads overall and dominates five domains, yet stumbles on Finance (41.03%), trailing GLM-5.1 (56.41%) and GPT-5.5 (53.85%). Conversely, GPT-5.5 suffers in Business (38.00%) while dominating Law (62.86%) and Technology (61.82%). This demonstrates that reasoning capability does not transfer uniformly across domain corpora.
- Evidence grounding dominates failure modes: Error taxonomy across wrong trajectories reveals that grounding failures account for the overwhelming majority of errors: Incomplete Inspection (II), Evidence Misidentification (EM), and Instruction Non-Following (INF) account for >85% of frontier failures. Pure Hallucination remains under 12% for frontier models (though reaching ~40% in smaller models). Resource Exhaustion is heavily model-dependent (24.59% for GPT-5.5 vs \(\le 1.10\%\) for the DeepSeek-V4 family).
- Extended trajectories signal disorientation: Turn distribution analysis demonstrates that successful episodes resolve within low-to-moderate turn counts (10–30 turns). Trajectories extending past 60–100 turns are almost exclusively incorrect, indicating that lengthy exploration reflects unproductive thrashing rather than incremental progress toward the answer.
Highlights & Insights¶
- Proactive leakage obfuscation provides anti-shortcut defense: By implementing attack suites that rewrite specific entity names into contextual business roles, AGORA effectively blocks simple lexical matching and forces models to execute the full multi-hop evidence chain.
- Deterministic verification eliminates evaluation drift: Grounding all complex multi-document reasoning tasks in exact, verifiable numeric values provides a completely objective, automated, and low-cost evaluation without reliance on noisy LLM judges.
- Demonstration of model-domain interaction: The benchmark empirically disproves the assumption of uniform capability transfer, demonstrating that document reasoning difficulty is an emergent property of the model-domain pairing.
Limitations & Future Work¶
- Author-admitted limitations: The benchmark size (362 tasks) is kept relatively compact to ensure exhaustive multi-stage filtering and human verification. Additionally, as frontier pretraining corpora continue to expand, closed-book filtering guarantees may erode if public documents are absorbed into future training sets.
- Self-identified limitations: The archive is entirely normalized to plain-text Markdown, omitting visual and multimodal challenges such as complex multi-column layout artifacts, low-resolution scan degradation, and graphical signatures.
- Future directions: Expanding the AGORA pipeline into an active, continuously refreshed evaluation platform, while extending the document reasoning setting to multimodal PDF artifacts with native chart and figure interpretation.
Related Work & Insights¶
- vs HotpotQA / MuSiQue: Wikipedia-based multi-hop QA focuses on short passage chains without tabular arithmetic; AGORA grounds reasoning in heterogeneous enterprise archives with complex metric and unit reconciliation.
- vs TAT-QA / FinanceBench: Document and table QA typically provide pre-selected documents in a reading comprehension paradigm; AGORA requires active, multi-turn Bash exploration across massive multi-file collections.
- vs OfficeQA Pro: OfficeQA Pro evaluates retrieval and calculation over a single corporate corpus (Treasury bulletins); AGORA spans eight diverse economic and technical domains, uncovering critical domain-specific blind spots and rank inversions.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering archive-grounded agentic reasoning with strict anti-leakage obfuscation and deterministic verification.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 8 models, human baselines, scope ablations, multiple harnesses, and detailed failure taxonomy.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent, lucid methodology and formalization of enterprise reasoning challenges.
- Value: ⭐⭐⭐⭐⭐ A vital, unsaturated testbed for driving LLM agents toward reliable real-world workplace utility.