docfollow a hierarchical fine grained constraints following benchmark in long co
content_hash: fc7187a3c97323bf ---# DocFollow: A Hierarchical Fine-Grained Constraints Following Benchmark in Long Contexts for LLMs
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/leonbig/DocFollow
Area: LLM Evaluation
Keywords: Long Context Generation, Long-In-Long-Out, Fine-Grained Constraint Following, Entity-Level Constraints, Deterministic Evaluation Protocol
TL;DR¶
Addressing the critical gap in evaluating Long-In-Long-Out text generation and the unreliability of LLM-as-a-judge scoring, DocFollow introduces a benchmark of 2,436 expert-validated instances across 5 domains and 20 subdomains with hierarchical constraints and a deterministic rule-based evaluation protocol.
Background & Motivation¶
Large Language Models (LLMs) have achieved notable breakthroughs in processing extended contexts and generating long-form texts. However, existing benchmarks predominantly focus on two isolated and fragmented paradigms: Long-In-Short-Out tasks such as summarization and needle-in-a-haystack retrieval (e.g., L-Eval, LongBench), and Short-In-Long-Out tasks like open-ended creative writing (e.g., LongWriter). In practical professional writing scenarios—such as drafting academic research papers that synthesize multiple source materials, compiling investigative news reports from interview transcripts, or restructuring financial filings under strict corporate governance guidelines—models must simultaneously absorb thousands of tokens of contextual input and produce structured long-form outputs exceeding 1,000 words. This critical intersection of Long-In-Long-Out generation remains underexplored.
The fundamental challenge in developing benchmarks for this setting stems from two intertwined issues. First, there is an absence of authentic, controllable, fine-grained task design; existing benchmarks like LongGenBench restrict inputs to brief prompts with synthetic query types, while others like WritingBench examine only coarse stylistic or persona constraints. Second, automatic evaluation frameworks in long-form generation are fundamentally unreliable. Relying on LLMs as judges to directly grade adherence introduces systematic inconsistency, position bias, and halluncinated scoring, whereas comprehensive human manual evaluation is prohibitively expensive, and surface n-gram metrics (e.g., BLEU, ROUGE) fail to capture deep structural and semantic constraint satisfaction.
DocFollow bridges these gaps through task formalization and evaluation decoupling. It frames the challenge as a "Document-Following" task, where an original document and target topic pair undergo controlled topic modifications (R1–R3) and cumulative constraint escalation (L1–L4). On the evaluation side, overall uncertainty is decomposed into atomic-level classification tasks: LLMs are restricted strictly to information extraction (named entity recognition and relation extraction), while all constraint satisfaction scoring is computed through deterministic rule-based programs. Core idea: DocFollow establishes the first comprehensive Long-In-Long-Out benchmark featuring hierarchical fine-grained constraints (topic, length, sentence relations, entity types, and entity relations) evaluated via deterministic rule-based execution over atomic extractions, demonstrating that current LLMs suffer catastrophic degradation on micro-level entity constraints despite maintaining macro-level topic coherence.
Method¶
Overall Architecture¶
The Document-Following task is formally defined as follows: given an original document \(X\) centered on topic \(T\), a modified target topic \(T'\), and a set of multi-level constraints \(\{c_1, c_2, \dots, c_n\}\), the model must generate a new long-form document \(X'\) that maintains topical consistency with \(T'\) while strictly satisfying all constraints derived from \(X\) and \(T'\).
The benchmark establishes a two-dimensional difficulty space combining horizontal topic modification levels (R1–R3) with vertical cumulative constraint levels (L1–L4). In the evaluation stage, LLMs never serve as scoring arbiters; instead, they function exclusively as information extractors for mature atomic subtasks (NER, sentence alignment, and RE). The structured outputs are subsequently fed into deterministic rule-based scripts to compute exact mathematical satisfaction scores.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Long Input Document X + Target Topic T' + Constraints {ci}"] --> B["Orthogonal Difficulty Gradients<br/>R1-R3 Topic Shifts × L1-L4 Constraint Levels"]
B --> C["Expert-Driven Dual-Sided Validation<br/>GPT-3.5 Lower-Bound Filtering + PhD Feasibility Review"]
C --> D["LLM Long-Form Rewriting<br/>Generates Document X' (Avg. 2,610 tokens)"]
D --> E["Atomic Information Extraction<br/>LLM Extracts Named Entities & Relation Triples"]
E --> F["Deterministic Rule-Based Assessment<br/>Mathematical Scoring across SC1 to SC5"]
Key Designs¶
1. Orthogonal Two-Dimensional Task Design: Decoupling Topic Modification (R1–R3) and Constraint Complexity (L1–L4)
To systematically profile model capabilities without confounding topical shift and constraint density, DocFollow structures tasks along two orthogonal axes. Topic Modification spans three progressive levels: R1 (word substitution using antonyms or near-synonyms), R2 (component replacement swapping parallel concepts within the same subdomain), and R3 (cross-subdomain transfer retaining the high-level argumentative frame within the broader domain). Orthogonally, Constraint Levels stack cumulatively from macro to micro granularity: - L1 (Document-level coarse constraints): Combines C1 (Topic Coherence, ensuring all entities strictly serve target topic \(T'\)) and C2 (Length Constraint, matching both sentence count and word count per sentence with original text \(X\)); - L2 (Discourse-level logical constraints): Extends L1 with C3 (Sentence Relation Constraint, preserving the discourse relation type—such as causal, contrastive, or temporal—between adjacent sentences); - L3 (Entity-level category constraints): Extends L2 with C4 (Entity Type Constraint, enforcing that each entity preserves the ontological category of its corresponding entity in \(X\)); - L4 (Entity-level relation constraints): Extends L3 with C5 (Entity Relation Constraint, requiring semantic relation triples between entity pairs to mirror the structural relationship graph of \(X\)).
2. Expert-Driven Dual-Sided Filtering Pipeline: Lower-Bound Calibration and Human Feasibility Verification
The dataset encompasses 5 primary domains (Academic & Engineering, Education, Finance & Business, Literature & Art, Politics & Law) and 20 specialized subdomains curated from authoritative repositories including the UN Parallel Corpus, arXiv, Project Gutenberg, SEC EDGAR, PubMed Central, and WWC. A team of 15 domain-expert PhD candidates (arranged into 5 three-person teams) guided curation and review, achieving high inter-annotator agreement (Fleiss' Kappa \(\kappa = 0.82\text{--}0.88\)). To balance difficulty, the authors established a two-sided filtering pipeline: an automated pre-test using ChatGPT-3.5-Turbo filters out trivially easy instances where a weaker model readily satisfies all constraints, preventing ceiling effects; subsequently, domain experts review the remaining instances to eliminate prohibitively difficult cases where topic transformation renders discourse or entity preservation unnatural or infeasible for human writers. The finalized dataset comprises 2,436 balanced instances (~120 per subdomain) with input lengths ranging from 750 to 8,323 tokens (averaging 2,610 tokens).
3. Deterministic Rule-Based Assessment Protocol: Reducing Subjectivity to Verifiable Atomic Extraction
DocFollow replaces opaque LLM scoring with an objective, modular evaluation protocol. For length adherence (C2), scores are calculated directly through absolute normalized errors in sentence count and word count: $\(S_{C2} = \left[1 - \left(a \frac{|N'_S - N_S|}{N_S} + (1 - a) \frac{|N'_W - N_W|}{N_W}\right)\right] \times 100\%\)$ where \(N_S, N'_S\) denote sentence counts and \(N_W, N'_W\) denote word counts in \(X\) and \(X'\), with \(a = 0.5\). For Topic Coherence (C1), an LLM inspects each sentence to detect topically incoherent entities \(N_{inco}^i\), normalized by sentence length \(N'^i_W\): $\(S_{C1} = \left[1 - \left(\frac{1}{N'_S} \sum_{i=1}^{N'_S} \frac{N_{inco}^i}{N'^i_W}\right)\right] \times 100\%\)$ For fine-grained structural constraints C3 (sentence relations), C4 (entity types), and C5 (entity relations), evaluation follows a three-step protocol: (1) map corresponding pairs between \(X\) and \(X'\); (2) extract types and relation triples using standardized taxonomies via LLM; and (3) verify exact type equality through deterministic comparison scripts: $\(S_{C3/C4/C5} = \frac{N'_{g3/g4/g5}}{N'_{3/4/5}} \times 100\%\)$ where \(N'_{g}\) represents the count of matched pairs and \(N'\) is the total count in \(X'\). Empirical validation confirms strong consistency between LLM extractors and human experts (intraclass correlation coefficient ICC \(\approx 0.86\)), and the resulting scores achieve 85% prediction accuracy against pairwise human expert preferences.
Key Experimental Results¶
Main Results¶
Thirteen state-of-the-art LLMs were evaluated with generation temperature set to 0.9 (averaged over 3 runs) and deterministic information extraction (\(t = 0.0\)). Table 1 summarizes model performance across Topic Modification levels (R1–R3) for all five constraint scores (SC1–SC5) and overall averages (directly quoting numbers from Table 3 in the paper):
| Model | R1: SC1/SC2/SC3/SC4/SC5 | R2: SC1/SC2/SC3/SC4/SC5 | R3: SC1/SC2/SC3/SC4/SC5 | Overall Avg. |
|---|---|---|---|---|
| Claude-4.5-Sonnet | 95.2 / 79.9 / 63.8 / 66.8 / 58.6 | 97.1 / 82.6 / 65.6 / 66.6 / 59.6 | 91.7 / 83.2 / 65.4 / 67.6 / 62.5 | 73.7% |
| DeepSeek-V3.2 | 96.4 / 79.8 / 62.2 / 65.7 / 57.2 | 95.3 / 82.3 / 66.3 / 66.8 / 59.2 | 91.3 / 80.3 / 65.1 / 65.4 / 56.6 | 72.4% |
| DeepSeek-R1 * | 90.9 / 79.3 / 61.6 / 68.2 / 58.2 | 92.6 / 82.7 / 54.4 / 65.7 / 60.3 | 82.7 / 82.6 / 61.2 / 62.8 / 51.7 | 70.4% |
| Qwen3-235B (thinking) * | 91.1 / 80.4 / 51.6 / 62.7 / 58.2 | 91.7 / 82.6 / 55.2 / 63.5 / 66.3 | 82.3 / 84.1 / 63.4 / 63.2 / 54.7 | 69.9% |
| Qwen3-235B (no thinking) | 96.3 / 79.7 / 67.3 / 61.8 / 61.1 | 93.8 / 85.4 / 63.6 / 67.8 / 60.3 | 91.7 / 80.6 / 52.1 / 39.9 / 17.9 | 67.9% |
| ChatGPT-5.2 | 93.3 / 69.1 / 58.1 / 52.8 / 48.7 | 93.5 / 74.1 / 66.5 / 60.7 / 50.7 | 92.7 / 72.2 / 65.6 / 54.7 / 51.7 | 66.9% |
| ChatGPT-5.1 | 89.6 / 60.4 / 56.3 / 51.8 / 49.6 | 95.2 / 61.8 / 63.3 / 56.5 / 52.3 | 92.1 / 63.6 / 64.1 / 55.2 / 54.8 | 64.6% |
| LLaMA 4 Scout | 92.6 / 72.4 / 52.1 / 52.6 / 45.6 | 91.9 / 75.7 / 49.9 / 52.3 / 50.6 | 84.1 / 76.4 / 52.8 / 59.6 / 49.3 | 63.9% |
| Qwen3-8B (thinking) * | 95.2 / 72.4 / 43.6 / 40.7 / 27.4 | 98.1 / 79.2 / 47.7 / 47.6 / 41.8 | 92.4 / 84.6 / 62.2 / 59.3 / 53.7 | 63.1% |
| Claude-4.5-Haiku | 90.1 / 65.6 / 44.3 / 34.2 / 26.2 | 92.2 / 66.4 / 43.3 / 29.6 / 22.2 | 90.7 / 69.5 / 45.4 / 31.3 / 25.7 | 51.8% |
(Note: Models marked with * are reasoning-enabled. SC1: Topic Coherence; SC2: Length; SC3: Sentence Relation; SC4: Entity Type; SC5: Entity Relation.)
Ablation & Generation Strategies Study¶
The paper evaluates various prompting paradigms alongside the proposed graph-guided DocFollower generation framework across representative LLMs (directly quoting numbers from Table 6):
| Model Family | Generation Strategy | R1: SC1 / SC5 | R2: SC1 / SC5 | R3: SC1 / SC5 | Overall Avg. |
|---|---|---|---|---|---|
| DeepSeek-V3 | DocFollower (Ours) | 94.7 / 67.8 | 94.0 / 64.6 | 96.5 / 68.7 | 77.5% |
| Direct Prompting | 91.0 / 44.4 | 91.5 / 49.7 | 82.4 / 49.5 | 63.0% | |
| Chain-of-Thought (CoT) | 89.9 / 37.8 | 90.8 / 42.1 | 82.3 / 48.3 | 58.8% | |
| Self-Critic | 92.1 / 52.5 | 92.3 / 55.5 | 87.2 / 55.3 | 66.0% | |
| Few-shot | 97.4 / 54.0 | 99.3 / 49.5 | 83.1 / 59.4 | 66.5% | |
| Claude-3.5-Sonnet | DocFollower (Ours) | 93.5 / 58.7 | 94.6 / 67.3 | 95.4 / 64.6 | 74.4% |
| Direct Prompting | 92.1 / 21.2 | 91.3 / 25.1 | 91.5 / 34.6 | 53.5% | |
| CoT | 89.1 / 17.2 | 91.3 / 16.9 | 91.7 / 32.1 | 51.6% | |
| Few-shot | 97.3 / 25.1 | 92.2 / 28.6 | 95.4 / 42.4 | 58.0% |
Key Findings¶
- Granularity-Dependent Degradation: A consistent hierarchical drop occurs across all models as constraints shift from coarse to fine granularity. While leading models (Claude-4.5-Sonnet, DeepSeek-V3.2) achieve 86%–97% on document-level topic coherence (SC1), performance plummets to 50%–62% on entity relations (SC5). Lightweight or smaller models drop even more drastically under complex settings (e.g., Claude-4.5-Haiku records 16.2%–25.7% on SC5).
- Reasoning Model Trade-offs: Enabling reasoning mechanisms offers substantial advantages in complex constraint adherence at the expense of simpler formatting. Qwen3-235B without thinking excels at coarse constraints in simple settings (SC3 = 67.3% at R1) but suffers catastrophic failure in R3 cross-domain transfer (SC4 drops to 39.9%, SC5 to 17.9%). In contrast, Qwen3-235B (thinking) maintains robust adherence (SC4 = 63.2%, SC5 = 54.7% at R3).
- Counter-Intuitive Prompting Effects: Chain-of-Thought (CoT) prompting degrades long-form constraint following across all models (dropping 4%–12% on average), showing that explicit reasoning tokens dilute context attention needed for simultaneous multi-objective control. Conversely, few-shot prompting provides structural exemplars that bolster constraint compliance, and the graph-guided DocFollower framework improves performance by up to 10.9%.
- Front-Loading Positional Bias: Positional analysis across generated texts reveals that constraint adherence is strongest in the initial 10% (topic coherence ~0.9) but steadily decays toward 0.65–0.7 by the final segments, reflecting progressive attention dispersion and working memory degradation in autoregressive models.
Highlights & Insights¶
- Paradigm Shift to Long-In-Long-Out: Expands long-context evaluation beyond unidirectional compression (summarization) or expansion (creative writing) to realistic, constraint-heavy professional document rewriting tasks.
- Decomposed Deterministic Evaluation: Solves the long-standing reproducibility and bias bottleneck of LLM-as-a-judge by delegating only atomic classification (NER/RE) to models while computing final metrics via verifiable mathematical rules (ICC ~0.86, human agreement 85%).
- Graph-Guided Controllable Generation: Introduces DocFollower, demonstrating that explicit entity-relation structural skeleton extraction and coarse-to-fine graph mapping significantly enhance fine-grained fidelity in long sequences.
Limitations & Future Work¶
- Restricted to Document Rewriting: The benchmark centers on rewriting reference texts under transferred topics, omitting ab initio creative generation or unstructured multi-document synthesis.
- Monolingual English Scope: Instances are restricted to English texts across 20 curated subdomains; extending to multilingual corpora and colloquial registers remains open.
- Inference Compute Overhead: The multi-pass information extraction evaluation across multiple seeds incurs substantial computational cost; future efforts could fine-tune specialized lightweight extractors to streamline execution.
Related Work & Insights¶
- vs. LongGenBench & WritingBench: Prior long-form generation benchmarks either use short query inputs or rely heavily on subjective styling constraints scored by LLM judges; DocFollow features authentic long inputs, fine-grained entity constraints, and deterministic evaluation.
- vs. FollowBench: FollowBench established fine-grained instruction following in short-form contexts; DocFollow scales this challenge to thousands of tokens where discourse structure and entity relations interact and decay over extended sequences.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first rigorous Long-In-Long-Out fine-grained constraint following benchmark with an elegant evaluation protocol.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 13 modern models with thorough granularity, positional decay, prompting strategy, and human correlation analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Formal problem setup, clean hierarchical definitions, and extensive documentation across tasks and guidelines.
- Value: ⭐⭐⭐⭐⭐ Sets an essential benchmark for evaluating structural and semantic controllability in long-context generative AI.