DocLayout-VL: A Foundational Model for Hierarchical, Open-set, and Promptable Document Layout Segmentation¶
Conference: ECCV 2026
Paper: ECCV 2026
Code: https://doclayoutvl.github.io/
Area: Segmentation
Keywords: Document Layout Analysis, Hierarchical Segmentation, Open-Vocabulary Segmentation, Promptable Grounding, Foundation Model
TL;DR¶
DocLayout-VL reframes document layout understanding as promptable open-set hierarchical segmentation, unifying serialized tree parsing in a VLM backbone, structural context modeling via a Hierarchical Grounding Contextualizer (HGC), and pixel-accurate SAM mask decoding to achieve new state-of-the-art results across 14 layout benchmarks.
Background & Motivation¶
Document Layout Analysis (DLA) establishes the structural organization of textual and graphical elements within a document. Because downstream workflows such as document question answering, information extraction, verification, and retrieval-augmented generation depend critically on this structural parsing, parsing errors propagate irreversibly through document intelligence pipelines. For years, document layout analysis has predominantly relied on closed-vocabulary object detectors that output flat, axis-aligned bounding boxes. However, real-world documents are inherently visually rich and hierarchically structured: layouts contain curved lines, irregular graphic regions, non-rectangular callouts, and multi-level nesting (such as articles containing titles, multi-column bodies, sub-sections, and captions). Flat bounding boxes inherently suffer from spatial ambiguity, overlap, and an inability to represent hierarchical reading flow.
The fundamental tension lies in the mismatch between modern document task demands and legacy DLA formulations: emerging applications require open-set concept recognition, flexible user-prompt conditioning, and pixel-accurate boundaries, whereas traditional models remain confined to static, closed-set bounding-box detection. Generic vision-language foundation models (VLMs) possess open-vocabulary knowledge and instruction-following abilities, but they localize regions primarily by generating discrete text coordinates, lacking dense pixel-level precision. Conversely, off-the-shelf segment-anything frameworks (like SAM) struggle under the significant domain shift and dense typography of documents, and they lack structural reasoning to model parent-child and sibling hierarchies.
To overcome these limitations, this paper reimagines document layout understanding as promptable, open-set, hierarchical segmentation. Rather than predicting flat boxes, the system simultaneously generates open-vocabulary layout trees, pixel-accurate segmentation masks, and textual reasoning. Core idea: decouple and synthesize structural symbolic tree parsing and dense spatial grounding by coupling a VLM backbone with a Hierarchical Grounding Contextualizer (HGC) that imposes tree-constrained local attention, backed by structure-aware containment and non-overlap losses.
Method¶
Overall Architecture¶
The DocLayout-VL architecture forms a unified vision-language-segmentation pipeline comprising three key stages: autoregressive serialized layout tree generation via a VLM backbone, structural context refinement via the Hierarchical Grounding Contextualizer (HGC), and dense mask decoding via a promptable segmentation module (SM).
An input document image \(d\) and user instruction prompt \(P\) are first fed into the VLM backbone (e.g., Qwen3-VL or InternVL3.5). The VLM autoregressively emits a structured layout string representing the document hierarchy, interleaving special [LAY] spatial anchor tokens after every semantic layout node (both leaf elements and structural container spans). Once generation finishes, the string is parsed into an explicit layout tree \(\hat{T} = (\hat{V}, \hat{E})\), and the hidden states \(\tilde{h}_i\) of all [LAY] tokens are extracted. These raw token states are passed to the HGC, which derives a dynamic sparse attention mask from \(\hat{T}\) to restrict attention strictly within each node's local structural neighborhood (parent, children, and siblings). The contextualized layout embeddings \(h_i\) are then mapped by a Prompt Encoder into spatial prompts \(p_i\), which are fused with multi-scale dense visual features \(z\) in the Mask Decoder to predict pixel-level binary masks \(\hat{M}_i \in \{0, 1\}^{H \times W}\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Input document image d + instruction prompt P"] --> VLM["VLM Backbone Autoregressive Generation<br/>Generates layout tree string and [LAY] tokens"]
VLM --> HGC["Hierarchical Grounding Contextualizer HGC<br/>Refines [LAY] hidden states with tree attention"]
VLM --> PARSE["Tree Parsing & Construction<br/>Builds explicit layout hierarchy T=(V, E)"]
PARSE -.->|Construct parent-child-sibling mask| HGC
HGC --> PE["Prompt Encoder<br/>Projects contextual vectors to spatial prompts pi"]
IN --> VE["SAM Vision Encoder<br/>Extracts high-resolution dense features z"]
PE --> DEC["Mask Decoder<br/>Fuses dense visual features and prompt tokens"]
VE --> DEC
DEC --> OUT["Final Output<br/>Explicit layout tree + pixel masks + explanations"]
Key Designs¶
1. Serialized Layout Tree Generation with Interleaved [LAY] Anchors: Bridging Symbols and Regions
Prior hierarchical methods rely on post-hoc heuristics or disjoint graph neural networks to reconstruct trees from detected boxes, risking severe error accumulation. DocLayout-VL casts hierarchical parsing into an autoregressive sequence generation task. Compound structural elements are enclosed in bracketed tags (e.g., <header> child_1[LAY] ... child_k[LAY] </header>[LAY]), while individual leaf nodes are generated as label[LAY]. Each emitted [LAY] token establishes an exact one-to-one correspondence with a semantic node \(v_i \in \hat{V}\). In this way, the autoregressive backbone seamlessly captures open-set category semantics, global reading order, and nesting grammar within a unified language generation process.
2. Hierarchical Grounding Contextualizer (HGC): Enforcing Structural Locality via Tree-Aware Attention
Directly using the raw VLM hidden states \(\tilde{h}_i\) as segmentation prompts causes layout predictions to ignore spatial dependencies across hierarchy levels. The HGC is a lightweight multi-layer transformer designed to inject structural constraints via Hierarchical Contextual Attention. Given the parsed tree \(\hat{T} = (\hat{V}, \hat{E})\), the HGC dynamically constructs a structural attention mask: for any node \(v_i\), attention is restricted strictly to its parent, its direct children, and its siblings sharing the same parent. This structural locality prevents distant, unrelated document blocks from diluting regional features while ensuring that parent containers and their nested components remain mutually coherent in feature space.
3. Structure-Aware Spatial Regularization Losses: Enforcing Containment and Non-Overlap
To ensure that predicted continuous masks respect the geometric realities of document hierarchies, the model augments the standard language cross-entropy loss \(\mathcal{L}_{lm}\) and segmentation loss \(\mathcal{L}_{seg}\) (Dice + BCE) with explicit geometric alignment penalties: - Background Exclusion Penalty (\(\mathcal{L}_{bg}\)): Penalizes mask activations that spill over into the document whitespace outside the global ground-truth foreground union \(M_{fg} = \bigvee_{v \in V} M_v\): $\(\mathcal{L}_{bg} = \frac{1}{|V|} \sum_{v \in V} \frac{\sum_{m \in \Omega} \hat{M}_v(m) [1 - M_{fg}(m)]}{\sum_{m \in \Omega} \hat{M}_v(m) + \epsilon}\)$ - Structural Loss (\(\mathcal{L}_{struct} = \lambda_{contain}\mathcal{L}_{contain} + \lambda_{overlap}\mathcal{L}_{overlap}\)): For every directed parent-child relation \((u, v) \in E\), the predicted child mask \(\hat{M}_v\) is penalized if it extends beyond its parent mask \(M_u\). Concurrently, for all sibling pairs \((v, w) \in S\) sharing a parent, mutual spatial overlap is penalized: $\(\mathcal{L}_{contain} = \frac{1}{|E|} \sum_{(u, v) \in E} \frac{\sum_{m \in \Omega} \hat{M}_v(m) [1 - M_u(m)]}{\sum_{m \in \Omega} \hat{M}_v(m) + \epsilon}\)$ $\(\mathcal{L}_{overlap} = \frac{1}{|S|} \sum_{(v, w) \in S} \frac{\sum_{m \in \Omega} \hat{M}_v(m) M_w(m)}{\sum_{m \in \Omega} \hat{M}_v(m) + \epsilon}\)$
Loss & Training¶
The model is trained end-to-end with the combined objective: $\(\mathcal{L}_{total} = \lambda_{lm}\mathcal{L}_{lm} + \lambda_{seg}\mathcal{L}_{seg} + \lambda_{bg}\mathcal{L}_{bg} + \lambda_{struct}\mathcal{L}_{struct}\)$ with weights set to \(\lambda_{lm}=1.0, \lambda_{seg}=2.0, \lambda_{bg}=0.5, \lambda_{struct}=0.5, \lambda_{contain}=1.0\), and \(\lambda_{overlap}=1.0\).
Training follows a three-stage curriculum on the HOPLay-Docs-80M corpus (2.5M pages, 80M masks): 1. Stage 1 (Flat Open-Set Grounding, FlatLay-Docs-25M): All layout elements are treated as siblings with dense HGC attention. The VLM visual encoder is frozen, while the LLM, HGC, and SAM components are optimized to ground open-vocabulary labels into spatial masks. 2. Stage 2 (Hierarchical Structure Learning, HierLay-Docs-40M): Trains on nested documents with HGC tree-constrained sparse attention and the structural loss \(\mathcal{L}_{struct}\) activated to master hierarchical nesting. 3. Stage 3 (Instruction Tuning, PromptLay-Docs-15M): Tunes on 5M prompt-grounded QA pairs. The SAM vision and prompt encoders are frozen, updating only the LLM and mask decoder to specialize in interactive, targeted region selection.
Key Experimental Results¶
Main Results¶
On the comprehensive HOPLay-Docs-Bench evaluation suite, DocLayout-VL-8B-it significantly outperforms specialized layout pipelines, open-source VLMs, and closed-source frontier models across all tracks:
| Type | Model | FB: Open mIoU (Om) | HB: Open mIoU (Om) | HB: mST | CB: Open mIoU (Om) | PB: cI | PB: g-eval (GE) | Overall Score (OS) |
|---|---|---|---|---|---|---|---|---|
| Pipeline Hierarchical | Dolphin [20] | 19.4 | 2.6 | 7.4 | 7.8 | 1.4 | 5.8 | 6.6 |
| Small Open-Source VLM | Qwen3-VL-8B-it [74] | 63.9 | 18.6 | 12.7 | 36.0 | 20.7 | 62.7 | 31.6 |
| Large Open-Source VLM | Qwen3-VL-235B-it [74] | 61.3 | 20.0 | 13.2 | 35.6 | 16.6 | 74.1 | 31.5 |
| Closed-Source VLM | Claude Sonnet 4.6 [3] | 68.4 | 12.1 | 11.2 | 38.7 | 11.7 | 68.7 | 30.5 |
| Closed-Source VLM | Gemini 3 Pro [22] | 72.2 | 6.9 | 11.1 | 40.3 | 28.9 | 77.6 | 37.2 |
| Fine-Tuned Baseline | Qwen3-VL-8B-ft | 72.1 | 32.5 | 40.3 | 39.7 | 42.4 | 74.3 | 45.8 |
| Ours | DocLayout-VL-4B-it | 79.6 | 69.5 | 71.7 | 83.6 | 57.5 | 73.1 | 71.8 |
| Ours | DocLayout-VL-8B-it | 86.7 | 77.5 | 79.2 | 87.9 | 70.2 | 87.2 | 79.4 |
On ten closed-set layout benchmarks evaluated in a fully zero-shot setting (mAP@50:95), DocLayout-VL demonstrates outstanding generalization across diverse domains, achieving 76.9 on M6Docs (compared to 70.0 for RoDLA) and 77.5 on the multilingual IndicDLP benchmark (where IndicDLP-YOLO achieves 55.0 and Gemini 3 Pro reaches only 50.6).
Ablation Study¶
The ablation experiments quantify the contributions of backbone choices, structural modules, and geometric loss constraints:
| Config | FB: Om | FB: OA | HB: Om | HB: mST | CB: Om | PB: cI | PB: GE | Closed-Set: IDLP |
|---|---|---|---|---|---|---|---|---|
| Full Model (DocLayout-VL-Q, 8B) | 86.7 | 61.0 | 77.5 | 79.2 | 87.9 | 70.2 | 87.2 | 77.5 |
| Backbone: DocLayout-VL-I (InternVL3.5) | 82.4 | 52.9 | 71.3 | 73.6 | 84.4 | 65.4 | 76.5 | 67.1 |
| w/o HGC Module (VLM + SAM) | 85.1 | 58.4 | 58.3 | 71.4 | 86.8 | 63.5 | 87.1 | 72.9 |
| w/o SAM & HGC (Base VLM alone) | 72.1 | 26.7 | 32.5 | 40.3 | 39.7 | 42.4 | 74.3 | 58.3 |
| w/o Structural Loss \(\mathcal{L}_{struct}\) | 87.2 | 61.1 | 72.3 | 74.2 | 86.5 | 65.9 | 86.7 | 74.5 |
| w/o Background & Struct Losses (\(\mathcal{L}_{bg} + \mathcal{L}_{struct}\)) | 84.8 | 57.8 | 69.2 | 71.5 | 84.1 | 63.1 | 86.6 | 72.8 |
Key Findings¶
- HGC is vital for hierarchical reasoning: Removing the HGC module causes HB Om to collapse from 77.5 down to 58.3 (a 19.2-point drop) and degrades mST from 79.2 to 71.4, proving that standard dense attention fails to capture parent-child boundaries.
- Mask-level decoding resolves the coordinate prediction bottleneck: Stripping both SAM and HGC (Base VLM predicting box text) degrades curved layout segmentation CB Om to 39.7 (vs. 87.9 for the full model) and drops open-set average precision OA from 61.0 to 26.7, confirming that autoregressive coordinate generation cannot match pixel-level mask grounding in text-dense documents.
- Architectural inductive biases surpass pure data scaling: Training with HGC on only 25% data yields higher hierarchical performance (HB Om 59.8) than scaling a VLM+SAM baseline to 100% data (HB Om 58.3). Furthermore, the three-stage curriculum outperforms joint single-stage training by 8.8 points on HB Om (77.5 vs. 68.7).
Highlights & Insights¶
- Graph-structured attention compiled directly from language tokens: HGC translates the generated language parse tree into a dynamic sparse attention mask, elegantly bridging symbolic grammatical structure with continuous neural representation learning.
- Self-consistent geometric containment without post-processing: By formulating parent-child containment and sibling mutual exclusion directly as differentiable loss penalties, the model inherently enforces topological validity without requiring complex geometric rule-based post-processing.
- Unified paradigm spanning closed-set, open-set, and interactive DLA: The framework seamlessly unifies classic closed-set parsing, domain-customized ontology extraction, and interactive promptable reasoning (e.g., PII redaction and spatial relational queries) within a single foundational checkpoint.
Limitations & Future Work¶
- Lack of cross-page multi-document context: The current architecture operates on single-page document images, limiting its capacity to resolve split tables, cross-page references, and extended reading flows.
- Inference latency on ultra-dense documents: Autoregressive tree generation becomes computationally demanding on documents with hundreds of micro-regions, where long token sequences increase latency beyond the single-page 602 ms baseline.
- Future Directions: Exploring parallelized non-autoregressive tree decoding and extending hierarchical promptable segmentation to multi-page document graphs.
Related Work & Insights¶
- vs Specialized Detectors (e.g., DocLayout-YOLO, RoDLA): Traditional detectors are constrained to rigid, closed-set bounding boxes and lack structural hierarchy awareness; DocLayout-VL supports arbitrary open-set taxonomies and pixel-level masks, achieving superior zero-shot transfer across complex benchmarks.
- vs General Multimodal Models (e.g., Gemini 3 Pro, Qwen3-VL): While general VLMs exhibit broad reasoning capabilities, their autoregressive coordinate generation struggles with fine-grained spatial boundaries and dense text layouts; DocLayout-VL bridges this gap via explicit mask decoding and hierarchical contextualization.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers a foundational model paradigm for promptable, open-set, hierarchical document layout segmentation with the novel HGC architecture]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Introduces the massive HOPLay-Docs-80M corpus and HOPLay-Docs-Bench, with extensive zero-shot and ablation evaluations across 14 benchmarks]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, mathematically rigorous loss formulations, well-organized architecture diagrams, and comprehensive empirical analysis]
- Value: ⭐⭐⭐⭐⭐ [Sets a new benchmark for document intelligence, bridging the gap between symbolic layout parsing and dense visual grounding for downstream RAG and VQA]