HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding¶

Conference: ECCV 2026
arXiv: 2602.12957
Code: https://github.com/whlscut/HSD
Area: Multimodal VLM / LLM Efficiency
Keywords: Hierarchical Speculative Decoding, Document Parsing, Vision-Language Models, Inference Acceleration, Training-Free

TL;DR¶

This paper proposes Hierarchical Speculative Decoding (HSD), a training-free end-to-end acceleration method for document parsing. It first utilizes a lightweight pipeline to generate rough drafts for each region, followed by a two-stage verification process: Stage 1 performs parallel local verification on cropped regions to achieve high throughput, while Stage 2 conducts global verification on the full page to restore cross-region coherence. HSD achieves close to zero loss in accuracy with a 2.78x end-to-end speedup on HunyuanOCR, and up to a 7.04x speedup in long-document scenarios.

Background & Motivation¶

End-to-end document parsing with VLMs has become a mainstream paradigm. Taking page images directly as input, these models autoregressively output structured tokens (text, formulas, tables, reading order), proving robust under complex layouts, noise, and cross-region dependencies thanks to their powerful semantic and global context modeling capabilities. However, the autoregressive decoding of long sequences causes inference latency to grow linearly with the output length, a problem particularly severe when processing multi-page long documents. Existing hybrid methods (using a pipeline for layout segmentation followed by region-by-region VLM parallel decoding) improve efficiency via region-level parallelism, but decoding each region independently loses cross-region relationships (e.g., reading order and cross-column connections). Once layout segmentation or reading order priors are incorrect, the VLM is forced to decode in the wrong partitions or sequences, leading to error propagation.

The Key Challenge here is that region-level parallelism and page-level global coherence seem mutually exclusive—achieving parallelism requires partitioning regions and losing context, whereas maintaining coherence necessitates full-page autoregression, sacrificing speed. This paper demonstrates that this conflict is not inherent. The Key Insight lies in speculative decoding: where a fast drafter first "guesses" a draft, which is then verified in parallel by a target model. However, directly applying standard speculative decoding to document parsing faces unique challenges: in traditional speculative decoding, the drafter dynamically synchronizes with the target at each step, whereas in document parsing, the pipeline generates all drafts at once and remains fixed (decoupled), causing misalignment between the static drafts and the target model's sliding decoding prefix. The Core Idea is to employ hierarchical two-stage verification (parallel regional verification followed by global full-page verification) to balance speed and coherence, while using decoupled speculative verification (draft-target matching + prefix-tree batching) to address the prefix misalignment problem in decoupled scenarios.

Method¶

Overall Architecture¶

The input of HSD is a document page image, and the output is a structured parsing result in Markdown. The overall workflow consists of three steps: First, a lightweight pipeline (using PP-StructureV3 by default) performs layout analysis and element recognition on the page, exporting region divisions \(R = \{r_i\}\) and a rough draft \(\tilde{\mathcal{Y}}^{(i)}\) for each region (preliminary recognition results for text, formulas, and tables). Then, the end-to-end VLM parser verifies these drafts in two stages—Stage 1 crops each region and feeds them in parallel to the VLM for speculative verification, generating refined regional parsing results; Stage 2 aggregates all Stage 1 outputs into a page-level draft, which is then fed into the VLM for a single global speculative verification on the full-page image, correcting remaining structural errors (such as reading order, hierarchical relationships, and cross-region fractures) to output the final parsing results. The entire process is training-free, requiring only an off-the-shelf end-to-end parser and an arbitrary pipeline drafter.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Document Page Image"] --> B["Lightweight Pipeline Drafter<br/>Layout Analysis + Element Recognition"]
    B --> C["Region Draft Set<br/>One Rough Draft per Region"]
    C --> D["Stage 1: Region-Level Parallel Verification<br/>Cropped Regions → VLM Parallel SpecDecode"]
    D --> E["Stage 1 Refined Output<br/>Aggregated as Page-Level Draft"]
    E --> F["Stage 2: Page-Level Global Verification<br/>Full-Page Image → VLM SpecDecode"]
    F --> G["Final Parsing Result<br/>Structured Markdown"]

Key Designs¶

1. Hierarchical Speculative Decoding: Stage 1 Region-Level Parallel Verification

The main bottleneck is that end-to-end parsers exhibit extremely high latency when autoregressively decoding long sequences, whereas partitioning the document for region-by-region parallel decoding (hybrid methods) loses global context. HSD addresses this by extending the "draft-and-verify" of speculative decoding into two layers. In Stage 1, for each region \(r_i\) output by the pipeline, its cropped image \(z_i = x|_{r_i}\) and corresponding draft \(\tilde{\mathcal{Y}}^{(i)}\) are obtained, and the speculative decoding operator is run in parallel for verification:

\[\hat{\mathbf{y}}^{(i)} = \text{SpecDecode}(p_\theta, z_i, \tilde{\mathcal{Y}}^{(i)})\]

The key benefit here is parallelism: multiple regions undergo speculative verification simultaneously, and within each region, multiple tokens are verified at once via speculative decoding, significantly compressing the decoding steps. However, the cost is that cropped inputs lack full-page context, and layout segmentation errors from the pipeline (e.g., over-segmentation, missed detections, incorrect reading order) will propagate to this stage. Consequently, although Stage 1 outputs are already high-quality at the token level, structural issues (hierarchical errors, fragmented regions, disordered reading order) still exist and need Stage 2 to act as a safeguard.

2. Stage 2 Page-Level Global Verification: Restoring Cross-Region Coherence

The input to Stage 2 is the unordered set of all Stage 1 outputs, serving as the page-level draft:

\[\tilde{\mathcal{Y}}^{\text{pg}} = \{\hat{\mathbf{y}}^{(i)} \mid r_i \in \mathcal{R}\}\]

Then, a single speculative verification is performed on the full-page image \(x\):

\[\hat{\mathbf{y}}^{\text{pg}} = \text{SpecDecode}(p_\theta, x, \tilde{\mathcal{Y}}^{\text{pg}})\]

The key insight is that Stage 1 outputs are already high-quality page-level drafts (token-level errors have been corrected), and Stage 2 only needs to perform a small number of decoding steps for structural correction—rectifying the reading order, merging over-segmented text blocks, completing missing semantic regions, and correcting heading hierarchies. Although Stage 2 reverts to full-page autoregression, because the draft quality is high and the accepted lengths are long, the actual decoding steps are far fewer than autoregressing from scratch. The final reading order is determined by the VLM during the verification process, rather than relying on the pipeline's prior. Ablation studies show that removing Stage 2 and using only Stage 1 causes the accuracy of dots.ocr on OmniDocBench v1.5 to plunge from 88.41 to 70.47, illustrating that page-level global verification is indispensable for restoring cross-region coherence.

3. Decoupled Speculative Verification: Draft-Target Matching and Prefix-Tree Batching

In traditional speculative decoding, the drafter generates drafts under the premise of prefix synchronization—the drafter's output is a continuation of the target model's currently accepted tokens. However, the scenario in HSD is decoupled: the pipeline generates all drafts at once and remains static, without updating according to the target's decoding progress. This leads to misalignment between the drafts and the target's current sliding prefix—the target has accepted a certain sequence of tokens, but the content at the corresponding position in the drafts might differ.

HSD addresses this issue with a two-part mechanism. (a) Draft-Target Matching: At each decoding step \(t\), the most recent \(n\) tokens of the target-accepted sequence are taken as a reference window \(w = \hat{y}_{t-n+1:t}\) (default \(n=3\)), and \(w\) is slid across all drafts to locate matching positions \(j\), extracting the subsequent tokens after the matching window as candidate continuations \(C\). Formally, the set of matched positions is \(J(\tilde{y}) = \{j \mid \tilde{y}_{j:j+n-1} = w\}\), and the candidate set is \(C = \{\tilde{y}_{j+n:|\tilde{y}|} \mid \dots\}\). (b) Prefix-Tree Batching: When there are multiple candidates \(|C| > 1\), they are organized into a prefix tree \(T\)—candidates sharing common prefixes are merged into the same path, where each node \(v\) represents a unique prefix \(\pi(v)\), and its children set \(\text{Next}(v)\) contains all possible next tokens for that prefix. The prefix tree is linearized into a packed sequence \(P\), and a Tree Ancestor Attention Mask is applied—each token only attends to the accepted sequence \(\hat{y}_{1:t}\) and the tokens along its own ancestral path—allowing multiple candidate paths to be verified in a single forward pass without breaking autoregressive causal dependency.

Verification utilizes a greedy tree traversal: Starting from the root node, the token \(u^*\) with the highest probability under the model is chosen from \(\text{Next}(s)\). If it satisfies the acceptance criterion \(\log p_\theta(u^*|\dots) - \log p_\theta(\hat{u}|\dots) \geq \log \tau\) (where \(\hat{u}\) is the token with the highest probability across the entire vocabulary, \(\tau\) is the acceptance threshold, default 0.75), then \(u^*\) is accepted, and the traversal moves to its child node; otherwise, it stops. Once traversal terminates, the accepted path is appended to the accepted sequence, followed by one step of freely generated token \(\hat{u}\), concluding one round of speculative verification. Parallel verification via the tree structure allows accepting multiple tokens per step.

A Complete Example¶

Suppose the input is an academic paper page containing a title, double-column body text, a table, and a set of formulas. The pipeline first performs layout analysis, dividing the page into 6 regions (title, left-column body, right-column body, table, formula 1, formula 2) and generating rough drafts for each (e.g., HTML/Markdown draft for the table, LaTeX draft for the formulas). Stage 1 feeds the 6 cropped regions in parallel to the VLM (e.g., dots.ocr) for speculative verification: for the body text regions, the pipeline drafts might exhibit only minor OCR errors (e.g., "the" misspelled as "th"), which the VLM corrects after accepting about 4 tokens in a single speculative verification; for the table region, the pipeline drafts might fail to merge cells, and the VLM completes them during verification. After the 6 regions are processed in parallel, the outputs of Stage 1 are aggregated into a page-level draft. At this stage, however, structural issues persist: the last paragraph in the left column actually continues onto the right column but was incorrectly treated as an independent paragraph, and the table caption was partitioned into the body text region. During Stage 2 verification on the full-page image, the VLM detects these reading-order errors—observing that the semantics of the left column's last paragraph must flow into the right column—and consequently rejects the incorrect paths, rearranges the reading order, and merges the table caption back into the table region within a single speculative decoding step. This entire flow consumes approximately 3 decoding steps in Stage 1 parallel verification and roughly 5 decoding steps in Stage 2 global verification, translating to a near 2.4x end-to-end speedup compared to the baseline autoregressive decoding (which takes ~40 steps).

Key Experimental Results¶

Main Results¶

HSD is evaluated on three benchmarks: OmniDocBench v1.5 (1,355 pages, 9 document types), olmOCR-Bench (1,403 PDFs, 7,010 unit tests), and Ocean-OCR-Bench (200 Chinese-English bilingual pages), covering 6 models (Qwen2.5/3-VL series, dots.ocr, HunyuanOCR).

Model	Params	AAL	SR_decode	SR_e2e	Max Single-Class SR_e2e
Qwen2.5-VL-7B	8B	3.56	2.13x	2.10x	2.95x (Newspaper)
Qwen2.5-VL-3B	4B	2.52	2.14x	2.12x	2.80x (Newspaper)
Qwen3-VL-8B	9B	3.98	2.62x	2.61x	4.62x (Newspaper)
dots.ocr	3B	3.98	2.44x	2.42x	4.89x (Financial Report)
HunyuanOCR	0.9B	4.55	2.82x	2.78x	7.04x (Financial Report)

Comparison with existing speculative decoding methods (all using Qwen2.5-VL-3B as the target model, adapted fairly to document parsing data):

Method	Source	OmniDocBench SRe2e	olmOCR-Bench SRe2e	Ocean-OCR-Bench SRe2e
VSD	ICML 2023	1.01x	1.02x	1.12x
Medusa	ICML 2024	1.26x	1.28x	1.32x
EAGLE-2	EMNLP 2024	1.69x	1.86x	1.69x
ViSpec	NeurIPS 2025	1.75x	1.90x	1.72x
HSD	Ours	2.12x	2.64x	2.72x

Existing speculative decoding methods show limited speedups on document parsing (with the best baseline ViSpec obtaining only 1.72-1.90x), as their drafters (either independent small models or target-embedded modules) struggle to effectively predict structured outputs for long documents. HSD leverages a layout pipeline as a helper drafter, which naturally fits the document's layout structure. Coupled with hierarchical verification, it significantly outperforms all baselines.

Ablation Study¶

Accuracy ablation of the hierarchical design (using dots.ocr as the parser):

Configuration	OmniDocBench v1.5	olmOCR-Bench	Ocean-OCR-Bench
Pipeline Draft (Unverified)	86.73	65.80	85.20
dots.ocr Baseline (Autoregressive)	88.41	79.90	91.45
HSD Stage 1 only	70.47	67.30	86.92
HSD Stage 1+2 (Full)	88.81	79.40	92.56

Using only Stage 1 (region-parallel verification) causes accuracy to plummet, as it loses global context and inherits layout segmentation errors from the pipeline. Incorporating Stage 2 restores accuracy to levels comparable to or even slightly higher than the baseline (OmniDocBench 88.81 vs 88.41; Ocean-OCR 92.56 vs 91.45), demonstrating that Stage 2 effectively corrects structural errors from Stage 1.

Framework design ablation (dots.ocr, OmniDocBench v1.5):

Configuration	AAL	SR_decode	SR_e2e
Baseline Autoregressive	-	1.00x	1.00x
+ Page-Level Speculative Decoding Only	2.49	2.11x	2.09x
+ Hierarchical Speculative Decoding (Full HSD)	3.98	2.44x	2.42x

Using only pipeline drafts for page-level speculative decoding already achieves a 2.09x speedup. Integrating Stage 1 regional parallel verification improves the AAL from 2.49 to 3.98, and the end-to-end acceleration of SR_e2e from 2.09x to 2.42x, indicating that the two-stage design yields collaborative gains.

Key Findings¶

Stage 2 is the key safeguard for accuracy: Removing Stage 2 drops dots.ocr's OmniDocBench accuracy from 88.41 to 70.47 (-17.94 points), proving that while region-level parallel verification is efficient, it severely disrupts global coherence. Conversely, adding Stage 2 completely restores accuracy (88.81), indicating that a few steps of full-page verification are sufficient to correct structural errors.
Acceleration is strongly correlated with document type: Document types with longer texts and multiple semantic blocks (e.g., Newspaper, Academic Papers, Financial Reports) achieve the peak speedups (up to 7.04x) because decoding is the dominant bottleneck and region-level parallelism is maximized. Handwritten pages or degraded scans (e.g., Old Scans in olmOCR-Bench) see limited speedup (~1.4x) since poor pipeline draft quality leads to low acceptance rates.
Robustness to the choice of pipeline drafter: Even when noise is injected to degrade draft signficantly, HSD maintains a speedup of over 2.3x, illustrating that the method is insensitive to the sheer quality of the drafter. An acceptance threshold \(\tau=0.75\) and a reference window \(n=3\) yield the optimal trade-off between speed and accuracy; setting \(\tau\) too small (<0.75) noticeably degrades accuracy.
Orthogonal integration with visual token compression (VTC): Overlaying HSD on top of DeepSeek-OCR's VTC yields an additional 1.41-1.91x speedup, demonstrating the plug-and-play nature and stackability of the method.

Highlights & Insights¶

Exploiting layout structure to "free-ride" on speculative decoding's parallelism: Standard speculative decoding requires prefix synchronization between the drafter and the target. In contrast, documents naturally compartmentalize into layouts. This paper cleverly converts "partitioning" into the "natural granularity of parallel speculation," utilizing the pipeline's layout analysis results directly as multi-path drafts without step-by-step updates. The core insight is that the structural priors of documents (layout, reading order) can be exploited as free structural information for speculative decoding.
Exquisite division of labor between two verification layers: Stage 1 focuses on throughput (verifying regions in parallel to correct token-level errors), while Stage 2 focuses on quality (globally verifying the full page to correct structural errors). This "coarse-to-fine" partition matches the error types in document parsing—local OCR errors are easy to fix within regional boundaries, whereas reading-order and hierarchical errors require a global field of view. This division of labor can be transferred to other long-sequence generation tasks requiring local structures with global consistency (e.g., cross-file references in code generation, cross-paragraph coherence in long-document summarization).
Prefix-tree batching + tree ancestor attention mask is an elegant decoupled speculative decoding scheme: matching with the most recent \(n\)-token window resolves decoupled misalignment, sharing common paths via a prefix tree avoids redundant verification, and using the tree ancestor mask preserves autoregressive causality. This mechanism is independent of document parsing and can be ported to any speculative decoding scenario where a drafter generates multiple candidates statically without dynamic target synchronization.
Engineering value of being training-free and plug-and-play: It requires no modifications to the VLM architecture, no extra training data, and no dependency on a specific drafter. Simply swapping out the end-to-end parser or the pipeline drafter makes it ready to go. This design holds natural advantages for industrial deployment.

Limitations & Future Work¶

Limited acceleration in prefill-dominated scenarios: For short documents or pages with sparse output, the fixed overhead of visual encoding and prefill takes up a large portion of latency. Even if the decoding phase is highly compressed, the end-to-end speedup is diluted by prefill overhead (Table A1 in the paper shows SRe2e is only 1.05x on pages where prefill accounts for >80% of latency). A potential future direction is combining HSD with prefill-acceleration techniques such as visual token compression and KV-cache reuse.
Pipeline draft quality acts as a hard upper bound on acceleration: When the pipeline encounters handwriting or severely degraded scans, inferior draft quality results in a low matching window hit rate, short accepted lengths, and frequent rollbacks, degrading the speedup (only 1.40x on olmOCR-Bench's Old Scans). A concrete improvement direction is leveraging synthetic data to enhance the pipeline's robustness to handwritten/degraded documents, or introducing multi-drafter voting mechanisms.
Currently verified only on single-page documents: Multi-page documents (e.g., PDF books, scanned contracts) exhibit stronger cross-page dependencies (chapter continuity, figures spanning pages). The hierarchical paradigm of HSD could be extended to a three-tier structure of "intra-page region-parallel + inter-page context verification," but this requires managing multi-page image stitching and cross-page KV-cache reuse.
Global static configurations for acceptance threshold \(\tau\) and reference window \(n\): Adapting these parameters dynamically based on the document type or draft confidence (e.g., relaxing \(\tau\) for handwritten pages and tightening it for structured pages) may further improve the speedup ratio without compromising accuracy.

vs Standard Speculative Decoding (VSD / Medusa / EAGLE-2): These methods rely on prefix synchronization between the drafter and the target, refreshing drafts at each step. In document parsing, their drafters (whether independent small models or target-embedded heads) struggle to predict long structured sequences, leading to very low acceptance rates (VSD's AAL is high but its SRe2e is only 1.01x, indicating verification overhead completely offsets the step reduction). HSD diverges by exploiting the pipeline's layout analysis as a "free" source of structured drafts and transitioning from synchronous verification to decoupled matching-based verification.
vs Hybrid Document Parsing (MonkeyOCR / Dolphin / MinerU 2.5): Hybrid methods also utilize the "layout partitioning first, regional parallel parsing later" approach, but lack verification/correction mechanisms. Once layout partitioning fails, the VLM is forced to decode in the wrong context. Stage 2 full-page verification in HSD essentially adds a "global error correction" safety net to hybrid methods, representing a critical design that the hybrid paradigm can readily adopt.
vs ViSpec / HiViS (VLM-specific speculative decoding): These methods are designed for general VLM conversation scenarios, relying on visual-token-level draft prediction. In document parsing, however, outputs are highly structured long sequences (Markdown/LaTeX), where the benefits of token-level prediction are negated by the validation overhead amplified by the sequence length. HSD's insight is: for structured long-output tasks, drafts should be generated leveraging the task's structural priors (layout partitioning) rather than general token predictions.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Introducing speculative decoding to document parsing and designing a hierarchical decoupled verification paradigm is novel; however, the draft-verify framework itself of speculative decoding is established, with the core innovation lying in "how to adapt it to document scenarios" rather than a brand-new paradigm.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely thorough, covering 6 models, 3 benchmarks, multiple document types, along with ablation, hyperparameter, comparative, and stacking experiments. It compares against 10 speculative decoding baselines + 4 document parsing paradigms, accompanied by qualitative and Roofline model analyses in the appendix.
Writing Quality: ⭐⭐⭐⭐⭐ The methodology steps seamlessly from paradigms to operators; the notation is clear, with formulas closely matched to description; the experiments are reasonably grouped with clear-cut key findings; Figure 2 offers effective visualization of the DSV mechanism.
Value: ⭐⭐⭐⭐☆ Document parsing is a core scenario for VLM deployment. A training-free, plug-and-play method with nearly lossless 2-3x speedup holds strong engineering value. The paradigm of hierarchical verification and decoupled speculative decoding also possesses migration potential for other structured long-sequence generation tasks (table understanding, chart parsing, code generation).