P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling¶

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/5238
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/9547.pdf
Authors: Le Xiang, Chenxi Zhai, Shu Wei, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He
Area: VLM Efficiency
Keywords: Multi-token prediction, document parsing, progressive curriculum learning, speculative decoding, dynamic drafting

TL;DR¶

P-MTP trains a document parser to establish reliable short-range predictions before taking on more distant targets and adjusts speculative length by confidence, reducing native decoding latency from 26.71 to 5.10 for a 5.24× speedup with Qwen3-VL-2B on PubTabNet; this figure must not be substituted for vLLM deployment gains or used to characterize accuracy changes across all tasks.

Background & Motivation¶

Vision-language models can turn page images directly into Markdown, LaTeX, or table structures, but dense output makes token-by-token autoregressive decoding increasingly expensive. Table tags, formula fragments, and repeated layouts often have strong visual evidence, so running the full backbone for every output token may be unnecessary. Multi-token prediction (MTP) is therefore a natural fit: an inexpensive auxiliary module drafts subsequent content, which the primary model verifies, allowing an expensive computation to accept several tokens.

The bottleneck is not simply whether more tokens can be guessed, but whether deeper drafts remain trainable and accurate enough to be accepted. Serial drafting depends on preceding predictions. If nearby targets have not been learned, propagating equally weighted losses from distant targets imposes unreliable optimization pressure on the backbone. Fixed decay weights alleviate this problem but can leave distant predictions permanently undertrained. Meanwhile, fixed-length speculation is insufficiently ambitious on simple table tags and wastes drafts at content transitions.

The paper connects training and inference through path reliability: ground-truth probabilities determine which distant targets deserve training, while candidate probabilities determine whether drafting should continue at inference time. Core idea: expand the effective training horizon with two reliability weights that evolve during learning, then let the same shared drafting module vary its prediction length according to current confidence instead of blindly adding fixed prediction heads.

Method¶

Overall Architecture¶

The input consists of a document image and the generated prefix; the output is structured text accepted after primary-model verification. P-MTP adds a repeatedly callable lightweight module after the vision-language backbone. During training, progressive curriculum loss adjusts supervision across positions and prediction depths. During inference, confidence-gated dynamic drafting selects the candidate length for the current round before primary-model verification.

“Multi-token” does not mean that all positions are predicted independently in parallel. The auxiliary module recurs serially at the feature level, but is cheaper than repeatedly running the full decoder. Sharing parameters across depths also allows unrolling beyond the training horizon. The diagram connects the stage that learns this capability to the stage that uses it; it does not imply that training loss is computed during inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Document image and prefix"] --> B["VLM backbone hidden state"]
    B --> C["Lightweight shared drafting module"]
    C --> D["Progressive curriculum loss<br/>Training stage"]
    D --> E["Confidence-gated dynamic drafting<br/>Inference stage"]
    E --> F["Primary-model verification<br/>Accept text and continue"]

Key Designs¶

1. Lightweight shared drafting module: retain serial dependencies while reducing per-step cost

Each drafting step combines the preceding hidden state with the corresponding token embedding. RMSNorm is followed by concatenation and linear dimensionality reduction, then a residual MLP produces the hidden state for the next depth. Classification uses a shared LM Head rather than a separate large prediction head at each depth. Ground-truth tokens are available for supervision and alignment during training; inference follows a candidate sequence, allowing errors to accumulate and motivating the reliability controls that follow.

This structure addresses limitations of two baseline families. Parallel heads are inexpensive but do not explicitly use one candidate to constrain the next. A full Transformer DecoderLayer has stronger capacity but can consume the acceleration gains in drafting itself. Regularities in document output create room for a lightweight residual MLP: its role is short-range feature progression, not repeating the entire visual-understanding process. The architecture comparison also shows that a longer accepted sequence does not automatically imply lower latency.

Parameter sharing has another consequence: the number of training unrolls does not structurally fix the number of inference steps. Dynamic drafting can subsequently exceed the training horizon, but this extrapolation requires empirical support; shared weights alone do not guarantee correct distant predictions.

2. Progressive curriculum loss: strengthen distant supervision only when both path and target are reliable

Conventional MTP assigns a fixed weight to each prediction depth. P-MTP instead assigns weights to each combination of starting position and depth. Its first factor is the sequential path constraint: uncertainty about an earlier ground-truth token should weaken supervision farther ahead. The mechanism multiplies the probabilities assigned to ground-truth tokens at preceding depths rather than inspecting only the current depth's local confidence.

\[ \omega^{1}_{(t,k)}=\prod_{j=0}^{k-1}\hat p_t^j[x_{t+j+1}]. \]

Here \(t\) is the starting position, \(k\) is the look-ahead depth, \(x\) denotes a ground-truth token, and \(\hat p_t^j\) is the predictive distribution at depth \(j\); \(j=0\) corresponds to the primary model's next-token prediction. The product amplifies weak links along a path, so the model is not forced to master distant predictions while its nearby predictions remain unreliable.

The second factor is the retrospective target constraint. Instead of following subsequent predictions from one starting point, it fixes a target token and examines confidence in that target from other prediction distances. If a target is difficult even at closer range, its more distant prediction receives less training pressure. Multiplying the two factors gives the final weight \(\omega_{(t,k)}\), checking both “the path leading here” and “this particular target.” The product indices in cached Eq. (7) are inconsistent with the adjacent description of the primary head and closer distances. The mechanism is therefore explained without silently correcting the exact indices in Eq. (7).

Reconstructed from the legible content of Eq. (9), the curriculum term is weighted cross-entropy:

\[ \mathcal L_{\mathrm{P\text{-}MTP}}=-\sum_{k=1}^{K}\frac{1}{T-k}\sum_{t=0}^{T-k-1}\omega_{(t,k)}\log\hat p_t^k[x_{t+k+1}]. \]

Here \(T\) is sequence length and \(K\) is the maximum training look-ahead depth. Early in learning, low-confidence products concentrate the loss on simple, nearby positions. Once these improve, the weights naturally rise and distant targets receive more supervision. This is neither an epoch-based curriculum switch nor permanent suppression of distant losses. The paper supports this interpretation with changes in the loss proportions across depths during training. However, the readable text does not specify whether gradients are stopped through the weight computation. Reproduction should check the implementation rather than assume a particular gradient treatment.

3. Confidence-gated dynamic drafting: use the entire candidate path to decide when to stop

Training deeper predictions does not imply that every inference round should go equally far. P-MTP expands the maximum inference budget to \(H=2K\) while accumulating the product of generated candidate-token probabilities. Expansion stops when cumulative reliability no longer exceeds threshold \(\delta\). These are candidate probabilities, not the ground-truth probabilities available during training. High-confidence table tags can extend farther; uncertain content triggers an earlier return to primary-model verification, avoiding additional drafting expense on an already unreliable path.

The threshold does not replace the verifier. It only decides how much draft is worth preparing; candidates still require primary-model verification before acceptance. A longer draft and greater output progress are therefore different quantities. The reported average acceptance length measures effective progress per decoding round. Its baseline value is 1 because ordinary next-token decoding advances by one token per round; it must not be mistaken for a percentage acceptance rate.

The paper also calibrates the threshold using validation loss at convergence. Higher residual uncertainty calls for a more conservative gate, while better convergence brings the threshold closer to the empirical baseline of 0.3; the sensitivity coefficient is typically 2. The cached exponential expression has missing characters, so the damaged formula is not presented as directly implementable. Supplementary experiments compare thresholds and budgets, but support their selection only in the tested settings, not an optimal stopping theorem for arbitrary models.

A Worked Example¶

Consider an image that must be converted into an HTML table. Ordinary decoding emits tags and cell contents one step at a time. P-MTP instead repeatedly invokes the shared drafting module from the current hidden state. For predictable closing tags, the product of candidate probabilities may remain above the gate, allowing further expansion. Upon reaching uncertain cell text, the product falls, drafting stops early, and the primary model verifies the candidates. This illustrates control flow rather than measured probabilities for a particular sample.

In the paper's measured table configuration, training depth \(K=9\) and inference budget \(H=2K\) yield an average acceptance length of 8.60 with dynamic drafting, versus 7.29 with fixed-length drafting. The value 8.60 is an average across rounds, not a count accepted in every round, and does not imply that all budgeted candidates pass verification.

Loss & Training¶

The total objective retains the primary model's next-token cross-entropy and adds the curriculum-weighted MTP loss, maintaining basic parsing capability while training the drafting module. Training lasts 2 epochs: the first uses standard supervised fine-tuning, and the second introduces MTP. The backbone is not permanently frozen, so the accelerated model and the original SFT baseline do not have identical parameters, and task accuracy can change.

The main experiments use InternVL3.5-1B and Qwen3-VL-2B. Peak learning rates are 1e-5 for the backbone and 5e-5 for the MTP module, with a warmup ratio of 0.05 and global batch size of 8. The optimizer is reported as Adam following the main text; no particular implementation is inferred from its reference number. Formula and table tasks use their official training sets, while full-document parsing uses 0.42M public samples from LightOnOCR-2. Timing uses a single NVIDIA A100 40GB, with native forward execution and vLLM measured separately.

Key Experimental Results¶

Main Results¶

The following selection from Table 5 compares SFT baselines with P-MTP. Throughput TPS denotes output tokens per second, where higher is better. Formula recognition uses CDM, table recognition uses TEDS, and full-document parsing uses the official Overall score, all reported as higher-is-better in the source. TEDS compares structural and content similarity between predicted and ground-truth tables. Scores are not comparable across tasks, and the detailed aggregation definition for Overall is not expanded in the cached paper.

Backbone / Task	Dataset	Metric	Baseline score	P-MTP score	Score change	Baseline TPS	P-MTP TPS
InternVL3.5-1B / Formula	UniMERNet	CDM ↑	95.64	92.77	-2.87	1136	1684
InternVL3.5-1B / Table	PubTabNet	TEDS ↑	83.40	86.26	+2.86	2475	5816
InternVL3.5-1B / Document	OmniDocBench	Overall ↑	73.30	71.34	-1.96	1324	1822
Qwen3-VL-2B / Formula	UniMERNet	CDM ↑	95.88	94.58	-1.30	1649	2663
Qwen3-VL-2B / Table	PubTabNet	TEDS ↑	85.09	84.78	-0.31	2774	4315
Qwen3-VL-2B / Document	OmniDocBench	Overall ↑	86.31	81.28	-5.03	582	891

Score changes are absolute differences, not relative percentages. The critical counterexample is full-document parsing with Qwen3-VL-2B: throughput improves, but Overall falls by 5.03 points. A blanket claim of lossless accuracy is therefore inappropriate.

Table 4 separately reports deployment performance across vLLM batch sizes. These gains are substantially smaller than the 5.24× native decoding result:

Batch size	Baseline TPS ↑	P-MTP TPS ↑	Speedup ↑
4	647	1028	1.59×
16	1894	2921	1.54×
32	2774	4315	1.56×
64	3719	5638	1.52×

Ablation Study¶

All results below use Qwen3-VL-2B on PubTabNet, with training depth 9 for non-baseline rows. Values are selected from Tables 2 and 3. Lower latency is better; the source table does not label its unit, so the original values are retained without adding one. Average acceptance length \(\tau\) measures the mean number of tokens by which a decoding round effectively advances.

Config	Drafting	TEDS ↑	Latency ↓	Average acceptance length ↑	Speedup ↑
NTP baseline	No drafting	85.09	26.71	1	1
Uniform loss	Fixed	82.54	7.82	5.85	3.42
Static-decay loss	Fixed	85.33	5.95	7.05	4.49
Sequential path constraint only	Fixed	84.16	6.33	6.95	4.22
Retrospective target constraint only	Fixed	84.37	6.09	7.06	4.39
Product of both constraints	Fixed	84.78	5.53	7.29	4.83
Product of both constraints	Dynamic	84.78	5.10	8.60	5.24

Key Findings¶

Combining the constraints improves on either alone: with fixed drafting, speedup rises from 4.22 or 4.39 to 4.83, with higher TEDS as well. However, static decay achieves TEDS 85.33 versus 84.78 for curriculum loss, so the latter should be described as a speed–accuracy trade-off rather than the best on every metric.
Dynamic drafting leaves TEDS unchanged in this table while increasing average acceptance length from 7.29 to 8.60 and reducing latency from 5.53 to 5.10. It improves inference scheduling over an already trained drafting capability, rather than adding another accuracy-improving training stage.
Deeper is not always faster. In supplementary Table 10, increasing training depth from 9 to 18 raises the dynamic variant's average acceptance length from 8.60 to 9.00, but reduces speedup from 5.24 to 5.13 and TEDS from 84.78 to 83.75.

Highlights & Insights¶

Curriculum signals come from current predictions rather than a manually specified schedule of which step to learn at which epoch. This turns the value of distant supervision into a path-reliability question and explains why simple depth decay may be too rigid.
The lightweight module and dynamic stopping must be considered together. More candidates do not imply more useful throughput; the contribution is to examine prediction quality, verified acceptance length, and drafting overhead along the same execution path.
The paper reports both native execution and an optimized inference framework. These results prevent deployment gains from being inferred solely from a slow baseline, and that distinction must be retained when interpreting the paper.

Limitations & Future Work¶

The authors acknowledge substantial degradation in full-document scores with public data and attribute it mainly to data scale and quality. With approximately 1.5M private samples, Qwen3-VL-2B Overall still decreases from 88.71 to 87.19 while TPS rises from 716 to 1437. This supports the possibility that better data improves the trade-off, but does not causally isolate the data factor and also limits reproducibility.
Training depth 18 already shows saturation and accuracy loss, and the maximum draft budget also requires restraint. In supplementary Table 9, expanding the budget from 2K to 3K increases acceptance length only from 8.60 to 8.75 while latency rises from 5.10 to 5.24.
Some cached equations are damaged. Retrospective-weight indexing, exact threshold calibration, and gradient handling for the weights need checking against the original PDF or implementation. The empirical benefit of threshold 0.3 cannot be promoted directly to a cross-model statistical optimality guarantee.
Main timing experiments use only a single A100, and quality degradation on full documents shows that gains depend on content regularity. Further validation should prioritize accuracy-sensitive cases and different deployment loads rather than merely increasing prediction depth.

vs Medusa: Parallel heads and static-decay weights represent a different design choice. P-MTP uses a serial shared module and adjusts losses with sample confidence. The paper's Medusa-style weighting ablation is not a faithful end-to-end reproduction of the complete Medusa system.
vs DeepSeek-V3-style MTP: Both retain sequence dependencies, but this paper uses a lightweight residual MLP and curriculum weights rather than relying on heavier projection modules and fixed weights. These conclusions concern the document experiments here and do not establish a comparison of the original models' overall capabilities.
vs document task decomposition and visual-token compression: The former changes the page-processing workflow, while the latter mainly reduces input-side burden. P-MTP addresses the token-by-token output bottleneck. It is not a quantization, pruning, or knowledge-distillation method; its focus remains VLM inference efficiency.

Rating¶

Novelty: 4/5. The dual-reliability curriculum connects clearly to dynamic drafting, while building on existing serial MTP and speculative decoding.
Experimental Thoroughness: 4/5. Tasks, backbones, model scales, and vLLM loads are covered, but rigorous attribution of public-data accuracy degradation is missing.
Writing Quality: 3/5. The overall narrative is clear, but negligible-accuracy-impact claims are too broad and some mathematical descriptions need checking.
Value: 4/5. Useful for regular, output-dense document parsing, with practical gains best judged jointly by deployment framework and quality requirements.