Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild

Conference: ECCV2026
Paper: ECCV Official Page
Dataset: PaddlePaddle/Real5-OmniDocBench
Area: Multimodal VLM / OCR and Document Parsing
Keywords: document parsing, physical degradation, paired benchmark, structural recognition, robustness evaluation

TL;DR

Real5-OmniDocBench reconstructs all 1,355 OmniDocBench v1.5 pages under five physical acquisition scenarios, using 6,775 content-matched images to test parsing robustness; digital-document proficiency does not guarantee physical-world reliability, and the 0.9B PaddleOCR-VL-1.6 leads the listed large generalist models with an Overall score of 93.19 in this evaluation.

Background & Motivation

Document parsing involves more than recognizing characters: it converts page images into structured content while preserving text, tables, formulas, and reading order. OmniDocBench provides fine-grained annotations across document types, but predominantly uses ideal images obtained directly from digital documents, with clear rows and columns, flat pages, and characters untouched by paper, camera lenses, or screen recapture. In deployment, the same table may be compressed by book-spine curvature or contaminated by moire patterns after screen photography; locally readable characters do not guarantee recoverable cell relationships or multi-column reading order.

Existing real-world datasets expose these failures but do not necessarily include clean digital references with identical content, making it difficult to separate intrinsic page difficulty from acquisition conditions. Physically acquiring only a subset of pages can also change the distribution of document types and difficulty, confounding the digital-to-physical score difference with sample selection. This work leverages OmniDocBench's complete annotations and invests acquisition effort in reproducing every page, instead of creating a different set of documents that cannot be compared directly; identical content can then be tested under scanning, warping, screen photography, illumination, and skew. Core Idea: keep document content and evaluation targets fixed while systematically varying physical acquisition scenarios, using page-level correspondence to turn an aggregate real-world performance drop into diagnosable recognition and structural failures.

Method

Overall Architecture

This is a benchmark paper, not a new parsing network, and it introduces no trainable image-correction module. Its inputs are the 1,355 original digital pages and JSON annotations from OmniDocBench v1.5, covering nine document types such as academic papers, books, notes, financial reports, and magazines. Its outputs are five physical-scenario test sets containing 1,355 pages each, or 6,775 images in total; every new image has an original digital page as its content reference. Evaluated systems receive an image and produce structured parsing results, which are scored with the OmniDocBench evaluation framework for text, formulas, tables, and reading order.

Construction connects four designs: complete pairing, standardized physical acquisition, quality auditing, and consistent evaluation. Here, physical reconstruction means printing, scanning, handheld photography, or photographing a screen, not recovering a 3D mesh or synthesizing distortions with a renderer. The paper emphasizes reusing original JSON annotations without modification, so the comparison concerns the content and logical structure that should be recovered from the same page; this does not mean warped or perspective-distorted pixel coordinates remain aligned with the digital original. The benchmark introduces no new training split: the 6,775 images serve as a test set, and the model committee used to screen images is not a newly trained model either.

Key Designs

1. Complete Page-Level Pairing: keep content difficulty comparable across scenarios

Every original page appears once in each of the five main scenarios, rather than selecting only pages convenient to capture. Consequently, a complex financial report does not appear exclusively in the skew group while a simple single-column page appears only in scanning; the main scenarios share the original document set with complete coverage. Original annotations provide text, tables, formulas, layout, and reading order, avoiding independent answer annotation for the physical versions and reducing annotation-induced differences. The benefit is not simply a larger image count, but the ability to associate changes in a parser's output with acquisition conditions while semantic content stays the same.

The pairing granularity matters: each page has five main-scenario variants, not a capture under every subcondition of every scenario. The total is therefore 6,775, without multiplying again by the five subconditions within each scenario. This design supports main-scenario comparisons and preserves the original document-type distribution, but subcondition allocation, device allocation, and continuous severity do not automatically form a strictly balanced factorial experiment. A photograph can still combine warping, shadows, and reduced clarity, so factor-wise diagnosis should be understood as controlled scenario comparison rather than a pure causal measurement isolating every imaging variable.

2. Standardized Physical Acquisition: cover distinct imaging mechanisms instead of applying a uniform filter

Paper reproduction uses a Canon C5840 with printing and scanning configured at 1200 dpi; A3 or A4 paper is selected according to original page dimensions to preserve font-size ratios and layout density as far as possible. The scanning scenario includes standard acquisition, low quality produced through repeated print-scan cycles, slanted placement, stapling, and book-style binding, covering paper texture, digitization noise, and local edge effects. Warping uses folding, rolling, crumpling, curled corners, and book-spine curvature to change spatial relationships between text elements; unlike simple rotation, these non-rigid changes can alter scale and orientation within a single line. This acquisition strategy includes real materials and imaging processes instead of treating the portion reproducible by software perturbations as a complete account of physical degradation.

Screen photography covers office monitors, professional displays, laptops, tablets, and phones, introducing interference from the interaction between display pixel structure and camera sampling. Illumination covers low light, shadows, color casts, localized flashlight overexposure, and refraction through transparent media; skew includes different axes, compound poses, and extreme views rather than only in-plane rotation. The four handheld scenarios use Apple, Xiaomi, and OPPO devices at an approximate ratio of 3:1:1 to incorporate different image signal processing, sensor, and lens characteristics. High-resolution printing reduces source reproduction loss but cannot eliminate the clarity gap between handheld capture and digital originals; the authors deliberately retain ordinary clarity degradation because it occurs in deployment.

3. Audit-and-Recapture Loop: remove acquisition accidents while retaining meaningful difficulty

The authors first run several advanced vision-language models (VLMs) on captured images, flagging candidates for which every committee member performs catastrophically, such as zero-score cases. The committee locates suspicious images: it can quickly identify severe motion blur, hardware overexposure, or framing errors, but cannot by itself decide whether an image belongs in the benchmark. Removing samples solely because models score poorly could discard genuinely diagnostic hard cases, making subsequent human verification the critical quality boundary.

Three rounds of full-set human inspection check source-page identity, content completeness, and unintended 90/180-degree rotations; irrecoverable problems such as severe defocus trigger recapture and replacement. Real moire patterns, mild noise, and ordinary clarity loss are retained so that cleaning does not turn the physical benchmark into another ideal digital collection. Every scenario ultimately retains all 1,355 pages, although the authors acknowledge possible residual local blur, shadows, and slight corner clipping, without a complete quantitative human-readability assessment for every sample and scenario. The audit therefore supports usability and reliable correspondence, not a claim that every remaining error must originate in the model.

4. Consistent Multidimensional Evaluation: distinguish character errors from structural errors

For baselines with official results, the authors first reproduce performance on the original digital test set and report physical-benchmark results only after successful alignment; newer models are added as supplementary evaluations under a consistent protocol. This ordering aims to avoid confusing configuration mismatch with physical degradation, but the paper does not fully specify every model's input resolution, prompt, decoding budget, or latency, so a consistent protocol does not establish equal compute budgets. Text error, TextE, and reading-order error, ROE, both use normalized edit distance, applied to text and ordering sequences respectively; lower values are better:

\[ \mathrm{NED}(s_1,s_2)=\frac{\operatorname{Lev}(s_1,s_2)}{\max(|s_1|,|s_2|)}. \]

Formulas are evaluated using character detection matching, CDM, where higher is better, to avoid penalizing equivalent expressions solely on the basis of their LaTeX strings; the paper does not detail the underlying implementation, so this should not be presented as a new matching algorithm introduced here. Tables use tree edit distance-based similarity, TEDS, to compare predicted and ground-truth table trees in structure and content; TEDS-Struct distinguishes structural from recognition errors, although the presented scenario tables primarily report TEDS. Overall averages only the text score, table TEDS, and formula CDM, with the latter two expressed on a percentage scale; reading order is excluded from this average:

\[ \mathrm{Overall}=\frac{100(1-\mathrm{TextE})+\mathrm{TEDS}+\mathrm{CDM}}{3}. \]

Overall is therefore not the percentage of completely correct pages, and a high aggregate score does not replace checking reading order. For example, recognizing characters while scrambling column order is not sufficient evidence of successful parsing; ROE helps establish whether the output remains useful for downstream retrieval or reading.

Key Experimental Results

Main Results

The following selection comes from the paper's Table 1 and reports Real5-OmniDocBench test results; S/W/SP/I/SK denote scanning, warping, screen photography, illumination, and skew, each containing 1,355 pages. All scores are higher-is-better, and parameter counts follow the original table; "Not disclosed" means neither zero parameters nor equal resource consumption.

| Model | Parameters | Overall | S | W | SP | I | SK | |---|---|---|---|---|---|---|---|---| | PP-StructureV3 | Not disclosed | 64.45 | 84.68 | 59.34 | 66.89 | 73.38 | 37.98 | | Qwen3-VL-235B | 235B | 88.90 | 89.43 | 89.99 | 89.27 | 89.27 | 86.56 | | Gemini-3 Pro | Not disclosed | 89.24 | 89.47 | 88.90 | 88.86 | 89.53 | 89.45 | | MinerU2.5-Pro | 1.2B | 88.94 | 92.11 | 88.72 | 91.29 | 91.31 | 81.26 | | GLM-OCR | 0.9B | 90.32 | 92.67 | 90.68 | 91.75 | 91.12 | 85.39 | | PaddleOCR-VL-1.5 | 0.9B | 92.05 | 93.43 | 91.25 | 91.76 | 92.16 | 91.66 | | PaddleOCR-VL-1.6 | 0.9B | 93.19 | 94.74 | 92.48 | 92.78 | 93.28 | 92.66 |

PaddleOCR-VL-1.6 exceeds Qwen3-VL-235B by 4.29 points and Gemini-3 Pro by 3.95 points; these are results for the evaluated models and configurations, not a parameter-scaling experiment controlling training data. The introduction states "17 models," whereas Table 1 lists 20 model/tool entries; this note preserves the internal version inconsistency rather than treating the introductory count as verified.

Ablation Study

The paper contains no training ablation that removes network modules; this analysis instead uses the scenario diagnostics from Tables 2 and 6, comparing scanning with skew for the same model and original page set. TextE and ROE are lower-is-better, while CDM, TEDS, and Overall are higher-is-better; this is a scenario comparison, not a single-variable ablation of pure skew severity.

Model	Scenario	Overall	TextE	CDM	TEDS	ROE
PP-StructureV3	Scanning	84.68	0.094	84.34	79.06	0.092
PP-StructureV3	Skew	37.98	0.557	44.37	25.27	0.417
Qwen3-VL-235B	Scanning	89.43	0.059	89.01	85.19	0.066
Qwen3-VL-235B	Skew	86.56	0.077	83.96	83.41	0.091
PaddleOCR-VL-1.6	Scanning	94.74	0.036	93.65	94.12	0.043
PaddleOCR-VL-1.6	Skew	92.66	0.045	91.44	91.04	0.058

Key Findings

• PP-StructureV3 loses 46.70 points from scanning to skew, with table TEDS falling from 79.06 to 25.27; this is not merely blurrier character recognition, because structure and reading order deteriorate as well.
• PaddleOCR-VL-1.6 loses only 2.08 points in the same comparison, but its lowest scenario score is 92.48 under warping; skew is not universally the hardest scenario for every model.
• Small specialized models do not universally outperform large ones: Deepseek-OCR 2 scores 73.01 Overall in Table 1. The evidence supports the importance of particular training and system designs, not prediction of rankings from the specialized label alone.
• The paper enables paired digital-to-physical evaluation, but its result tables do not provide complete per-model digital scores and paired confidence intervals, so they cannot establish every model's exact digital-to-physical decline.

Highlights & Insights

• Complete content pairing is more diagnostically valuable than simply enlarging a real-image collection. It reduces page-selection confounding and gives scenario changes a shared reference.
• Reusing structural annotations extends error analysis beyond OCR. Table and reading-order failures reveal cases where characters are recognized but the document remains unusable.
• Separating acquisition accidents from authentic degradation is a reusable practice. Human inspection should determine whether a hard image is readable, rather than rejecting it merely because all current models fail.

Limitations & Future Work

• Author-acknowledged: residual local blur, shadows, and cropping may remain, and sample-level human-readability assessment is incomplete. Readable-region and missing-content labels could distinguish model errors from irrecoverable information loss.
• Note assessment: the five scenarios are not strictly orthogonal physical factors, and devices and clarity may vary together. Publishing subcondition assignments, severity, and device metadata would support finer matched comparisons.
• Note assessment: parameter count does not replace latency, memory, or input budgets, and cross-model results do not control training data. Explanations involving domain-specific inductive biases and augmentation should therefore be treated as hypotheses rather than completed mechanism ablations.
• Note assessment: screening anomalies with existing models may favor retaining samples they can handle. Independent human-readability evaluation, paired digital/physical errors, and confidence intervals would further strengthen the benchmark.

Related Work & Insights

• vs OmniDocBench: inherits its pages, annotations, and evaluation framework while extending digital originals into physical acquisitions. The contribution is primarily environmental coverage and comparability, not new semantic questions.
• vs WildDoc: natural acquisition better represents an open distribution, whereas this work emphasizes same-content digital references for diagnosis. Open-environment performance and controlled scenario changes are complementary questions, not substitutes.
• vs DocPTBench and scene synthesis: compared with photographed subsets, this work covers the complete source collection; compared with rendered degradation, physical capture introduces real imaging processes, but is more costly and harder to disentangle into isolated variables.

Rating

• Novelty: 4/5, complete digital/physical pairing adds a clear evaluation capability, but is not a new parsing algorithm.
• Experimental Thoroughness: 4/5, covers multiple models, scenarios, and structural subtasks, while lacking fine-grained readability and controlled resource analysis.
• Writing Quality: 3/5, acquisition and metrics are clearly organized, but model counts conflict and some mechanistic interpretations exceed the experimental evidence.
• Value: 4/5, useful for assessing document-parsing deployment risks, particularly structural failures obscured by digital-only leaderboards.