ECCV2026 Multimodal VLM Unified Text and Formula Recognition Multi-Level Recognition Hierarchical Supervision Semantic-Decoupled Tokenization Lightweight Models

UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters¶

Conference: ECCV2026
Paper: ECCV Official Page
Code: https://github.com/Topdu/OpenOCR
Area: OCR / Document Parsing
Keywords: Unified Text and Formula Recognition, Multi-Level Recognition, Hierarchical Supervision, Semantic-Decoupled Tokenization, Lightweight Models

TL;DR¶

UniRec-0.1B trains a 0.1B encoder-decoder from scratch using approximately 40 million multi-level samples, explicit line and paragraph supervision, and decoupled text/formula tokenization, achieving an average edit distance of 0.100 on UniRec-Bench and an average block recognition time of 0.37 seconds under the paper's specified single-GPU conditions.

Background & Motivation¶

Not every component of document parsing is equally difficult. The paper's analysis of OmniDocBench finds that text, formulas, and mixed content account for 97.43% of page regions; with MinerU2.5, they consume 36839 seconds, or 87.90% of total parsing time. Assigning these frequent, relatively saturation-prone recognition tasks to the same large vision-language model (VLM) used for complex structures such as tables imposes substantial per-token decoding costs. In the authors' scaling experiment, text and formula performance saturates around 0.1B parameters, whereas tables continue to benefit from larger models. This is an empirical observation under their settings, not a universal scaling law for all OCR tasks.

Simply shrinking the recognizer creates two specific difficulties. A character, a line, a paragraph spanning several lines, and an exam question mixing text with formulas have very different spatial structures. Flattening their annotations into strings loses the distinction between a visual line wrap and a paragraph ending. Meanwhile, a shared tokenizer may split the ordinary word sum and the LaTeX command \sum into shared pieces, leaving a capacity-constrained model to resolve their meanings from context. Conventional line recognizers are small, but typically require additional detection to process multi-line paragraphs and do not directly cover text-formula mixtures.

The paper retains a standard encoder-decoder and concentrates on training data and output representation: it supplies genuine supervision at multiple levels while reducing structural and semantic ambiguity in the target sequence. The 0.1B model is not expected to replace layout analysis, table recognition, and reading-order processing on its own. Instead, it handles the dominant text and formula regions. Core Idea: reduce the learning burden on a small model through explicit line/paragraph boundaries and formula-aware token representations, then train a unified recognizer on large-scale, multi-domain, multi-level data for integration into existing parsing systems.

Method¶

Overall Architecture¶

The input is an image region containing text, formulas, or both; the output is a sequence containing text, formula markup, and paragraph structure. The visual encoder is FocalNet. Inputs retain their aspect ratio, with width and height capped at 960 and 1408 pixels, respectively. Spatially downsampled, 768-dimensional features are flattened into visual tokens for a 6-layer Transformer decoder with cross-attention.

During training, UniRec40M supplies labels augmented with line and paragraph boundaries. SDT converts these labels into discrete target sequences, and cross-entropy trains next-token prediction. At inference time, the model receives only the image and its generated prefix, not ground-truth labels. Generated hierarchical tokens are converted into ordinary paragraph structure during post-processing. Dashed edges below denote training supervision; solid edges show data preparation and inference flows.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    dataset["Multi-Level Data Construction<br/>UniRec40M"] --> hierarchy["Hierarchical Supervision HST<br/>Label line and paragraph boundaries"]
    hierarchy --> tokenizer["Semantic-Decoupled Tokenization SDT<br/>Encode target sequences"]
    image["Input image region"] --> encoder["FocalNet<br/>Visual features"]
    encoder --> decoder["Autoregressive decoder<br/>Cross-attention"]
    tokenizer -.->|Training labels| loss["Token cross-entropy"]
    decoder -.->|Training predictions| loss
    decoder --> output["SDT decoding and boundary processing<br/>Text, formulas, paragraphs"]

Unification here concerns tasks and granularity, not general question answering. In the full-page system experiments, MinerU2.5 or PaddleOCR-VL still provides layout analysis and table recognition; only second-stage recognition of text and formula regions is replaced. Thus, eliminating a separate text-line detector means that the recognizer can process multi-line regions directly, not that the entire system requires no layout detection.

Key Designs¶

1. Multi-Level Data Construction: obtain aligned text and formula supervision from source documents

UniRec40M matters not only because of its size, but because one data pipeline covers text, formulas, mixed content, and multiple recognition levels. The authors collect approximately 2 million arXiv TeX sources and TeX files converted from Wikipedia HTML. They insert unique color identifiers for valid text or formula tokens and render the sources into PDFs. Matching source tokens to rendered regions by color yields word- and line-level supervision, while further LaTeX parsing supplies paragraph structure. Training labels therefore specify not just which characters are present, but which lines and paragraphs they belong to, providing the foundation for HST.

Rendered data alone can overrepresent clean typography. The pipeline therefore also uses PyMuPDF to extract text blocks and image regions from digital-born research reports and newspapers, and incorporates public datasets including LSVT, MTWI, HierText, CASIA-HWDB, TAL, and K-12. Handwritten notes receive annotations assisted by Qwen3VL-235B-A30B and refined manually; this is a data-labeling step, not knowledge distillation into the recognition network. The collection contains roughly 30 million English and 10 million Chinese samples, or approximately 19 million text-only, 13 million formula-only, and 8 million mixed samples. Table 2 gives the more precise total of 39.60M, whereas proportion-balanced subsampling or resampling yields 12.63M samples per epoch. Training for 10 epochs should not be interpreted as traversing all 40 million examples each time.

2. Hierarchical Supervision HST: distinguish visual line wraps from paragraph endings

HST represents spatial structure as learnable symbols in the target sequence: <|ln|> marks a line break within a paragraph, and <|pn|> marks the end of a paragraph. Ordinary character supervision specifies content without explicitly indicating when the decoder should move to another visual line or begin a new paragraph. These tokens let the same decoder learn character recognition and line/paragraph boundaries together. No additional layout prediction head or separate structural loss is introduced. Structural symbols receive the same next-token supervision as ordinary content, changing the learning target without adding a complex inference module.

After generation, <|ln|> is removed and <|pn|> is replaced with two newline characters. A paragraph visually wrapped across two lines is therefore not incorrectly split into two paragraphs, while genuine paragraph boundaries remain. Training uses finer structural information than the final output exposes, and the resulting text remains suitable for reading and downstream processing. In Table 3, paragraph edit distance drops from 0.049 without HST to 0.032 with it, supporting a connection between its gains and structural granularity. However, boundary supervision depends on annotation quality; special tokens alone do not solve reading order for arbitrary layouts.

3. Semantic-Decoupled Tokenization SDT: avoid forcing formula commands to reuse ordinary word pieces

SDT trains separate tokenizers on plain text and mathematical formulas, then adds formula tokens absent from the text vocabulary as special tokens in the text tokenizer. In Figure 3, Dolphin separates the backslash from pieces such as sum, whereas SDT can preserve formula units such as \sum, \infty, and \frac, avoiding an identical representation for the corresponding ordinary text fragments. The procedure learns vocabularies separately and merges them into a single output vocabulary. It does not run two decoders at inference time or force every character into two disjoint token-ID sets; the paper explicitly excludes tokens already present in the text vocabulary.

This design assigns part of semantic disambiguation to the discrete representation instead of requiring the small model to rediscover the distinction in every context. The decoder still generates ordinary text, formula content, and structural markers in a single sequence, supporting paragraphs that alternate between text and formulas. It adds neither a mathematical solver nor a formula syntax validator: the objective is faithful visual transcription. In the ablation without HST, adding SDT reduces formula edit distance from 0.255 to 0.144, a substantially larger gain than for plain text. This supports formula representation as an important bottleneck, but the experiment alone cannot distinguish semantic decoupling from potential effects such as tokenized sequence length.

A Worked Example¶

Consider an image region containing two lines of English explanation followed by a separate formula paragraph. The explanation mentions the ordinary word sum, while the formula contains \sum. This is an illustrative input for explaining the pipeline, not an additional experimental sample.

Data preparation records the line wrap inside the first paragraph with <|ln|>, and marks the ends of the explanation and formula paragraphs with <|pn|>. SDT encodes the ordinary word and formula command into distinguishable tokens. FocalNet supplies spatial visual features from the same image, and the decoder generates the target sequence from these features and the preceding tokens. During training, incorrectly predicting a boundary counts as a token prediction error rather than an ignored formatting difference.

At inference time, these labels are unavailable. The model predicts content and boundaries from the image, SDT maps generated IDs back to text and LaTeX, and post-processing removes within-paragraph line markers while restoring paragraph separation. The reader receives a continuous explanation paragraph and a separate formula paragraph. The system does not first need to crop the explanation into two line images and stitch their outputs together, nor does it treat the English word sum as an instruction to perform mathematics.

Loss & Training¶

The network performs autoregressive decoding with a causal mask and applies cross-entropy to all target tokens, including text, formulas, and structural markers. No separate HST-specific loss or distillation objective is reported. Several mathematical expressions in the cached text are corrupted by extraction, so this note explains the training mechanism from the prose rather than reconstructing them as exact author equations.

Training starts from scratch without pretrained weights. The decoder hidden size is 768; the stated 64 dimensions per head imply 12 attention heads. The vocabulary contains 56371 tokens, and the maximum target length is 1024. AdamW uses a learning rate of 0.0001, weight decay of 0.01, and global batch size of 64 for 10 epochs. The prose specifies a one-cycle schedule with linear warm-up over the first 0.5 epochs. Augmentations include rotation, distortion, motion blur, and Gaussian noise. Training takes approximately 80 hours on 8 A800 40GB GPUs. Lightweight deployment parameters do not imply small training-data or training-compute requirements.

Key Experimental Results¶

Main Results¶

UniRec-Bench extracts regions from OmniDocBench, comprising 14301 text blocks, 620 formula blocks, and 1314 mixed blocks, with five granularity levels, Chinese/English categories, and nine document domains. The table below preserves the Edit values from the paper's Table 3. Edit measures differences between predicted and ground-truth strings, with lower values indicating better performance. Avg is the arithmetic mean of the three modality columns, not a sample-count-weighted dataset accuracy. These values are not recast as exact-sequence match accuracy.

Model	Parameters	Avg Edit	Text Edit	Formula Edit	Mixed Edit
Dolphin-1.5	0.3B	0.206	0.050	0.365	0.202
MinerU2.5	1.2B	0.154	0.167	0.140	0.155
PaddleOCR-VL	0.9B	0.100	0.041	0.125	0.135
UniRec-0.1B	0.1B	0.100	0.038	0.134	0.128

UniRec matches PaddleOCR-VL on Avg, improves text and mixed-content recognition, but is worse by 0.009 on formulas. It does not lead on every task. In the full-page experiments of Table 5, replacing only text/formula recognition reduces MinerU2.5 Overall Edit from 0.143 to 0.120 and PaddleOCR-VL from 0.115 to 0.113, with layout analysis and table recognition unchanged.

Ablation Study¶

The following results come from Table 3. All three configurations contain 0.1B parameters; the paragraph column concerns only the paragraph-level text subset.

Config	Avg Edit	Text Edit	Formula Edit	Mixed Edit	Paragraph Edit
Without HST, without SDT	0.159	0.062	0.255	0.161	0.053
Without HST, with SDT	0.113	0.050	0.144	0.143	0.049
Full model	0.100	0.038	0.134	0.128	0.032

Without HST, SDT reduces formula Edit by 0.111. With SDT, HST reduces paragraph Edit by 0.017. These are absolute differences, not relative percentages of error reduction. The paper does not provide a configuration with HST but without SDT, preventing a complete separation of their interaction effects.

The efficiency results below are from Table 6. All models are measured on one A800 40GB GPU, with batch size 1, Torch or PaddlePaddle dynamic graph execution, KV Cache enabled, and no additional inference acceleration.

Model	Blockavg / seconds	Pageavg / seconds
MinerU2.5	2.54	42.72
PaddleOCR-VL	1.88	31.92
Dolphin-1.5	0.78	13.16
UniRec-0.1B	0.37	6.20

Relative to PaddleOCR-VL, the block-level speed ratio in Table 6 is approximately 5.08. Its Pageavg should not be conflated with the system timing in Figure 1(d), which retains table recognition: the substituted system still spends 5069 seconds on tables, and the authors report approximately 4-fold system acceleration. Elsewhere, the prose uses 42.72/6.20 to describe nearly 7-fold page acceleration, reflecting different timing scopes. This note preserves the reported values and their boundaries rather than claiming 6.20 seconds as the end-to-end latency of any complete hybrid system.

Key Findings¶

Structural supervision particularly benefits paragraphs, multi-paragraph content, and complex layouts, whereas tokenization changes most strongly benefit formulas. The two components address different problems.
In Table 4, Edit on the handwritten Note subset is 0.055 versus PaddleOCR-VL's 0.067. Domain coverage in the training data matters, so gains cannot all be attributed to network architecture.
The small model's advantage concerns its specialized text/formula recognition scope, not the other capabilities of general-purpose VLMs.

Highlights & Insights¶

Limited capacity is prioritized for recognition instead of learning every output convention implicitly. HST and SDT remove avoidable ambiguity from supervision representations rather than adding complex modules.
Allocating compute according to document-region frequency is more targeted than assigning the same model size to every task. This depends on reliable region routing and experts for the harder tasks.

Limitations & Future Work¶

The authors' conclusion mainly identifies uneven cross-granularity capabilities in existing models, without a comprehensive taxonomy of UniRec failures. Those broader observations should not be presented as established limitations unique to UniRec.
This note's assessment: coverage is primarily Chinese and English, and the maximum target length is 1024. Very long documents and other languages require additional evaluation; multi-level results do not establish those capabilities.
This note's assessment: the ablation is not a complete two-by-two design and does not isolate dataset scale, vocabulary size, or sequence length. Evidence is insufficient to identify semantic decoupling as the sole cause of improvement.
This note's assessment: single-sample dynamic-graph timing does not replace high-throughput deployment evaluation. End-to-end comparisons should also align batching, inference engines, layout processing, and table costs.

vs PP-Recv5 / OpenOCR-Rec: conventional experts focus on characters, words, and lines, while UniRec expands to multi-line and text-formula mixtures. Its value lies in broader coverage, not a parameter-count-based claim that it replaces every fast line recognizer.
vs Dolphin-1.5: both can recognize document regions, while this work emphasizes formula representation and multi-domain data. Cross-model comparisons do not exclude training-data differences, so SDT ablations are important for interpreting the contribution.
vs MinerU2.5 / PaddleOCR-VL: the proposal replaces high-frequency recognition modules rather than requiring an entirely new parsing system. A transferable principle is to measure task frequency and capacity saturation before deciding which capabilities need separate experts.

Rating¶

Novelty: 3/5. The combination of data and representation design is targeted, while the network architecture is conventional.
Experimental Thoroughness: 4/5. Evaluations cover multiple levels and domains, component ablations, and system substitution, but lack a complete interaction ablation and a unified deployment timing scope.
Writing Quality: 4/5. Problems and designs align clearly, although speed claims require careful attention to measurement scope.
Value: 4/5. The work offers practical lessons in data pipelines, target representations, and system integration for low-parameter document recognition.