Table-MCR2TR: Merged-Cell-Aware Table Recognition via Reinforced Multimodal Language Models¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Reinforcement Learning / Multimodal VLM
Keywords: table recognition, merged-cell recognition, reinforcement learning, collaborative alignment, multimodal large language models

TL;DR¶

Table-MCR2TR addresses the critical failure of table recognition models on complex merged cells by constructing the large-scale MCT-350K dataset and introducing a progressive multi-task paradigm coupled with GRPO-based collaborative reinforcement learning (M-CRL), enabling a 3B model to achieve state-of-the-art performance and outperform leading proprietary models such as Gemini-2.5-Pro across multiple benchmarks.

Background & Motivation¶

Tables serve as a foundational semi-structured data representation across scientific literature, corporate financial statements, insurance policies, and digital documents. Contemporary table recognition (TR) approaches are broadly divided into traditional specialized small models (e.g., RapidTable, PP-StructureV3) and multimodal large language models (e.g., Qwen2.5-VL, InternVL3.5, Dolphin). While these methods demonstrate competent performance on simple, uniformly structured grids, their capability degrades significantly when confronted with complex tables containing dense multi-row and multi-column merged cells ($\text{colspan} > 1$ or $\text{rowspan} > 1$). Inaccurate merged-cell predictions disrupt the underlying hierarchical coordinates of the table, causing cascading failures in downstream table question answering and information extraction pipelines.

The inability of existing methods to resolve complex merged cells stems from two fundamental bottlenecks: severe training data imbalance and optimization metric insensitivity. On the data side, over 95% of table samples in established open-source benchmarks (such as SciTSR, PubTabNet, and FinTabNet) contain fewer than 10 merged cells, leaving models unexposed to dense spanning structures. On the optimization side, mainstream MLLMs rely on end-to-end autoregressive supervised fine-tuning (SFT) to generate complete HTML sequences in one shot, inevitably missing localized fine-grained spanning tags during long-sequence decoding. Furthermore, existing reinforcement learning attempts typically rely on Tree-Edit-Distance-based Similarity (TEDS), which is fundamentally insensitive to merged-cell errors—empirical analysis demonstrates that even when all merged-cell span attributes are predicted incorrectly (0% merged-cell accuracy), the global TEDS score can still remain as high as 72.7%, failing to supply effective gradient feedback.

The entry point of this paper is to shift away from black-box monolithic generation by explicitly decoupling structural spanning priors into verifiable intermediate representations. The core idea is to establish merged-cell recognition as an explicit contextual prior that guides full HTML table generation, coupled with the MCT-350K dataset and a tripartite collaborative reward under GRPO incorporating TEDS, MCR F1, and cross-stage alignment.

Method¶

Overall Architecture¶

The Table-MCR2TR pipeline is grounded upon two primary pillars: a high-fidelity data construction pipeline that remedies the scarcity of complex merged-cell samples, and the MCATR (Merged-Cell-Aware Table Recognition) optimization framework unifying supervised pre-training with collaborative reinforcement learning. During inference, rather than directly outputting a lengthy HTML string, the model executes a chain-of-thought style progressive reasoning procedure: given an input table image, it first predicts an explicit list of merged-cell attributes (rowspan, colspan, and text content) in Task 1, and subsequently leverages this recognized structural prior as input context in Task 2 to autoregressively decode the complete, structurally sound HTML sequence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    Input["Input Table Image<br/>Scans / Real Photographed Geometries"] --> Stage1["High-Fidelity Complex Merged Data Pipeline<br/>MLLM Synthesis + Web Crawling Multi-Check + UVDoc Reverse Warping"]

    subgraph MCATR["MCATR Progressive Training & Inference Framework"]
        direction TB
        Stage1 --> SFT["3-Task Joint Supervised Fine-Tuning<br/>TR Task + MCR Task + MCR2TR Alignment Task"]
        SFT --> Design1["Progressive MCR2TR Alignment Paradigm<br/>Explicit Spanning Triplet Prediction → Context-Conditioned HTML Generation"]
        Design1 --> Design2["Collaborative Reinforcement Fine-Tuning<br/>GRPO-Based Sequence-Level Multi-Hypothesis Exploration"]
        Design2 --> Reward["Tripartite Collaborative Reward Calculation<br/>r_teds (Global Tree) + r_f1 (MCR Match) + r_align (Cross-Stage Consistency)"]
    end

    Reward --> Output["High-Fidelity Structured HTML Output<br/>Intact Topology / Zero Span Deviation / Empowering Downstream QA"]

Key Designs¶

1. High-Fidelity Complex Merged Data Pipeline: Dual-Stream Synthesis and Tripartite Verification

To resolve the extreme scarcity of complex merged tables in public datasets, the authors construct a dual-branch generation and auditing pipeline. The first branch employs frontier MLLMs (e.g., Gemini-2.5-Pro) with parameterized prompts explicitly governing merged-cell counts, rowspans, and colspans to synthesize high-complexity HTML code, subsequently rendered into scanned-style images. The second branch collects unannotated real-world tables across financial, accounting, insurance, and customs domains, utilizing an ensemble of MLLMs (e.g., Qwen-VL-max, PaddleOCR-VL, MinerU) to produce candidate HTML sequences.

To purge pseudo-label hallucination and noise, a three-stage quality checker is established: - Cross-Model Consistency Checker: Pairwise average TEDS across $K$ recognition models is computed: $$\text{TEDS}_{avg} = \frac{2}{K(K-1)} \sum_{1 \le i < j \le K} \text{TEDS}(H_i, H_j)$$ Samples with $\text{TEDS}_{avg} < \mu_{TEDS} = 0.7$ are discarded to filter out degraded or ambiguous geometries. - HTML Structure Checker: The candidate HTML is parsed into a 2D logical grid matrix; if unoccupied fragmented cells exceed $\tau_{frag} = 0.05$ of the total area, the sequence is filtered out. - Merged-Cell Count Checker: All samples containing more than 10 merged cells are retained, while the rest are sub-sampled, ensuring that tables with $>20$ merged cells constitute 16.7% of the resulting 350K dataset (MCT-350K). Finally, real-world 3D geometries and UV displacement fields from UVDoc are applied in reverse to warp clean scans into realistic photos with natural distortions and variable illumination.

2. Progressive MCR2TR Alignment Paradigm: Explicit Structural Priors Mitigating Long-Sequence Burden

Monolithic SFT directly maps images to thousands of HTML tokens, causing attention drift where the decoder overlooks local spanning constraints. MCATR decomposes table modeling into three synergistic objectives: - Table Recognition (TR): Maps an image directly to full HTML. - Merged-Cell Recognition (MCR): Parses all spanning regions into an explicit attribute list $M = \{(r_i, c_i, t_i)\}_{i=1}^N$, outputting rowspan $r_i$, colspan $c_i$, and textual content $t_i$. - MCR2TR Alignment: Demands that the model output the <task1> MCR sequence </task1> first, followed immediately by <task2>, where the model generates the complete <table>...</table> conditioned on both the image and the generated MCR predictions.

This progressive formulation functions as an explicit visual scratchpad, substantially narrowing the search space during full HTML generation.

3. Collaborative Reinforcement Fine-Tuning: Tripartite Rewards Rescuing Metric Insensitivity

Token-level cross-entropy loss optimizes immediate probability distributions but remains oblivious to two-dimensional topological validity. To achieve sequence-level structural optimization, the authors deploy GRPO (Group Relative Policy Optimization) with 8 rollout candidates per sample. Addressing the insensitivity of TEDS to localized spanning errors, a tripartite collaborative reward structure is introduced: - Global Structure Reward $r_{teds}$: Normalized minimum HTML tree edit distance: $$r_{teds} = 1 - \frac{\text{EditDist}(H_{pred}, H_{gt})}{\max(|H_{pred}|, |H_{gt}|)}$$ - Fine-Grained Spanning Reward $r_{f1}$: An optimal bipartite matching algorithm matches prediction set $M_{pred}$ with ground truth $M_{gt}$. A match requires identical row/col spans and ANLS text similarity $> \tau_{sim} = 0.5$, producing an exact MCR F1 score. Because MCR F1 relies on discrete matching logic, it exhibits high variance across rollouts, enlarging advantage differentials and driving policy exploration. - Cross-Stage Alignment Reward $r_{align}$: Evaluates consistency between spanning attributes parsed from the final HTML in Task 2 and the predicted MCR attributes from Task 1, preventing the model from ignoring its own generated context. The task-specific rewards are formulated as: $r_{TR} = r_{teds}$, $r_{MCR} = r_{f1}$, and $r_{MCR2TR} = r_{teds} + r_{f1} + r_{align}$.

A Worked Example¶

Consider a balance sheet with a 2-column header and a 2-row sub-metric: 1. The model ingests the image and generates the <task1> block:

<task1>
| Rowspan | Colspan | Cell content |
| ------- | ------- | ------------ |
|    1    |    2    |   Revenue    |
|    2    |    1    |   Region A   |
</task1>

2. In <task2>, conditioned on the above textual context, the model decodes the HTML table:

<task2>
<table>
  <tr><th colspan="2">Revenue</th></tr>
  <tr><td rowspan="2">Region A</td><td>Q1: $10M</td></tr>
  <tr><td>Q2: $12M</td></tr>
</table>
</task2>

3. During RL exploration, the reward module verifies that the extracted spanning cells in <task2> match <task1> ($r_{align} = 1.0$) and ground truth ($r_{f1} = 1.0, r_{teds} = 1.0$), maximizing policy advantages.

Loss & Training¶

The framework executes in two stages: 1. Full-Parameter 3-Task Joint SFT: Initialized from Qwen2.5-VL-3B, with maximum image token length of 2048 and context length of 10,800. Trained on 2 nodes (16$\times$ A100 GPUs) with effective batch size 16 (batch size 8, 2 gradient accumulation steps). 2. Reinforced Fine-Tuning (M-CRL): Implemented using the MS-Swift library on 7K unique table images expanded to 21K multi-task samples. 8 rollout trajectories are sampled per instance ($T=1.0, \text{top\_k}=50, \text{top\_p}=0.9$), maximum sequence length 6144 tokens, and KL penalty weight $\beta = 0.04$.

Key Experimental Results¶

Main Results¶

On 7 established benchmarks, the 3B Table-MCR2TR outperforms specialized small models, open-source general MLLMs, document-specific models, and proprietary commercial models in both merged-cell F1 score and overall TEDS.

Model	Type / Scale	SciTSR (F1/TEDS)	PubTabNet (F1/TEDS)	FinTabNet (F1/TEDS)	SynthTabNet (F1/TEDS)	CC-OCR Scan (F1/TEDS)	CC-OCR Photo (F1/TEDS)	Average AVG (F1/TEDS)
RapidTable	Specialized Small	88.3 / 87.7	81.5 / 86.8	67.0 / 77.1	16.6 / 74.2	56.3 / 66.4	37.3 / 39.0	60.2 / 72.2
PP-StructureV3	Specialized Small	87.5 / 82.7	68.9 / 73.1	42.9 / 58.6	13.1 / 59.3	49.7 / 65.0	35.7 / 33.6	54.3 / 63.7
Qwen2.5-VL-32B	General Open MLLM	80.2 / 92.9	51.2 / 78.6	49.4 / 73.6	0.5 / 61.7	53.4 / 84.8	53.1 / 79.4	51.8 / 78.8
InternVL3.5-38B	General Open MLLM	88.2 / 94.2	62.7 / 87.5	71.8 / 92.8	6.4 / 81.0	56.7 / 84.8	43.6 / 71.6	57.5 / 84.2
MinerU2-VLM	Specialized MLLM	89.5 / 93.0	68.7 / 90.3	60.7 / 84.4	73.1 / 94.4	73.4 / 75.1	54.3 / 46.2	70.7 / 80.9
PaddleOCR-VL	Specialized MLLM	90.6 / 92.7	68.1 / 87.1	59.1 / 83.5	28.1 / 79.7	62.9 / 81.3	59.2 / 74.7	65.2 / 83.6
GPT-4o	Commercial Proprietary	81.9 / 88.9	59.3 / 73.2	56.8 / 79.8	2.8 / 50.1	57.7 / 65.5	47.3 / 57.6	54.0 / 70.8
Gemini-2.5-Pro	Commercial Proprietary	91.4 / 92.9	72.9 / 90.9	79.7 / 92.8	8.7 / 54.4	69.6 / 89.4	54.6 / 82.2	66.3 / 84.2
Table-MCR2TR	Ours (3B)	95.5 / 97.7	80.6 / 93.7	95.7 / 97.0	87.6 / 98.7	76.6 / 90.5	71.2 / 83.0	83.9 / 92.3

Ablation Study¶

Ablation of reward components within the MCR2TR alignment task on the challenging CC-OCR benchmark:

Config #	Optimization & Reward Setup (Align / TEDS / F1)	Scan (F1 / TEDS)	Photo (F1 / TEDS)	AVG (F1 / TEDS)	Note
1	3-task Joint SFT baseline (no RL)	73.3 / 88.6	62.5 / 80.9	67.9 / 84.8	Lacks sequence-level exploration
2	RL with $r_{align} + r_{teds}$	74.9 / 89.7	66.4 / 82.2	70.7 / 86.0	Lacks discrete F1 advantage signal
3	RL with $r_{align} + r_{f1}$	75.5 / 89.5	69.4 / 81.9	72.5 / 85.7	Lacks global tree structural penalty
4	RL with $r_{teds} + r_{f1}$ (w/o $r_{align}$)	76.0 / 90.1	70.4 / 82.6	73.2 / 86.4	Degraded cross-task prior coupling
5	Full Collaborative Tripartite Reward ($r_{align} + r_{teds} + r_{f1}$)	76.6 / 90.5	71.2 / 83.0	73.9 / 86.8	Optimal synergy across local & global metrics

Key Findings¶

SynthTabNet reveals catastrophic blind spots in frontier models: On SynthTabNet where merged cells are pervasive, InternVL3.5-38B achieves only 6.4% MCR F1 despite an 81.0% TEDS score, and Gemini-2.5-Pro achieves merely 8.7% F1. Table-MCR2TR achieves 87.6% F1, proving that standard SFT completely neglects dense spanning tags.
Discrete F1 reward is essential for RL optimization: Moving from Config #2 to #5 increases MCR F1 by 3.2% (70.7% $\to$ 73.9%). Continuous TEDS suffers from low reward variance across rollout paths, whereas discrete ANLS-based F1 yields sharp discriminative gradients.
Accurate MCR directly bounds downstream QA accuracy: Supplying ground-truth MCR inputs further boosts TEDS by +3.4% (Scan) and +4.6% (Photo). In downstream QA across WTQ (77.6%), TabFact (89.8%), and PubHealthTab (75.8%), the two-stage pipeline (Table-MCR2TR $\to$ Qwen3-32B) substantially surpasses direct multimodal QA with GPT-4o (64.5% / 79.0% / 65.5%).

Highlights & Insights¶

MCR as an explicit visual scratchpad: Rather than discarding intermediate structure detection as an auxiliary training-only branch, generating explicit spanning triplets upfront provides contextual conditioning that simplifies long HTML decoding.
De-biasing the illusion of high TEDS: The paper exposes how global tree edit distance masks catastrophic cell-merging errors, establishing MCR F1 as a rigorous standard for fine-grained document evaluation.
Multi-reward synergy in multimodal RL: Harmonizing continuous global structure (TEDS), discrete localized accuracy (F1), and inter-stage consistency (Align) prevents policy collapse into reward-hacking shortcuts.

Limitations & Future Work¶

Loss of general conversational capability: Intensive task-specific fine-tuning on massive table datasets causes catastrophic degradation in following general instruction prompts, specializing the model strictly into a table parser.
OCR error sensitivity in MCR matching: MCR F1 calculation enforces ANLS text matching ($\tau_{sim} = 0.5$); blurred images with minor typographical OCR errors can lead to rejected cell matches despite perfect coordinate boundaries.
Future Directions: Exploring parameter-efficient fine-tuning (PEFT/LoRA) to preserve base model general reasoning, and expanding the progressive alignment formulation to nested hierarchical structures like org-charts and nested mind-maps.

vs RapidTable / PP-StructureV3: Specialized small architectures fail on long sequences and distorted captures due to limited representational capacity; Table-MCR2TR leverages a 3B foundation model to gain a 10-20% margin on average TEDS.
vs Dolphin / MonkeyOCR / MinerU2-VLM: Document MLLMs rely on monolithic autoregression without explicit spanning constraints; the progressive MCR2TR paradigm mitigates structural hallucination.
vs Table2LaTeX-RL / Infinity-Parser: Earlier RL frameworks for document analysis optimized only single end-to-end rendering rewards; this work introduces the first multi-task collaborative reward linking intermediate spanning extraction to final output.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Decouples merged-cell attributes into explicit contextual prompts and introduces tripartite collaborative reinforcement learning.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across 7 diverse benchmarks, comparing against 15+ small, open, and proprietary baselines with thorough ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Convincing motivation, clear formulation, and rigorous empirical analysis of metric insensitivity.
Value: ⭐⭐⭐⭐⭐ Establishes a new state of the art in complex table parsing and directly eliminates cascading errors in downstream multimodal table reasoning.