ECCV2026 Multimodal VLM text image machine translation global-local dual perception multimodal large language model high-resolution text-rich images cross-region consistency

Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation¶

Conference: ECCV2026
Paper: ECCV
Area: Multimodal VLM
Keywords: text image machine translation; global-local dual perception; multimodal large language model; high-resolution text-rich images; cross-region consistency

TL;DR¶

GLoTran recasts high-resolution text-rich image translation as "detect text regions, cut them into local slices in reading order, then let an MLLM translate region by region under the contextual reference of a low-resolution global image," injecting global semantics into local tokens through early-layer cross-attention with source-type and spatial biases and keeping regions consistent via an \(\eta\)-window translation replay and a four-part structured prompt; it lifts average BLEU on MCiTon English-to-Chinese translation from 46.7 (Qwen3-VL 8B) to 50.2.

Background & Motivation¶

Text image machine translation (TIMT) has to translate text that "lives inside" an image: the words on a poster, menu, screenshot, road sign or document can neither be fed to a text-only MT system as a clean sequence nor be handled like image captioning, where the gist suffices. TIMT must jointly handle text localization, recognition and translation. The field has two dominant paradigms. Cascade methods run OCR region by region and hand the recognized strings to a neural MT model; they are modular and easy to swap components in and out, but a single recognition error propagates through the whole sentence, and the serial multi-model pipeline adds latency. End-to-end models jointly encode vision and language and map the image directly to a translation, which curbs error propagation but remains task-specific — a new layout, font or language direction means retraining — so they generalize poorly to the diversity of real-world scenarios.

Multimodal large language models (MLLMs) with strong language priors and visual understanding have recently been brought into TIMT, for example InImageTrans, which fine-tunes an MLLM with preference optimization to suppress hallucination, and MT3, which improves translation quality via multi-task reinforcement learning. Their training and evaluation, however, largely stay on low-resolution images with relatively simple visual content. On genuinely high-resolution text-rich images the problems surface quickly: MLLMs have limited input resolution and their visual encoders use a rigid square-patch design, so feeding the full image either incurs superlinear computation and an excessive number of visual tokens, or forces a downsample that smears the glyphs. Worse, the many non-textual elements in posters and documents — icons, decorations, cluttered backgrounds — disperse visual attention, so the model spends capacity on meaningless regions. The observed symptoms are local text omission, misrecognition, fragmented semantic understanding, and, from there, cross-region semantic drift and hallucination.

The core tension is that TIMT is not shaped like general visual question answering: VQA only needs to assemble enough evidence to answer one question, whereas TIMT demands that all text in the image be translated in the correct reading order and mutually consistent in meaning. Existing high-resolution VLM strategies follow the former logic. mPLUG-DocOwl2 and URaDeR partition a high-resolution image into fixed-size sub-images with shape-adaptive cropping; ZoomEye picks local patches by tree-based search and V* by guided visual search. All of them treat local patches as auxiliary evidence for answering a holistic question, and their random partitioning cuts sentences in half while their search-based selection misses small or low-contrast text instances. On top of that, they all process the image as a single "one image, one translation" unit, with no modeling of terminology consistency or coreference across regions. This paper takes the opposite angle: since text regions can be located fairly reliably by a detector, the model should not be asked to find the text on the full image at all. Translation is instead rewritten as an ordered, region-by-region progressive decoding, with one cheap global view always present to supply scene context (layout, theme, inter-region relations), and cross-region consistency enforced explicitly on the decoding side. Core idea: replace whole-image input with a dual visual input of "one low-resolution global image + ordered local text slices," let local tokens actively query the global view through early-layer cross-attention with source-type and spatial biases, then enforce cross-region consistency with an \(\eta\)-window translation replay and a four-part structured prompt, backed by the 510K-scale GLoD global-local paired dataset used for fine-tuning.

Method¶

Overall Architecture¶

GLoTran takes one source-language high-resolution text-rich image and outputs target-language translations arranged in reading order. The pipeline is "locate → group into slices → dual-path encoding → context injection → region-by-region decoding": a self-trained text region detector proposes block- or paragraph-level candidate boxes, which are cropped and merged into compact local slices using geometric and typographic cues, with a content ordering model fixing the reading order; the original image is simultaneously downsampled into a low-resolution global view; both views are encoded by a shared ViT encoder plus vision projector into visual tokens and fed to the LLM together with the ⟨image_g⟩ / ⟨image_l⟩ type-identifier tokens; hierarchical global-to-local cross-attention is inserted at early Transformer layers so every local token can selectively draw context from global tokens; finally the LLM generates the translation region by region in reading order, with the prompt for region \(i\) carrying the translations already produced for the previous \(\eta\) regions (the replay window). Notably, GLoTran performs no explicit OCR: the detector only answers "where is the text," while recognition and translation are both done by the MLLM directly on each slice; OCR appears only in the data construction stage.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["High-resolution text-rich image"] --> B["Text region detection and slice grouping<br/>block/paragraph slices in reading order"]
    A --> D["Global-local dual visual encoding<br/>downsampled global view + slices, shared ViT"]
    B --> D
    D --> E["Hierarchical cross-attention with source-type bias<br/>local tokens query global semantics"]
    E --> F["Region-by-region progressive decoding with replay<br/>structured prompt + η-window translations"]
    F --> G["Per-region translations in reading order"]

The data backing this dual-perception paradigm is a contribution in its own right: the authors build GLoD, a TIMT dataset designed for global-local dual input, covering more than 40 real-world scenes (documents, posters, menus, road signs, receipts, handwriting, etc.) in 5 languages. Curation has five stages: scene conceptualization defines 40+ translation scenarios; large-scale web crawling plus existing text-image datasets are filtered strictly to remove privacy-, confidentiality- or copyright-sensitive content as well as low-resolution, blurred, distorted, low-OCR-confidence (PaddleOCR), visually uniform and text-sparse images; two complementary detectors, PaddleOCR and Qwen3-VL-Plus, cross-verify and fuse text regions, which are then hierarchically grouped into semantically coherent blocks (phrases, sentences, paragraphs) by reading order, spatial adjacency and visual heuristics; recognition combines dual OCR with bidirectional translation fusion (PaddleOCR captures fine-grained local content while Qwen3-VL-Plus supplies global context for cross-line disambiguation), and the recognized text is translated by GPT-4o and DeepSeek-R1 through multi-round mutual back-translation and fusion, with human experts verifying semantics, grammar and layout correspondence, yielding roughly 2M global-local image-text pairs; a two-stage quality control then removes low-confidence OCR segments, ultra-short translations and non-linguistic symbols, and quantifies semantic faithfulness between global and local translations with multilingual embedding similarity (LaBSE / SimCSE) plus round-trip translation consistency, discarding regions that drift or mismatch their global reference, converging to about 510K pairs (the paper reports 510K in one place and 517,354 in another — see Limitations).

Key Designs¶

1. Text region detection and slice grouping: make the partitioning text-centered and order-preserving

Existing high-resolution VLMs split a large image into sub-images by random partitioning or search-based selection. That is good enough for VQA but disastrous for TIMT: a sentence cut in two, or an entire block of small text missed, becomes an omission in the translation. GLoTran inverts the order and detects first: a self-trained text region detector proposes candidate boxes \(\mathcal{B}=\{b_i\}\) on image \(I\), each corresponding to a block- or paragraph-level region, i.e. it groups the sentences belonging to the same coherent text segment rather than cutting line by line or word by word. After cropping and normalization, adjacent or semantically related regions are merged according to geometric and typographic cues such as overlap, line height and alignment, giving \(N_s \ll N_b\) compact grouped slices. This step is essential: it turns "fragmented detector boxes" into "text units that can be translated independently." The merged slices are then deterministically ordered by a content ordering model (e.g. MinerU) so the output order matches human reading order. The price of "detect first, then group" rather than "split first, then recognize" is dependence on detector quality (see Limitations); what it buys is slices that align with semantic text units and, more importantly, a finite, ordered, individually checkable translation list.

2. Global-local dual visual encoding: an extremely low-resolution full image for scene context, slices for glyph detail

Constrained by fixed input resolution and square patches, an MLLM fed a full high-resolution image hits two problems at once: the visual token count grows superlinearly with resolution, and background, icons and decorations crowd out attention, diluting the characters that actually need reading. GLoTran therefore splits the visual input into two paths. One downsamples the original image into a low-resolution global view \(I_g\) that only carries coarse priors — layout structure, scene theme, inter-region relations. The other keeps the local slices \(I_i\) at their original crop resolution for fine-grained glyph detail. The two paths share one ViT encoder and one vision projector that align visual features into the text embedding space, and identifier tokens tell the LLM which tokens belong to which path:

\[\hat{Y}_i = \text{LLM}\big([E_g; V_g; E_l; V_i; \mathcal{P}_{<i}]\big)\]

where \(E_g / E_l\) are the text embeddings of ⟨image_g⟩ and ⟨image_l⟩, \(V_g, V_i\) are the global and local visual tokens, and \(\mathcal{P}_{<i}\) is the preceding prompt and translations (⚠️ rewritten from the corrupted PDF extraction; refer to the original paper). The experiments justify the design: the global view is 224×224 by default and raising its resolution to 448/896/1792 actually hurts scenes with small, scattered text such as Introduction and Leaflet, because a larger global view brings more visual clutter rather than more usable glyph information. What the global path needs is "knowing what kind of scene and layout this is," not "reading the characters." That also makes it the cheap path: GLoTran needs only 4.9K visual tokens at 224×224, whereas sibling models process 164K tokens of full-resolution input to reach comparable accuracy.

3. Hierarchical cross-attention with source-type bias: let local tokens fetch context from the global view at early layers only

Simply concatenating the two token streams does not guarantee that local tokens actually use global information; the model may well treat them as two unrelated visual contexts and decode in silence. GLoTran therefore explicitly inserts global-to-local cross-attention at early Transformer layers \([0, 8, 16, 24]\), letting each local token act as a query over global tokens, and adds a learnable scalar bias to the attention weight, conditioned jointly on source type (global vs. local) and token spatial proximity:

\[\text{Att}(q_i,k_j)=\text{softmax}\!\left(\frac{q_i k_j^{\top}}{\sqrt{d_t}} + b_{ij}\right)\]

The practical effect is disambiguation plus focus. Whether "House" on a menu should become a restaurant or a legislative chamber depends on the scene theme carried by the global image, while irrelevant decorative regions should be ignored; \(b_{ij}\) is precisely the knob that learns which kind of global token to look at and which spatial positions to skip (⚠️ formula rewritten from the corrupted extraction; the exact parameterization of \(b_{ij}\) should be checked against the original). Restricting the injection to layers 0/8/16/24 rather than mixing throughout is deliberate: after global context has been written into the local representation early, later layers stay free to model the internal structure of the local text. In the ablation, removing the global image drops BLEU on the short-text Title and Sign scenes from 54.3/46.3 to 41.0/37.8, confirming that this context injection pays off most where there are few characters and high ambiguity.

4. Region-by-region progressive decoding with translation replay: put cross-region consistency into the decoding process

Translating an image region by region naturally loses the discourse relations between regions: a recurring column name on the same poster may be rendered differently each time, and pronouns or ellipses may be resolved wrongly because the model cannot see the preceding text. GLoTran makes translation progressive along the reading order: the decoding condition for region \(i\) is that region's slice, the global image, and the prefix prompt \(\mathcal{P}_{<i}\), which explicitly carries the translations of the previous \(\eta\) regions (pre-region translation replay). The replayed translations supply target-language-side context — terminology uniformity, sentence continuation, coreference grounding — so the mechanism is not "showing the source again" but keeping the model consistent inside the translation it has already produced. During training, teacher forcing is applied to the replayed history, replacing the model's own predictions with ground truth so that early mistakes do not snowball across regions (⚠️ the paper states that \(\hat{Y}_{<i}\) is replaced by \(Y_{<i}\) within the replay window; the exact boundary should be checked against the original). The companion piece is a four-part structured prompt: a Global Understanding Instruction makes the model first extract layout and scene semantics from the global image; a Local Focus Instruction pins attention on the textual content of the current slice; a Global-Local Consistency Rule asks the model to verify, before generating, that the local translation agrees with the global context; and a Translation Instruction states the task explicitly and includes the previous region's translation as a reference — for example, when translating the second slice the prompt already contains 「斯克里布纳五月刊」 from the first. The window size \(\eta\) is the one knob that needs tuning: BLEU peaks at 43.54 when \(\eta=4\); a smaller window lacks target-language context, a larger one accumulates noise and drift while adding overhead.

A Worked Example¶

Take an English magazine cover or poster at 1573×2000 — far beyond the comfortable input range of an MLLM. Feeding the whole image to Qwen3-VL 8B, one sees it miss the small footer text, render the masthead and column names inconsistently across the page, and even invent content from language priors. Through GLoTran: the self-trained detector first proposes roughly 12 block-level text regions (masthead, date line, column titles, a lead paragraph, price tag, …); the grouping stage merges three or four overlapping boxes with matching line heights into two paragraph-level slices, leaving, say, six slices sorted in reading order; in parallel the original image is downsampled to a 224×224 global view. After encoding, progressive decoding begins. Translating slice 1, the masthead 「斯克里布纳五月刊」, the prompt has no history and the model relies only on the global view telling it "this is the cover of a periodical." Translating slice 2, the price line 「每份25美分」, the prompt already contains slice 1's translation, so the model keeps the masthead and column naming consistent. By slices 5 and 6 the replay window holds the previous four translations, which resolve cross-region coreference. When all six slices are done, they are concatenated in reading order into the full translation, and each region keeps its correspondence to the image so a human can check for omissions — precisely the individually checkable translation list that neither cascade pipelines nor whole-image MLLMs provide.

Loss & Training¶

The training objective is cross-entropy over the target-language tokens of each region, summed over regions and over the dataset:

\[\theta^{*}=\arg\min_{\theta}\sum_{(I,Y)\in\mathcal{D}}\sum_{i=1}^{N_s}-\log p\big(\hat{Y}_i \mid I_g, I_i, \mathcal{P}_{<i};\theta\big)\]

(⚠️ rewritten from the corrupted extraction; the symbolic form should be checked against the original.) In practice existing MLLMs are fully fine-tuned: backbones are InternVL2.5 4B and Qwen3-VL 8B, GLoD is split 99:1 into 512,180 training pairs and 5,174 validation pairs (no image overlap with MCiTon or MTIT6), all inputs are resized to 224×224, batch size is 128 with gradient accumulation over 8 steps, AdamW with an initial learning rate of \(1.0\times10^{-5}\) for 10,000 iterations on 8 NVIDIA A100 (80GB) GPUs. The authors also train a GLoTran-LoRA variant for comparison, which performs clearly worse than the fully fine-tuned version — evidence that a method which restructures the visual input needs to move the visual pathway itself, and that low-rank adaptation alone is not enough.

Key Experimental Results¶

Main Results¶

English-to-Chinese translation is evaluated on MCiTon (1,450 image-text pairs spanning documents, natural scenes and posters, with eight sub-scenarios: Paper, News, Novel, Title, Sign, Introduction, Cover, Leaflet) using BLEU / COMET. The table gives representative baselines and GLoTran per scenario (the full set of 17 baselines is in the original Table 2):

Method	Size	Paper	News	Novel	Sign	Introduction	Leaflet	Avg
GPT-4o	~200B	62.1 / 86.3	51.6 / 85.6	44.2 / 83.4	46.1 / 83.3	40.1 / 75.3	43.9 / 71.7	46.8 / 79.9
Qwen-VL-Max	~72B	60.5 / 85.7	58.3 / 85.7	44.8 / 83.5	43.2 / 80.5	37.8 / 74.6	43.7 / 69.9	46.5 / 78.6
Qwen2.5-VL-Instruct	7B	60.7 / 85.8	53.4 / 85.5	27.4 / 78.1	44.0 / 82.2	36.2 / 72.8	43.6 / 72.4	42.9 / 77.7
InternVL3	8B	59.8 / 86.0	48.9 / 83.9	31.2 / 79.8	41.8 / 81.6	32.8 / 72.6	35.8 / 70.9	41.4 / 77.9
InImageTrans	8B	59.0 / 85.0	46.5 / 80.9	33.5 / 76.5	28.9 / 66.9	32.0 / 71.7	32.1 / 70.8	35.9 / 72.7
Qwen3-VL-Instruct	8B	60.9 / 86.1	54.3 / 86.2	43.6 / 83.1	44.2 / 83.0	33.6 / 75.8	48.2 / 75.1	46.7 / 80.2
Qwen3-VL-Instruct⋆ (GLoD SFT only)	8B	60.8 / 86.2	56.0 / 86.4	43.8 / 83.9	44.7 / 83.8	35.5 / 77.4	48.5 / 75.9	46.5 / 80.8
GLoTran-LoRA (Qwen3-VL 8B) ⚠️	8B	64.8 / 85.9	56.0 / 86.7	45.5 / 84.8	45.0 / 82.7	40.0 / 74.5	49.5 / 75.8	42.2 / 78.6
GLoTran (Qwen3-VL 8B)	8B	66.3 / 87.9	57.5 / 88.6	46.8 / 86.6	46.3 / 84.8	41.3 / 76.4	51.1 / 77.9	50.2 / 82.2

(⚠️ The GLoTran-LoRA row is numerically identical to the "w/o Replay Mechanism" row in the ablation table, and its reported average of 42.2/78.6 disagrees with the ≈48.8 obtained from its own eight scenario scores — apparently a typesetting error in the original; refer to the paper.) Multilingual results are given as bar charts on MTIT6 (jp→zh / ko→zh / zh→en / zh→jp / zh→ko, about 200 high-resolution images per direction) in Fig. 5(a)(b): GLoTran is best on every direction, ahead of Qwen3-VL 8B and InternVL3 8B.

Ablation Study¶

Component-level ablation on MCiTon with the Qwen3-VL 8B backbone (BLEU / COMET):

Config	Paper	Novel	Sign	Introduction	Leaflet	Note
Qwen3-VL 8B (baseline)	60.9 / 86.1	43.6 / 83.1	44.2 / 83.0	33.6 / 75.8	48.2 / 75.1	translate the whole image
Qwen3-VL♢ (fed detected regions)	60.6 / 85.8	42.5 / 82.5	37.4 / 76.4	38.2 / 75.1	45.4 / 73.9	accurate localization, no dual path / replay
GLoTran (PSC)	57.1 / 83.4	36.8 / 78.4	36.2 / 75.3	33.2 / 73.3	43.8 / 72.7	parameter-free shape-adaptive cropping replaces the detector
GLoTran (w/o Global Image)	60.7 / 85.9	44.1 / 82.9	37.8 / 77.4	33.1 / 74.6	44.9 / 73.1	local slices only
GLoTran (w/o Replay Mechanism)	64.8 / 85.9	45.5 / 84.8	45.0 / 82.7	40.0 / 74.5	49.5 / 75.8	no cross-region translation replay
GLoTran	66.3 / 87.9	46.8 / 86.6	46.3 / 84.8	41.3 / 76.4	51.1 / 77.9	full model

Backbone generalization, reported on MTIT6 directions and selected MCiTon scenarios (baseline → GLoTran; the relative gain in parentheses is the one reported in the paper):

Backbone	zh→en	ko→zh	jp→zh	Novel	Introduction	Leaflet
InternVL2.5 4B	4.02 → 6.68 (+66.2%)	2.57 → 5.97 (+132.3%)	28.37 → 29.93 (+5.5%)	18.48 → 25.80 (+39.6%)	26.16 → 29.47 (+12.7%)	21.22 → 23.27 (+9.7%)
Qwen2.5-VL 7B	9.54 → 10.99 (+15.2%)	20.23 → 22.39 (+10.7%)	35.83 → 43.81 (+22.3%)	27.43 → 44.59 (+62.6%)	36.18 → 38.93 (+7.6%)	43.66 → 45.64 (+4.5%)
Qwen3-VL 8B	11.27 → 11.87 (+5.3%)	17.29 → 20.69 (+19.7%)	37.71 → 43.54 (+15.5%)	43.56 → 46.84 (+7.5%)	33.63 → 41.32 (+22.9%)	48.21 → 51.18 (+6.2%)

Key Findings¶

Removing the global image hurts most, especially on short-text, high-ambiguity scenes: Sign drops from 46.3 to 37.8 and Title from 54.3 to 41.0, which says the value of the global view is not "seeing characters better" but giving isolated short texts a scene and topic to belong to. By contrast, removing only the replay mechanism costs just 1.5/1.3 BLEU on Paper/Novel. The comparison suggests the global context is the main term of dual perception while replay is an additive gain (though it still contributes a steady 1.3–1.6 BLEU on multi-paragraph small-text scenes such as Introduction and Leaflet).
Accurate localization alone is not enough: feeding GLoTran's detector output directly to InternVL3 / Qwen2.5-VL / Qwen3-VL (the ♢ rows) mostly does not help and even hurts — Sign falls from 44.2 to 37.4 — because region-level input without global context and cross-region consistency constraints amplifies the model's misjudgment of isolated text blocks. Replacing the detector with parameter-free random cropping (PSC) degrades GLoTran across the board (Paper 57.1), showing that text-centered slice grouping is a necessary condition of the method rather than a replaceable detail.
Parameter scale does not correlate with TIMT performance: InternVL2 8B and 40B differ by only 2.2 BLEU / 1.8 COMET, and Qwen2.5-VL 32B scores lower BLEU than its 7B variant (42.2 vs. 42.9). The authors read this as a possible scaling saturation on TIMT: stacking parameters does not fix the attention dilution caused by high-resolution text-rich images.
Plain SFT on GLoD alone brings little: Qwen3-VL⋆ (8B, same data, ordinary supervised fine-tuning) averages 46.5/80.8, essentially level with the untuned 46.7/80.2 and worse on some complex scenes. GLoTran gains 3.5 BLEU (7.49% relative) and 2.0 COMET (2.49% relative) over the same untuned backbone. The data is not the bottleneck; restructuring the visual input into global-local dual perception and coupling it with replay decoding is.
Both hyper-parameters have clear working ranges: global image resolutions of 224 and 448 are best, and beyond 448 Introduction/Leaflet fall noticeably; the replay window peaks at BLEU 43.54 with \(\eta=4\), with \(\eta\in[3,5]\) also steadiest on COMET — larger windows add accumulated noise and translation drift while linearly increasing decoding cost.
Cheaper in tokens, slower in latency: GLoTran averages 8.4K visual tokens at 448×448 (4.9K at 224×224, about 18GB of memory), whereas sibling models need full-resolution processing for comparable accuracy — 164K visual tokens and roughly 215× the FLOPs on Novel. But GLoTran's first-token latency is much higher (3.002s on Novel vs. 0.099s for Qwen3-VL♡), because decoding is serial across regions: the method trades "one expensive high-resolution encoding" for "cheap encoding plus many region decodings," buying accuracy with throughput. It is not suitable for low-latency settings in its current form.

Highlights & Insights¶

Reformulating TIMT from "one image, one translation" to "streaming translation over ordered regions" is the most valuable part of the paper. Once the translation unit is an ordered list of text regions, cross-region consistency stops being mystical and becomes explicitly modelable (replay window, consistency rule, a checkable per-region translation list), and omission turns from "something feels missing" into "which slice was not translated" — a locatable defect.
A 224px global image suffices for context — counterintuitive but coherent: the MLLM community habitually scales resolution to preserve detail, while this paper shows the other path can be extremely cheap. The global path needs layout and scene theme, not glyphs; high resolution is reserved for the slices that actually need it, cutting visual tokens from 164K to 4.9K.
Cross-attention with source-type and spatial bias is a reusable small knob: marking provenance with ⟨image_g⟩/⟨image_l⟩ and letting a learnable scalar bias let local tokens read global tokens selectively transfers unchanged to any MLLM scenario with multi-scale, multi-image, or "main image plus evidence image" inputs — multi-page documents, video keyframes with thumbnails, remote-sensing overview plus sub-regions.
The data recipe is worth reusing: 2M raw pairs converge to about 510K high-quality pairs through dual-OCR fusion, bidirectional back-translation, multilingual-embedding consistency filtering and human spot checks. This "scale up first, then converge strictly" pattern applies to any task needing region-level translation annotation (document translation, subtitles, product images).
The replay window idea moves directly to long-document and multi-page translation: generalizing \(\eta\) from "the previous \(\eta\) regions" to "the previous \(\eta\) segments or pages" buys terminology consistency without external memory, at the cost of longer prompts.

Limitations & Future Work¶

Dependence on the detector is understated: slice boundaries come entirely from a self-trained detector whose details sit in the appendix. Once handwriting, vertical text, artistic fonts or very low-contrast text are missed, they can never be translated (the collapse of PSC shows slice quality is a hard gate). A natural fix is to jointly optimize detection and translation end to end, or at least give the MLLM a gap-filling channel that re-checks the global image for uncovered text regions.
Latency is much higher than whole-image approaches: 3.0s first-token latency on Novel vs. 0.099s, FPS 0.035 vs. 0.041, with serial per-region decoding as the bottleneck. Limited parallelism by semantic group (parallel within a paragraph, ordered across paragraphs), an adaptive \(\eta\), or draft-and-verify decoding for small regions are plausible remedies.
The metrics do not match the headline claim of "completeness": the paper repeatedly emphasizes omission as the core pain point, yet reports only BLEU / COMET, neither of which penalizes a missing sentence nearly enough, and gives no omission rate or region recall. A region-level coverage / omission metric would be the honest addition.
Statistics and tables are not self-consistent: GLoD's size is 510K pairs in the abstract, "510K images and 517,354 image-text pairs" in the body, and 517,354 (512,180 + 5,174) in Table 1 and the training split — these need unifying; the GLoTran-LoRA row disagrees with its own average (see the note under the main table) and the LoRA rank and target modules are unspecified, so the fairness of that comparison cannot be judged.
Language directions and scenarios remain narrow: English-to-Chinese dominates, MTIT6 has only about 200 images per direction, and performance on non-Latin scripts (Arabic, Thai, …) or very long documents is not reported; the 5 languages and 40+ scenes of GLoD are not fully exercised on the evaluation side.
Not yet combined with preference-based optimization: InImageTrans has shown that DPO suppresses hallucination, while GLoTran only does full-parameter SFT. Turning the consistency signal inside the replay window into preference pairs (consistent vs. drifted translations) for DPO/RL is a promising direction for pushing hallucination down further.

vs cascade methods (PaddleOCR + NMT, Nougat-style document processing, manga translation pipelines): cascades decouple recognition from translation, making components swappable and interpretable, but errors accumulate stage by stage and the serial multi-model pipeline is slow; GLoTran lets the MLLM emit translations directly from slices so recognition and translation share one context, at the cost of losing the debuggability that an OCR intermediate would provide.
vs traditional end-to-end TIMT (Translatotron-V, LayoutDIT, etc.): they jointly encode vision and language and map straight to text — efficient but task-specific, needing retraining for a new layout; GLoTran fine-tunes an off-the-shelf MLLM and generalizes over 5 languages and 40+ scenes, paying for it with training cost (8×A100, 510K training pairs).
vs InImageTrans: also an MLLM-based TIMT method evaluated on MCiTon, but it takes the multi-condition DPO route to suppress hallucination; GLoTran leads clearly on average (50.2/82.2 vs. 35.9/72.7). The two are orthogonal, and stacking DPO on top of the dual-perception framework is a natural follow-up.
vs MT3 (multi-task RL) and M4Doc (document image translation): MT3 changes the training objective and M4Doc narrows to the document domain; GLoTran changes how the visual input is organized and how decoding proceeds, so it composes with the former and divides labor with the latter (general text-rich scenes vs. documents).
vs general high-resolution VLM strategies (shape-adaptive cropping in mPLUG-DocOwl2 / URaDeR, tree search in ZoomEye, guided visual search in V*): they treat local patches as auxiliary evidence for a holistic question, with random or search-driven partitioning that truncates text and misses small instances. GLoTran's position is that TIMT requires "everything, in order, semantically consistent," which forces text-region-centered extraction and order-preserving grouping, with cross-region consistency pushed into decoding — a methodological difference that follows from the task's shape, not a mere engineering detail.

Rating¶

Novelty: ⭐⭐⭐⭐ Reformulating TIMT as dual-path progressive translation with a low-resolution global view, ordered local slices and translation replay is more valuable than any single module; the attention bias and the dual-OCR data pipeline are more engineering than novelty.
Experimental Thoroughness: ⭐⭐⭐⭐ Eight English-to-Chinese scenarios, five multilingual directions, three backbones, hyper-parameter and efficiency analyses, plus the "feed detected regions to an off-the-shelf MLLM" control; but only BLEU/COMET, no omission-rate metric matching the headline claim, and a few table numbers are inconsistent.
Writing Quality: ⭐⭐⭐ The method narrative and figures are clear, but the PDF formulas are corrupted, the GLoTran-LoRA row disagrees with its average, and the dataset size is reported inconsistently across the paper.
Value: ⭐⭐⭐⭐ It gives a token-efficient, reproducible and easily transferable answer to "how should high-resolution text-rich images be fed to an MLLM," and the accompanying GLoD dataset and curation recipe are directly reusable for future TIMT work.