Large-scale Multilingual News Image Captioning with LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/tharindu/LaMuN
Area: Multimodal VLM
Keywords: Multilingual News Image Captioning, Vision-Language Large Models, Chain-of-Thought Fine-tuning, Entity Grounding, Multimodal Benchmark
TL;DR¶
To break the monolingual English bottleneck in news image captioning, this work presents LaMuNโthe first large-scale benchmark comprising over 800,000 news image-article pairs in 59 languagesโand proposes Spatial-CoT fine-tuning, which decomposes image captioning into spatial segmentation, event retrieval, entity mapping, and synthesis to achieve state-of-the-art multimodal entity grounding.
Background & Motivation¶
News image captioning is a distinctive and challenging branch of multimodal vision-language generation, tasked with generating context-dependent, entity-rich captions by integrating the visual content of an image with its accompanying news article. In stark contrast to generic image captioning, which merely describes visible common objects in broad categorical terms (such as "a crowd of people standing around" or "a woman holding a trophy"), news captions are inherently tied to journalistic narratives. They must accurately pinpoint specific individuals by name, organizations, geographical locations, and ongoing newsworthy events (for instance, identifying Chelsea footballer Maren Mjelde lifting the Women's Super League trophy). Such information cannot be retrieved from visual pixels alone and must be faithfully grounded across modalities. This capability is of paramount importance for assistive technologiesโenabling screen readers to convey factual information to visually impaired usersโas well as for content-based indexing and retrieval across extensive digital media archives.
Despite its practical significance, the current research ecosystem for news image captioning remains almost entirely restricted to English. Well-established benchmarks, such as Visual News, GoodNews, and NYTimes800k, are exclusively monolingual. While MUNICHUS recently took an initial step toward multilingual captioning, it remains constrained to nine languages and a single news provider. More critically, when contemporary open-weight vision-language large models (VLLMs) are deployed with basic zero-shot or few-shot prompts on long-context news, they struggle with severe hallucination and omission of named entities, often collapsing into vague generic descriptions. Without dedicated mechanisms to bridge visual regions with specific article contexts, scaling models to diverse languages with limited supervision remains a major open problem.
The opportunity to overcome these limitations emerges from two key developments: the accumulation of vast multilingual archival reporting by major international public broadcasters, and recent advancements in modular vision-language toolsโincluding dense multilingual text retrieval models and zero-shot spatial segmenters. Rather than treating multilingual news captioning as an unconstrained end-to-end black-box generation task, complex multimodal grounding can be decomposed into an interpretable chain of reasoning. Core idea: We establish LaMuN, a comprehensive multilingual benchmark spanning 59 languages and 800k+ images, and propose Spatial-CoTโa decoupled reasoning framework combining SAM3 spatial segmentation, BGE-M3 event clustering, and language-specific NERโwhich dramatically elevates entity grounding when adapted via supervised LoRA fine-tuning.
Method¶
Overall Architecture¶
The proposed methodology addresses the cross-modal grounding challenge by decomposing news caption generation into four modular reasoning stages: visual scene analysis, dense article event retrieval, named entity extraction, and multimodal context synthesis. Instead of prompting an LLM with the entire unstructured article text and an unparsed image, the framework systematically constructs a structured prompt that equips the model with high-purity event context and precise entity-to-region associations.
The authors formulate two primary variants depending on the granularity of visual analysis: Holistic-CoT, which utilizes a holistic natural language image caption as a text query, and Spatial-CoT, which incorporates instance-level foreground-background decomposition via SAM3 alongside heuristic entity assignments. The complete pipeline and its adaptation via parameter-efficient fine-tuning are illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: News Image and Accompanying Article"] --> D1["Global Event Retrieval & Entity Extraction<br/>Florence-2 proxy query + BGE-M3 dense embeddings + HDBSCAN event clustering + Language-specific NER"]
In --> D2["Spatial Decomposition & Region-Entity Grounding<br/>SAM3 foreground/background segmentation + Semantic concept labelling + Type-based heuristic entity mapping"]
D1 --> D3["Structured Chain-of-Thought Adaptation<br/>LoRA parameter-efficient fine-tuning + Multi-step reasoning trace supervision"]
D2 --> D3
D3 --> Out["Output: Entity-grounded Journalistic Caption"]
Key Designs¶
1. Global Event Retrieval & Entity Extraction: Long-article contextual distraction and modal divergence -> Florence-2 proxy query with BGE-M3 and HDBSCAN event clustering -> Extracts compact, highly relevant event-centric factual context
Feeding an entire news article directly into a vision-language model inevitably disperses cross-modal attention over hundreds of irrelevant sentences. To solve this, the framework first leverages Florence-2 to produce a holistic natural language description of the image, functioning as a linguistic proxy for the visual content. Next, the multilingual dense retrieval model BGE-M3 encodes all candidate sentences from the article into continuous vectors. Rather than employing fixed-\(k\) clustering like K-Means, the method utilizes HDBSCAN (hierarchical density-based clustering) to automatically infer the natural number of sub-events discussed in the article while discarding peripheral outlier sentences. The sentence cluster whose centroid exhibits the highest cosine similarity to the Florence-2 query embedding is selected as the representative event context. Finally, a dedicated, high-precision language-specific NER model parses this targeted passage to extract a structured entity pool spanning persons (PER), organizations (ORG), locations (LOC), dates (DATE), and events (EVENT), effectively filtering out journalistic noise before generation.
2. Spatial Decomposition & Region-Entity Grounding: Global visual representations omit spatial layout and blur multiple person entities -> SAM3 fine-grained segmentation with heuristic entity assignment -> Establishes structured, foreground-to-background spatial grounding
Although a holistic visual description identifies high-level context, it lacks geometric layout information, leading to entity confusion when multiple individuals or landmarks appear in a single frame. Spatial-CoT eliminates this ambiguity by employing SAM3 (Segment Anything Model 3) to segment the input image into discrete foreground subjects and background areas, assigning each region a semantic concept tag (e.g., person or building). A type-based heuristic mapping is then applied: extracted person entities are routed to person regions, while location and organization entities are mapped to building or background regions. The multimodal model is subsequently provided with the original image, cropped visual region patches, retrieved event context, and the explicit entity-region assignment manifest. The structured prompt instructs the model to describe the scene progressively from foreground to background, strictly referencing the mapped entity names verbatim. This provides an effective inductive bias that anchors names to their exact visual counterparts without requiring costly manual bounding box annotations.
3. Structured Chain-of-Thought Adaptation: Small open-weight models collapse under zero-shot complex multi-step reasoning -> Full reasoning trace alignment via LoRA fine-tuning -> Internalizes step-by-step grounding and activates low-resource entity recall
When multi-step CoT prompts are presented to instruction-tuned models in an out-of-the-box zero-shot setting, performance paradoxically degrades across all automated metrics compared to simple direct prompting. Smaller open-weight models (7B-8B parameters) struggle to follow intricate multi-part prompting formats without explicit training. To resolve this, the authors implement supervised fine-tuning on the structured CoT templates. Utilizing LoRA across query, key, value, and output projection matrices (\(q, k, v, o\)) with rank \(r=16\), alpha \(\alpha=32\), and AdamW optimizer over 2 epochs on an NVIDIA H200 GPU, each target language is trained independently. Under this structured supervision, the model internalizes the step-by-step cognitive trajectoryโanalyzing visual regions, reviewing retrieved event context, grounding mapped entities, and formulating journalistic phrasingโthereby unlocking significant gains in both lexical generation quality and entity accuracy.
Benchmark & Evaluation Protocol¶
To benchmark multilingual multimodal performance, the paper introduces LaMuN (Large-scale Multilingual News Image Captioning Benchmark), aggregated from BBC News, Voice of America (VOA), and Deutsche Welle (DW) spanning from 2001 to December 2025. Following rigorous quality filtering (discarding images with dimensions below 180 pixels or captions shorter than four words), LaMuN comprises 778,556 training pairs and 57,880 test pairs across 59 distinct languages. The benchmark covers diverse linguistic families and resource tiers, ranging from high-resource (English, Arabic, Mandarin) to mid-resource (Hindi, Indonesian, Swahili) and low-resource (Bengali, Nepali, Sinhala).
The empirical evaluation investigates four modern open-weight VLLMs: Aya-Vision-8B, Gemma-4-E4B-it, Qwen2.5-VL-7B, and Qwen3-VL-8B across nine representative languages spanning all three resource tiers. Evaluation metrics include standard n-gram overlap BLEU-4, consensus metric CIDEr-R, and an exact-span NER F1 score computed via language-specific NER models to quantitatively evaluate entity grounding fidelity.
Key Experimental Results¶
Main Results: Cross-Lingual Captioning Performance Across Resource Tiers¶
The table below reports BLEU-4 and CIDEr-R scores across nine representative languages under various prompting and fine-tuning regimes (compiled from Table 2 in the paper):
| Configuration / Model | English (eng) | Arabic (ara) | Chinese (cmn) | Hindi (hin) | Swahili (swa) | Bengali (ben) | Nepali (nep) | Sinhala (sin) | Overall (All) |
|---|---|---|---|---|---|---|---|---|---|
| BLEU-4 Metric | |||||||||
| Zero-shot (Qwen2.5-VL-7B) | 2.94 | 2.62 | 4.32 | 3.19 | 4.55 | 2.53 | 1.07 | 0.80 | 2.72 |
| Zero-shot (Qwen3-VL-8B) | 2.51 | 1.79 | 4.13 | 3.31 | 4.26 | 2.62 | 0.78 | 0.71 | 2.57 |
| Holistic-CoT Prompting (Qwen2.5-VL) | 2.21 | 1.69 | 4.95 | 2.32 | 4.45 | 0.75 | 0.40 | 0.45 | 2.12 |
| Spatial-CoT Prompting (Qwen2.5-VL) | 1.89 | 1.23 | 5.36 | 1.76 | 3.98 | 0.71 | 0.31 | 0.44 | 1.93 |
| Basic FT (Qwen2.5-VL-7B) | 5.70 | 4.32 | 7.22 | 4.39 | 5.62 | 2.85 | 1.48 | 1.42 | 4.14 |
| Holistic-CoT FT (Qwen2.5-VL-7B) | 7.08 | 5.65 | 8.84 | 5.58 | 6.98 | 3.79 | 2.02 | 2.01 | 5.32 |
| Spatial-CoT FT (Qwen2.5-VL-7B) | 7.95 | 6.51 | 9.77 | 6.42 | 7.93 | 4.31 | 2.52 | 2.43 | 6.24 |
| Spatial-CoT FT (Qwen3-VL-8B) | 7.52 | 6.08 | 9.31 | 6.05 | 7.48 | 3.98 | 2.18 | 2.12 | 5.82 |
| CIDEr-R Metric | |||||||||
| Zero-shot (Qwen2.5-VL-7B) | 25.82 | 21.91 | 10.05 | 21.80 | 26.71 | 23.52 | 21.95 | 15.12 | 20.78 |
| Basic FT (Qwen2.5-VL-7B) | 54.74 | 44.44 | 69.41 | 43.76 | 56.62 | 22.44 | 12.36 | 11.61 | 39.55 |
| Holistic-CoT FT (Qwen2.5-VL-7B) | 74.51 | 61.82 | 91.92 | 60.11 | 76.14 | 35.12 | 19.42 | 19.31 | 55.92 |
| Spatial-CoT FT (Qwen2.5-VL-7B) | 81.92 | 68.78 | 100.52 | 66.82 | 83.92 | 39.31 | 22.91 | 21.62 | 63.12 |
| Spatial-CoT FT (Qwen3-VL-8B) | 78.13 | 65.52 | 96.24 | 63.42 | 80.23 | 36.24 | 20.42 | 19.81 | 59.84 |
Ablation & Analysis: Entity Grounding Evaluation (NER F1)¶
The following table details the exact-span NER F1 scores between human ground-truth captions and model outputs across language tiers (from Table 3 in the paper):
| Prompting & Fine-tuning Regimes | English (eng) | Arabic (ara) | Chinese (cmn) | Hindi (hin) | Swahili (swa) | Bengali (ben) | Nepali (nep) | Sinhala (sin) | Overall (All) |
|---|---|---|---|---|---|---|---|---|---|
| Zero-shot (Qwen2.5-VL-7B) | 7.12 | 5.89 | 8.34 | 9.23 | 13.45 | 2.89 | 4.23 | 2.12 | 6.46 |
| Zero-shot (Qwen3-VL-8B) | 7.45 | 6.12 | 8.67 | 9.56 | 13.89 | 3.12 | 4.56 | 2.34 | 6.76 |
| Holistic-CoT Prompting (Qwen3-VL) | 16.23 | 13.78 | 18.95 | 20.87 | 28.12 | 6.23 | 10.12 | 4.92 | 14.43 |
| Spatial-CoT Prompting (Qwen3-VL) | 16.12 | 13.23 | 18.52 | 19.43 | 27.73 | 7.39 | 12.61 | 5.37 | 14.50 |
| Basic FT (Qwen2.5-VL-7B) | 20.45 | 17.34 | 23.56 | 26.78 | 34.56 | 8.89 | 12.89 | 6.45 | 18.31 |
| Basic FT (Qwen3-VL-8B) | 21.23 | 18.12 | 24.34 | 27.89 | 35.89 | 9.23 | 13.34 | 6.78 | 19.02 |
| Holistic-CoT FT (Qwen3-VL-8B) | 39.45 | 35.12 | 45.67 | 50.89 | 64.23 | 17.89 | 24.34 | 12.89 | 35.49 |
| Spatial-CoT FT (Qwen2.5-VL-7B) | 39.89 | 35.34 | 46.12 | 51.34 | 64.89 | 18.56 | 25.34 | 13.45 | 36.02 |
| Spatial-CoT FT (Qwen3-VL-8B) | 41.23 | 36.78 | 47.56 | 53.12 | 67.12 | 19.34 | 26.45 | 14.12 | 37.33 |
Key Findings¶
- CoT Prompting Failure vs. Fine-tuning Efficacy: Zero-shot CoT prompting underperforms standard zero-shot prompting across all languages on overlap metrics (e.g., Qwen2.5-VL drops from 2.72 to 2.12 on BLEU-4). Sub-10B models struggle to follow complex reasoning chains without task adaptation. Conversely, when fine-tuned on the structured CoT format, Spatial-CoT FT surges to 63.12 CIDEr-R (vs. 39.55 for Basic FT and 20.78 for Zero-shot), demonstrating that explicit reasoning representations provide immense leverage during parameter updates.
- Pronounced Disparities Across Resource Tiers: Performance strongly correlates with linguistic resource availability. High-resource languages achieve extraordinary fidelity (Chinese CIDEr-R reaches 100.52; English reaches 81.92), and mid-resource Swahili achieves a competitive 83.92. However, low-resource languages face significant bottlenecks, with Sinhala peaking at 21.62 CIDEr-R and 14.12 NER F1, underscoring the severity of multimodal data scarcity.
- Substantial Entity Grounding Gains: In contrast to surface n-gram metrics, NER F1 demonstrates that CoT prompting immediately enhances entity extraction (rising from 6.76 in zero-shot to 14.50 in Spatial-CoT prompting). Under Spatial-CoT fine-tuning, NER F1 achieves a dramatic leap to 37.33, indicating that spatial-semantic binding directly targets and remedies entity hallucination.
- Model Decoupling Between Fluency and Entity Accuracy: Qwen2.5-VL-7B attains the highest BLEU-4 and CIDEr-R across most settings, yet Qwen3-VL-8B consistently outperforms it on NER F1 (37.33 vs. 36.02 overall). Furthermore, model superiority in high-resource tiers does not consistently transfer to low-resource languages, demonstrating that monolingual rankings cannot substitute for comprehensive multilingual evaluations.
Highlights & Insights¶
- First Massive-Scale Multilingual News Captioning Benchmark: Expands the scope of the field from English-centric corpora to 59 diverse languages and over 800,000 image-article pairs, establishing an open benchmark for equitable multilingual multimodal research.
- Heuristic Spatial-Entity Grounding Without Dense Box Labels: Cleverly marries zero-shot instance segmentation from SAM3 with entity typing rules (persons to foreground person masks, locations/orgs to building masks) to enforce spatial grounding without requiring labor-intensive bounding box annotations.
- Shifting the CoT Paradigm from Inference to Training: Uncovers an essential behavior of small-to-midsize VLLMs: complex multi-stage CoT acts as a distraction at inference time, but serves as a powerful scaffolding format when utilized as structured supervision during fine-tuning.
Limitations & Future Work¶
- Evaluation Restricted by Upstream NER Tooling: While the dataset spans 59 languages, comprehensive evaluation was constrained to nine languages due to the lack of reliable, high-performing monolingual NER models in many low-resource tongues.
- Automated Metric Limitations vs. Human Editorial Nuance: Experiments rely strictly on BLEU-4, CIDEr-R, and exact-span NER F1. These lexical metrics do not fully capture journalistic tone, objectivity, or nuanced factual alignment, and cross-lingual human evaluation was omitted due to annotator scaling challenges.
- Vulnerability to Non-Literal and Figurative Contexts: News reporting frequently involves irony, metaphors, and humor, where image content correlates figuratively rather than literally with article text. The current grounding pipeline assumes direct factual correspondence and may stumble on rhetorical expressions.
Related Work & Insights¶
- vs. VISUAL NEWS & NYTIMES800K: Monolingual English benchmarks that pioneered entity-aware news captioning but fail to generalize to cross-lingual contexts. LaMuN matches their scale (>800k) while democratizing evaluation across 59 languages.
- vs. MUNICHUS: The initial multilingual attempt covering nine languages from a single news provider, which relied on basic zero-shot prompting and reported poor feasibility. This work expands coverage six-fold across three international outlets and demonstrates that structured fine-tuning yields high-quality captions.
- vs. JO-GANIC & MERGE: Prior methods designed task-specific architectures for entity retrieval and layout encoding on English data. This work adapts general-purpose open-source VLLMs via decoupled toolchains (SAM3 + BGE-M3) and parameter-efficient LoRA tuning, offering greater modularity and lower training overhead.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering 59-language 800k-scale benchmark combined with spatial-semantic decomposed CoT fine-tuning]
- Experimental Thoroughness: โญโญโญโญโ [Evaluates four VLLMs across three prompting and three fine-tuning regimes on nine languages, though 50 languages remain unbenchmarked]
- Writing Quality: โญโญโญโญโญ [Exemplary clarity, compelling narrative structure, and rigorous analysis of the interplay between CoT prompting and fine-tuning]
- Value: โญโญโญโญโญ [Provides a foundational asset for multilingual vision-language research and a practical blueprint for factually grounded multimodal generation]