Large-scale Multilingual News Image Captioning with Fact-Grounded Context Retrieval¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/tharindu/LaMuN
Area: Multimodal VLM
Keywords: Multilingual News Image Captioning, Vision-Language Models, Entity Grounding, Visual Chain-of-Thought, Retrieval-Augmented Generation
TL;DR¶
Addressing the severe English-only limitation and entity hallucination in news image captioning, this work establishes LaMuNโthe first benchmark spanning 59 languages and 800K+ news imagesโand introduces a multi-stage Chain-of-Thought fine-tuning paradigm integrating visual decomposition, dense event retrieval clustering, and region-level entity grounding to achieve state-of-the-art results across diverse resource tiers.
Background & Motivation¶
News image captioning fundamentally diverges from generic image captioning in both intent and semantic specificity. While generic captioning focuses on describing observable visual elements in broad categories (such as "a crowd of people standing" or "a woman holding a trophy"), news captions are inherently fact-grounded and context-dependent. They situate the depicted visual scene within a concrete real-world narrative by identifying exact individual names, political or corporate affiliations, geographical locations, and specific historical or breaking news events that cannot be deduced from pixels alone. Practically, accurate news image captions serve as an essential assistive medium for conveying visual news content to visually-impaired citizens via screen readers, and they provide fundamental indexing keys for cross-modal digital archive search and disinformation verification across international press agencies.
Despite its high practical value and the rapid evolution of Vision Large Language Models (VLLMs), the research landscape in news image captioning has long been stifled by an acute monolingual bottleneck. Established datasets such as Visual News, NYTimes800k, and GoodNews collectively contain hundreds of thousands to millions of image-text pairs, yet they are exclusively restricted to English. Even recent pioneer efforts like MUNICHUS only include nine languages gathered from a single outlet, and simply test elementary zero-shot or few-shot prompts with discouraging outcomes. Crucially, directly prompting modern VLLMs with an entire unstructured news article leads to severe distraction: the models struggle to ground visually salient figures amidst multi-topic news text, frequently hallucinating unrelated entities or collapsing into generic descriptions stripped of informative named entities.
Resolving this tension requires bridging the cross-modal gap between visual scene layout and unstructured textual narratives across heterogeneous linguistic families, particularly in low-resource environments lacking paired alignment data. The core idea is to build LaMuNโthe first large-scale multilingual news image captioning benchmark comprising over 800,000 images across 59 languagesโand decompose captioning into a structured four-stage Chain-of-Thought (CoT) fine-tuning pipeline combining visual region parsing, dense multilingual event clustering, language-specific named entity extraction, and progressive spatial-to-text synthesis.
Method¶
Overall Architecture¶
The proposed system resolves the long-context entity distraction problem by decomposing end-to-end news captioning into four interpretable, specialized reasoning stages. The input comprises an unannotated news image alongside its full accompanying multi-sentence news article. In the visual analysis stage, an open-world vision foundation model (Florence-2 for global summary or SAM3 for fine-grained segmentation) extracts structural visual priors. In the contextual retrieval stage, sentence embeddings are derived via the multilingual dense retriever BGE-M3 and clustered into discrete event candidates using HDBSCAN; the event cluster exhibiting the highest cosine similarity to the visual representation is selected as the grounded context. In the entity extraction stage, dedicated language-specific Named Entity Recognition (NER) models isolate key entities (persons, locations, organizations, dates). Finally, in the caption synthesis stage, the filtered event context, the grounded entities, and the regional visual crops are assembled into an ordered structural prompt fed into an adapted VLLM (such as Qwen2.5-VL, Qwen3-VL, Aya-Vision, or Gemma) to generate factually accurate captions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multilingual News Image + Full Article Text"] --> B["Visual Semantic Parsing and Region Decomposition"]
A --> C["Multilingual Dense Retrieval and Event Clustering"]
B --> D["Language-Specific Entity Extraction and Cross-Modal Mapping"]
C --> D
D --> E["Spatially-Guided Progressive Caption Synthesis"]
E --> F["Fact-Grounded Multilingual News Caption"]
Key Designs¶
1. Visual Semantic Parsing and Region Decomposition: Bridging modalities via language-space queries and semantic segmentation Directly supplying an unformatted high-resolution image and a lengthy article forces the language model to search for needle-in-a-haystack visual-textual alignments without focal anchors. The authors devise two complementary visual analysis mechanisms. In Holistic-CoT, the unified vision model Florence-2 generates a concise global description of the image, functioning as an expressive language-space proxy query that allows cross-modal event retrieval to operate entirely within a shared semantic space. In Spatial-CoT, targeting scenes with multiple individuals or complex backgrounds where entity-swapping errors occur, SAM3 performs instance-level foreground-background segmentation. Each segmented mask is tagged with coarse semantic labels (e.g., person, building), providing precise spatial bounding boxes and cropped regional images that serve as localized anchors for subsequent entity attachment.
2. Multilingual Dense Retrieval and Event Clustering: Unsupervised adaptive identification of focal narrative events News articles routinely intertwine primary event descriptions with historical recaps, tangential commentaries, and contextual background. Simple keyword overlap or fixed-window text truncation inevitably injects confounding textual noise. The pipeline embeds every sentence in the article into a dense vector space using BGE-M3, chosen for its demonstrated cross-lingual alignment across high- and low-resource languages alike. Rather than forcing a predetermined number of clusters through standard \(k\)-means, the authors deploy HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise). HDBSCAN dynamically discovers an arbitrary number of coherent event topics based on vector density while effectively casting off isolated outlier sentences. The cluster centroid \(\bar{E}_{C_k}\) maximizing the cosine similarity against the Florence-2 visual embedding \(E_{\text{image}}\) is retrieved: $\(\text{Sim}(E_{\text{image}}, C_k) = \frac{E_{\text{image}} \cdot \bar{E}_{C_k}}{\|E_{\text{image}}\| \|\bar{E}_{C_k}\|}\)$ This ensures that only the textual segment describing the exact event captured in the frame is forwarded to downstream generation.
3. Language-Specific Entity Extraction and Cross-Modal Mapping: Mitigating boundary drift and enforcing type-based topological bindings Monolingual or general-purpose multilingual NER systems exhibit significant performance degradation when applied to low-resource languages. The method integrates independently pre-trained, high-accuracy language-specific NER models for each evaluated language (such as RoBERTa-large-NER for English, MacBERT4NER for Chinese, and XLM-RoBERTa-large-masakhaner for Swahili) to extract structured entity sets spanning persons, organizations, locations, dates, and events directly from the retrieved event passage. Under the Spatial-CoT paradigm, a type-based soft alignment heuristic is deployed: identified person entities are mapped directly to foreground person crops, while organization and location entities are bound to background architectural regions. This rule-based mapping avoids expensive manual visual-grounding annotations while enforcing strict topological correspondence between named actors and visual objects.
4. Spatially-Guided Progressive Caption Synthesis: Internalizing multi-step reasoning through structured instruction tuning Small-to-medium open-weight VLLMs (ranging from 4B to 8B parameters) struggle to execute complex multi-step instructions during zero-shot or few-shot inference, often yielding degraded outputs when exposed to lengthy Chain-of-Thought prompts. To overcome this limitation, the authors formalize the entire inference sequence into a structured instruction prompt requiring the model to describe the scene progressively from foreground to background along the sorted region order and mandating the verbatim inclusion of assigned named entities. Crucially, this structured format is leveraged during supervised instruction fine-tuning via Low-Rank Adaptation (LoRA). By exposing the language backbone to identical intermediate reasoning structures during training, the model internalizes the alignment trajectory and learns to condition fluent text generation on isolated, high-precision contextual facts.
Loss & Training¶
The fine-tuning phase employs standard autoregressive causal language modeling cross-entropy loss applied exclusively to the output caption tokens. The visual encoders and upstream reasoning modules (Florence-2, BGE-M3, SAM3, and language-specific NER models) remain completely frozen. Parameter-efficient adaptation is conducted via LoRA on the underlying attention projections (\(q\_proj, k\_proj, v\_proj, o\_proj\)) across four base architectures: Aya-Vision-8B, Gemma-4-E4B-it, Qwen2.5-VL-7B, and Qwen3-VL-8B. Hyperparameters are held uniform across all models and languages: LoRA rank \(r = 16\), scaling factor \(\alpha = 32\), and dropout rate of \(0.05\). Optimization utilizes AdamW (\(\beta_1 = 0.9, \beta_2 = 0.999\), weight decay \(0.01\)) with a peak learning rate of \(2 \times 10^{-4}\) decayed via a cosine schedule preceded by a 3% linear warmup. Training is executed on a single NVIDIA H200 GPU (141 GB VRAM) using bfloat16 mixed precision, a per-device batch size of 4 with 4 gradient accumulation steps (effective batch size 16), a maximum sequence length of 8192 tokens, and spans 2 training epochs per language.
Key Experimental Results¶
Main Results¶
Experiments were conducted on nine representative target languages spanning high-resource (English eng, Arabic ara, Chinese cmn), mid-resource (Hindi hin, Indonesian ind, Swahili swa), and low-resource tiers (Bengali ben, Nepali nep, Sinhala sin) from the LaMuN benchmark. The table below presents the quantitative comparison of BLEU-4 and CIDEr-R across prompting and fine-tuning configurations:
| Evaluation Paradigm | Backbone Model | High-Resource (eng / ara / cmn) CIDEr-R | Mid-Resource (hin / ind / swa) CIDEr-R | Low-Resource (ben / nep / sin) CIDEr-R | Overall BLEU-4 | Overall CIDEr-R |
|---|---|---|---|---|---|---|
| ยง4.1 Zero-shot Prompting | Aya-Vision-8B | 26.12 / 24.90 / 10.33 | 22.45 / 25.44 / 25.33 | 22.11 / 21.52 / 12.24 | 2.05 | 20.94 |
| ยง4.1 Zero-shot Prompting | Gemma-4-E4B-it | 23.54 / 22.28 / 9.42 | 20.22 / 21.00 / 24.53 | 24.02 / 21.40 / 16.14 | 2.14 | 20.41 |
| ยง4.1 Zero-shot Prompting | Qwen2.5-VL-7B | 25.82 / 21.91 / 10.05 | 21.80 / 23.93 / 26.71 | 23.52 / 21.95 / 15.12 | 2.72 | 20.78 |
| ยง4.1 Zero-shot Prompting | Qwen3-VL-8B | 24.03 / 21.24 / 10.42 | 21.13 / 23.00 / 25.73 | 24.38 / 20.67 / 15.78 | 2.57 | 21.31 |
| ยง4.3 Basic Finetuning | Aya-Vision-8B | 49.13 / 40.72 / 63.32 | 38.55 / 37.16 / 51.83 | 18.67 / 9.33 / 9.75 | 3.69 | 35.21 |
| ยง4.3 Basic Finetuning | Gemma-4-E4B-it | 45.82 / 38.64 / 59.24 | 35.55 / 33.13 / 49.14 | 16.95 / 8.44 / 8.32 | 3.47 | 32.93 |
| ยง4.3 Basic Finetuning | Qwen2.5-VL-7B | 54.74 / 44.44 / 69.41 | 43.76 / 41.56 / 56.62 | 22.44 / 12.36 / 11.61 | 4.14 | 39.55 |
| ยง4.3 Basic Finetuning | Qwen3-VL-8B | 52.14 / 43.64 / 66.62 | 41.93 / 39.23 / 55.45 | 20.64 / 11.22 / 10.69 | 3.97 | 38.12 |
| ยง4.3 Holistic-CoT Finetuning | Qwen2.5-VL-7B | 74.51 / 61.82 / 91.92 | 60.11 / 58.24 / 76.14 | 35.12 / 19.42 / 19.31 | 5.32 | 55.92 |
| ยง4.3 Holistic-CoT Finetuning | Qwen3-VL-8B | 69.82 / 58.82 / 87.13 | 56.55 / 53.85 / 72.53 | 31.52 / 17.55 / 16.24 | 5.07 | 53.26 |
| ยง4.3 Spatial-CoT Finetuning (SOTA) | Aya-Vision-8B | 74.24 / 62.52 / 92.22 | 58.78 / 57.64 / 76.44 | 32.85 / 18.23 / 17.51 | 5.26 | 54.33 |
| ยง4.3 Spatial-CoT Finetuning (SOTA) | Gemma-4-E4B-it | 69.15 / 58.24 / 86.54 | 55.64 / 52.34 / 72.83 | 29.64 / 16.25 / 14.83 | 4.95 | 51.23 |
| ยง4.3 Spatial-CoT Finetuning (SOTA) | Qwen2.5-VL-7B | 81.92 / 68.78 / 100.52 | 66.82 / 64.14 / 83.92 | 39.31 / 22.91 / 21.62 | 6.24 | 63.12 |
| ยง4.3 Spatial-CoT Finetuning (SOTA) | Qwen3-VL-8B | 78.13 / 65.52 / 96.24 | 63.42 / 61.24 / 80.23 | 36.24 / 20.42 / 19.81 | 5.82 | 59.84 |
Ablation Study¶
The table below details the impact of prompting strategies versus supervised instruction tuning on exact named entity grounding performance (measured by exact span-matching NER F1 scores):
| Method Setting | Backbone Model | English (eng) NER F1 | Chinese (cmn) NER F1 | Swahili (swa) NER F1 | Bengali (ben) NER F1 | Sinhala (sin) NER F1 | Overall NER F1 | Configuration Description |
|---|---|---|---|---|---|---|---|---|
| ยง4.1 Zero-shot Prompting | Qwen2.5-VL-7B | 7.12 | 8.34 | 13.45 | 2.89 | 2.12 | 6.46 | Unstructured raw article and direct instruction |
| ยง4.1 Zero-shot Prompting | Qwen3-VL-8B | 7.45 | 8.67 | 13.89 | 3.12 | 2.34 | 6.76 | Baseline prompting without structural entities |
| ยง4.2.1 Holistic-CoT (Prompting) | Qwen2.5-VL-7B | 15.67 | 18.28 | 27.09 | 5.94 | 4.70 | 13.89 | Dense retrieval and entity list injected during inference |
| ยง4.2.2 Spatial-CoT (Prompting) | Qwen2.5-VL-7B | 15.58 | 17.88 | 26.72 | 7.06 | 5.13 | 13.97 | Regional decomposition added during inference |
| ยง4.3 Basic Finetuning | Qwen2.5-VL-7B | 20.45 | 23.56 | 34.56 | 8.89 | 6.45 | 18.31 | LoRA adaptation using basic instruction template |
| ยง4.3 Holistic-CoT Finetuning | Qwen2.5-VL-7B | 38.23 | 44.23 | 62.34 | 17.23 | 12.34 | 34.30 | LoRA tuning conditioned on event passages and entities |
| ยง4.3 Spatial-CoT Finetuning | Qwen2.5-VL-7B | 39.89 | 46.12 | 64.89 | 18.56 | 13.45 | 36.02 | Full spatial region and entity mapping adaptation |
| ยง4.3 Spatial-CoT Finetuning | Qwen3-VL-8B | 41.23 | 47.56 | 67.12 | 19.34 | 14.12 | 37.33 | Top-performing entity grounding across all benchmarks |
Key Findings¶
- The Prompting vs. Fine-Tuning Paradox in Chain-of-Thought: When applied strictly at inference time via prompting, both Holistic-CoT and Spatial-CoT underperform standard zero-shot prompting across all languages in n-gram overlap metrics (CIDEr-R drops from 20.78 to 13.20 and 14.82 on Qwen2.5-VL-7B). This stems from the inability of 4B-8B instruction-tuned models to follow multi-step formatting without task-specific exposure. Conversely, when fine-tuned on the structured CoT prompts, Spatial-CoT achieves massive gains, elevating CIDEr-R to 63.12 (a 59.6% relative boost over basic fine-tuning's 39.55).
- Substantial Disparity Across Resource Tiers: While high-resource languages reach peak CIDEr-R marks above 81.9 (English 81.92, Chinese 100.52) and mid-resource languages show resilient performance (Swahili reaching 83.92), low-resource languages face severe performance barriers. Under the best configuration, Sinhala peaks at only 21.62 CIDEr-R and Nepali at 22.91, underscoring the scarcity of foundational cross-modal representations for low-resource scripts.
- Model Specialization: Text Fluency vs. Fact Grounding: While Qwen2.5-VL-7B marginally leads in surface language generation metrics (BLEU-4 and CIDEr-R), Qwen3-VL-8B consistently surpasses all competitors in exact entity grounding, achieving the highest overall NER F1 score of 37.33 (reaching 67.12 on Swahili and 47.56 on Chinese). This demonstrates that evaluation rankings on high-resource languages do not uniformly generalize across grounding dimensions or resource tiers.
Highlights & Insights¶
- Establishment of the First Truly Multilingual News Captioning Benchmark: LaMuN fundamentally addresses the historical English bias by curating 800,000+ verified journalistic pairs across 59 languages from three global outlets, creating a reproducible evaluation foundation for diverse linguistic communities.
- Unsupervised Event Clustering via Density Trajectories: Employing Florence-2 as a language-space visual proxy combined with HDBSCAN density clustering provides an elegant solution for isolating local event narratives from verbose news articles without requiring heuristic paragraph truncations or fixed cluster counts.
- Strategic Value of Task-Decomposed Adaptation: The paper delivers an empirical revelation: small open-weight VLMs cannot reliably execute multi-hop reasoning off-the-shelf via complex prompts, but they excel when the structured reasoning trajectory is internalized through parameter-efficient fine-tuning.
Limitations & Future Work¶
- Constrained Empirical Evaluation Subset: Although LaMuN encompasses 59 languages, resource constraints and the requirement for dependable language-specific NER models restricted fine-tuning experiments to nine target languages, leaving diverse African, Caucasian, and indigenous language families unmeasured.
- Lexical Bias of Automated Evaluation Metrics: The benchmark relies primarily on n-gram overlap metrics (BLEU-4, CIDEr-R), which penalize factually sound and semantically accurate paraphrases that deviate from the ground truth reference style. Furthermore, human evaluations could not be conducted across the vast multilingual landscape.
- Vulnerability to Figurative and Non-Literal Discourse: News content often incorporates journalistic humor, irony, metaphor, and visual satire, which remain profound failure modes for current vision-language models and warrant dedicated non-literal grounding research.
Related Work & Insights¶
- vs. MUNICHUS (Chen et al., 2026): MUNICHUS first broached multilingual news captioning but was limited to nine languages from a single outlet and only assessed rudimentary prompting; LaMuN expands linguistic coverage over six-fold across three global agencies and establishes an effective CoT fine-tuning methodology.
- vs. Monolingual Benchmarks (Visual News, NYTimes800k): Traditional English-only datasets overlook cross-lingual entity translation and morphological diversity; this work reveals that model superiority established in English does not reliably translate to low-resource multilingual environments.
- vs. Complex Multi-Stage Retrieval Architectures (MERGE, ICECAP): Prior specialized frameworks rely on complex offline face-name banks or dedicated multi-modal indexers; the proposed method establishes that off-the-shelf generalist vision models (SAM3, Florence-2) coupled with dense cross-lingual text retrievers (BGE-M3) provide a more modular, scalable path forward.
Rating¶
- Novelty: โญโญโญโญโ Groundbreaking scale in multilingual news captioning benchmark construction paired with a decoupled visual-contextual CoT formulation.
- Experimental Thoroughness: โญโญโญโญโ Thorough evaluation across four contemporary open-weight VLMs, nine languages, and multiple prompting and LoRA paradigms.
- Writing Quality: โญโญโญโญโญ Clear pedagogical narrative, meticulous methodological documentation, and comprehensive appendix statistics.
- Value: โญโญโญโญโญ A transformative open-source contribution that establishes a pivotal milestone for global multilingual multimodal research.