Skip to content

TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Morty-Xu/TCC-Bench
Area: LLM (Other)
Keywords: MLLM evaluation, traditional Chinese culture, multimodal VQA, cultural bias, visual grounding

TL;DR

To assess the vulnerabilities of multimodal large language models (MLLMs) in non-Western cultural understanding, this paper introduces TCC-Bench, the first comprehensive bilingual VQA benchmark spanning eight distinct knowledge domains with 770 curated images and 1,015 questions, revealing persistent deficits across both open- and closed-source models when grounding nuanced cultural knowledge in fine-grained visual evidence.

Background & Motivation

Recent years have witnessed remarkable breakthroughs in Multimodal Large Language Models (MLLMs), drastically enhancing their ability to process complex cross-modal information, follow visual instructions, and conduct general commonsense reasoning. However, an accumulating body of empirical evidence indicates that mainstream foundation MLLMs, predominantly pre-trained on massive Western-centric and contemporary internet-scale corpora, experience severe performance degradation and implicit cultural bias when encountering non-Western languages and deep heritage knowledge. Traditional Chinese culture, with an unbroken history spanning several millennia, incorporates an exceptionally rich yet intricate network of astronomical calendars, ritual social customs, structural architectural techniques, and specialized material artifacts. Far from being archaic relics, these cultural foundations continue to profoundly shape the contemporary social fabric and collective consciousness of Chinese communities worldwide, rendering robust cultural comprehension an indispensable requirement for truly inclusive and globally competent multimodal artificial intelligence.

Despite the growing urgency of cross-cultural AI evaluation, existing multimodal benchmarks suffer from two major bottlenecks. First, while general multicultural datasets such as MaRVL, CVQA, and CultureVQA incorporate a handful of Chinese cultural samples, their coverage remains disproportionately superficial, largely confined to over-represented, high-frequency celebratory symbols like red envelopes and dragon dances. Conversely, specialized benchmarks like the Pun Rebus Art Dataset or CII-Bench concentrate narrowly on traditional painting metaphors or fine-art aesthetics, falling short of offering a systematic and multifaceted evaluation across diverse life spheres. Second, contemporary Visual Question Answering (VQA) benchmarks are widely plagued by severe language priors and shortcut learning. Question stems frequently reveal the exact cultural entity name (e.g., explicitly asking "What is a sundial used for?"), which allows text-only large language models to exploit linguistic memory and bypass visual processing altogether, thereby rendering the visual evaluation spurious. Furthermore, fully automated question generation with multimodal foundation models risks severe data leakage and artificially inflated scores, while purely manual data creation remains prohibitively costly.

To overcome these structural limitations, the authors introduce a systematic taxonomy of traditional Chinese culture spanning eight balanced domains that bridge classical material necessities (Clothing, Diet, Architecture, Transportation) and refined intellectual/ritual dimensions (Astronomy, Music, Custom, Artifact). To balance high curation quality with scalable dataset construction, the authors devise a human-in-the-loop semi-automated pipeline that explicitly restricts GPT-4o to text-only mode during candidate question generation, subsequently pairing this with rigorous human-guided visual grounding rewriting and two rounds of independent expert verification. Core idea: develop TCC-Bench, the first systematic bilingual multimodal benchmark covering eight core cultural domains, engineered through text-only question drafting and strict de-concept visual grounding rewriting to eliminate language priors and reliably diagnose the cultural perception and reasoning boundaries of current MLLMs.

Method

Overall Architecture

TCC-Bench is specifically architected to evaluate the fine-grained visual recognition, cultural concept mapping, and situated contextual reasoning of multimodal foundation models. The curated benchmark comprises 770 culturally authentic and visually diverse images sourced from open-licensed digital archives, museum collections, and everyday cultural scenes, yielding 1,015 high-quality four-choice VQA problems with bilingual questions, answer options, and comprehensive cultural explanations. To systematically mitigate language shortcuts and eliminate model self-evaluation leakage, the construction workflow is structured into four sequential stages: cultural concept extraction (Stage I), text-only candidate question generation (Stage II), human-directed de-concept visual refinement (Stage III), and two-round independent human verification (Stage IV).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Cultural Image Collection<br/>1,830 raw images filtered to 770"] --> B["Cultural Concept Extraction<br/>Expert extraction of visual entities & domain tags"]
    B --> C["Text-Only Question Generation<br/>Text-only GPT-4o generates options & distractors"]
    C --> D["De-Concept Visual Refinement<br/>Strip entity names & enforce visual grounding"]
    D --> E["Two-Round Human Verification<br/>Independent cross-check & strict zero-tolerance filtering"]
    E --> F["Final TCC-Bench Dataset<br/>8 domains·1,015 bilingual VQA instances"]

Key Designs

1. Cultural Concept Extraction: Ensuring Authentic Representation Across Eight Cultural Dimensions To avoid superficial modern recreations and guarantee cultural authenticity, the dataset construction process begins with six native annotators holding bachelor's degrees or higher who possess long-term cultural immersion. Annotators inspect each curated image and extract one unambiguous, visually verifiable cultural concept. Guided by traditional Chinese cultural taxonomies, these concepts are organized into eight balanced domains: Astronomy (Tiānwén, covering the 24 solar terms, ancient armillary spheres, and the 28 Mansions), Music (Yīnyuè, covering pentatonic scales, Guqin, Pipa, and classical opera), Custom (Mínsú, covering life-cycle rites and festival etiquettes), Architecture (Jiànzhù, encompassing Dougong brackets and Siheyuan courtyards), Transportation (Jiāotōng, including historic sedan chairs and watercraft), Diet (Yǐnshí, including regional cuisines and tea ceremonies), Clothing (Fúshì, spanning Hanfu garments and ceremonial robes), and Artifact (Qìwù, covering bronze vessels, porcelain, and the Four Treasures of the Study). Annotators enforce three strict filtering principles: cultural relevance, unambiguous visual evidence within the image frame, and substantial utility for meaningful cultural inquiries.

2. Text-Only Question Generation: Eliminating Multimodal Self-Evaluation Data Leakage Addressing the pervasive issue of dataset contamination and evaluation inflation in LLM-assisted dataset generation, the authors intentionally restrict GPT-4o to a text-only mode during candidate generation. Rather than inputting images directly into vision-language models, annotators feed only the extracted textual concepts into a tailored prompt requiring three multiple-choice questions, four distinct options (one ground-truth and three plausible, confusing distractors), and concise explanatory rationales in both Chinese and English. Disabling the multimodal capability of the generation model is critical: allowing GPT-4o to inspect the images would inevitably yield questions implicitly aligned with its own proprietary visual feature preferences and recognition strengths, creating an unfair evaluation bias favoring frontier closed-source architectures. The text-only prompting protocol ensures that candidate questions are derived strictly from cultural ontology rather than idiosyncratic vision-language representations.

3. De-Concept Visual Refinement: Eradicating Shortcut Language Priors Because text-only generated questions inherently state the underlying cultural concept directly in the prompt (e.g., asking "What is a sundial used for?"), passing them directly to evaluators would allow pure language models to answer questions using text memorization alone without processing the visual modality. In the manual refinement phase, four expert annotators systematically rewrite each question into a strictly visually grounded format. All explicit references to the cultural concept are replaced with direct visual deictic pointers, transforming the question into "What is the instrument in the picture used for?" or asking for the specific symbolic meaning of patterns on an attire's collar. This transformation forces MLLMs to first perform accurate, fine-grained visual perception to identify the depicted entity before retrieving relevant cultural knowledge, successfully insulating the benchmark against language prior exploitation.

4. Two-Round Independent Human Verification: Enforcing Strict Quality Assurance To guarantee zero-defect factual accuracy and eliminate visual ambiguity, each candidate instance undergoes two subsequent rounds of fully independent, blind verification conducted by the remaining two annotators. Evaluators cross-check each entry against four rigorous criteria: domain consistency, visual-factual question correctness, cultural explanation precision, and complete absence of concept leakage in the text prompt. Any sample failing any single check in either round is irrevocably discarded without further leniency. Through this stringent quality control funnel, 13 substandard questions were discarded, finalizing a benchmark of 1,015 resilient, highly validated cultural VQA problems.

Benchmark & Evaluation Protocol

The evaluation suite spans diverse parameter scales and model families, including open-source small models (DeepSeek-VL2-3B, LLaVA-1.6-7B, Qwen2.5-VL-7B, GLM-4V-9B, InternVL3.5-8B), open-source medium/large models (DeepSeek-VL2-16B/27B, CogVLM2-19B, LLaVA-1.6-34B, InternVL3.5-38B, Qwen2.5-VL-32B/72B), and frontier closed-source APIs (GPT-4.1, Gemini-2.5 Flash, Gemini-2.5 Pro). Open-source models are deployed using LMDeploy on A800 GPUs. In addition, the authors establish a native human baseline (three native Chinese speakers evaluating 15 randomly sampled questions per domain, totaling 120 questions) and a text-only blind test with GPT-4.1 (stripping visual inputs to quantify language prior bias). Analyses further incorporate Chain-of-Thought (CoT) prompting, domain-aware prompting, explanation consistency via LLM-as-a-Judge (scored 0–3 using GPT-4.1), and manual error typology classification.

Key Experimental Results

Main Results

The comprehensive evaluation results on TCC-Bench across bilingual prompts and eight cultural domains are summarized below:

Model Group / Model Parameter / Architecture Overall Acc (ZH, %) Overall Acc (EN, %) Strongest Domain (ZH, %) Weakest Domain (ZH, %)
Human Performance 3 Native Chinese Speakers 85.00 - Custom / Diet (93.33) Astronomy (73.33)
Open-source (<10B) DeepSeek-VL2-3B 3B (MoE) 60.69 55.67 Artifact (70.52) Astronomy (38.57)
Open-source (<10B) LLaVA-1.6-7B 7B (Vicuna) 42.66 46.01 Artifact (51.00) Astronomy (24.29)
Open-source (<10B) Qwen2.5-VL-7B 7B 73.60 69.36 Architecture (81.62) Astronomy (54.29)
Open-source (<10B) GLM-4V-9B 9B 51.23 46.11 Custom (61.59) Astronomy (39.29)
Open-source (<10B) InternVL3.5-8B 8B 69.06 60.49 Artifact (78.88) Astronomy (50.71)
Open-source (>10B) DeepSeek-VL2-16B 16B (MoE) 67.68 63.74 Diet (81.33) Astronomy (37.86)
Open-source (>10B) DeepSeek-VL2-27B 27B (MoE) 60.00 55.27 Diet (70.67) Astronomy (40.00)
Open-source (>10B) CogVLM2-19B 19B 57.64 54.88 Artifact (68.92) Astronomy (36.43)
Open-source (>10B) Qwen2.5-VL-32B 32B 75.27 70.44 Diet (85.33) Astronomy (52.14)
Open-source (>10B) LLaVA-1.6-34B 34B 63.05 58.42 Music (73.08) Astronomy (23.57)
Open-source (>10B) InternVL3.5-38B 38B 73.40 66.90 Transportation (86.49) Astronomy (51.43)
Open-source (>10B) Qwen2.5-VL-72B 72B 75.17 71.03 Custom (84.11) Astronomy (56.43)
Closed-source GPT-4.1 Commercial API 76.65 73.89 Diet (89.33) Astronomy (44.29)
Closed-source Gemini-2.5 Flash Commercial API 84.93 80.89 Diet (94.67) Astronomy (77.14)
Closed-source Gemini-2.5 Pro Commercial API 87.00 83.55 Diet (96.00) Transportation (81.08)
Text-only Blind Baseline GPT-4.1 (Text-only) Vision Stripped 23.35 34.09 Clothing (32.65) Astronomy (10.00)

Ablation & Mechanism Study

To investigate how prompting mechanisms and error modalities affect cultural reasoning, the authors conducted targeted ablations across CoT reasoning, domain-aware prompting, and fine-grained error taxonomy:

1. Effect of Chain-of-Thought (CoT) Prompting Across Model Tiers

Evaluated Model Standard Accuracy (w/o CoT, %) CoT Accuracy (w/ CoT, %) Delta (%) Empirical Mechanism
LLaVA-1.6-7B 42.66 35.37 -7.29 Hallucinates reasoning chains due to weak visual grounding
Qwen2.5-VL-72B 75.17 77.64 +2.47 Effectively leverages step-by-step cultural deduction
GPT-4.1 76.65 76.95 +0.30 Marginal gain; visual perceptual bottleneck persists
Gemini-2.5 Pro 87.00 86.31 -0.69 Near ceiling; explicit CoT introduces redundant noise

2. Effect of Domain-aware Prompting on Model Accuracy

Model Domain-agnostic Acc (%) Domain-aware Acc (%) Relative Change (%)
LLaVA-1.6-7B 42.66 43.74 +1.08
InternVL3.5-8B 69.06 66.50 -2.56
DeepSeek-VL2-27B 60.00 51.53 -8.47
Qwen2.5-VL-72B 75.17 74.68 -0.49
GPT-4.1 76.65 76.26 -0.39
Gemini-2.5 Pro 87.00 87.78 +0.78

3. Error Taxonomy Distribution for Qwen2.5-VL-72B (under CoT)

Error Category Percentage (%) Typical Failure Mechanism
Lack of Cultural Knowledge 55.9% Correctly identifies visual instrument but lacks liturgical ritual knowledge, misclassifying a sacrificial instrument as a banquet flute
Visual Perceptual Error 33.9% Misses transient visual actions (e.g., fails to spot Sichuan Opera face-changing gesture, mistaking it for Beijing Opera costume)
Contextual Interpretation Error 8.4% Captures visual elements but overlooks key discriminatory semantics (e.g., mistaking a relief-coiled dragon pillar for a standard Huabiao)
Reasoning Error 1.8% Gathers correct visual perception and facts but fails in the final deduction step (contradicting its own preceding reasoning)

Key Findings

  • Definitive Mitigation of Language Priors: Disabling image access causes GPT-4.1's accuracy to plunge from 76.65% to 23.35% under Chinese prompts (below the 25% random-chance threshold for 4-choice questions) and to 34.09% under English prompts. This massive performance gap confirms that visual understanding is strictly required to answer TCC-Bench questions.
  • Extreme Cross-Domain Knowledge Imbalance: Astronomy is universally the most challenging domain across all evaluated models (averaging below 50%, with LLaVA-1.6-7B achieving only 24.29% and GPT-4.1 scoring 44.29%), whereas Customs and Diet consistently exhibit high scores (surpassing 85%–95%). This disparity stems from the heavy representation of living folklore in web pre-training data compared to the extreme rarity of classical astronomical and ritual archives.
  • Systematic Linguistic Disparity (ZH vs. EN): Almost all models attain superior accuracy under Chinese prompts compared to English prompts, with margins typically ranging from 3% to 9%. This confirms that classical terminology (e.g., "Dou-gong", "Armillary sphere", "Kesi silk") undergoes semantic drift during cross-lingual translation, degrading visual-semantic grounding.
  • Explanation Quality Divergence: In the LLM-as-a-Judge explanation consistency evaluation (0–3 scale), Gemini-2.5 Pro secures a leading score of 2.60, while LLaVA-1.6-7B languishes at 0.85 (and 0.43 in Astronomy). Smaller open-source models frequently pick the correct option by spurious correlation while their generated rationales remain factually detached from the image.

Highlights & Insights

  • Modality-Blind Question Generation Paradigm: Deliberately turning off the visual modality of frontier generative models during VQA question authoring establishes an elegant design template that prevents generative evaluation leakage.
  • De-Concept Deictic Grounding Rule: Substituting explicit conceptual entities with deictic expressions ("the object in the image") provides a standardized methodological safeguard for domain-specific benchmark curation.
  • Cultural Deficits Are Primarily Knowledge-Bound: The error distribution reveals that 55.9% of failures stem from absent cultural facts rather than visual misidentification, demonstrating that enhancing cross-cultural MLLMs requires fine-grained knowledge ingestion rather than vision encoder scaling alone.

Limitations & Future Work

  • High Human Curation Costs: Despite leveraging text-only GPT-4o for initial candidate generation, meticulous manual rewriting and two rounds of expert cross-checking remain labor-intensive, limiting dataset size to 1,015 instances.
  • Visual Scarcity of Ancient Historical Domains: Many cultural concepts and classical architectural styles predate photography, leading to an inherent scarcity of high-resolution, open-licensed visual artifacts in domains such as ancient Transportation (3.6%).
  • Regional and Ethnic Diversity Underrepresentation: The present benchmark predominantly reflects mainstream Han cultural traditions, leaving regional folklore and ethnic minority cultural heritage under-represented. Future work should expand toward comprehensive non-heritage and regional folk art databases.
  • vs MaRVL / CVQA / CultureVQA: Prior multicultural benchmarks spread their scope across dozens of global cultures, resulting in shallow, stereotypical coverage for each specific culture; TCC-Bench delivers an in-depth, structured taxonomy specifically focused on traditional Chinese heritage.
  • vs CII-Bench / Pun Rebus Art: Earlier Chinese-specific cultural datasets restricted their domains to artistic paintings or linguistic puns; TCC-Bench spans eight realistic daily, ritual, and material domains with broader everyday applicability.
  • vs Fully Automated VQA Pipelines: Fully synthetic datasets suffer from severe language bias and visual triviality; TCC-Bench's hybrid pipeline pairs text-only generation with rigorous visual de-concept grounding, establishing a reliable benchmark methodology.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Pioneers a rigorous eight-domain bilingual cultural VQA benchmark with an innovative blind-generation anti-leakage pipeline.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 14 diverse open/closed models, incorporating CoT ablations, domain-aware prompting, text-only blinding, and fine-grained error taxonomy.
  • Writing Quality: ⭐⭐⭐⭐⭐ Extremely lucid narrative structure, insightful error case analyses, and well-organized experimental tables.
  • Value: ⭐⭐⭐⭐⭐ Sets a high-fidelity evaluation foundation for diagnosing and mitigating cultural blind spots in multimodal artificial intelligence.