TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Morty-Xu/TCC-Bench
Area: Multimodal VLM
Keywords: Multimodal Large Language Models, Traditional Chinese Culture, Visual Question Answering Benchmark, Cultural Understanding, Language Bias Mitigation
TL;DR¶
Addressing the severe Western data bias and lack of non-Western cultural understanding in current vision-language models, this paper introduces TCC-Bench, the first comprehensive bilingual multimodal VQA benchmark spanning eight cultural domains, equipped with a text-only generation and visual decoupling pipeline that eliminates language shortcuts and highlights key cognitive gaps across modern MLLMs.
Background & Motivation¶
Recent breakthroughs in Multimodal Large Language Models (MLLMs, such as GPT-4.1, Gemini-2.5, Qwen2.5-VL, and InternVL3.5) have dramatically elevated artificial intelligence capabilities in visual understanding, cross-modal reasoning, and complex instruction-following. However, existing foundational models are overwhelmingly pre-trained on Western-centric web corpora and paired image-text datasets, resulting in severe performance drops, factual hallucinations, and deep cultural blind spots when deployed in non-Western linguistic and cultural settings. Traditional Chinese culture spans millennia of civilizational evolution, encapsulating intricate iconography, institutional rituals, philosophical metaphors, and material folklore that often require specialized human expertise to accurately interpret. The persistent lack of deep cultural grounding severely circumscribes the global inclusivity and real-world applicability of current multimodal systems.
While several recent multilingual and multicultural benchmarks (such as MaRVL, CVQA, and CultureVQA) have sought to broaden evaluation scopes beyond standard Western imagery, their coverage of Chinese culture remains superficial, predominantly restricted to contemporary landmarks or high-frequency festive tags. A few domain-specific datasets (such as Pun Rebus Art Dataset focusing exclusively on visual pun rebuses in ancient paintings, and CII-Bench targeting artistic aesthetic perception) are narrowly focused and structurally restricted. Creating a comprehensive, authentic, and rigorously controlled benchmark in this space presents severe methodological challenges: historically authentic artifacts and ritual depictions are visually scarce and require specialized annotation, and standard Visual Question Answering (VQA) setups frequently suffer from pervasive language priors, where models exploit cultural entity hints leaked within the text query to guess correct answers without engaging in genuine visual perception.
To bridge this fundamental research gap, this paper introduces TCC-Bench, the first systematic bilingual VQA benchmark specifically constructed to evaluate MLLMs on traditional Chinese cultural knowledge. Drawing upon canonical scholarship and folkloric taxonomy, the authors formulate eight balanced domains encompassing daily material culture ("Clothing, Diet, Architecture, Transportation") and high-level intellectual rituals ("Astronomy, Music, Custom, Artifact"), curated from 770 verified, open-licensed cultural images across museums and historical records. To reconcile annotation scalability with benchmark integrity, the authors engineer a semi-automated pipeline that combines text-only LLM generation, human visual decoupling, and two-round cross-verification. Core idea: by strictly disabling the multimodal input of the generator model to prevent self-alignment leakage and manually replacing textual entity names with visual deictic references, the benchmark eliminates language bias shortcuts and establishes an authentic vision-dependent testbed for deep cultural reasoning.
Method¶
Overall Architecture¶
TCC-Bench evaluates the capacity of MLLMs to perceive culturally grounded visual elements, interpret ritual and artistic significance, and reason about historical implications. The benchmark construction workflow is organized into four interconnected stages: cultural concept extraction, text-only question candidate generation, manual visual decoupling refinement, and two-round independent human verification. The entire architecture enforces two core principles—language shortcut elimination and strict vision-knowledge interdependence—ensuring that every test instance requires joint fine-grained visual perception and cultural knowledge retrieval.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: 1,830 Open/Museum Candidate Images"] --> B["Cultural Taxonomy Mapping & Concept Extraction<br/>Curate 770 authentic images across 8 domains; extract core concepts"]
B --> C["Text-Only Constrained Generation & Leakage Prevention<br/>Disable GPT-4o vision; prompt text-only model to generate 3 MCQs"]
C --> D["Manual Visual Decoupling & Shortcut Mitigation<br/>Rewrite explicit entity names into visual deictic references"]
D --> E["Two-Round Independent Cross-Verification<br/>Double-blind review by cultural experts yielding 1,015 bilingual MCQs"]
E --> F["Benchmark & Evaluation Protocol<br/>Bilingual evaluation, CoT analysis, LLM-as-a-judge consistency, domain prompting"]
Key Designs¶
1. Cultural Taxonomy Mapping & Concept Extraction: Addressing domain fragmentation and verifying visual evidence Traditional Chinese culture is remarkably broad, and without a systematic ontological structure, any benchmark risks degenerating into an arbitrary collection of trivia. Grounded in authoritative historical literature and expert consultations, the authors formalize an eight-domain taxonomy. This begins with the classical four pillars of everyday social existence—Yi-Shi-Zhu-Xing (Clothing, Diet, Architecture, Transportation)—and extends to four foundational intellectual and ritual domains: Astronomy, Music, Custom, and Artifact. Six native annotators holding bachelor's degrees or higher and possessing lifelong cultural immersion gathered an initial corpus of 1,830 images from open-source repositories and museum digital archives (e.g., National Palace Museum, Wikimedia Commons). Applying strict filtering criteria regarding unambiguous cultural representation, verified visual evidence, and open-use licensing, they distilled 770 pristine images and manually extracted exactly one core, visually grounded cultural entity per image (e.g., sundial, Vermilion Bird, bianzhong chime bells, horse-face skirt).
2. Text-Only Constrained Generation & Leakage Prevention: Preventing artificial performance inflation from multimodal self-alignment Generating questions via multimodal models (e.g., GPT-4o Vision) frequently results in implicit data leakage: the model naturally poses questions that align with its own visual encoder features and recognized cues, artificially inflating its subsequent benchmark performance. To eliminate this confounding factor, the authors explicitly disable GPT-4o's multimodal capabilities, operating purely in text-only mode. The model is fed only the human-extracted cultural concept (e.g., "sundial") alongside a system prompt instructing it to act as an expert cultural examiner. It generates at least three candidate multiple-choice questions per concept, complete with one correct answer, three confusing distractors, and concise explanatory rationales in both Chinese and English. This design ensures that candidate questions are generated in a controlled, modality-independent manner while drastically curbing human drafting overhead.
3. Manual Visual Decoupling & Shortcut Mitigation: Neutralizing language priors to mandate genuine visual grounding VQA benchmarks are notoriously vulnerable to language priors, where text-only LLMs can solve questions by exploiting lexical cues without ever seeing the image (e.g., "What is a sundial used for?" trivially gives away the answer). To enforce genuine multimodal reasoning, four dedicated annotators conduct rigorous manual refinement. Crucially, they strip all explicit cultural concept names from the question stems, transforming them into visually grounded deictic references (e.g., rewriting "What is a sundial used for?" into "What is the instrument in the picture used for?"). Furthermore, annotators hand-craft additional challenging questions capturing subtle visual nuances overlooked by LLMs, verify the plausibility of distractors, and polish bilingual translations, ensuring the final questions strictly necessitate visual evidence to solve.
4. Two-Round Independent Cross-Verification: Enforcing factual precision and bilingual consistency Cultural and historical facts demand absolute accuracy, as inaccuracies in dynasties, technical terminology, or ritual contexts severely damage benchmark validity. The remaining two annotators act as independent verifiers, performing two full rounds of double-blind review across all refined samples. Evaluators judge four strict criteria: domain alignment, question/answer factual correctness, explanation validity, and complete absence of concept leakage. Any question failing either round is discarded without revision, filtering out 13 sub-standard candidates. The finalized TCC-Bench comprises 770 images and 1,015 high-quality four-choice questions (averaging 1.32 questions per image) with full bilingual parity. Question stems average 14.8 Chinese characters / 11.9 English words, options average 3.3 characters / 2.4 words, and explanations average 19.3 characters / 16.2 words, yielding a robust evaluation standard.
Benchmark & Evaluation Protocol¶
The benchmark assesses 12 open-source MLLMs (ranging from 3B to 72B parameters across DeepSeek-VL2, LLaVA-1.6, Qwen2.5-VL, InternVL3.5, GLM-4V, and CogVLM2) and 3 leading proprietary API models (GPT-4.1, Gemini-2.5 Flash, and Gemini-2.5 Pro) on standardized zero-shot multiple-choice prompts with temperature set to 0. The evaluation framework encompasses: - Bilingual Accuracy: Evaluating overall and per-domain accuracy under native Chinese vs. translated English prompts; - Text-Only Baseline Comparison: Testing GPT-4.1 without image inputs to empirically quantify visual dependency and language prior resistance; - Chain-of-Thought (CoT) Prompting: Examining whether eliciting step-by-step reasoning enhances or degrades cultural inference across model scales; - Explanation Consistency via LLM-as-a-Judge: Prompting models to generate rationales alongside answer choices, scored on a 0-3 rubric (3=Excellent, 2=Good, 1=Fair, 0=Poor/Hallucinatory) by GPT-4.1 against ground-truth cultural explanations; - Domain-Aware Prompting: Augmenting input prompts with explicit cultural domain labels to examine the effect of category-level priors.
Key Experimental Results¶
Main Results: Comprehensive MLLM Evaluation Across Cultural Domains¶
The main experiment benchmarks all 15 MLLMs against a human baseline established by 3 native Chinese speakers (evaluating 15 random questions per domain, 120 total).
| Category | Model | Overall Acc (ZH) | Overall Acc (EN) | Astr. (ZH/EN) | Music (ZH/EN) | Custom (ZH/EN) | Arch. (ZH/EN) | Tran. (ZH/EN) | Diet (ZH/EN) | Clot. (ZH/EN) | Arti. (ZH/EN) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Human Baseline | Human Avg. | 85.00 | — | 73.33 / — | 80.00 / — | 93.33 / — | 80.00 / — | 86.67 / — | 93.33 / — | 86.67 / — | 86.67 / — |
| Text-Only Baseline | GPT-4.1 (Text-only) | 23.35 | 34.09 | 10.00 / 20.00 | 26.32 / 2.63 | 22.52 / 35.76 | 30.88 / 39.71 | 29.73 / 40.54 | 13.33 / 20.00 | 32.65 / 34.01 | 25.90 / 42.63 |
| Open-Source (<10B) | DeepSeek-VL2-3B | 60.69 | 55.67 | 38.57 / 37.14 | 60.26 / 50.00 | 68.87 / 60.93 | 58.09 / 43.38 | 56.76 / 48.65 | 64.00 / 61.33 | 58.50 / 58.50 | 70.52 / 68.92 |
| LLaVA-1.6-7B | 42.66 | 46.01 | 24.29 / 25.71 | 29.49 / 46.15 | 48.34 / 49.67 | 46.32 / 39.71 | 35.14 / 40.54 | 42.67 / 46.67 | 45.58 / 48.30 | 51.00 / 57.77 | |
| Qwen2.5-VL-7B | 73.60 | 69.36 | 54.29 / 41.43 | 73.08 / 65.38 | 77.48 / 75.50 | 81.62 / 75.00 | 67.57 / 72.97 | 81.33 / 76.00 | 65.99 / 62.59 | 80.88 / 80.88 | |
| GLM-4V-9B | 51.23 | 46.11 | 39.29 / 26.43 | 46.15 / 44.87 | 61.59 / 56.29 | 51.47 / 38.24 | 56.76 / 51.35 | 54.67 / 53.33 | 43.54 / 40.82 | 55.78 / 55.78 | |
| InternVL3.5-8B | 69.06 | 60.49 | 50.71 / 43.57 | 71.79 / 61.54 | 75.50 / 68.87 | 63.97 / 50.00 | 70.27 / 51.35 | 78.67 / 73.33 | 61.22 / 57.14 | 78.88 / 69.72 | |
| Open-Source (>10B) | DeepSeek-VL2-16B | 67.68 | 63.74 | 37.86 / 41.43 | 69.23 / 65.38 | 74.17 / 74.17 | 67.65 / 52.21 | 62.16 / 54.05 | 81.33 / 73.33 | 64.63 / 59.18 | 78.49 / 76.89 |
| DeepSeek-VL2-27B | 60.00 | 55.27 | 40.00 / 35.00 | 60.26 / 53.85 | 68.87 / 66.23 | 52.21 / 48.53 | 62.16 / 59.46 | 70.67 / 64.00 | 53.74 / 46.26 | 70.12 / 66.14 | |
| CogVLM2-19B | 57.64 | 54.88 | 36.43 / 32.14 | 53.85 / 47.44 | 60.93 / 57.62 | 54.41 / 45.59 | 67.57 / 54.05 | 57.33 / 62.67 | 57.82 / 53.06 | 68.92 / 72.11 | |
| Qwen2.5-VL-32B | 75.27 | 70.44 | 52.14 / 45.00 | 71.79 / 67.95 | 84.11 / 77.48 | 82.35 / 76.47 | 70.27 / 72.97 | 85.33 / 81.33 | 64.63 / 61.22 | 84.06 / 79.68 | |
| LLaVA-1.6-34B | 63.05 | 58.42 | 23.57 / 32.14 | 73.08 / 61.54 | 72.85 / 68.21 | 63.97 / 52.94 | 62.16 / 59.46 | 72.00 / 68.00 | 63.95 / 55.78 | 72.51 / 67.73 | |
| InternVL3.5-38B | 73.40 | 66.90 | 51.43 / 41.43 | 76.92 / 74.36 | 82.78 / 70.20 | 72.06 / 69.12 | 86.49 / 75.68 | 81.33 / 78.67 | 67.35 / 58.50 | 78.88 / 75.70 | |
| Qwen2.5-VL-72B | 75.17 | 71.03 | 56.43 / 47.86 | 80.77 / 65.38 | 84.11 / 81.46 | 76.47 / 75.00 | 75.68 / 81.08 | 76.00 / 78.67 | 70.07 / 61.22 | 80.48 / 79.28 | |
| Proprietary | GPT-4.1 | 76.65 | 73.89 | 44.29 / 44.29 | 82.05 / 80.77 | 85.43 / 81.46 | 80.88 / 77.94 | 81.08 / 70.27 | 89.33 / 93.33 | 72.11 / 62.59 | 83.67 / 82.87 |
| Gemini-2.5 Flash | 84.93 | 80.89 | 77.14 / 72.14 | 87.18 / 84.62 | 92.72 / 88.08 | 88.97 / 84.56 | 83.78 / 83.78 | 94.67 / 93.33 | 72.11 / 64.63 | 86.45 / 83.67 | |
| Gemini-2.5 Pro | 87.00 | 83.55 | 85.00 / 83.57 | 88.46 / 85.90 | 95.36 / 90.07 | 88.97 / 87.50 | 81.08 / 75.68 | 96.00 / 93.33 | 72.79 / 65.99 | 88.05 / 85.26 |
Ablation & In-Depth Analysis¶
1. Explanation Consistency Scores (LLM-as-a-Judge: 0-3 Scale)¶
Evaluating model rationales against verified cultural reference explanations:
| Model Name | Overall Consistency | Astr. | Music | Custom | Arch. | Tran. | Diet | Clot. | Arti. |
|---|---|---|---|---|---|---|---|---|---|
| LLaVA-1.6-7B | 0.85 | 0.43 | 0.51 | 0.94 | 0.88 | 1.00 | 0.69 | 1.07 | 1.03 |
| DeepSeek-VL2-27B | 1.53 | 0.94 | 1.56 | 1.90 | 1.44 | 1.54 | 1.89 | 1.37 | 1.64 |
| InternVL3.5-8B | 1.84 | 1.25 | 1.69 | 2.11 | 1.90 | 1.51 | 2.29 | 1.73 | 2.01 |
| Qwen2.5-VL-72B | 2.14 | 1.65 | 2.01 | 2.42 | 2.24 | 2.16 | 2.36 | 1.91 | 2.33 |
| GPT-4.1 | 2.23 | 1.22 | 2.29 | 2.50 | 2.39 | 2.32 | 2.77 | 2.12 | 2.43 |
| Gemini-2.5 Pro | 2.60 | 2.61 | 2.57 | 2.83 | 2.66 | 2.48 | 2.74 | 2.31 | 2.59 |
2. Domain-Aware Prompting Impact¶
Comparing standard domain-agnostic prompting against prompts explicitly injecting domain labels:
| Model | Domain-Agnostic Acc (%) | Domain-Aware Acc (%) | Delta (Δ) | Observation |
|---|---|---|---|---|
| LLaVA-1.6-7B | 42.66 | 43.74 | +1.08% | Marginal gain; baseline capacity remains low |
| Gemini-2.5 Pro | 87.00 | 87.78 | +0.78% | Stable high-tier performance |
| GPT-4.1 | 76.65 | 76.26 | -0.39% | Virtually unchanged |
| Qwen2.5-VL-72B | 75.17 | 74.68 | -0.49% | Domain label provides no empirical benefit |
| InternVL3.5-8B | 69.06 | 66.50 | -2.56% | Explicit label distracts cross-modal attention |
| DeepSeek-VL2-27B | 60.00 | 51.53 | -8.47% | Severe degradation from negative categorical bias |
Key Findings¶
- Language Shortcut Immunity: Disabling visual inputs causes GPT-4.1's accuracy to drop precipitously by 53.30% in Chinese (plummeting to 23.35%, below random guessing) and by 39.80% in English (to 34.09%). This confirms that TCC-Bench successfully prevents language prior exploitation; models cannot solve items through textual memorization alone.
- Pronounced Proprietary vs. Open-Source Divide: Closed-source models form a superior, compact cluster. Gemini-2.5 Pro establishes the state-of-the-art across all benchmarks, reaching 87.00% in Chinese and 83.55% in English, outperforming the native human average (85.00%). Among open-source alternatives, the Qwen2.5-VL family dominates, with Qwen2.5-VL-32B (75.27% ZH) and 72B (71.03% EN) leading, whereas sub-10B models fluctuate wildly and trail proprietary leaders by over 15%.
- Extreme Disparities Across Domains: All models achieve high accuracy in daily-life domains like Custom (up to 95.36%) and Diet (up to 96.00%), which frequently appear in modern digital training corpora. Conversely, performance collapses in specialized, archival domains like Astronomy (e.g., LLaVA-1.6-7B achieves only 24.29%, and open-source models average 24%–56%), reflecting severe historical data sparsity in pre-training.
- Double-Edged Nature of Chain-of-Thought: CoT prompting produces scale-dependent divergence. In under-parameterized models like LLaVA-1.6-7B, CoT causes severe degradation (dropping from 42.66% to 35.37%, -7.29%) due to compounding perceptual hallucination. In Qwen2.5-VL-72B, it provides a +2.47% boost (75.17% to 77.64%), while in leading proprietary models (GPT-4.1 and Gemini-2.5 Pro), its impact is negligible (+0.30% and -0.69%).
- Fine-Grained Error Attribution (Qwen2.5-VL-72B CoT Audit): Manual error breakdown reveals that Lack of Cultural Knowledge is the primary failure mode, accounting for 55.9% of all errors (the model recognizes the physical object but misidentifies its historical function or ritual role). Visual Perceptual Errors constitute 33.9% (e.g., mistaking Sichuan Opera face-changing for Peking Opera folding fan dance). Contextual Interpretation Errors make up 8.4% (e.g., misidentifying a Huabiao due to overlooking coiling dragon reliefs), while pure Reasoning Errors account for only 1.8%.
Highlights & Insights¶
- Modality-Decoupled Benchmark Generation: Enforcing text-only candidate generation prevents multimodal generator models from embedding visual inductive biases, while systematic entity-masking in question stems depresses text-only performance to chance level (23.35%), setting a rigorous precedent for cultural VQA design.
- Beyond Multiple-Choice: Explanation Consistency Scoring: Introducing LLM-as-a-judge rationale scoring demonstrates that smaller models often pick correct options via superficial heuristics without possessing the factual capability to generate culturally coherent explanations.
- Methodological Implications for Multimodal Alignment: The failure of domain-aware prompting and the instability of CoT prove that cultural comprehension cannot be patched via surface-level prompting tricks; foundational progress demands diversifying pre-training corpora with non-Western historical-cultural data and specialized multimodal fine-tuning.
Limitations & Future Work¶
- High Human Curation Costs: Despite leveraging GPT-4o for initial candidate drafting, maintaining high cultural fidelity and zero entity leakage requires extensive expert intervention, constraining low-cost scaling to tens of thousands of instances.
- Historical Visual Data Scarcity: Many foundational cultural elements predate photography, necessitating reliance on museum artifacts, archeological relics, and classical paintings. This introduces visual domain shifts and leaves regional and ethnic minority folk cultures underrepresented.
- Format Constraints: The benchmark is currently restricted to four-option multiple-choice VQA, lacking open-ended visual dialogue, fine-grained cultural grounding, and multi-image historical sequencing tasks. Future research should target automated high-fidelity generation and open-ended evaluation protocols.
Related Work & Insights¶
- vs MaRVL / CVQA / CultureVQA: While existing multicultural datasets span broad geographic areas, their Chinese cultural coverage is limited to contemporary surface imagery. TCC-Bench delves deeply into 5,000 years of civilization across eight structured domains, establishing an unprecedented depth of cultural evaluation.
- vs Pun Rebus Art Dataset / CII-Bench: Earlier Chinese culture datasets focus exclusively on visual puns or traditional painting aesthetics. TCC-Bench spans the entirety of material, daily, and intellectual culture with a verified anti-shortcut pipeline that eliminates language prior artifacts.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First systematic multimodal benchmark targeting traditional Chinese culture across eight balanced domains with an innovative text-only decoupled generation pipeline.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 15 proprietary and open-source models, paired with text-only ablations, CoT analysis, LLM-as-a-judge rationale scoring, and four-way manual error categorization.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly structured, articulate prose with precise taxonomy descriptions and clean qualitative case studies.
- Value: ⭐⭐⭐⭐⭐ Crucial benchmark for exposing Western-centric bias in contemporary multimodal foundation models, accelerating the development of globally inclusive AI.