Skip to content

Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models

Conference: ACL 2026
arXiv: 2601.07984
Code: https://github.com/yha9806/VULCA-Framework
Area: Multimodal VLM / Cultural Evaluation / Art Critique
Keywords: Cross-cultural evaluation, art critique, VLM-as-Judge, human calibration, Vulca-Bench

TL;DR

The paper proposes the Vulca-Bench three-tier evaluation framework (automated metrics + single-judge scoring + human sigmoid calibration), covering 6 major art traditions, 165 cultural dimensions, and L1–L5 levels ranging from "visual description" to "cultural interpretation." For the first time, it quantitatively reveals that across 15 VLMs, "models experience significant performance drops in deep cultural interpretation and exhibit a systematic preference for Western art."

Background & Motivation

Background: Current evaluations of VLM cultural capabilities primarily focus on the perception layer (VQAv2, POPE, MME, SEED-Bench). Third-generation cultural probes (CulturalBench, CulturalVQA, GIMMICK), while introducing multi-national backgrounds, remain closed-ended QA, testing whether a model "recognizes a cultural symbol" rather than "interprets a painting like an art critic."

Limitations of Prior Work: When VLMs are applied to open-ended generation tasks like art critique, evaluation methods fail completely. Automated metrics (like BLEU/ROUGE) only match superficial keywords; LLM-as-Judge using the average of two judges suffers from severe scale inconsistency (the authors measured a cross-judge ICC(2,1) as low as \(-0.50\)); and mono-cultural studies (e.g., evaluating only Chinese paintings) cannot isolate "culture-specific difficulty" from "systematic Western bias."

Key Challenge: The evaluation construct (depth of cultural understanding) and the evaluation mechanism (automated metrics / judges / humans) are confounded. A score of 0.x is often used to claim "the model understands culture" without clarifying which capability level is being measured or verifying the reliability of the measurement itself.

Goal: (1) Provide a verifiable hierarchical definition of cultural understanding; (2) Verify the reliability of various evaluation proxies (automated metrics / LLM judge) across cultural depths; (3) Use these validated tools to diagnose the true performance of 15 SOTA VLMs across 6 cultures.

Key Insight: The authors borrow from classic art theories: Panofsky's three stages of iconology (pre-iconographic description / iconographic analysis / iconological interpretation) and Goodman's theory of notation. These correspond to five empirically separable capability levels: "L1 Visual Perception → L2 Technical Analysis → L3 Cultural Symbols → L4 Historical Context → L5 Philosophical Aesthetics." Essentially, L1–L2 focus on whether the VLM can "see," while L3–L5 focus on whether it can "understand."

Core Idea: Strictly distinguish between "Levels (L1–L5, the target construct)" and "Tiers (Tier I/II/III, the measurement mechanism)," explicitly stating which L-levels each Tier measures. Furthermore, aggregate scores from a single judge are anchored to human expert ratings via sigmoid calibration, avoiding the non-convergence trap of multi-judge averaging.

Method

Overall Architecture

The input to Vulca-Bench is a triplet: (i) an image of an artwork, (ii) a cultural label \(k\) (chosen from Chinese, Western, Japanese, Korean, Islamic, or Indian), and (iii) a bilingual expert reference critique. After receiving images compressed to \(\leq 3.75\)MB, the target VLM generates bilingual (CN/EN) critiques \(c\) across levels L1–L5 using a unified prompt. \(c\) then enters three parallel/serial Tiers:

  • Tier I (Automated Metrics): Four metrics (DCR / CSA / CDS / LQS) are calculated on \(c\) without requiring judges, serving as "risk signals" rather than ranking indicators.
  • Tier II (Single-Judge Scoring): Using Claude Opus 4.5 as the sole judge and the expert critique as a reference anchor (rather than a gold answer), the model is scored 1–5 across 5 dimensions: Coverage, Alignment, Depth, Accuracy, and Quality.
  • Tier III (Human Calibration): A sigmoid function \(S_{\text{II}}^{*}=1+4\sigma(a\cdot S_{\text{II}}+b)\) is fitted using 295 human-scored samples to map Tier II aggregate scores back to the \([1,5]\) range and align them with human scoring standards.

The pipeline finally outputs: (a) calibrated aggregate scores \(S_{\text{II}}^{*}\), (b) fine-grained diagnostics across 5 dimensions, and (c) Tier I risk flags (e.g., low cultural coverage, weak semantic alignment, high templating risk).

Key Designs

  1. Orthogonal Decomposition of L1–L5 Levels × Tier I/II/III Mechanisms:

    • Function: Explicitly separates the "depth of capability tested" from the "measurement method." Each metric in Table 2 is explicitly labeled with the L1–L2 / L3–L5 layers it covers.
    • Mechanism: DCR/CSA are keyword matches providing surface signals for L1–L5; CDS uses weighting \(w_\ell=\ell/15\) (L1 weight \(1/15\), L5 weight \(5/15\)) to emphasize deep layers; LQS measures linguistic fluency orthogonal to cultural depth. Tier II similarly distinguishes: Depth and Alignment target L3–L5, Coverage and Accuracy span all layers, and Quality is layer-agnostic.
    • Design Motivation: Previous cultural evaluations mixed these dimensions into a single score, allowing high L1–L2 scores to mask failures in L3–L5. This orthogonal decomposition enables the quantitative diagnosis of the "sees accurately but understands poorly" phenomenon.

  2. Single Judge + Sigmoid Calibration (Replacing Multi-Judge Averaging):

    • Function: Circumvents scale drift issues from multi-judge averaging while maintaining the scalability of LLM-as-Judge.
    • Mechanism: The authors tested 8 candidate judges and found systematic shifts: OpenAI models (GPT-4o mean 4.52) tilted lenient, while Anthropic models (Claude Opus 4.5 mean 3.42) tilted strict. Cross-judge ICC(2,1) drifted between \(-0.50\) and \(0.12\), all below the 0.6 threshold. By using only Claude Opus 4.5 and fitting \(a,b\) on 295 samples to minimize MSE against human scores, they ensured stability. Claude Opus 4.5 was selected for its stable rank discrimination, consistent cultural bias direction, and lack of self-favoritism.
    • Design Motivation: Averaging strict and lenient judges results in a shift toward the median rather than true consensus. The sigmoid transformation is monotonic and reversible, preserving rankings while explicitly anchoring "model scores" to "expert scores" to provide interpretable benchmarks.

  3. 165 Culture-Specific Dimensions + Reference-Guided Bilingual Critiques:

    • Function: Allows the judge to determine "correct attribution" rather than mere "mention" by providing both the VLM output and an expert bilingual critique of the same theme.
    • Mechanism: The authors identified 165 culture-specific dimensions across 6 traditions (e.g., 30 for China, 30 for India, 27 for Japan). Each dimension uses explicit tags like CN_L1_D1 in expert references. VLMs must output in both Chinese (to preserve untranslatable terms like "Qiyun") and English. The judge checks if the VLM correctly utilizes the specific cultural terminology at the appropriate level.
    • Design Motivation: Standard LLM-judges scoring free-form generations essentially measure "fluency." Using a reference anchor with dimensional tags anchors the score to "alignment with an expert's specific interpretation at layer L," turning subjective scoring into objective alignment evaluation.

Loss & Training

Ours does not train the VLM; it only trains the two sigmoid parameters \((a,b)\) for Tier III by minimizing \(\text{MSE}(S_{\text{II}}^{*}, S_h)\), where \(S_h\) is the mean human score for the 295 training samples. The individual 5-dimension scores in Tier II are not calibrated to maintain diagnostic granularity. The judge temperature is set to the provider default (\(T=1.0\)), which introduces some non-determinism, though the JSON-restricted integer scoring template suppresses variance.

Key Experimental Results

Main Results

15 VLMs × 294 samples × 6 cultures = 4,405 model-sample evaluations (5 excluded due to parsing failure, 0.11%). The table below shows Tier II dimensions + calibrated \(S_{\text{II}}^{*}\) scores (top/mid/bottom selection):

Model \(S_{\text{II}}^{*}\) Coverage Alignment Depth Accuracy Quality
Gemini-2.5-Pro 4.27 4.49 4.26 4.38 3.56 4.55
Qwen3-VL-235B 4.21 4.49 4.10 4.41 3.33 4.51
Claude-Sonnet-4.5 4.11 4.29 4.05 4.00 3.44 4.48
GPT-5 4.00 4.23 3.48 4.04 3.85 4.08
Llama4-Scout 3.67 4.21 3.48 3.36 2.96 4.10
GPT-4o 3.57 3.88 3.38 3.21 3.09 4.10
GPT-4o-mini 3.24 3.76 2.94 2.93 2.90 3.76
DeepSeek-VL2 3.01 3.50 2.74 2.64 2.72 3.78
Var \(\sigma\) — 0.33 0.48 0.56 0.35 0.24
  • Top-tier (top 3) and bottom-tier (bottom 3) models do not overlap under bootstrap 95% CI (\(p<0.001\), permutation test). Middle rankings with overlapping CIs should be viewed as "performance tiers" rather than strict rankings.
  • Depth and Alignment exhibit the highest discriminative power (\(\sigma=0.56\) / \(0.48\)), while Quality has the lowest variance (\(\sigma=0.24\)), confirming that fluency is not the bottleneck, but deep cultural understanding is.

Ablation Study

Config Key Metric Note
Single Judge + Sigmoid (Ours) MAE 0.446 (held-out \(n=155\)) Reduced MAE by 1.7% compared to uncalibrated aggregate score
Double Judge (Claude-Opus + GPT-5) cross-judge ICC(2,1) = \(-0.50\) Systematic scale inconsistency make scores unreliable
Double Judge (Claude-Sonnet + GPT-5) ICC(2,1) = \(0.12\) Still significantly below the 0.6 threshold
DCR\(_\text{auto}\) vs Tier II Judge ICC = 0.02 / Pearson \(r=0.53\) Keyword coverage is nearly uncorrelated with semantic understanding
CSA\(_\text{auto}\) vs Judge ICC = 0.17 / \(r=0.44\) Weak correlation with both judge and human gold standards
CDS\(_\text{auto}\) vs Judge ICC = 0.18 / \(r=0.51\) Moderate correlation but underestimates true cultural alignment
LQS\(_\text{auto}\) vs Judge ICC = \(-0.17\) / \(r=0.27\) Fluency direction is opposite to cultural depth

Key Findings

  • Monotonic Collapse from L1–L2 to L3–L5: All 15 VLMs are strong at the perception layer (Coverage \(\geq 3.50\)) but drop sharply in interpretation (Alignment/Depth). DeepSeek-VL2 shows a gap of nearly 1 point between Coverage (3.50) and Depth (2.64), validating the hypothesis that image-caption training provides description skills but fails at cultural grounding.
  • Systematic Western Bias in 13/15 Models: The mean difference between Chinese and Western scores is \(-0.39\) (Cohen's \(d=-0.74\), \(p<0.001\)). GPT-4o-mini is the most biased (\(\Delta=-1.08\)), while GPT-5.2 is the most neutral (\(\Delta=+0.07\)).
  • Dual Confirmation for Bias: The gap widens to \(d=-0.93\) in the landscape genre subset (controlling for subject matter). In a blind-culture setting (removing cultural labels), the gap \(\Delta_{\text{blind}}=-0.61\) is larger than \(\Delta_{\text{std}}=-0.54\). These counter-proofs indicate that "Western bias" originates from VLM training distributions rather than genre distribution or judge leakage.
  • Automated Metrics vs. Judges Measure Different Constructs: All 4 Tier I metrics show ICC \(< 0.2\) against Tier II, with DCR at only 0.02. This strongly critiques the use of BLEU/ROUGE for evaluating cultural generation.

Highlights & Insights

  • Orthogonal Decomposition of Construct/Mechanism is an Underestimated Methodology: In subjective tasks like culture/alignment, research often conflates "what is being measured" with "how it is measured." Table 2 in this paper maps every indicator to L1–L5, providing a blueprint for future VLM cultural evaluations.
  • Single Judge + Sigmoid is an Excellent Engineering Trade-off: While multi-judge setups theoretically reduce variance, the measured negative ICC proves that scale mismatch outweighs variance gains. Using one judge and calibrating for bias retains scalability while obtaining an "absolute scale," a trick transferable to almost any LLM-as-Judge scenario.
  • Increased Bias in Blind-Culture Settings is a Crucial Counter-Intuitive Finding: Using a small 50-sample ablation to show that bias increases when labels are removed clarifies that the fault lies in the VLM's training distribution rather than the evaluation design.
  • 165-Dimension Fine-Grained Tags are a Data-Level Achievement: Attaching hierarchy-dimensional tags (e.g., CN_L1_D1) to expert critiques allows Tier I keyword matches and Tier II alignment. This "schema-driven evaluation data" approach can be extended to any open-ended task requiring hierarchical assessment (e.g., medical diagnosis, legal opinions).

Limitations & Future Work

  • Sample imbalance: Chinese and Western art account for 91% of samples. For minority cultures (Korean \(n=16\), Islamic \(n=18\)), calibration MAE rises by over 6%, indicating degradation in sparse data scenarios.
  • Multi-lingual limitations: Bilingualism is currently restricted to Chinese-English. Native terms from Japanese/Korean/Arabic/Hindi (e.g., wabi-sabi, jeong, rasa) are only partially preserved via romanization, resulting in systematic translation loss.
  • Reliance on Claude Opus 4.5: There is a single-point failure risk of the judge. Temperature \(T=1.0\) causes score fluctuations of \(\pm 0.02\), which may affect the stability of closely ranked models.
  • Rubric grain size: The 1–5 integer rubric limits expressiveness. A pilot with a 0–5 scale showed MAE rising to 0.4870, suggesting rubric design is a non-trivial hyperparameter.
  • ICL failure: Adding 1–3 expert critiques as few-shot exemplars caused performance drops, likely due to attention dilution or style overfitting. This implies that simple ICL is insufficient for cultural interpretation; specialized reasoning scaffolds (e.g., L1→L5 retrieval-augmented exemplars) are needed.
  • Self-assessment: This method is currently a diagnostic tool rather than an improvement tool. The next step is to integrate these metrics into RLHF/SFT reward signals.
  • vs. CulturalBench/CulturalVQA/GIMMICK: These focus on closed QA (identification), while Ours focuses on open generation (expert-level critique), providing the only framework covering cross-culture + L1–L5 full-spectrum + human calibration.
  • vs. GalleryGPT/Strafforello et al. 2024: Previous art VLMs focused on L1–L2 (style classification). This work pushes evaluation to L3–L5 and quantifies the universal failure to reach L3.
  • vs. G-Eval/MT-Bench/Prometheus-Vision: These LLM-as-Judge papers were validated on general English tasks. This work reveals that multi-judge averaging collapses on culturally sensitive tasks, necessitating single-judge calibration. This conclusion should be adopted by the cross-cultural NLG evaluation community.
  • vs. Mono-cultural Yu et al. 2025: Previous works found VLM-expert discrepancies only in Chinese or non-Western paintings but could not decouple "cultural difficulty" from "Western bias." Using 6 cultures and blind control makes this the paradigmatic model for upgrading mono-cultural studies to cross-cultural research.

Rating

  • Novelty: ⭐⭐⭐⭐ The combination of cross-culture × L1–L5 × 3-tier evaluation is a first; the approach is solid but not purely disruptive.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ 15 VLMs × 6 cultures × 294 samples + 450 human ratings + bootstrap CI + blind-culture + genre-control ablations; the empirical rigor is top-tier.
  • Writing Quality: ⭐⭐⭐⭐⭐ RQ–Contribution correspondence is clear, Tier vs. Level is strictly distinguished, and appendices are comprehensive.
  • Value: ⭐⭐⭐⭐⭐ Provides a reusable evaluation protocol, 165 cultural dimensions, and hard evidence regarding the unreliability of multi-judge averaging.