C3-Bench: A Context-Aware Change Captioning Benchmark¶

Conference: ECCV2026
arXiv: 2606.25445
Code: https://github.com/AutoCompSysLab/C3-Bench
Area: Multimodal VLM / Image Generation
Keywords: Change Captioning, Context-Aware, Benchmark, LLM Evaluation, Reversibility

TL;DR¶

C3-Bench is a comprehensive benchmark for the context-aware change captioning task. It features 4,996 human-annotated image pairs spanning 51 real-world change contexts across 4 major domains. It introduces the LLM-as-Judge framework and a reversibility metric to change captioning evaluation for the first time. Benchmarking 32 models reveals critical blind spots, such as the complete failure of traditional models in open-world scenarios and systematic perceptual, spatial, and positional biases in large multimodal models (LMMs).

Background & Motivation¶

The task of change captioning is to automatically generate textual descriptions of changes between a pair of "before" and "after" images. This capability is crucial in downstream applications such as disaster monitoring, autonomous driving, remote sensing analysis, and media forensics. However, the concept of "change" is inherently ambiguous—given the same pair of images, the correct answer to "what changed" depends entirely on the context. From a weather monitoring perspective, describing "snow covering the ground, clouds clearing" is valid. From a railway monitoring perspective, describing "a train appearing on the left track" is correct, while weather differences should be treated as distractor pseudo-changes. Currently, dominant benchmarks (e.g., LEVIR-CC, SpotTheDiff) mostly ignore this context dependency. They either construct image pairs within a single, narrow domain (e.g., LEVIR-CC only covers block-level residential areas in remote sensing) or allow annotators to describe changes arbitrarily without a well-defined criterion. Consequently, models trained on them only learn datasets' implicit semantic biases rather than genuinely understanding changes. Meanwhile, existing evaluation metrics primarily rely on fixed reference-matching like BLEU, ROUGE, or BERTScore, failing to capture correct but semantically equivalent expressions—a problem particularly pronounced for flexible Large Multimodal Models (LMMs).

This dual limitation of "narrow context coverage + coarse evaluation granularity" has prevented a systematic validation of the real-world generalization capabilities of change captioning systems. The core insight of this paper is that for change captioning evaluation to be practically meaningful, one must reformulate the problem—shifting the definition of "change" from an unconditional, implicit assumption to an explicitly specified, controllable context criterion. Based on this formalization, the authors propose C3-Bench (Context-aware Change Captioning Benchmark), addressing gaps in data coverage, evaluation dimensions, and model benchmarking simultaneously. Core Idea: Formulate change captioning with explicit change criteria, construct an annotated benchmark spanning 51 real-world change contexts across 4 major domains, and introduce fine-grained LLM-as-Judge evaluation alongside reversibility metrics, thereby systematically revealing existing models' systematic blind spots in open-world change understanding.

Method¶

Overall Architecture¶

C3-Bench is not an algorithm, but a repository of benchmark designs and evaluation frameworks. Built on the theoretical foundation of a new context-aware change captioning formulation, it features a dataset of 4,996 annotated pairs covering 51 real-world change contexts, a set of fine-grained automatic evaluation metrics via LLM-as-Judge (Correctness, Specificity, Fluency, and Relevance), and a unique Reversibility metric that tests if the model's descriptions remain symmetric and consistent when the order of the images is swapped. The entire benchmark is constructed via an efficient pipeline that outputs high-quality annotations and leverages unified prompt templates to ensure fair comparisons between traditional change captioning models and LMMs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Problem Formulation<br/>Context-aware CC"] --> B["Data Construction Pipeline"]
    B --> C["4 Domains × 51 Contexts<br/>4,996 Image Pairs + Criteria"]
    C --> D["Evaluation Framework"]
    D --> E["LLM-as-Judge<br/>4 Dimensions Scoring"]
    D --> F["Reversibility Metric<br/>Symmetric Consistency"]
    C --> G["32 Model Benchmarking"]
    G --> H["Key Findings"]
    F --> G

Key Designs¶

1. Context-Aware Problem Formulation: Making "Change Criteria" an Explicit Parameter

The traditional formalization of change captioning, \(y_{A\rightarrow B} = \arg\max p(y\mid I_A, I_B)\), carries an implicit assumption: that the model inherently knows "what qualifies as a change." In practice, this delegates the semantic boundaries to the implicit distribution of the training data—for instance, a model trained on LEVIR-CC might ignore color differences, whereas one trained on CLEVR-CC would flag them as major changes. C3-Bench reformulates this as \(y^c_{A\rightarrow B} = \arg\max p(y\mid I_A, I_B; C_c)\), explicitly introducing the context criterion \(C_c\) as an input parameter. Each \(C_c\) consists of three structured fields: (i) Change to Detect—which objects or state changes are the targets; (ii) Change to Ignore—which differences (e.g., lighting, perspective, season) should be treated as pseudo-changes; (iii) Tone & Nuance—the expected narrative style. This formulation unifies traditional models (where \(C_c\) is empty, degenerating into a special case of implicit model bias) and LMMs (where \(C_c\) is explicitly appended via textual prompts) into a single framework, establishing a foundation for fair comparisons.

2. Multi-Level Data Construction Pipeline: From Cross-Community Context Distillation to Multi-Source Quality Control

The primary challenges in data construction are the high cost of change annotation and the difficulty of acquiring image pairs across diverse contexts while maintaining consistent control. The pipeline in C3-Bench consists of five distinct steps: (a) Context Conceptualization—reviewing literature across communities like change detection, image editing, and anomaly detection to distill 51 highly representative real-world contexts organized into four major domains: Natural Scenes, Remote Sensing, Image Editing, and Anomaly Analysis; (b) Context Formalization—encoding the criterion \(C_c\) for each context using the aforementioned three-field template; (c) Data Collection and Pairing—extracting image pairs from over 15 public benchmarks (e.g., MVTecAD, SECOND, MagicBrush, PASLCD, ChangeVPR) and supplementing missing contexts with high-fidelity, feature-matched pairs from Google Earth and Google Street View via UTM-guided sampling; (d) Annotation—providing annotators with auxiliary metadata (change masks, structural damage polygons, anomaly categories) alongside context criteria to ensure high semantic consistency; (e) Filtering and Quality Control—applying multi-round verification (redundancy filtering, temporal order validation, and description-image alignment checks) before rebalancing contexts to the target volume. This pipeline enables C3-Bench to vastly exceed prior benchmarks in context diversity: LEVIR-CC covers only a single context (remote sensing residential areas), and IER covers 3 contexts, while C3-Bench scales to 51.

3. LLM-as-Judge Evaluation and Reversibility Metrics: From Coarse Word-Overlap to Semantically Aligned Fine-Grained Evaluation

C3-Bench pioneers the integration of the LLM-as-Judge framework into change captioning evaluation. Using GPT-5.2 as the evaluator, the model's generated descriptions are compared with ground-truth references and rated on a 1–10 scale across four dimensions: Correctness (semantic consistency with ground truth), Specificity (detail richness vs. generic descriptions), Fluency (linguistic naturalness), and Relevance (direct addressing of the query), alongside an overall Aggregation score. User studies (with 40 participants) show that the Pearson correlation of these four dimensions with human ratings (\(r \in [0.62, 0.73]\)) dramatically outperforms traditional metrics like BLEU-4 (\(0.33\)) and BERTScore (\(0.45\)). In addition, the Reversibility metric evaluates the model's symmetric understanding by requesting descriptions for both \(A\rightarrow B\) and \(B\rightarrow A\) directions. It uses GPT-5.2 to determine if the reports represent the reverse operations of the same set of modifications (scored binary 0 or 1). While human reversibility reaches \(0.93\), the best LMM (Gemini-3-Pro) only scores \(0.73\), and the best traditional model scores a mere \(0.28\)—uncovering a severe positional bias in models.

A Complete Example: The Highway Context¶

Consider the "highway" context under the Natural Scenes domain. The criterion \(C_c\) explicitly instructs: changes to detect include vehicle movement/entry/exit/modification, structural changes to lanes/guardrails/road signs, and activation/deactivation of electronic signs; changes to ignore include illumination transitions, minor perspective offsets, small photometric/compression artifacts, and weather/seasonal/shadow shifts; the tone should remain objective and neutral. During evaluation, the prompt template injects \(C_c\) into the system prompt of the LMM, labeling the images with <before> and <after> tags to indicate temporal order. The model must produce its description within <output>...</output> tags, which is subsequently evaluated against human annotations. If the model incorrectly describes "the sky getting darker" (which falls under weather changes to ignore), it is penalized for a pseudo-change error.

Loss & Training¶

Not applicable (This is a benchmark paper; traditional models were trained independently on four representative datasets, while LMMs were evaluated zero-shot without fine-tuning).

Key Experimental Results¶

Main Results¶

Model Category	Model	Correctness	Specificity	Aggregation	Reversibility
Human	Human	6.96	7.12	7.45	0.93
Commercial LMM	GPT-5.2	5.47	5.16	5.51	0.62
Commercial LMM	Gemini-3-Pro	5.45	5.15	5.50	0.73
Commercial LMM	GPT-4o	4.95	4.34	4.95	0.49
Open-source LMM	Qwen3-VL-32B	5.18	4.93	5.16	0.47
Open-source LMM	InternVL3-78B	4.86	4.51	4.94	0.45
Traditional Model	DIRL	1.36	1.53	2.30	0.20
Traditional Model	VARD	1.44	1.56	2.40	0.28

Ablation Study¶

(a) Prompt Component Ablation (GPT-5.2)

Configuration	Change Criteria	Temporal Labels	Aggregation	Reversibility
No Criteria, No Labels	✗	✗	4.93	0.58
Criteria Only	✓	✗	5.35	0.60
Criteria + Labels + Forward	✓	✓	5.51	0.62
Criteria + Labels + Backward	✓	✓	5.31	0.62

(b) Criteria Sub-component Ablation

Configuration	Aggregation	Description / Note
Complete Three-Field Template	5.51	Detect + Ignore + Tone
Detect + Ignore Only	5.44	Removing tone has minor impact
Detect + Tone Only	5.49	Removing ignore-changes has minor impact
Detect Only	5.12	Dropped significantly when ignoring changes and tone are omitted

Key Findings¶

Traditional models collapse entirely on C3-Bench (Correctness around \(1.4/10\)), and even the latest model DIRL—which attains SOTA performance on its native training set—is no exception. This demonstrates that they merely capture implicit dataset-specific distributions rather than developing a genuine understanding of visual changes.
LMMs perform significantly better than traditional models in end-to-end evaluation, yet the absolute performance of the best model (GPT-5.2, aggregation score of \(5.51/10\)) remains far below human levels (\(7.45\)). Semantic accuracy remains the primary bottleneck (with fluency being closest to human standards).
Error analysis reveals that Perceptual Errors dominate, accounting for \(63.3\%\) of total failures. Spatial errors are distinct due to perspective discrepancies in natural scenes. In terms of reversibility, Inconsistency Errors are most common overall, except in the anomaly detection domain where Change Collapse Errors dominate—meaning models tend to fixate on anomaly cues and ignore criteria "revisiting the original state".
LMM positional biases are present even with a single pair of images (not restricted to long sequences). The highest reversibility score achieved is only \(0.73\) (Gemini-3-Pro), indicating that the causal attention deficit related to input sequence order is structural.
Multi-dataset joint training actually degrades traditional model performance (aggregation score dropping from \(1.79\rightarrow 1.31\)) because the implicit change criteria of different datasets directly conflict (e.g., color shifts are valid changes in CLEVR-CC but treated as pseudo-changes in LEVIR-CC). This underscores the fundamental necessity of context-aware formalization before attempting multi-source data fusion.

Highlights & Insights¶

Structured Context Criteria: Deconstructing the vague definition of "change" into a structured three-field template (Change to Detect, Change to Ignore, Tone) is simple yet highly scalable. Future extensions only need to define criteria matching this template without redesigning the annotation protocols.
Power of the Reversibility Metric: Evaluating models on forward and backward runs of the same image pair successfully diagnoses systematic positional bias at near-zero cost. This diagnostic is highly transferrable to other pair-wise image comparison tasks.
Fine-Grained Error Attribution: Categorizing failures into perceptual, pseudo-change, reasoning, and spatial errors (complemented by two reversibility error types) provides targeted feedback. The discovery that perceptual errors account for \(63\%\) of total failures calls for prioritizing visual perception accuracy over purely linguistic capability in next-generation LMM development.
Unified Prompting Scheme: Practical results demonstrate that explicitly providing structured context criteria, temporal ordering tags, and prompting for forward-directional inference substantially improves model output, offering a template that easily transfers to other multi-image comparison tasks.

Limitations & Future Work¶

While C3-Bench covers 51 contexts, the average number of image pairs per context is fewer than 100, and the distribution across the four domains is imbalanced (natural scenes and remote sensing have more data, while anomaly analysis and image editing are sparser), which might affect the statistical stability of domain-level insights.
The LLM-as-Judge framework heavily relies on a single model (GPT-5.2). Even though the paper cross-validates this choice using Claude-Haiku-4.5, the judge's own visual and linguistic capability imposes an inherent ceiling on evaluation, potentially introducing bias if a model produces equivalent descriptions that the judge fails to recognize.
The evaluation protocol for traditional models requires training on four source datasets separately before testing across all 51 contexts (demanding \(51\times 4\) experiments). However, the training sets contain no C3-Bench data. Does the conclusion of "complete failure of traditional models" apply equally to LMMs? LMMs were not explicitly exposed to C3-Bench either, but their vast pre-training corpora may have encountered the public source datasets. This potential training discrepancy is not fully explored in the paper.
The Reversibility metric employs a binary (0 or 1) scoring mechanism. This is relatively coarse-grained and might mask intermediate levels of symmetric comprehension when the generated descriptions partially overlap but are not perfectly symmetric. Future designs could introduce continuous, weighted reversibility metrics.
The synthesis of change criteria currently relies on manual drafting. It remains uncertain if these rules fully encompass all semantic transformations within each context, and overlapping boundaries between certain contexts (e.g., terrain alterations in natural scenes vs. remote sensing) might introduce semantic ambiguity.

vs. CLEVR-Change / LEVIR-CC: The former two benchmarks are restricted to synthetic scenes (blocks) and a single remote sensing environment (residential plots), relying on static token-matching metrics like BLEU or ROUGE. In contrast, C3-Bench spans 51 real-world contexts across 4 domains, expanding the evaluation space to 4 multidimensional semantic criteria and a reversibility metric, achieving 1–2 orders of magnitude higher diversity.
vs. SpotTheDiff / IER: Prior works offer very few change contexts (1 and 3 respectively) and lack explicit evaluation criteria—leaving annotators to describe changes arbitrarily, which limits semantic reproducibility. C3-Bench makes "what constitutes a change" explicit through structural criteria, fostering reproducible and traceable evaluations.
vs. BLINK / MMTBench: These multimodal benchmarks also assess image pairs but depend on closed-ended, multiple-choice questioning. C3-Bench evaluates open-ended generation, aligning closer with actual deployment scenarios while pioneering both LLM-as-Judge and reversibility assessment.
vs. MuirBench: While MuirBench evaluates multi-image comprehension, it primarily targets spatial/temporal relationships (e.g., ordering or correspondence) rather than the sequential semantics of "before-and-after" changes. C3-Bench specifically targets fine-grained, context-aware change captioning.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneers context-aware change captioning formulation + LLM-as-Judge evaluation + reversibility metric, offering a distinct methodological contribution to the community.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Systematic evaluation of 32 models (6 traditional, 9 commercial LMMs, 17 open-source LMMs) across 51 contexts, complemented by human studies, error classification, prompt ablations, and multi-dataset training analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Strong, logical flow clarifying the motivation for task reformulation. Provides detailed data curation schemes and supports every core observation with quantifiable metrics and targeted error analysis.
Value: ⭐⭐⭐⭐⭐ Delivers a methodologically sound and reality-aligned evaluation framework for change understanding. More importantly, the uncovered bottlenecks (pseudo-generalization of traditional models, perceptual caps of LMMs, and structural positional bias) chart a clear path for future optimizations.