The Telephone Game: Evaluating Semantic Drift in Unified Models¶
Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://github.com/mollahsabbir/telephone-game-semantic-drift
Area: Image Generation
Keywords: unified multimodal models, semantic drift, cyclic evaluation, cross-modal consistency, compositional semantics
TL;DR¶
The paper evaluates unified models through a telephone game that alternates visual understanding and image generation, using embedding-based semantic retention and object-constraint preservation to expose accumulated errors missed by single-pass evaluation; BAGEL achieves the highest full-benchmark MCDavg (0.4414) and MGG (78.2) among seven systems.
Background & Motivation¶
Unified multimodal models allow one system to describe images and generate images from text, yet these capabilities are usually evaluated separately. FID concerns image distributions, GenEval checks object and relation constraints in a single generation, and MME or MMBench assess understanding. These results do not directly answer a practical question: can a model accurately draw what it has just understood from an image? The paper illustrates the gap through BAGEL's inconsistent understanding and generation of chess positions: recognizing a semantic condition does not guarantee the ability to realize it in another modality.
The gap becomes more consequential in multi-step workflows. A caption may omit a count, or an image generation may change a color; subsequent steps then consume an altered input rather than the original scene. Each output can remain fluent or visually plausible while moving further from the user's request. Rather than introducing another generator, the paper connects understanding and generation into a feedback loop without human intervention and observes when objects, attributes, positions, and counts disappear. Starting separately from text and images avoids treating a model's preference for one starting modality as overall reliability.
The central measurement problem is separating a broadly similar theme from preservation of the actual constraints: embeddings may consider two street scenes similar despite the disappearance of a particular object or a count error. Core Idea: repeatedly convert the model's own output into the other modality, always measure retention against the original input, and combine holistic embedding similarity with object-level constraint checks.
Method¶
Overall Architecture¶
The Semantic Drift Protocol (SDP) treats each evaluated model as two callable functions: image-to-text (I2T) for description and text-to-image (T2I) for synthesis. Given an image-text pair, it constructs text-first and image-first chains, feeding only the current output into the next conversion. The original input remains the evaluator's reference rather than being reintroduced at every step to correct the model.
The output is not a newly trained model but semantic-retention curves over conversion steps and aggregate scores. Cyclic trajectories enter two parallel scoring paths: MCD measures overall content retention, while MGG checks detectable object constraints. A separate single-pass human evaluation checks whether the automated scores correspond to perceived fidelity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Original image-text pair<br/>or GenEval prompt"] --> Chain["Dual-Start Cyclic Protocol"]
Chain -->|Current output becomes next input| Chain
Chain --> Retention["Multi-Space Semantic<br/>Retention MCD"]
Chain --> Objects["Multi-Generation Object<br/>Constraints MGG"]
Retention --> Human["Single-Pass Human<br/>Cross-Validation"]
Objects --> Human
Input -->|Generate independent single-pass outputs| Human
Human --> Output["Retention curves, model rankings,<br/>and failure modes"]
MCD and MGG in the diagram are parallel analyses: the system does not compute MCD and then modify outputs before computing MGG. Human evaluation also uses independent single-pass outputs rather than repairing the cyclic outputs. Full-benchmark MCD and MGG use different input collections, so they are not the same accuracy measured on the same images.
Key Designs¶
1. Dual-Start Cyclic Protocol: expose errors in both directions within a trajectory
A text-first chain proceeds from original text to a generated image, then a new description, then another image. An image-first chain reverses the starting order by describing the original image before generating from that description. One generation step denotes only one I2T or T2I conversion. Thus, the main experiment's \(G=20\) means 20 one-way conversions, or 10 complete round trips, not 20 round trips. A text-first chain produces images at odd steps and text at even steps; an image-first chain does the opposite.
This design measures retention under repeated self-composition, not merely the quality of the first generated image. Later images in a text-first chain must be compared with the initial prompt, not just the preceding caption: that caption may already have replaced "four clocks" with "a group of clocks," allowing good local image-text agreement to conceal global divergence. The two starting modalities also provide four reference relationships: original text to later text, original text to later images, original image to later images, and original image to later text.
2. Multi-Space Semantic Retention MCD: anchor the entire trajectory to the original input
For each reference relationship, the protocol first computes embedding similarity between each sample's output and its original input at every applicable evaluation step. Averaging across the dataset yields a similarity curve over steps; averaging its evaluated generation positions gives MCD for that relationship. MCDavg then averages the four relationship-specific scores. "Cumulative" does not mean summing distances between adjacent outputs, nor does it mean using only the last step. A model that fails early and subsequently maintains an incorrect scene still receives low scores through repeated comparison with the original input.
Text-to-text comparison uses MPNet, image-to-image uses DINO, and both cross-modal directions use CLIP, reducing reliance on a single representation space. Despite the name Mean Cumulative Drift, the aggregate measures similarity: higher values mean better semantic retention and less drift. The spaces have different scales and sensitivities, so MCDavg is a composite measure rather than object accuracy. Equation extraction in the cached paper is corrupted; the explanation here follows the prose definition in Section 3.1 rather than guessing the authors' exact notation from damaged equations.
3. Multi-Generation Object Constraints MGG: prevent broad similarity from hiding lost conditions
MGG extends GenEval from a single image generation to multiple image-producing positions in the cycle. The authors use rewritten GenEval prompts from the BLIP3-o resources: their longer descriptions better match the caption style produced during cycling. An OWLv2 detector checks objects and their constraints. Tasks cover single objects, two objects, counting, colors, positions, and attribute binding. At each image-producing position, task accuracies are averaged into an overall GenEval score; averaging these scores across image-producing positions gives MGG.
Evaluation remains tied to the initial task requirements. If a later caption drops an original color or position, the lost constraint therefore does not become a new correct answer. Higher MGG is better, supplementing embeddings that may overlook specific details, but the metric does not directly measure photorealism and depends on detector reliability. The paper places its human detector audit in the supplement; the supplied cache contains only the main paper, so detector accuracy or audit sample counts cannot be supplied from it.
4. Single-Pass Human Cross-Validation: separately assess faithful understanding and generation
The human study samples 100 text-first and 100 image-first examples from the MCD evaluation set, for 200 examples in total. It independently applies one I2T and one T2I operation to the original paired data, asking whether the generated caption matches the original image and whether the generated image matches the original text. Six annotators participate, with two independently evaluating each example. Model identities are hidden and presentation order randomized. Each output receives both a Good, Medium, or Poor fidelity rating and a comparative model ranking.
Rankings test agreement between automated metrics and human preferences, while the three-level ratings identify asymmetry, such as Good understanding paired with Poor generation. The authors retain individual annotations for consistency analysis rather than first merging the two judgments for each example; reported average ranks aggregate over samples. This is single-pass cross-validation, not human annotation of every step in all 20-step trajectories. Receiving Poor in both directions should not be mistaken for high-quality semantic retention merely because the ratings match.
A Worked Example¶
The initial prompt in Figure 1 specifies a suitcase to the left of one banana. The first image can satisfy this request, but feedback between subsequent images and descriptions can weaken the suitcase while increasing the banana count. The caption states that the suitcase disappears entirely after \(I^{(5)}\), while \(T^{(10)}\) mentions 14 bananas. These superscripts indicate conversion steps, not complete round trips.
Checking only whether a later image matches its later caption could accept this incorrect scene. SDP instead keeps the original suitcase, single banana, and left-right relation as the evaluation reference: MCD tracks holistic divergence, and MGG checks the relevant object, counting, and relation constraints for applicable tasks. The example explains why continually producing plausible outputs differs from preserving the original meaning. It is a qualitative example from the paper, not evidence that every model fails at identical steps.
Loss & Training¶
The paper introduces no training loss or fine-tuning procedure; it evaluates inference using existing models. The main study fixes the maximum chain length at 20 conversions because its weakest system, VILA-U, reaches a near-zero object-level score around step 19 in Figure 6. This horizon is intended to separate model behavior without excessively extending already failed, saturated trajectories.
The length is an empirical choice for the evaluated systems, not a theoretical threshold applicable to every model. The authors associate BAGEL's advantage with scale, unified architecture, and interleaved training data, but do not conduct controlled training experiments to isolate the causal contribution of these factors.
Key Experimental Results¶
Main Results¶
MCD uses Nocaps+Docci400: 200 image-text pairs each from NoCaps and DOCCI, totaling 400. The former stresses novel objects; the latter emphasizes fine-grained attributes, relations, counts, and other details. MGG uses rewritten GenEval prompts. The following reproduces the results in Table 1: both MCDavg and MGG are higher-is-better, and MGG retains the original table's percentage-scale presentation.
| Model | MCDavg | MGG |
|---|---|---|
| BAGEL | 0.4414 | 78.2 |
| BLIP3-o | 0.3984 | 62.2 |
| Show-o | 0.3691 | 57.8 |
| Janus-7B | 0.3778 | 52.5 |
| LLaVA+SDXL | 0.3298 | 32.3 |
| Janus-1.3B | 0.3470 | 27.0 |
| VILA-U | 0.2829 | 19.7 |
BAGEL leads both metrics. Show-o has lower MCDavg than Janus-7B but higher MGG, showing that holistic similarity and specific constraint retention do not produce identical rankings. LLaVA+SDXL likewise exceeds Janus-1.3B on MGG while scoring lower on MCDavg: retaining some detectable objects is not equivalent to preserving the full scene semantics.
Ablation Study¶
The relevant experiment is a diagnostic subset analysis, not a training ablation that removes model components. Each diagnostic subset contains 80 examples. Common objects come from GenEval-R single-object prompts, while novel objects come from NoCaps. Simple MGG tasks involve single objects and colors; complex tasks involve multiple objects, counting, positions, and attribute binding. The following selects three systems from Table 2. These subset values should not be treated as the same test distribution as the full results in Table 1.
| Model | Common-object MCD | Novel-object MCD | Simple-task MGG | Complex-task MGG |
|---|---|---|---|---|
| BAGEL | 0.3178 | 0.2969 | 92.4 | 75.9 |
| BLIP3-o | 0.3293 | 0.2897 | 87.2 | 49.5 |
| Show-o | 0.3152 | 0.2853 | 89.8 | 43.4 |
Subtracting the original table values gives simple-to-complex gaps of 16.5 percentage points for BAGEL, 37.7 for BLIP3-o, and 46.4 for Show-o. These are differences between difficulty subsets, not temporal declines on the same examples. They support the diagnosis that compositional constraints are more fragile, rather than independently proving that a particular component causes drift.
Key Findings¶
- Figure 5 identifies six failure modes: position inconsistency, object misidentification, style changes, quantity inconsistency, object hallucination, and color inconsistency. Later outputs often reinforce critical errors, motivating the main paper's emphasis on catastrophic loss rather than uniformly gradual degradation.
- In Figure 8, Pearson correlations between MCDavg and human generation and understanding ranks are -0.818 and -0.833, respectively. The signs are negative because lower ranks are better while higher automated scores are better.
- MGG correlates with human generation ranks at -0.821, compared with -0.753 for single-pass GenEval. Multi-generation evaluation agrees more closely with human rankings for these seven systems; the result does not establish the same correlation across all possible models.
- In Table 3, BAGEL's average understanding rank is 2.65 and generation rank is 1.63. Figure 9 still reports 7% Good-understanding/Poor-generation inconsistency and 5.9% in the reverse direction: leading the comparison does not eliminate the cross-modal gap.
Highlights & Insights¶
- The evaluation target shifts from individual capabilities to their repeated composition. This requires no new training but exposes interface failures overlooked by separate capability scores.
- The original input remains a fixed reference. Comparing only adjacent steps could make a chain settled on an incorrect scene appear stable; SDP distinguishes stability from fidelity.
- Embedding similarity and object constraints provide evidence at different granularities. Applications that require precise counts or spatial relations should examine both rather than substituting thematic resemblance for constraint satisfaction.
Limitations & Future Work¶
- MCD depends on representation models and MGG on a detector; combining them does not remove all shared blind spots. The main paper mentions representation substitutions and detector audits, but their detailed supplementary results are absent from the supplied cache.
- The primary ranking covers seven open systems, the full MCD set has 400 pairs, and the human study has 200 examples. Model-level correlation analysis has few observations, and the single-pass human study is not direct human validation of entire drift trajectories.
- Repeated captioning creates an information bottleneck. Some detail loss may arise from linguistic summarization and generation randomness rather than a complete failure to understand. A further experiment could control caption length, sampling parameters, and multiple random seeds to separate compression loss from cross-modal realization errors.
- A 20-step horizon and trajectory averages compress information about the first failure. The authors place survival-style collapse analysis in the supplement; applications could additionally report the first loss of a critical constraint and whether recovery is possible without reintroducing the original input.
Related Work & Insights¶
- vs GenEval: GenEval evaluates object and relation compliance for a single T2I output; MGG extends this across generations while changing prompt resources and the detector. Comparisons therefore require aligned evaluation implementations as well as chain lengths, rather than combining single-pass results from unrelated papers.
- vs Cycle Consistency as Reward: The related work uses a single cycle to construct preference data. This paper instead evaluates repeated self-composition and failure modes of unified models rather than training a new model with cyclic rewards.
- vs Uni-MMMU: Uni-MMMU jointly evaluates understanding and generation through bidirectionally linked tasks. SDP tracks preservation of the same semantics through repeated conversions, addressing stability as errors accumulate over steps.
- Transferable insight: Systems for repeated editing or visual-state description followed by synthesis can use task-critical conditions as fixed references in multi-step testing. This is an application of the paper's idea, not a reported downstream performance gain.
Rating¶
- Novelty: 4/5. Systematizes semantic retention under repeated self-composition, with the contribution centered on evaluation questions and protocol rather than architecture.
- Experimental Thoroughness: 4/5. Includes seven systems, two metric families, difficulty diagnostics, and human evaluation, although the main paper alone does not support independent verification of every supplementary claim.
- Writing Quality: 4/5. The examples and protocol are accessible, but the MCD name and actual score direction require explicit explanation.
- Value: 4/5. Offers a useful reliability check for multi-step unified-model workflows without replacing single-pass quality or real-task evaluation.