SpecV: Specification Verification for Robust Unified Multimodal Evaluation

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/yuyouxixi/SpecV
Area: Multimodal VLM
Keywords: unified multimodal models, evaluation drift, specification verification, multi-model ensemble filtering, unified benchmark

TL;DR

Addressing the severe vulnerability of unified multimodal models (UMMs) to evaluation drift under holistic VLM-as-a-judge scoring, SpecV reformulates open-ended multimodal assessment into atomic binary specification checking via the Specification Verification Protocol (SVP) backed by the multi-model Specification Ensemble and Refinement (SER) pipeline on the 1,200-instance SpecV-Bench, substantially enhancing cross-judge agreement and evaluation reproducibility.

Background & Motivation

Unified Multimodal Models (UMMs) are consolidating previously isolated capabilities—such as visual comprehension, text-to-image synthesis, visual editing, and interleaved text-image reasoning—into singular end-to-end architectures, exemplified by models like Nano Banana, Bagel, and Emu 3.5. However, this architectural consolidation has exacerbated rather than simplified model evaluation. Because the majority of core UMM tasks are open-ended without unique ground-truth references, conventional reference-based discriminative metrics (such as multiple-choice accuracy, FID, or CLIPScore) fail to capture fine-grained compositional semantics and suffer from choice-set biases. Consequently, the research community has increasingly embraced the VLM-as-a-Judge paradigm, where frontier vision-language models directly assign holistic scalar scores to model outputs.

Yet, this holistic scoring paradigm introduces a critical systemic instability: it compresses multi-dimensional visual and linguistic qualities into a single scalar value. In doing so, it conflates two fundamentally distinct responsibilities within the judge model—deciding which visual aspects matter and deciding how heavily to weight each aspect. Because these weighting preferences are implicit and unobservable, they depend heavily on the judge's pretraining data, prompt wording, and decoding strategies. When a judge model undergoes a version upgrade, system prompt revision, or provider switch, its internal weighting shifts unpredictably, resulting in dramatic ranking reversals among tested models. This structural vulnerability—which the authors term evaluation drift—undermines the reproducibility and credibility of unified multimodal benchmarking.

To resolve this fundamental tension, prompt engineering alone is insufficient to eliminate the latent degrees of freedom in implicit weighting. The evaluation criteria must be made explicit prior to invoking any judge, narrowing each judgment to an unambiguous, single-aspect determination. The core idea is to reformulate open-ended multimodal evaluation as an explicit, fine-grained specification verification problem: decomposing complex user prompts into atomic, binary verification criteria before evaluation, thereby replacing subjective holistic scalar scoring with discrete, verifiable specification satisfaction rates that eliminate the degrees of freedom for evaluation drift.

Method

Overall Architecture

The SpecV evaluation framework comprises two complementary core components: the Specification Verification Protocol (SVP) and the Specification Ensemble and Refinement (SER) pipeline. For any test instance \(x = (t, I)\) consisting of textual instruction \(t\) and optional reference image set \(I\), along with an under-test model output \(y\), SpecV does not ask the judge model for a global subjective score. Instead, it operates over a pre-constructed discrete set of atomic natural-language specifications \(\mathcal{S}_x = \{s_1, s_2, \dots, s_K\}\). During evaluation, the judge performs an independent binary check \(v_k \in \{0, 1\}\) for each individual specification, computing the instance score as the aggregate satisfaction rate. To prevent any single model's inductive biases from creating systematic evaluation blind spots, the offline specification sets are constructed via the multi-model SER pipeline through generation, semantic deduplication, multi-dimensional quality filtering, and human quality assurance, resulting in the comprehensive SpecV-Bench.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    subgraph SER["Specification Ensemble & Refinement (SER Pipeline)"]
        direction TB
        A["Multi-Model Independent Generation<br/>Gemini 3 Pro / GPT 5.2 / Claude Opus 4.5"] --> B["Semantic Consolidation & Paraphrase Merging<br/>Deduplicate variants and flag contradictions"]
        B --> C["Cross-Model Quality Filtering<br/>Retain items with average score >= 8 + Human QA"]
    end
    Input["Test Input x = (t, I)<br/>and Model Output y"] --> Eval["Specification Verification Execution"]
    C -->|Offline Grounded Spec Set S_x| Eval
    Eval --> D["Specification Verification Protocol (SVP)<br/>Perform binary verification v_k in {0, 1}"]
    D --> Output["Aggregate Specification Satisfaction Rate<br/>Score = (1/K) sum v_k"]

Key Designs

1. Specification Verification Protocol (SVP): Bounding Judgment Scope to Atomic Binary Checks Conventional holistic scoring compresses evaluation into a scalar rating \(Score_{\mathrm{H}}(y; x) = \mathcal{J}(x, y, p_{\mathrm{H}}) \in [0, N]\), forcing the judge to simultaneously decide which criteria to inspect and how to weight them. Even if two judges perceive visual details identically, divergent internal weighting can invert their model rankings. SVP decouples what to evaluate from how to evaluate it: prior to evaluation, each instance is formalized into \(K\) explicit natural-language specifications \(\mathcal{S}_x = \{s_1, \dots, s_K\}\) satisfying atomicity (targeting one single attribute), objectivity (evaluating factual presence rather than subjective aesthetics), and determinacy (ensuring consistent agreement across judges). For output \(y\), the judge executes a binary verification:

\[v_k = \mathcal{J}(x, y, s_k) \in \{0, 1\}\]

The final instance score is computed as the specification satisfaction rate:

\[\mathrm{Score}_{\mathrm{SVP}}(y; x) = \frac{1}{K} \sum_{k=1}^K v_k \in [0, 1]\]

This mechanism provides an explicit mathematical bound on evaluation instability: any single disagreement across judges alters the overall instance score by at most \(1/K\). Cross-judge discrepancies are bounded and attenuated through averaging, structurally immunizing the evaluation against evaluation drift.

2. Specification Ensemble and Refinement (SER): Multi-Model Consensus Against Blind Spots Constructing comprehensive specifications requires capturing all salient task constraints without introducing vague or unverifiable criteria. Relying on a single frontier VLM inevitably introduces inductive biases (e.g., some models disproportionately emphasize color/style while others focus on spatial layout or text rendering). The SER pipeline establishes consensus across diverse models in four sequential stages: - Stage 1 (Multi-model generation): Three frontier VLMs from distinct providers (Gemini 3 Pro, GPT 5.2, and Claude Opus 4.5) independently generate candidate specification sets \(\mathcal{S}^{(i)} = \mathrm{Generate}(\mathcal{M}_i, x)\) under identical structural prompts; - Stage 2 (Semantic consolidation): A dedicated model merges paraphrased or near-duplicate specifications across \(\bigcup_{i=1}^L \mathcal{S}^{(i)}\) into a canonical representation \(\mathcal{S}_{\mathrm{merged}}\), while flagging contradictions for review; - Stage 3 (Cross-model quality filtering): Each model independently rates every consolidated specification on a 0–10 scale across relevance, clarity, and binary verifiability, retaining only items whose cross-model average reaches threshold \(\tau\):

\[\mathcal{S}_q = \left\{ s \in \mathcal{S}_{\mathrm{merged}} \;\middle|\; \frac{1}{L} \sum_{i=1}^L \mathrm{Score}(\mathcal{M}_i, s) \geq \tau \right\}, \quad \tau = 8\]

Finally, trained annotators conduct a human quality assurance pass to eliminate ambiguous or redundant items, combining multi-model generative diversity with rigorous verifiability.

3. SpecV-Bench: Hierarchical Constraint Complexity Taxonomy SpecV-Bench operationalizes the specification framework across 1,200 test instances spanning six core capability axes of UMMs: text-to-image generation, image editing, many-to-one generation, interleaved image-text generation, thinking with images, and multimodal understanding. To enable targeted failure analysis, the benchmark is organized into 28 fine-grained sub-tracks with progressively increasing constraint complexity. Across the entire benchmark, 17,604 atomic specifications are established—averaging 14.67 binary checks per instance. This dense specification coverage exposes exact points of breakdown when handling compositional constraints, spatial relations, or visual reasoning.

A Worked Example

Consider an instance from the Many-to-One Generation task in SpecV-Bench, where the user provides an input photo of a person in casual attire and a second reference photo of a formal navy suit: "I want to use this photo as my formal resume headshot. Please replace the casual T-shirt with the deep blue suit from the second image." - Under holistic scoring, a judge model produces a single subjective rating (e.g., 7/10), with judges disagreeing on whether facial fidelity or suit realistic texture is more important; - Under SVP, the SER pipeline defines three atomic specifications: 1. \(s_1\): Is the man's face and hair completely preserved from the original image without distortion? 2. \(s_2\): Has the casual T-shirt been replaced by the deep blue suit jacket, white shirt, and burgundy tie from the reference image? 3. \(s_3\): Is the grey textured background preserved? - The judge evaluates each independently: \(v_1, v_2, v_3 \in \{0, 1\}\). If the model preserves the face (\(v_1=1\)) and adds the suit correctly (\(v_2=1\)), but replaces the background with pure white (\(v_3=0\)), the score is objectively \(\frac{2}{3} \approx 66.7\%\), providing both an immutable score and actionable diagnostic feedback on background preservation failure.

Key Experimental Results

Main Results

On the 1,200 instances of SpecV-Bench, 19 models across three categories were evaluated: 8 unified multimodal models (UMMs), 5 vision-language models (VLMs, understanding only), and 6 dedicated image generation/editing models. Among UMMs, Nano Banana 2 achieves the top overall performance, though architecture-dependent performance trade-offs remain substantial.

Category	Model	Understanding	T2I	Editing	Many to One	Interleave Gen.	Think w/ Images	Average
Unified Multimodal Models	Nano Banana 2	83.25	90.45	90.34	76.33	80.27	69.97	81.77
Unified Multimodal Models	Nano Banana Pro	83.25	90.23	88.42	74.31	74.63	61.81	78.78
Unified Multimodal Models	Nano Banana	78.21	68.03	74.95	68.74	82.04	51.00	70.50
Unified Multimodal Models	Qwen-Image-2	71.00	75.28	73.30	67.73	74.56	32.47	65.72
Unified Multimodal Models	Qwen-Image	57.00	64.60	69.16	60.82	34.03	19.84	50.91
Unified Multimodal Models	Bagel w. CoT	69.00	26.39	56.88	45.96	45.06	18.34	43.61
Unified Multimodal Models	Bagel w/o CoT	63.75	28.09	54.45	47.35	35.90	16.81	41.06
Unified Multimodal Models	Emu 3.5	45.99	67.14	64.53	66.53	62.23	14.18	53.43
Vision-Language Models	Gemini 3.1 Pro	89.00	-	-	-	-	-	-
Vision-Language Models	Gemini 3 Pro	88.25	-	-	-	-	-	-
Vision-Language Models	GPT 5.2	87.29	-	-	-	-	-	-
Vision-Language Models	Qwen 3.5 Plus	85.14	-	-	-	-	-	-
Vision-Language Models	Claude Opus 4.5	78.25	-	-	-	-	-	-
Image Generation / Editing Models	Seedream 5.0	-	81.30	85.14	73.10	-	-	-
Image Generation / Editing Models	GPT Image 1.5	-	80.24	81.14	77.64	-	-	-
Image Generation / Editing Models	GPT Image 1	-	66.79	68.64	73.23	-	-	-

In ranking stability assessments across five judge models (GPT 4.1, Gemini 2.5 Flash Lite, Gemini 2.5 Flash, Gemini 3 Flash, and Qwen 3.5 Plus), SVP significantly outperforms holistic scoring: - Intra-family stability: Holistic scoring yields average Kendall's \(\tau = 0.805, \rho = 0.920\), whereas SVP boosts stability to \(\tau = 0.867, \rho = 0.948\); - Cross-family stability: Holistic scoring reaches \(\tau = 0.856, \rho = 0.942\), whereas SVP raises it to \(\tau = 0.902, \rho = 0.965\). In the image editing task, SVP increases cross-family Kendall's \(\tau\) from 0.758 to 0.903.

Ablation Study

The authors ablated the SER pipeline by contrasting individual model specification generators and removing cross-model filtering against the complete pipeline, evaluating correlation with human judgments (Kendall's \(\tau\) / Spearman's \(\rho\)):

Configuration	Understanding \(\tau\) / \(\rho\)	T2I \(\tau\) / \(\rho\)	Editing \(\tau\) / \(\rho\)	Many to One \(\tau\) / \(\rho\)	Interleave Gen. \(\tau\) / \(\rho\)	Think w/ Images \(\tau\) / \(\rho\)	Average \(\tau\) / \(\rho\)	Note
Only Claude Opus 4.5	0.347 / 0.385	0.561 / 0.634	0.379 / 0.435	0.312 / 0.380	0.547 / 0.661	0.684 / 0.766	0.471 / 0.543	Strong on T2I but lags on M2O
Only GPT 5.2	0.368 / 0.402	0.480 / 0.560	0.386 / 0.441	0.290 / 0.344	0.516 / 0.620	0.637 / 0.733	0.446 / 0.517	Strong comprehension, weaker generation
Only Gemini 3 Pro	0.319 / 0.345	0.556 / 0.620	0.453 / 0.492	0.228 / 0.260	0.468 / 0.564	0.627 / 0.704	0.442 / 0.497	Leads in editing, uneven elsewhere
w/o filtering	0.318 / 0.352	0.541 / 0.599	0.375 / 0.435	0.308 / 0.374	0.470 / 0.574	0.655 / 0.750	0.444 / 0.514	Unfiltered union introduces noisy items
Complete SER Pipeline	0.329 / 0.360	0.516 / 0.585	0.408 / 0.455	0.442 / 0.494	0.548 / 0.640	0.716 / 0.794	0.493 / 0.555	Balanced and highest human correlation

Key Findings

• Asymmetric capability trade-offs in UMMs: The consolidation of multimodal capabilities entails stark architectural trade-offs. Bagel achieves 69.00 on visual understanding but collapses to 26.39 on text-to-image generation; conversely, Emu 3.5 scores 67.14 on generation but only 45.99 on understanding. Only Nano Banana 2 exhibits balanced capability across tasks (90.45 T2I vs. 83.25 understanding).
• Sub-track bottlenecks: Text rendering in T2I remains a universal failure mode across models (top model achieves 83.19 while others fall below 30). In image editing, reasoning-driven edits exhibit steep performance drops compared to explicit attribute swaps.
• Necessity of multi-model filtering: Merging specifications from multiple models without quality filtering degrades human correlation compared to Claude alone (average \(\tau\): 0.444 vs. 0.471), as uncurated unions accumulate tangential or ambiguous criteria. Cross-model consensus scoring is critical for retaining high-precision specifications.

Highlights & Insights

• Deconstructing holistic scoring into atomic binary verifications: By replacing continuous 0–10 scalar scores with \(K\) discrete binary channels, SVP bounds the maximum impact of any single judgment disagreement to \(1/K\), eliminating the uncontrolled degrees of freedom that cause evaluation drift.
• Critical empirical insight on evaluation drift: The study demonstrates that version iterations within a single model provider family (e.g., Gemini 2.5 vs. Gemini 3) can produce greater ranking volatility than baseline differences across providers, exposing an acute reproducibility crisis in current AI evaluation.
• Generalizable verification paradigm: The specification decomposition principle naturally generalizes beyond vision-language models to agent tool execution audits, long-form factual verification, and complex code generation evaluation where holistic grading fails.

Limitations & Future Work

• Uniform specification weighting assumption: SVP currently weights all specifications equally within an instance (\(1/K\)). In realistic deployment, critical failures (such as distorting primary human identities) are more severe than minor background discrepancies. Incorporating hierarchical or adaptive constraint weighting remains an open research avenue.
• High offline curation overhead: The SER pipeline necessitates candidate generation across three frontier models followed by cross-scoring and human review. While specifications are created once per benchmark version, this setup limits rapid, low-cost dynamic benchmark updates.
• Binarization challenges on continuous attributes: Enforcing strict binary decisions on continuous visual dimensions (such as photorealism, artistic style, or subtle lighting gradations) may introduce minor boundary threshold noise.

Related Work & Insights

• vs MME-Unify / RealUnify: Existing unified multimodal benchmarks employ fragmented evaluation protocols across tasks (multiple-choice for comprehension, CLIPScore/FID for generation) and neglect the evaluation drift caused by judge updates. SpecV delivers an overarching, unified specification verification protocol spanning all six UMM task families.
• vs TIFA / DSG: While TIFA and DSG decompose prompts into question-answering checks, they are confined to text-to-image faithfulness. SpecV introduces the multi-model SER consensus pipeline and expands coverage to unified tasks including editing, interleaved generation, and visual reasoning.
• vs VLM-as-a-Judge: Traditional holistic VLM judging conflates criteria selection with implicit weighting, rendering benchmarks vulnerable to evaluation drift. SpecV enforces explicit atomic criteria, yielding substantially higher cross-judge ranking stability and stronger human alignment.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Systematically defines and solves the evaluation drift phenomenon in unified multimodal models via specification verification]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluates 19 models across 6 tasks and 28 sub-tracks, with rigorous intra- and cross-family judge stability analyses]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formalization, cohesive structural narrative, and precise empirical analysis]
• Value: ⭐⭐⭐⭐⭐ [Provides a robust, drift-resistant benchmarking methodology for the rapidly evolving field of unified multimodal models]