UEval: A Benchmark for Unified Multimodal Generation

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: unified multimodal generation, rubric-based evaluation, image-text alignment, multi-step planning, automatic evaluation

TL;DR

UEval uses 1,000 questions requiring both images and text and 10,417 human-validated rubric criteria to test whether unified models can answer jointly across modalities; GPT-5-Thinking scores only 66.4 overall, while the strongest open-source model, Emu3.5, scores 49.1.

Background & Motivation

Visual question answering usually asks a model to inspect an image and produce text, while text-to-image generation asks it to render a description. These tests measure understanding and generation separately, without directly checking whether images and text in one response support each other. For example, explaining why visitors can stand inside the Statue of Liberty requires more than an image of the whole statue accompanied by correct prose: the visual answer should reveal the relevant crown interior. Likewise, a drawing tutorial cannot succeed merely by displaying an attractive final picture; intermediate images, step labels, and textual actions must agree.

Such open-ended outputs have no single answer suitable for string matching, and generic grading prompts can overlook the visual details that matter for a particular question. Pairwise comparisons require many judgments, and the resulting win rates depend on the set of competing models. UEval instead establishes an independent rubric for each question, making the evaluation criteria depend on the task rather than on an overall impression. Core Idea: convert reference image-text answers into human-reviewed, individually checkable requirements, then test whether generated images, text, and their relationships satisfy them.

Method

Overall Architecture

UEval is an evaluation dataset and scoring protocol, not a new generation network, and it introduces no model-training loss. During construction, reference images and text from real sources support question creation and rubric drafting; during testing, a multimodal judge checks model responses criterion by criterion. Reference answers help define reliable requirements rather than requiring models to reproduce reference images pixel for pixel. The resulting benchmark contains 1,000 questions across 8 tasks and 10,417 criteria, with scores that can be broken down by task and output modality.

The protocol distinguishes between including an image and having that image explain, demonstrate, or express the technical relationship needed by the question. Approximately half of the criteria assess images and the remainder assess text; specific criteria also constrain cross-modal correspondence and continuity across steps. A model with strong prose can therefore lose points for incorrect structures, omitted actions, or duplicate step labels. The following designs explain task construction, rubric drafting, human review, and scoring adaptation.

Key Designs

1. Task construction: separate grounded explanation from open-ended demonstration

The closed-ended tasks are Space, Textbook, Diagram, and Paper, emphasizing factual explanations with identifiable target content. Space starts with 20 seed questions about landmarks, expands candidates with GPT-5, and uses human review to verify real architectural features and retrieve suitable images and explanations. Textbook uses reference diagrams and text from TQA to generate questions about scientific phenomena, followed by checks for scientific validity, clarity, and answerability from the references. The visual requirement is therefore to depict the structure or mechanism involved, not merely to attach a decorative picture.

Diagram and Paper both involve research figures but make different demands on the evaluated model. Diagram selects figures from relatively recent papers and uses the full paper to construct a self-contained technical instruction, allowing the model to draw an architecture and provide a caption without accessing the original paper; familiar paper diagrams are deliberately avoided to reduce memorization risk. Paper uses figures and explanations from sources such as Transformer, ResNet, and D2L to formulate reader-oriented questions requiring an accessible image-text explanation of a technical concept. The open-ended Guide group contains Art, Life, Tech, and Exercise, drawing references from tutorials such as WikiHow and demonstration videos such as YouTube, and requires progressively organized visual instructions. Questions do not fix the number of steps and allow a composite image or multiple images, so evaluation should check a coherent process rather than enforce the reference answer's layout.

2. Rubric drafting: define correctness separately for each question

Gemini-2.5-Pro receives the question and its reference image-text answer and produces multiple fine-grained criteria. Closed-ended criteria emphasize essential reference information: a Statue of Liberty image should expose relevant crown features, and the text should explain how the internal staircase connects to an occupiable space. These requirements distinguish an approximately related object from an image that provides evidence for the answer. The reference image establishes factual requirements; it is not used to define an image-similarity scoring function.

Open-ended criteria emphasize higher-level properties, such as whether each image matches its numbered instruction, whether later images build logically on earlier ones, and whether the text explains a complete process. This accommodates multiple reasonable drawings while penalizing skipped stages, state reversals, and image-text mismatches. The paper's examples also include specific operations such as starting with a circle, so open-ended rubrics should not be interpreted as entirely independent of the reference procedure. Human judgment still determines the boundary between acceptable alternatives and required content; high-level wording alone cannot eliminate reference-answer preference.

3. Human review: remove double counting and recover missing requirements

Initial criteria undergo two rounds of review rather than entering the benchmark directly. A primary annotator first checks the correctness and alignment of the question, reference answer, and rubric; other co-authors then review independently, retaining only items unanimously considered unambiguous and appropriate for the task. Review covers the input question, reference output, and grading requirements, not just the wording of a scoring prompt. For example, requirements for sequential steps and logically ordered steps are merged so that the same property is not counted repeatedly.

Reviewers also add missing requirements, including correctly spelled, naturally rendered text in generated images without garbled or meaningless characters. This matters because a concept-only rubric could reward the textual explanation while overlooking unreadable labels inside the image. Human review thus provides quality control while also defining which failures the task must penalize. The 10,417 criteria are the validated total, not the number of unfiltered items initially drafted by the model.

4. Scoring adaptation: check individual criteria and expose modality-specific weaknesses

The default judge, Gemini-2.5-Pro, evaluates generated images and text against the question-specific rubric. The paper's verbal definition of the per-sample score can be written on a percentage scale as follows; this notation restates the definition rather than introducing a training loss from the paper.

\[ S_q = 100\frac{\sum_{k=1}^{K_q}\mathbf{1}[\text{criterion}_{q,k}\text{ is satisfied}]}{K_q}. \]

Here \(K_q\) is the number of criteria for the question, and each criterion contributes according to whether it is satisfied. The paper separately reports task scores, an overall Avg, and image/text criterion scores; an overall score neither establishes image quality by itself nor represents the proportion of entirely correct answers. Questions have different numbers of criteria, so the reported averages should not silently be replaced with a pooled pass rate over all criteria in the dataset.

In the evaluated configurations, Janus-Pro, Show-o2, MMaDA, and BAGEL cannot produce images and text together in one inference pass, so the authors run the same question separately for image generation and text generation and combine the outputs for grading. Systems supporting joint outputs, such as GPT-5 and Gemini, contribute their native multimodal responses; these are not identical inference-call protocols. Both GPT-5 variants are accessed through the official chat interface. There is no training stage; the cached main paper does not include the referenced appendix with grading prompts and complete generation hyperparameters, so temperature, seeds, and a uniform compute budget cannot be supplied from this source.

Key Experimental Results

Main Results

Table 1 evaluates 9 unified models on all 8 tasks with Gemini-2.5-Pro as the default judge. The following selection reports overall Avg; Reference denotes graded reference image-text answers, not another evaluated model or a guaranteed perfect human ceiling.

System	Type	Avg, percentage scale
Reference	Reference answers	92.2
BAGEL	Open-source	31.0
Emu3.5	Open-source	49.1
Gemini-2.0-Flash	Proprietary	55.1
Gemini-2.5-Flash	Proprietary	66.0
GPT-5-Instant	Proprietary	65.2
GPT-5-Thinking	Proprietary	66.4

GPT-5-Thinking leads Emu3.5 by 17.3 points but Gemini-2.5-Flash by only 0.4 points. The paper does not provide significance tests for these closely spaced overall scores, so small gaps should not be treated as stable capability rankings.

Ablation Study

There is no module-removal ablation; the following diagnostic analysis uses the modality breakdown from Table 2, with scores calculated from criteria for the corresponding modality.

System	Image score	Text score
Reference	87.8	96.8
BAGEL	13.6	48.5
Emu3.5	33.6	64.6
Gemini-2.5-Flash	56.4	75.5
GPT-5-Instant	52.8	77.7
GPT-5-Thinking	49.1	83.8

Key Findings

• GPT-5-Thinking scores 83.8 on text but only 49.1 on images; its leading overall score does not make it the strongest image generator, as Gemini-2.5-Flash scores 56.4 on images (Table 2).
• Human validation randomly samples 10% of GPT-5-Thinking outputs from each task, finding approximately 90% criterion-level agreement and a question-level Pearson correlation of 0.88 (Figures 7 and 8); this is not a human audit of every evaluated model.
• Changing judges can materially shift scores: the same GPT-5-Thinking responses receive 63.8 on Life from Gemini-2.5-Pro but 50.1 from GPT-5-Thinking as judge (Table 3), so judges are not freely interchangeable.
• Appending a GPT-5-Thinking reasoning trace to the original question improves the Statue of Liberty examples from GPT-5-Instant and Gemini-2.5-Flash but not BAGEL (Figure 9); the paper provides no aggregate improvement table for this intervention.

Highlights & Insights

• Question-specific criteria turn plausible-looking generation into a test of whether the required evidence is present. This transfers naturally to scientific illustrations and procedural tutorials, provided the rubric covers substantive errors rather than only stylistic properties.
• Separate image and text diagnostics expose weaknesses hidden by aggregate scores. Better language reasoning and better visual execution are different objectives, so model selection should consider the relevant modality scores.
• Leaving step counts unspecified tests a model's ability to organize its own response. Evaluation must consequently emphasize state continuity and image-text correspondence rather than simply compare image counts.

Limitations & Future Work

• The authors suggest that difficulty judging multiple images may depress open-ended reference scores; a reference average of 92.2 rather than a perfect score also leaves room for automatic grading error.
• The human-agreement study covers sampled GPT-5-Thinking outputs only, so its reliability does not automatically extend to weaker models or unusual outputs.
• Separate image/text calls coexist with native joint generation, and GPT-5 uses a chat interface; the leaderboard compares these specific system configurations rather than isolating the causal effect of architecture.
• Reasoning-trace transfer remains qualitative evidence without full-task, equal-length-prompt, or equal-budget controls; future work could separate additional knowledge, prompt length, and reasoning structure.
• Diagram deliberately avoids familiar paper figures, but Paper includes classic research and public teaching materials; this supports construction without ruling out training-data memorization, a risk inferred from the data sources.

Related Work & Insights

• BLINK and ScienceT2I: the former primarily tests visual understanding and the latter science-related image generation; UEval places joint image-text responding within one task, which separate-modality scores cannot directly replace.
• HealthBench: UEval borrows question-specific rubrics and extends them to image content, image-text correspondence, and continuity across steps; the transfer concerns evaluation targets rather than training objectives.
• OpenING and MMIE: the paper emphasizes that automatic rubric grading avoids per-sample human scoring or training an additional scoring model, while still requiring initial human review and a reliable multimodal judge.
• Dataset: zlab-princeton/UEval. The paper provides a dataset address, which is not presented here as a verified code repository.

Rating

• Novelty: 4/5. Question-specific evaluation of joint image-text responses is useful, although rubric-based evaluation builds on prior work.
• Experimental Thoroughness: 4/5. Model comparisons, modality breakdowns, human agreement, and judge comparisons are included, but reasoning transfer lacks quantitative controls.
• Writing Quality: 4/5. Task examples and rubric mechanisms are clear, though broad claims require careful reading alongside detailed scores and judge differences.
• Value: 4/5. Useful for diagnosing explanatory images and multi-step generation failures, rather than serving as a single ranking of all multimodal capabilities.