TIIF-Bench: How Does Your T2I Model Follow Your Instructions?

Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation
Keywords: instruction following, short and long prompts, binary verification, text rendering, style control

TL;DR

TIIF-Bench combines hierarchical short/long prompt pairs, attribute-level binary checks, and specialized text/style metrics, adapting Qwen3VL-8B into an interpretable evaluator that achieves 91.07% accuracy on a human-annotated validation set.

Background & Motivation

A generated image can look convincing without following its instructions: a color belongs to the wrong object, a numerical comparison is reversed, or an excluded object appears. CLIP-style global similarity can capture which concepts are present without reliably checking their relationships. Asking a vision-language model (VLM) for a holistic score also compresses multiple requirements into an opaque judgment. Including the full generation prompt in the evaluation query can further encourage the evaluator to mistake requested content for visible evidence.

The evaluation inputs introduce another blind spot. Short templates often recycle similar entities, limiting semantic coverage while leaving a practical question unanswered: does a model still satisfy the same requirement when it is expressed through a longer, more elaborate description? Text rendering, artistic style, and real designer requirements also cannot all be measured through object detection or attribute checks. TIIF-Bench therefore redesigns both the prompts and their scoring protocol instead of merely replacing an existing scorer with a larger model.

The benchmark treats instruction following as a collection of verifiable constraints while retaining separate measurements for different capabilities. Core Idea: check individual requirements under semantically equivalent short and long prompts, use a task-adapted reasoning evaluator for attributes and relations, and complement it with specialized text and style metrics.

Method

Overall Architecture

The inputs are a text-to-image model and a hierarchical prompt set; the outputs are dimension-level and overall scores under short and long prompts. Concept pools first support prompt construction and the creation of a checklist for each prompt. After the tested model generates an image, TIIF Evaluator answers its binary questions, while text and style tasks receive additional specialized measurements.

Two model roles must remain distinct: the text-to-image system is the subject, and the Qwen3VL-8B-derived TIIF Evaluator is the judge. The paper trains the latter. It neither introduces a new image-generation architecture nor demonstrates that optimizing a generator against this judge improves generation.

Key Designs

1. Hierarchical short/long prompts: separate compositional difficulty from linguistic length

The authors extract objects and attributes/relations from existing benchmarks, using GPT-4o to construct 10 concept pools organized into attribute, relation, and reasoning groups. They generate natural instructions through 36 combination policies with dedicated meta-prompts. Basic Following primarily combines concepts within a group; Advanced Following combines groups, for example requiring a numerical condition and a color condition simultaneously. Difficulty therefore comes from concurrent constraints rather than simply adding words.

Beyond composition, text-rendering prompts come from Lex-Art, style-control prompts come from AIGC communities, and 100 designer-level prompts are manually collected. Text and style belong to the advanced tier, while Designer Level is separate. Table 1 gives construction policies and category counts. Section 3.2 reports 5,000 prompts and 12,074 binary questions in the full benchmark, with 554 prompts and 1,446 questions in TestMini. These are the reported totals; short/long variants should not be multiplied into them again.

Each original prompt receives an expanded version intended to preserve its core semantics, testing the same constraint set at two linguistic lengths. Elaboration may clarify a style or bury an important requirement, so longer prompts are not inherently harder or better. The paper analyzes semantic uniqueness using CLIP text embeddings with a cosine-similarity deduplication threshold of 0.85, but that analysis is not a human verification of equivalence for every short/long pair.

2. Attribute-level binary checks: replace global impressions with constraint satisfaction

For ordinary compositional prompts, the pipeline extracts core concepts and uses a strong closed-source language model to produce corresponding yes/no questions and target answers. The evaluator receives the generated image and its checklist instead of being asked to score alignment with the entire prompt. Designer-level prompts receive longer manually constructed checklists that capture practical constraints. Target answers are not necessarily all yes: negative requirements must also have appropriate expected answers.

The image score is the fraction of predicted answers matching the targets. It measures satisfaction of the instructions, not the evaluator's classification accuracy on human-labeled images. Although these quantities have similar forms, their reference labels serve different purposes. The following restores the damaged equation layout from the prose in Section 3.3: \(N\) is the number of questions, \(a_i\) the predicted answer, and \(\hat a_i\) the answer required by the prompt.

\[ s=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[a_i=\hat a_i]. \]

Errors become localizable: an image can satisfy a color condition while violating a spatial relation instead of receiving an unexplained middling similarity score. However, averaging also permits many easy successes to conceal one critical failure. The score is not the probability that the entire prompt has been satisfied.

3. Specialized text and style metrics: avoid asking one judge to measure everything

Text rendering uses Global Normalized Edit Distance (GNED). Required words and OCR-recognized words form two collections. Character-level normalized edit distance supplies the cost of each possible word pair, and the Hungarian algorithm finds the minimum-cost matching. The absolute difference in word counts penalizes extra or missing words, and the larger word count normalizes the result. Generating many additional words therefore cannot cheaply inflate a recall-only score.

The following equation is restored from the textual definition in Section 3.3 because its cached mathematical layout is incomplete. Here \(m,n\) are the required and recognized word counts, \(\mathcal M\) is the optimal matching, and \(\operatorname{NED}\) is character-level normalized edit distance between words:

\[ \operatorname{GNED}(P,G)=\frac{\sum_{(i,j)\in\mathcal M}\operatorname{NED}(p_i,g_j)+|m-n|}{\max(m,n)}. \]

Lower GNED is better, with 0 indicating perfect agreement. The Text column in Table 2 uses \(1-\operatorname{GNED}\) so that higher scores are better throughout. This matching primarily evaluates textual content; it should not be assumed to verify word order, layout, or prescribed placement. OCR errors also propagate into the score.

Style evaluation supplies reference images at five aspect ratios: 16:9, 5:4, 1:1, 4:5, and 9:16. Scoring selects the reference closest to the generated image's aspect ratio. TestMini references are generated with Nano-Banana-Pro, whereas full-benchmark references use Qwen-Image, followed by manual verification. These references are scoring targets, not extra conditioning images supplied to the tested generator.

The source has an unresolved protocol discrepancy: Section 3.3 specifies the average of CSD and DINOv3 similarities, while earlier text describes a weighted combination of CSD, DINOv3, and SigLIP2. This note follows the method section without inventing weights; reproduction requires checking the released implementation. Style similarity is also a reference-based proxy, not a direct human aesthetic judgment.

4. Reasoning-based evaluator: adapt the judge for reliable fine-grained decisions

An off-the-shelf VLM is not sufficiently reliable. The authors generate two rollouts for every one of the 5,000 prompts with each of 30 text-to-image models, obtaining approximately 300K images and 724K binary-question instances. They manually annotate 300 images with 560 questions for validation, balanced between 280 yes and 280 no answers. GPT-4o achieves 90.18% accuracy, compared with 84.11% for unadapted Qwen3VL-8B. This supports task adaptation, but does not make GPT-4o an error-free source of labels.

After excluding the human validation set, GPT-4o supplies answers and chain-of-thought (CoT) reasoning for the remaining data. Filtering and balancing yield 109K positive and 109K negative questions. The authors compare three Qwen3VL-8B training formats: direct answer generation, a two-layer MLP classifier over the final hidden state, and reasoning followed by an answer. They choose the third not because it has the highest accuracy, but because it approaches the best classification result while explaining which requirement failed.

Loss & Training

All three paradigms train for 1 epoch. Direct and reasoning-based prediction supervise the answer or the reasoning-plus-answer sequence, respectively; representation-based classification applies binary supervision through the two-layer MLP. The main text does not provide a complete reproducible loss specification, learning rate, or optimizer configuration, so no additional hyperparameters are inferred here.

The reasoning judge can help inspect failures and is discussed as a candidate reward signal. The paper does not report generator training with this reward. Its potential use for optimization remains a proposed application rather than a demonstrated improvement.

Key Experimental Results

Main Results

Source: selected representative models from Table 2. Values retain the original percentage-style score scale, all higher-is-better; Text is the transformed rendering score, not GNED itself. Short/long denotes semantically paired prompts. Section 4.1 reports basic, advanced, and designer tiers, but the available main text does not explicitly identify Table 2 as full-set or TestMini evaluation, nor fully specify per-model sampling settings. Rankings should not be treated as comparisons under strictly matched compute budgets.

Model	Overall Short	Overall Long	Text Short	Text Long	Designer Short	Designer Long
Qwen-Image	87.0	87.5	79.0	81.0	89.9	90.6
FLUX.2 dev	87.2	87.3	71.0	73.0	91.3	90.9
SD 3.5	69.5	68.5	55.0	43.0	66.0	65.6
JanusPro	65.5	66.6	27.0	32.0	67.4	59.3
Emu3.5-Image	88.4	87.1	82.0	83.0	90.5	89.4
GPT-Image-1	86.1	87.6	79.0	81.0	90.8	92.5
NB-Pro	93.1	93.3	84.0	85.0	95.2	95.5

Ablation Study

Source: prose results in Section 3.4; these ablations have no separate table number. Every accuracy uses the same human-annotated 560-question validation set, not the generation-quality scores above. The final three rows use the same backbone and each train for 1 epoch.

Evaluator / Paradigm	Accuracy	Note
GPT-4o	90.18%	Teacher reliability reference
Qwen3VL-8B, unadapted	84.11%	Off-the-shelf evaluation
Qwen3VL-8B, direct prediction	89.46%	Binary answer only
Qwen3VL-8B, representation classification	91.43%	Final hidden state plus two-layer MLP
Qwen3VL-8B, reasoning-based prediction	91.07%	Selected evaluator with reasoning

Key Findings

• The reasoning evaluator improves over the unadapted backbone by 6.96 percentage points but trails representation classification by 0.36 percentage points. Its selection is motivated by interpretability, not the highest classification accuracy.
• NB-Pro scores 93.1/93.3 Overall with limited short/long variation. SD 3.5 drops from 55.0 to 43.0 on Text, showing that a small overall change can conceal substantial degradation in one capability.
• The claim that closed-source text rendering surpasses every open-source model is too broad: Emu3.5-Image scores 82.0/83.0 on Text, above GPT-Image-1 at 79.0/81.0. Conclusions should follow the specific models and columns.
• Section 4.2 additionally compares CompBench++, GenAI Bench, and TIIF-Bench, inviting 10 volunteers to blindly rank outputs from 8 models. Exact Spearman correlations appear in an appendix absent from the available cache, so no coefficient is quoted.

Highlights & Insights

• Short/long pairing makes robustness to prompt rewriting measurable. It is more diagnostic than merely adding long prompts because the same semantic requirement can be tracked under additional description.
• Checklists localize errors to individual conditions. Subsequent model development should preserve both per-condition outcomes and whole-prompt success rates to prevent averages from hiding critical failures.
• GNED penalizes omissions and additions together. Its global-matching idea transfers to other set-valued outputs, but spatial layout still requires additional constraints.

Limitations & Future Work

• The authors acknowledge that prompts predominantly involve common objects, use English exclusively, and do not systematically vary formal versus conversational language. Length pairing does not replace these generalization tests.
• The evaluator validation set contains only 560 questions, and the main text reports no confidence intervals. A 0.36-percentage-point gap does not establish a stable accuracy ordering between training paradigms.
• Training and validation outputs originate from the benchmark's prompt system. Validation images are excluded from training, but the main text does not establish complete prompt- or generator-level separation; judge generalization needs testing on new prompts and models.
• GPT-4o pseudo-labels, OCR errors, and generated style references can introduce systematic bias. The inconsistent style-metric description particularly requires implementation, reference-image, and aggregation checks before reproduction.

Related Work & Insights

• vs CompBench++ / GenEval: TIIF-Bench explicitly adds short/long prompts, text, and designer tasks beyond composition, expanding diagnostic coverage rather than merely increasing prompt count.
• vs CLIPScore / VQAScore: Global matching or coarse question answering can miss local constraints. Attribute-level binary verification addresses that limitation but still depends on the judge's visual understanding.
• vs Lex-Art: The benchmark reuses its text-prompt source and introduces GNED to penalize both additions and omissions. It is not a new text-rendering generator.
• Implications for reward modeling: Interpretable binary checks can provide component-level feedback, but turning an offline evaluator into an optimization reward requires further tests of exploitability and human agreement.

Rating

• Novelty: 4/5. Length pairing, specialized metrics, and an adapted judge are complementary, although several underlying techniques are established.
• Experimental Thoroughness: 4/5. Multiple generator families and evaluator paradigms are covered, but larger human validation and explicit generalization splits remain necessary.
• Writing Quality: 3/5. The problem is clearly motivated, but the style metric and some model-comparison claims are inconsistent.
• Value: 4/5. Useful for diagnosing text-to-image instruction following, provided dimension-level outcomes and evaluator uncertainty remain visible.