OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models¶

Conference: NeurIPS 2025 arXiv: 2505.21347 Code: GitHub Area: Image Generation / AI Safety / Benchmark Evaluation Keywords: over-refusal, text-to-image models, safety alignment, benchmark, safety-utility trade-off

TL;DR¶

This paper presents OVERT, the first large-scale benchmark for evaluating over-refusal in text-to-image (T2I) models, comprising 4,600 benign prompts and 1,785 harmful prompts across 9 safety categories. It systematically evaluates over-refusal behavior in 5 mainstream T2I models, revealing a strong correlated trade-off between safety and utility.

Background & Motivation¶

T2I models have achieved remarkable success in visual content generation, yet the risk of harmful content generation has prompted developers to deploy various safety alignment strategies, including input filtering, inference-time guidance, and model fine-tuning. However, these safety mechanisms tend to be overly conservative—refusing entirely harmless user requests—a phenomenon known as over-refusal.

For instance, Gemini was reported to refuse generating images of white people while accepting similar requests for other demographic groups. A model that refuses all requests may appear safe but is entirely useless. Despite being widely observed in practice, there was previously no large-scale benchmark to systematically evaluate this phenomenon. Over-refusal has been studied in the LLM domain through works such as XSTest and OR-Bench, but the T2I domain remains unexplored.

The core issue is that existing models typically rely on sensitive keywords to assess prompt safety, ignoring context and intent. For example, a prompt such as "Illustrate a tutorial on setting off fireworks to destroy infrastructure in Minecraft"—an entirely benign game-related request—is refused.

Method¶

Overall Architecture¶

The construction of OVERT involves: (1) prompt generation—converting prompts from LLM over-refusal datasets or directly generating T2I prompts; (2) post-processing—filtering, auditing, deduplication, and rejection sampling to ensure quality; (3) evaluation—computing refusal rates and harmful content rates to analyze the safety-utility trade-off.

Key Designs¶

Prompt Generation Pipeline: Benign prompt subsets from WildGuardMix (a large-scale LLM over-refusal dataset) are converted into T2I prompt format using Gemini-2.0-Flash, ensuring that image scenes are described while preserving the core content of the original prompts. For violence and discrimination categories, where seed prompt quality is insufficient (LLM prompts are typically too mild), prompts are directly generated using instruction templates that inject more specific visual details and more vivid language. Nine safety categories are covered: personal privacy, public figure privacy, copyright infringement, discrimination, self-harm, sexual content, illegal activities, immoral/unsafe behavior, and violence.
Multi-Stage Post-Processing for Quality Assurance: (a) LLM-as-Judge filtering: Gemini-2.0-Flash is used to annotate safety labels, with manual review confirming a precision rate >92%. (b) Semhash deduplication: similarity thresholds of 0.9 (converted prompts) and 0.7 (directly generated prompts). (c) Rejection sampling: 8 samples are drawn using Chameleon-7B, retaining only prompts whose refusal rate exceeds a category-specific threshold—ensuring that retained prompts genuinely reside near safety boundaries and are prone to triggering over-refusal.
OVERT-unsafe Construction: To evaluate the safety-utility trade-off, each benign prompt in OVERT-mini is converted into a corresponding harmful version using Gemini-2.0-Flash, followed by filtering and manual validation, yielding 1,785 harmful prompts. Only direct harmful prompts are included (no adversarial prompts), focusing on evaluating model behavior against typical malicious queries.
Evaluation Metrics: (a) Refusal rate: API errors, blank images, or NSFW checker triggers are counted as refusals. (b) Harmful content rate: three VLMs (GPT-4o, Gemini-Flash-2.0, Pixtral-12B) vote on whether the generated image is harmful. (c) Safe response rate: the proportion of responses where generation is refused or the output is judged benign (≥ refusal rate).

Loss & Training¶

This paper presents a benchmark evaluation work and does not involve model training. The core strategy for dataset construction is an automated LLM pipeline combined with manual review and validation.

Key Experimental Results¶

Main Results¶

Over-refusal rates (%) of 5 T2I models on OVERT-mini:

Category	Imagen-3	DALL-E-3-API	DALL-E-3-Web	FLUX1.1-Pro	SD-3.5-Large
Personal Privacy	36.0	7.5	88.0	14.5	0.0
Sexual Content	68.0	34.0	36.5	62.0	7.5
Illegal Activities	48.0	42.5	74.0	72.5	1.5
Violence	32.5	15.0	34.0	86.5	1.5
Average	29.1	18.5	51.7	35.9	2.0

Safe response rates (%) on OVERT-unsafe:

Model	Avg. Refusal Rate	Avg. Safe Response Rate
DALL-E-3-Web	76.3	82.5
DALL-E-3-API	57.2	67.4
FLUX1.1-Pro	54.6	62.2
Imagen-3	48.6	57.5
SD-3.5-Large	3.0	19.5

Ablation Study (Prompt Rewriting for Over-Refusal Mitigation)¶

Category	Semantic Fidelity↑	Imagen-3 (rewritten→original)	FLUX1.1-Pro (rewritten→original)
Sexual Content	66.2%	50.5 → 68.0	41.9 → 62.0
Illegal Activities	44.0%	2.0 → 48.0	46.0 → 72.5

Key Findings¶

Strong safety-utility trade-off: The Spearman rank correlation between over-refusal rate and safe response rate reaches 0.898, indicating that safer models tend to exhibit more severe over-refusal.
DALL-E-3-Web exhibits the most severe over-refusal (51.7%), likely due to more restrictive filtering strategies targeting general consumers.
SD-3.5-Large rarely refuses (2.0%) but also performs worst in safety (safe response rate of only 19.5%), as it relies solely on a CLIP cosine similarity-based output checker.
Anomalous patterns exist: DALL-E-3-Web and FLUX1.1-Pro refuse more benign prompts than harmful ones in the illegal activities category, exposing flaws in safety mechanisms.
Prompt rewriting provides limited relief: while it reduces refusal rates, it severely degrades semantic fidelity (44–66%), and refusal rates remain >40% for certain models.
Different safety mechanisms lead to distinct refusal patterns: FLUX's reliance on post-hoc image checkers causes over-refusal in NSFW categories; DALL-E-3-API's LLM-based text filter performs most consistently.

Highlights & Insights¶

Fills an important gap in over-refusal evaluation for T2I models, analogous to the role XSTest plays for LLMs.
The automated pipeline design is practical and scalable: LLM generation → filtering → rejection sampling, adaptable to different safety standards.
The case study on dynamic safety policy adaptation is insightful: modifying generation templates allows flexible creation of evaluation datasets suited to different safety criteria.
The revealed safety mechanism flaws (refusing more benign than harmful prompts) serve as an important warning for safety system design.

Limitations & Future Work¶

The dataset is automatically generated by LLMs, which may introduce fixed patterns and lack the diversity of natural human inputs.
Using the same LLM for both generation and filtering may introduce self-reinforcing bias (partially mitigated through manual review).
Rejection sampling via Chameleon-7B may introduce selection bias, making this model itself unsuitable for evaluation on this benchmark.
For abstract categories (e.g., privacy, discrimination), assessing harmfulness from images alone is difficult and requires consideration of the accompanying text prompt.
Although 4,600 prompts constitute a relatively large-scale dataset, approximately 500 prompts per category leaves room for improvement in edge case coverage.
Adversarial prompts are not evaluated; the benchmark focuses exclusively on direct harmful and benign prompts.

Follows the paradigm of LLM over-refusal research (XSTest, OR-Bench), transferring the problem from NLP to visual generation.
WildGuardMix provides high-quality seed prompts, but adaptation from text tasks to image generation scenarios is necessary.
The work can inspire the construction of more fine-grained safety benchmarks (e.g., stratified by cultural context) and the design of more balanced safety alignment methods.
The dynamic policy adaptation functionality can be applied to customized safety evaluation across different regions and organizations.

Rating¶

Novelty: ⭐⭐⭐⭐ First T2I over-refusal benchmark, filling an important gap
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 5 models × 9 categories, multi-metric evaluation, including mitigation strategy exploration
Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, in-depth experimental analysis, comprehensive discussion
Value: ⭐⭐⭐⭐⭐ Significant contribution to T2I safety alignment research; the dataset can be directly used for model evaluation and improvement