Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Images

Conference: AAAI2026 arXiv: 2502.05066
Code: sprintml/ToxicBench / sprintml/SafeTextGen
Area: Image Generation Keywords: Diffusion Models, NSFW Safety, Text-in-Image, Safety Fine-tuning, benchmark

TL;DR

This paper identifies a novel threat of NSFW text embedded in diffusion-model-generated images, proposes NSFW-Intervention — a targeted LoRA fine-tuning method applied to text-rendering layers — and releases the ToxicBench benchmark.

Background & Motivation

• Mainstream diffusion models (SD3, SDXL, Flux, DeepFloyd IF) can render not only realistic visual content but also legible text within generated images (e.g., signage, captions, typography).
• Prior NSFW safety work has focused primarily on visual-level harmful content (violence, nudity), successfully mitigated via safety filters and concept erasure techniques.
• However, text rendered within generated images can equally contain slurs, racial epithets, and sexually suggestive terms — constituting a previously overlooked safety threat.
• Such harmful text can escalate to hate speech or ideological propaganda, posing particular risk in the open-weight model setting.

Core Problem

1. Novel Threat Identification: All mainstream diffusion models can embed NSFW text in generated images, and existing safety mechanisms offer no effective defense.
2. Failure of Naïve Approaches:
   • Input-side text filtering: lacks visual context (e.g., "Penetrating" carries different meanings in different scenes) and is ineffective against white-box access to open-source models.
   • Output-side OCR detection: models frequently generate misspelled text that humans can still interpret but OCR+toxicity detectors fail to flag (detection rate below 50% on SDXL).
3. Insufficiency of Existing Methods: Methods such as AURA, ESD, and Safe-CLIP suppress NSFW text but equally degrade benign text generation quality, lacking targeted precision.

Method

ToxicBench Benchmark

• Dataset Construction: 218 prompt templates (adapted from CreativeBench) × 437 NSFW terms (each paired with a semantically similar benign substitute generated by GPT-4).
  • Training split: 337 NSFW–benign word pairs → 73,466 prompt pairs.
  • Test split: 100 held-out word pairs → 21,800 prompt pairs, evaluating generalization to unseen NSFW terms.
• Evaluation Pipeline:
  • Image generation → EasyOCR text extraction → computation of text/image quality metrics.
  • Supports two modes: pre/post-mitigation comparison & standalone NSFW detection.
• Novel Metric NGramLD (N-gram Levenshtein Distance):
  • Extracts all k-gram substrings (\(k \in [1, n+1]\)) from OCR output and computes the minimum edit distance to the target word.
  • Addresses the over-penalization of long text outputs by standard Levenshtein distance, focusing on the most relevant substring match.

NSFW-Intervention Method

Step 1: Constructing the Safety Fine-tuning Dataset

• For each NSFW prompt, generate an image \(I_{\text{NSFW}}\) containing harmful text.
• Replace the NSFW term with its benign substitute and regenerate \(I_{\text{benign}}\) using intermediate activation caching-based image editing.
• The two images differ only in the embedded text, with all other visual elements preserved.
• This yields training triplets \((x_{\text{NSFW}}, I_{\text{NSFW}}, I_{\text{benign}})\).

Step 2: Targeted Safety Fine-tuning

• Building on findings by Staniszewski et al. that text rendering in diffusion models is controlled by a small subset of attention layers:
  • SD3: joint attention layers.
  • SDXL / DeepFloyd IF: cross-attention layers.
• LoRA updates are applied exclusively to these text-rendering layers, substantially reducing the number of trainable parameters.
• Training objective: the model receives text encodings \(\phi(x_{\text{NSFW}})\) from the NSFW prompt but learns to output the benign image \(I_{\text{benign}}\).

Training Loss:

\[\mathcal{L} = \|w(t) \cdot (f_\theta(I_{\text{NSFW}}(t), t, \phi(x_{\text{NSFW}})) - I_{\text{benign}})\|^2\]

• \(w(t)\) adopts logit-normal timestep weighting, prioritizing mid-to-late denoising steps where text is most legible.
• \(I_{\text{NSFW}}(t)\) denotes the NSFW image with Gaussian noise added at timestep \(t\).

Core Advantage: Directly modifies model weights, making it applicable to white-box/open-source settings and bypass-resistant to input/output filtering.

Key Experimental Results

Model	Benign NGramLD Δ↓	NSFW NGramLD Δ↑	Differential Gain	KID
SD3	2.07	3.63	+1.56	0.059/0.061
DeepFloyd IF	3.78	4.72	+0.94	0.059/0.060
SDXL	4.75	5.69	+0.94	0.063/0.065

• Baseline Comparison: AURA achieves a NSFW/benign NGramLD differential of only 0.36 (2.56 vs. 2.20); Safe-CLIP achieves 0.22 (2.87 vs. 2.65) — both far less targeted than the proposed method.
• User Study: The human recognition rate of harmful text in NSFW-prompted images decreases from 78.67% to 26.56%; misspelled NSFW variants drop from 76.41% to 11.45% (unseen during training), while benign text is maintained at 55.40%.
• Ablation: Fine-tuning all attention layers instead of only text-rendering layers yields a NGramLD improvement of only +0.49 on SD3 (vs. +3.63 for the targeted approach), validating the importance of layer selection.
• Image Quality: KID increases by at most 9%; FID degradation is minimal.

Highlights & Insights

1. Novel Threat Definition: The paper is the first to systematically expose the safety blind spot of NSFW text embedded in generated images, with a clear and practically relevant problem formulation.
2. Precise Layer-Level Intervention: By exploiting the locality of text-rendering layers, LoRA fine-tuning is applied to a minimal set of parameters, balancing safety and generation quality.
3. Ingenious Data Construction: Activation-cache-based image editing produces image pairs differing only in rendered text, providing precise supervision signals for training.
4. Complete Benchmark Ecosystem: ToxicBench encompasses a dataset, evaluation metric (NGramLD), and pipeline, offering a standardized toolkit for future research.
5. Strong Generalization: The method is effective on unseen NSFW terms and spelling variants, and is extensible to VAR-based models (e.g., Infinity).

Limitations & Future Work

• NSFW text suppression is not fully complete; some samples remain human-readable (e.g., highly sensitive terms such as "giant cocks").
• Benign text quality incurs some degradation (CLIP-Score on SD3 drops from 91.42 to 85.10), leaving room for optimization.
• The method relies on prior knowledge of text-rendering layers; new architectures require re-identification of these layers.
• Evaluation depends on OCR accuracy; artistic fonts or severely distorted text may evade detection.
• The dataset is based on English NSFW vocabulary; applicability to multilingual settings is not validated.
• Robustness against adversarial attacks (e.g., deliberately eliciting variant spellings) requires further investigation.

Related Work & Insights

Method	Type	NSFW Suppression	Benign Preservation	Applicable Setting
SLD	Inference guidance	Effective for visual NSFW, not for text	Good	API/black-box only
ESD	Full model fine-tuning	Moderate effect on text	Poor (overall quality degraded)	SD1.4 only
AURA	Neuron suppression	Effective but untargeted	Poor (equal harm to benign)	Requires text encoder
Safe-CLIP	Encoder fine-tuning	Effective but untargeted	Poor (equal harm to benign)	Requires CLIP architecture
NSFW-Intervention	Targeted layer LoRA	Strong and generalizable	Good	General white-box/open-source

• The layer-wise controllability paradigm is transferable to targeted editing of other generative attributes (e.g., style, watermarks, copyright text).
• Image–text alignment safety is a universal concern for multimodal models; the benchmark design methodology proposed here is applicable to video generation models.
• This work is complementary to concept erasure methods (e.g., Selective Amnesia, UCE): the latter erase visual concepts, while this paper erases textual concepts.
• The NGramLD metric is broadly applicable to any task involving text rendering quality evaluation.

Rating

• Novelty: ⭐⭐⭐⭐ — Novel problem definition; the first systematic study of NSFW text generation in images.
• Experimental Thoroughness: ⭐⭐⭐⭐ — Covers 4 models, 3 baselines, ablation studies, and a user study; fairly comprehensive.
• Writing Quality: ⭐⭐⭐⭐ — Clear structure; problem–solution–evaluation logic is rigorous.
• Value: ⭐⭐⭐⭐ — Fills a gap in the safety literature; the benchmark is expected to have lasting impact.

Related Papers

• [AAAI 2026] CausalCLIP: Causally-Informed Feature Disentanglement and Filtering for Generalizable Detection of Generated Images
• [NeurIPS 2025] Detecting Generated Images by Fitting Natural Image Distributions
• [ECCV 2024] Zero-Shot Detection of AI-Generated Images
• [CVPR 2026] The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment
• [CVPR 2025] OpenSDI: Spotting Diffusion-Generated Images in the Open World