Q-REAL: Towards Naturalness and Distortion Evaluation for AI-Generated Content

Conference: ECCV 2026
Paper: ECCV Official
Dataset: Hugging Face
Area: AIGC Detection
Keywords: AIGC Quality Assessment, Fine-grained Evaluation, Naturalness and Distortion, Multimodal Large Language Model, Grounding with Reasoning

TL;DR

To overcome the uninterpretable single-scalar score paradigm in text-to-image quality assessment, this work establishes Q-Real—a 10K-image fine-grained benchmark with 32.8K entities annotated across Naturalness and Distortion dimensions—alongside a progressive three-stage MLLM training pipeline that achieves entity-level judgment, spatial grounding with causal attribution, and consistent quality score prediction.

Background & Motivation

With the rapid emergence of generative diffusion and autoregressive models such as Flux, Kolors, and SD3.5, text-to-image synthesis has achieved impressive photorealism. However, generated outputs still frequently exhibit anatomical anomalies, unnatural physical textures, or the recognizable "AI look". Existing image quality assessment (IQA) datasets and benchmarks—such as AGIQA-3K, Pick-a-Pic, and Q-Eval-100K—primarily compress perceptual quality into scalar mean opinion scores (MOS) or pairwise preferences. Such scalar feedback fails to indicate the exact spatial locations and specific root causes of visual degradation, providing little actionable diagnostic guidance for optimizing generative backbones.

The primary obstacle to fine-grained evaluation lies in disentangling orthogonal perceptual dimensions while managing the prohibitive cost of localized annotations. On the one hand, defects in photorealistic synthesis span both high-level perceptual plausibility (overall realism) and low-level physical deformations. Prior efforts like RichHF-18K rely on coarse pixel heatmaps lacking semantic reasoning, while works such as FakeXplain and X-AIGD either lack structured question answering or omit descriptive rationale generation. On the other hand, exhaustively annotating entity-level bounding boxes and multi-faceted diagnostic analyses across thousands of images is labor-intensive, while fully automated LLM generation suffers from severe hallucinations.

This paper tackles these challenges by decoupling photorealistic evaluation into two complementary dimensions—naturalness and distortion—and constructing a human-in-the-loop automated annotation and adjudication workflow. Core idea: construct Q-Real, a 10K-image, 32.8K-entity benchmark covering naturalness and distortion, and establish a three-stage progressive MLLM training pipeline—object judgment pre-training, image-level grounding with reasoning, and training-inference consistent CoT score regression—to empower models with end-to-end defect localization, explanation, and precise scoring.

Method

Overall Architecture

The Q-Real framework consists of a semi-automated fine-grained annotation pipeline and a three-stage progressive MLLM training methodology. The benchmark introduces two complementary tasks: ObjectQA, which evaluates entity-level binary discriminative judgment, and ImageQA, which assesses full-image defect localization and attribution reasoning. Correspondingly, the model progresses from local binary defect classification to open-vocabulary grounding and causal explanation, and finally to global quality scoring conditioned on dynamically generated Chain-of-Thought (CoT) defect contexts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["T2I Image Pool<br/>Q-Eval-100K subset + recent SOTA generation"] --> B["Hybrid Annotation Pipeline<br/>GPT-4o naming + Grounding DINO + 3-annotator voting"]
    B --> C["Q-Real Benchmark & Dataset<br/>10K images / 32,815 entities / naturalness & distortion"]
    C --> D["Stage 1: Object-Level Judgment Training<br/>Entity-cropped 6 binary QA tuning for defect perception"]
    D --> E["Stage 2: Fine-Grained Evaluation Training<br/>Full-image defect filtering + bounding box + rationale"]
    E --> F["Stage 3: Quality Score Prediction Training<br/>Dynamic Stage-2 CoT context + 5 rating token expectation"]
    F --> G["Unified Output Capabilities<br/>Judgment + spatial grounding + reasoning + continuous score"]

Key Designs

1. Semi-Automated Collaborative Annotation Pipeline: Decoupled Dimensions and Multi-Annotator Adjudication

To overcome the prohibitive cost and hallucination risks of fine-grained multi-entity annotation, Q-Real introduces a five-step hybrid annotation protocol. First, GPT-4o extracts major entity names by jointly processing images and textual prompts. Grounding DINO then localizes these entities with bounding boxes, filtering out low-confidence and visually ambiguous candidates. Next, the pre-trained Q-Eval-Score model assesses the quality score of each detected object, providing a strong conditioning prior for GPT-4o to generate initial objective questions and explanatory descriptions across naturalness and distortion dimensions. Finally, human verification ensures data integrity: annotators correct bounding boxes and label mismatches, while three independent reviewers inspect the descriptions. If inaccuracies are found, reviewers independently rewrite the descriptions and vote on the objective questions (majority voting), synthesized via an LLM. An additional 400 human-centric images receive deeper annotations across face, hands, limbs, torso, and clothing.

The naturalness dimension inspects color authenticity, lighting consistency, and material realism to determine whether an entity exhibits an artificial "AI look". The distortion dimension identifies anatomical deformities, geometric collapse, abnormal textures, and garbled text. In total, 32,815 entities across 10K images are annotated with 6 objective questions and subjective attribution rationales.

2. Spatial Matching and Causal Evaluation Benchmark: Q-Real Bench

To rigorously benchmark MLLMs on fine-grained quality evaluation, Q-Real Bench defines two complementary tasks. ObjectQA presents 2 naturalness and 4 distortion questions per entity, measuring basic binary discrimination via Accuracy, Precision, F1, and AUC.

ImageQA requires the model to identify all defective entities, predict bounding boxes, and generate diagnostic explanations. To reliably match predicted boxes \(P=\{p_i\}_{i=1}^N\) with ground-truth boxes \(G=\{g_j\}_{j=1}^M\), a bipartite cost matrix \(C \in \mathbb{R}^{N \times M}\) based on spatial overlap is constructed:

\[C_{ij} = 1 - \text{IoU}(p_i, g_j)\]

The Hungarian matching algorithm resolves the optimal assignment. Crucially, the benchmark establishes a joint spatial-semantic metric, Detection Rate: a predicted entity is counted as successfully detected only if its matched \(\text{IoU} > 0.5\) and the semantic agreement of its generated description against the ground truth (\(\text{LLM-Score}\)) exceeds 0.5.

3. Progressive Three-Stage Tuning and Training-Inference Consistent (TIC) Quality Scoring

Because off-the-shelf MLLMs tend to ground all entities indiscriminately without recognizing quality flaws, the paper designs a progressive three-stage training pipeline. Stage 1 fine-tunes MLLMs on cropped object-level binary QA pairs to instill basic sensitivity to visual degradation. Stage 2 trains on full images, prompting the model to detect only defective entities and output <label><coordinate><description>, successfully transitioning local discrimination into full-image grounding and reasoning.

Stage 3 leverages the acquired diagnostic capability to predict global quality scores (Q-Real-Score). The fine-grained defects identified by the Stage-2 model are structured into a Chain-of-Thought (CoT) prompt (<detected_issues>) as reference information. Critically, to prevent distribution shift, Training-Inference Consistency (TIC) is enforced: during Stage 3 training, the model does not receive ground-truth defect annotations; instead, the Stage-2 model runs inference dynamically to generate the prompt context, aligning training conditions with real-world deployment.

Loss & Training

To predict continuous quality scores without disrupting the language model's latent representation, the continuous MOS scale \([1, 5]\) is partitioned into five discrete rating tokens: \(\mathcal{R} = \{\text{Bad}, \text{Poor}, \text{Fair}, \text{Good}, \text{Excellent}\}\) with assigned weights \(w_j \in \{0, 0.25, 0.5, 0.75, 1.0\}\).

At inference time, the logits \(z_j\) of the five rating tokens are normalized via Softmax to obtain probabilities \(p_j = \frac{\exp(z_j)}{\sum_{k=1}^5 \exp(z_k)}\), and the continuous score is computed as the expected value \(\hat{r} = \sum_{j=1}^5 p_j \cdot w_j\). The model is optimized using LoRA with a joint loss function:

\[\mathcal{L} = \alpha \mathcal{L}_{CE} + \beta \mathcal{L}_{MSE}\]

where \(\mathcal{L}_{CE} = -\sum_{i=1}^{N_{ans}} y_i \log(p_i)\) supervises response generation and \(\mathcal{L}_{MSE} = (\hat{r} - r_{MOS})^2\) minimizes the squared residual between expected rating score \(\hat{r}\) and true MOS \(r_{MOS}\) (default \(\alpha = 1, \beta = 1\)).

Key Experimental Results

Main Results

Evaluations were performed across commercial closed-source MLLMs (GPT-5.4, Gemini3.1-Pro), dedicated quality models (Q-Eval-Score), and open-source models (Qwen and InternVL families).

Table 1: Performance comparison on ImageQA task on Q-Real Bench (Table 2 in the paper)

Model	Naturalness IoU	Naturalness Detection Rate	Naturalness LLM Score	Distortion IoU	Distortion Detection Rate	Distortion LLM Score
GPT-5.4	0.700	0.069	0.439	0.661	0.063	0.197
Gemini3.1-Pro	0.752	0.481	0.376	0.820	0.136	0.431
Q-Eval-Score	0.835	0.165	0.353	0.838	0.038	0.249
Qwen3-VL-8B (Zero-Shot)	0.848	0.313	0.435	0.848	0.206	0.341
Qwen3-VL-8B (Q-Real-tuned)	0.834	0.548 (+23.5%)	0.722 (+28.7%)	0.832	0.523 (+31.7%)	0.639 (+29.8%)
InternVL3-8B (Zero-Shot)	0.888	0.060	0.230	0.878	0.274	0.413
InternVL3-8B (Q-Real-tuned)	0.829	0.527 (+46.7%)	0.737 (+50.7%)	0.825	0.481 (+20.7%)	0.645 (+23.2%)

Table 2: Cross-dataset general quality assessment performance (Table 3 in the paper)

Model	Q-Real (SRCC ↑)	Q-Real (PLCC ↑)	AGIQA-3K (SRCC ↑)	AGIQA-3K (PLCC ↑)	ImageReward (Acc ↑)
CLIP-IQA	0.264	0.262	0.074	0.088	0.493
IPCE	0.631	0.636	0.074	0.057	0.495
Q-Align	0.651	0.644	0.587	0.582	0.601
Q-Eval-Score	0.690	0.693	0.572	0.558	0.556
Q-Real-Score (Ours)	0.710	0.720	0.608	0.611	0.607

Ablation Study

Ablations examine prompt components in Stage 3: naturalness CoT (N-CoT), distortion CoT (D-CoT), and Training-Inference Consistency (TIC).

Table 3: Ablation study of prompt design and TIC strategy in Stage 3 (Table 6 in the paper)

N-CoT	D-CoT	TIC	Q-Real SRCC	Q-Real PLCC	AGIQA-3K SRCC	AGIQA-3K PLCC	ImageReward Acc
-	✓	✓	0.700	0.694	0.570	0.598	0.558
✓	-	✓	0.696	0.700	0.576	0.601	0.560
✓	✓	-	0.692	0.686	0.550	0.574	0.557
✓	✓	✓	0.710	0.720	0.608	0.611	0.607

Key Findings

• Distortion details correlate more strongly with quality scores: Incorporating distortion CoT alone achieves higher SRCC than naturalness CoT alone on Q-Real (0.700 vs 0.696), indicating that severe anatomical or structural deformations exert a harsher penalty on human perceptual quality than subtle color or lighting artificiality.
• Training-Inference Consistency (TIC) prevents catastrophic test-time degradation: Omitting TIC (training on ground-truth flaws and testing on predicted flaws) results in the lowest correlation across all datasets (SRCC drops from 0.710 to 0.692 on Q-Real and from 0.608 to 0.550 on AGIQA-3K), confirming that exposure to model-generated defect patterns during training is essential.
• Fine-tuning bridges the gap between grounding and discrimination: Zero-shot frontier models (including GPT-5.4 with Detection Rate < 0.07) suffer from low recall and indiscriminate bounding box output. After fine-tuning on Q-Real, open-source models achieve Detection Rates above 0.50 and LLM-Scores above 0.72, dramatically surpassing off-the-shelf commercial baselines.

Highlights & Insights

• Decoupled Naturalness vs. Distortion Taxonomy: Systematically separates semantic-level artificiality ("AI look") from physical-level degradation (anatomical failure, blur), giving generative model developers precise diagnostic signals.
• Dynamic Reasoning Injection with TIC: Demonstrates that conditioning quality scoring on dynamically predicted spatial defects provides more robust guidance than relying on static ground-truth annotations during training.
• Strict Joint Metric (Detection Rate): By enforcing both geometric bounding box alignment (\(\text{IoU} > 0.5\)) and semantic agreement (\(\text{LLM-Score} > 0.5\)), this benchmark prevents false-positive rewards for empty boxes or hallucinated descriptions.

Limitations & Future Work

• Author-admitted limitations: Defining the boundary of naturalness in stylized or abstract artistic generations remains partially subjective; the dedicated human portrait breakdown covers 400 images and warrants further expansion.
• Observed limitations: The benchmark focuses exclusively on single-image 2D photorealism; it does not yet evaluate temporal coherence, physical causality, or frame-to-frame distortions in text-to-video (T2V) models.
• Future directions: Integrating the localized defect bounding boxes into diffusion cross-attention loss or direct preference optimization (DPO) as dense reward signals to automatically rectify flaws during image synthesis.

Related Work & Insights

• vs RichHF-18K: RichHF-18K maps plausibility and aesthetics to coarse pixel heatmaps without structured natural language rationales; Q-Real provides bounding boxes, structured QA, and fine-grained attribution texts.
• vs Q-Eval-100K: Q-Eval-100K focuses on macro-level image-text alignment and scalar MOS ratings; Q-Real enriches a subset with fine-grained multi-entity annotations to diagnose why an image received a specific score.
• vs FakeXplain & X-AIGD: FakeXplain focuses on forensic AI detection without multi-turn question answering; X-AIGD provides region classification without causal explanation. Q-Real unifies detection, localization, causal reasoning, and continuous scoring.

Rating

• Novelty: ⭐⭐⭐⭐ [First comprehensive benchmark decoupling naturalness and distortion with full-spectrum entity-level grounding and reasoning]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across commercial MLLMs, cross-dataset evaluations, Hungarian-matched detection rates, and prompt ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural taxonomy, well-motivated problem formulation, and precise mathematical definitions]
• Value: ⭐⭐⭐⭐⭐ [Provides an actionable diagnostic tool and fine-grained benchmark for next-generation generative image evaluation]

Related Papers

• [ECCV 2026] Beyond Artifacts: Real-Centric Envelope Modeling for Reliable AI-Generated Image Detection
• [NeurIPS 2025] Can LLMs Write Faithfully? An Agent-Based Evaluation of LLM-generated Islamic Content
• [ECCV 2026] Trustworthy Image Authentication using Forensic Knowledge Graphs
• [ECCV 2026] Efficient Document Tampering Localization with Multi-Level Discrepancy Features and Unified DCT-Quantization Embedding
• [ACL 2026] C-ReD: A Comprehensive Chinese Benchmark for AI-Generated Text Detection Derived from Real-World Prompts