Skip to content

title: >- [Paper Note] IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment description: >- [ECCV 2026][VLM Reasoning][Image Quality Assessment] Augmenting MLLMs with specialized perceptual tools to generate structured visual evidence, achieving SOTA interpretable IQA via SFT and GRPO. tags: - ECCV 2026 - VLM Reasoning - Image Quality Assessment - Tool Use - Reinforcement Learning date: 2026-09-19 content_hash: 1b1592fbb735cc50

IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/zibuyu-02/IQA-T1
Area: VLM Reasoning / Image Quality Assessment
Keywords: Image Quality Assessment, Multimodal Large Language Model, Tool Use, Visual Evidence Reasoning, Reinforcement Learning

TL;DR

IQA-T1 presents the first tool-based visual evidence reasoning framework for Image Quality Assessment (IQA), equipping MLLMs with a library of 15 specialized perceptual analysis tools and the Q-Tool reasoning dataset to autonomously gather structured visual evidence, substantially alleviating semantic bias and achieving state-of-the-art average performance across seven benchmarks.

Background & Motivation

Image Quality Assessment (IQA) aims to evaluate perceptual image quality in alignment with human visual judgment, serving as a critical referee in image restoration, enhancement, compression, and open-world autonomous vision systems. Driven by the rapid advancement of multimodal large language models (MLLMs), recent research has progressively shifted IQA from conventional black-box numerical regression toward language-driven frameworks that can articulate textual rationales behind perceived degradations.

However, existing MLLM-based IQA paradigms suffer from a fundamental flaw: their judgments rely almost exclusively on high-level, semantically biased internal representations acquired during general pretraining, rendering them blind to low-level physical distortions. Whether employing pure-text chain-of-thought (CoT) reasoning (such as Q-Insight and VisualQuality-R1) or patch-level visual cropping (such as Zoom-IQA and Q-Probe), models still fail to bridge this representational gap. Pure-text reasoning hallucinate plausible stories detached from actual degradation, while cropped image patches remain unstructured raw pixels that lack explicit attribution to specific physical defects. Formally, when an image \(I\) is corrupted by severe noise into \(I_d\), human perceptual scores degrade dramatically \(|s(I) - s(I_d)| \gg \epsilon\), yet the MLLM visual embedding distance \(\|z_v(I) - z_v(I_d)\|_2\) remains negligibly small, creating a catastrophic perceptual bottleneck.

This paper tackles the challenge from a novel perspective: since frozen or generic visual encoders inherently discard low-level degradation cues in favor of semantic invariance, the model should be equipped with external perceptual tools that extract deterministic, structured visual evidence to ground its reasoning chain. Core idea: build a specialized perceptual tool library covering 15 low-level attributes alongside the Q-Tool multimodal reasoning dataset, inject tool invocation syntax and evidence interpretation via SFT, and train an adaptive invocation policy using multi-reward GRPO reinforcement learning to enable verifiable, evidence-grounded IQA reasoning.

Method

Overall Architecture

IQA-T1 formulates image quality assessment as a conditional sequence generation task guided by dynamically acquired, tool-generated visual evidence. Given an input image \(I \in \mathbb{R}^{H \times W \times 3}\) and a scoring prompt \(P\) (e.g., instructing the system to rate quality on a 1-to-5 scale), the model does not rely solely on its semantic vision tokens \(z_v = \mathcal{M}(E_\phi(I))\). Instead, it dynamically queries a specialized perceptual tool library \(\mathcal{T}\) to collect an evidence subset \(E_r \subseteq \mathcal{E}\).

The training pipeline operates in two distinct stages. In Stage I (Evidence-Grounded Reasoning Learning), the model undergoes supervised fine-tuning (SFT) on the newly curated Q-Tool dataset (11k multimodal reasoning chains) to master structured analysis templates, tool calling syntax, and evidence-to-judgment deduction. In Stage II (Adaptive Tool Invocation Policy Optimization), Group Relative Policy Optimization (GRPO) aligns the model with perceptual quality targets through four collaborative rewards (format, score accuracy, tool usage efficiency, and repetition penalty), empowering the model to decide which tools to call and when to terminate evidence collection.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Prompt"] --> B["Structured Perceptual Analysis Tool Library<br/>Extracting noise residual/gradient stats/frequency spectra"]
    B --> C["Evidence-Driven Dynamic CoT Reasoning<br/>Autonomous tool invocation & visual observation integration"]
    C --> D["Multi-Objective Reinforcement Learning Policy<br/>GRPO optimizing accuracy/usage efficiency/repetition penalty"]
    D --> E["Output Evidence-Grounded Rationale & Quality Score"]

Key Designs

1. Structured Perceptual Analysis Tool Library: Decoupling Low-Level Degradation from High-Level Semantics

To eliminate the semantic bias where generic vision encoders overlook low-level distortions, IQA-T1 identifies seven core perceptual attributes governing image quality: Structure, Sharpness, Noise, Artifacts, Luminance, Color, and Naturalness. Around these attributes, the framework establishes 15 specialized, deterministic analysis tools.

Each tool \(T_k \in \mathcal{T}\) maps the input image into a degradation-oriented representation space \(T_k: \mathbb{R}^{H \times W \times 3} \to \mathcal{D}_k\). Specifically, NoiseResidualMap isolates sensor or compression noise by subtracting a smoothed filter output from the input; FourierMagnitudeSpectrum applies a 2D discrete Fourier transform to expose periodic artifacts, compression ringing, and high-frequency energy loss; GradientOrientationCoherenceMap and GradientMagnitudeHistogram quantify boundary edge sharpness and directional consistency, diagnosing blur and over-smoothing; and LuminanceHistogram alongside PerceptualColorfulnessMap capture under/over-exposure and color saturation anomalies. All tools share unified invocation interfaces and produce deterministic, moderately downsampled representations \(e_k = \psi(T_k(I))\), preserving spatial structure while conserving visual token budgets.

2. Evidence-Driven Dynamic CoT Reasoning: Grounding Judgments in Verifiable Visual Observations

To teach the MLLM how to orchestrate external tools and integrate evidence, the authors created Q-Tool, the first dedicated dataset comprising 11k verified multimodal reasoning chains. Instead of unconstrained prompting that induces hallucinated or redundant calls, Q-Tool synthesizes reasoning traces using expert-curated templates (Structure \(\to\) Noise \(\to\) Color/Exposure \(\to\) Final Assessment) guided by GPT-4o, followed by rigorous multi-stage verification on tool appropriateness, visual grounding, and logical consistency.

During Stage I (SFT), the training sequence contains the original image \(I\), the ground-truth invoked visual evidence \(V\), and the textual reasoning chain \(T\) terminating in score \(s_\text{gt}\). Crucially, the model is not tasked with generating the evidence images themselves; loss masking is applied over all visual evidence tokens: $\(\mathcal{L}_\text{SFT} = -\sum_{t=1}^{L} \log p_\theta(y_t \mid y_{<t}, I, V, T)\)$ This forces the model to focus purely on structured reasoning, valid invocation tags, and the causal link between visual evidence and quality deductions. At inference time, the model dynamically gathers its evidence subset \(E_r \subseteq \mathcal{E}\): $\(\hat{Y} = \arg\max_{Y} \sum_{t=1}^{T} \log p_\theta(y_t \mid \mathbf{h}_p, y_{<t}, E_r)\)$ Every analytical claim (e.g., "the background suffers from elevated high-frequency noise") directly references a preceding tool-generated visual token, ensuring all quality rationales are grounded in verifiable visual evidence.

3. Multi-Objective Reinforcement Learning Policy: Adaptively Balancing Evidence Sufficiency and Invocation Efficiency

While SFT instills template execution, it produces rigid invocation patterns incapable of adapting to diverse real-world images exhibiting varied degradation complexity. To optimize an adaptive invocation policy, Stage II employs Group Relative Policy Optimization (GRPO). For each input \(x\), the policy samples a group of \(n\) reasoning chains \(\{y_1, \dots, y_n\}\) and calculates advantages relative to the group mean and variance: $\(\mathcal{L}_\text{GRPO}(\theta) = -\mathbb{E}_{x \sim \mathcal{D}, y_i \sim \pi_\theta(\cdot \mid x)} \left[ \hat{A}_i \frac{\pi_\theta(y_i \mid x)}{\pi_\text{ref}(y_i \mid x)} - \beta \mathbb{D}_\text{KL}\left(\pi_\theta \parallel \pi_\text{ref}\right) \right]\)$ where \(\hat{A}_i = (R_i - \mu_R) / \sigma_R\), and \(\pi_\text{ref}\) denotes the SFT baseline. The composite reward function \(R = \lambda_\text{fmt} R_\text{fmt} + \lambda_\text{score} R_\text{score} + \lambda_\text{tool} R_\text{tool} + \lambda_\text{rep} R_\text{rep}\) balances multiple objectives: - Format Reward \(R_\text{fmt}\): Binary reward (1 or 0) enforcing correct XML tags and JSON answer structures. - Scoring Reward \(R_\text{score}\): Exponential decay metric rewarding accurate numerical quality prediction: \(R_\text{score} = \exp(-\alpha |s_\text{pred} - s_\text{gt}|)\). - Tool Usage Reward \(R_\text{tool}\): A weak constraint balancing evidence sufficiency against over-invocation (\(N = |E_r|\), upper bound \(M = 4\)): $\(R_\text{tool}(N) = \begin{cases} 0, & N = 0 \\ 1, & 1 \le N \le M \\ 1 - \gamma(N - M)^2, & N > M \end{cases}\)$ - Repetition Penalty \(R_\text{rep}\): Suppresses repetitive tool calling without information gain, returning -1 if any tool \(T_k\) is invoked more than once (\(n_k > 1\)), and 0 otherwise.

This policy guides the model to adaptively select tools: simple low-light pictures trigger only 1-2 exposure checks, while complex blurry/noisy samples evoke coordinated gradient and residual analysis, minimizing computation while maximizing accuracy.

A Worked Example

Consider an underexposed, blurry photograph with sensor noise captured on a beach: 1. Initial Structural Inspection: The model opens its <think> trace, inspects the overall composition, and noticing softened contours, emits <tool_call> GradientMagnitudeMap </tool_call>. The system executes the tool and injects the resulting gradient map as visual tokens. Observing faint edge responses, the model deduces: "structural clarity is severely compromised, with weak edge sharpness across the foreground subjects." 2. Frequency and Noise Diagnostic: The model initiates a second tool call <tool_call> NoiseResidualMap </tool_call>. The returned residual map highlights dense, high-frequency granular speckles in uniform sky and sand areas. The model infers: "significant high-frequency noise is evident throughout flat regions, deviating from clean natural statistics." 3. Synthesis and Final Scoring: Grounding its rationale in both the gradient and residual maps, the model aggregates the evidence, closes its thought trace, and outputs <answer_start>{"score": 2.67}<answer_end>. The score matches human ground truth (2.66) while providing an indisputable, visually verified audit trail.

Key Experimental Results

Main Results

All models (except handcrafted baselines) are trained on the KonIQ-10k dataset and evaluated zero-shot across seven diverse benchmarks: real-world wild images (KonIQ, SPAQ, LiveW), synthetic distortions (KADID, CSIQ), algorithm-induced distortions (PIPAL), and AI-generated content (AGIQA). Evaluation metrics are Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank Correlation Coefficient (SRCC).

Model Category Method KonIQ (PLCC/SRCC) SPAQ (PLCC/SRCC) LiveW (PLCC/SRCC) CSIQ (PLCC/SRCC) PIPAL (PLCC/SRCC) AGIQA (PLCC/SRCC) AVG. (PLCC/SRCC)
Handcrafted BRISQUE 0.225 / 0.226 0.490 / 0.406 0.361 / 0.313 0.740 / 0.556 0.267 / 0.232 0.541 / 0.497 0.436 / 0.369
Deep Learning MUSIQ 0.924 / 0.929 0.868 / 0.863 0.789 / 0.830 0.771 / 0.710 0.431 / 0.431 0.722 / 0.630 0.726 / 0.707
MLLM (w/o reasoning) Q-Align 0.941 / 0.940 0.886 / 0.887 0.853 / 0.860 0.671 / 0.737 0.403 / 0.419 0.772 / 0.735 0.743 / 0.752
MLLM (w/o reasoning) DeQA 0.953 / 0.941 0.895 / 0.896 0.892 / 0.879 0.787 / 0.744 0.472 / 0.478 0.809 / 0.729 0.786 / 0.765
MLLM (w/ reasoning) Q-Insight 0.918 / 0.895 0.903 / 0.903 0.870 / 0.839 0.685 / 0.640 0.458 / 0.435 0.816 / 0.766 0.765 / 0.740
MLLM (region zoom) Zoom-IQA 0.938 / 0.922 0.902 / 0.900 0.887 / 0.870 0.797 / 0.754 0.468 / 0.465 0.816 / 0.765 0.787 / 0.768
Proposed IQA-T1 0.942 / 0.925 0.905 / 0.908 0.888 / 0.860 0.850 / 0.840 0.480 / 0.490 0.816 / 0.754 0.795 / 0.784

Ablation Study

1. Component and Reward Ablation (Trained on KonIQ, Evaluated on 7 Benchmarks)

Config Tool Evidence SFT Stage \(R_\text{tool}\) \(R_\text{rep}\) CSIQ (PLCC/SRCC) PIPAL (PLCC/SRCC) AVG. PLCC AVG. SRCC Note
(1) Placeholder only \(\times\) \(\checkmark\) \(\times\) \(\times\) 0.675 / 0.640 0.412 / 0.418 0.705 0.680 Tool calls return empty placeholders w/o evidence
(2) SFT Baseline \(\checkmark\) \(\checkmark\) \(\times\) \(\times\) 0.768 / 0.730 0.464 / 0.455 0.756 0.738 Tool evidence enabled, no RL policy optimization
(3) w/o \(R_\text{rep}\) \(\checkmark\) \(\checkmark\) \(\checkmark\) \(\times\) 0.841 / 0.835 0.479 / 0.481 0.790 0.777 Tool usage reward only, occasional repetitive calls
(4) w/o \(R_\text{tool}\) \(\checkmark\) \(\checkmark\) \(\times\) \(\checkmark\) 0.838 / 0.832 0.481 / 0.485 0.789 0.775 Repetition penalty only, unstable invocation count
(5) Full Model \(\checkmark\) \(\checkmark\) \(\checkmark\) \(\checkmark\) 0.850 / 0.840 0.480 / 0.490 0.795 / 0.784 Full IQA-T1 framework

2. Tool Invocation Strategy and Efficiency Analysis (Evaluated on KonIQ)

Invocation Strategy PLCC SRCC Avg. Tools Avg. Time (s) Peak Mem. (GB)
No Tools (Qwen3-VL-4B) 0.785 0.744 0.00 4.62 8.52
All Tools (Force all 15) 0.865 0.841 15.00 6.06 9.13
Fixed-K (K=3) 0.895 0.869 3.00 5.32 8.70
Random-K (K=3) 0.871 0.854 3.00 5.42 8.74
Dynamic (IQA-T1) 0.942 0.925 2.34 5.09 8.67

Key Findings

  • Visual Evidence Acts as the Core Differentiator: Comparing Config (1) and (2) shows that holding reasoning text and SFT data identical, merely replacing placeholder tokens with real tool evidence yields an immediate +0.051 / +0.058 surge in average PLCC/SRCC (from 0.705/0.680 to 0.756/0.738), and leaps by +0.093 on CSIQ. This proves that external perceptual operators effectively rectify the intrinsic semantic blindness of MLLMs.
  • Adaptive Invocation Wins in Accuracy and Efficiency: Invoking all 15 tools brute-force degrades PLCC to 0.865 while escalating latency to 6.06s due to token redundancy and visual clutter. In contrast, the RL-tuned dynamic policy calls an average of only 2.34 tools, maintaining a lightweight 8.67GB peak VRAM footprint while boosting PLCC to 0.942.
  • Superior Generalization on Synthetic & Algorithmic Distortions: On the challenging synthetic CSIQ benchmark, IQA-T1 reaches 0.850 / 0.840 PLCC/SRCC, heavily outperforming Zoom-IQA (0.797 / 0.754) and Q-Insight (0.685 / 0.640). It also leads the algorithmic PIPAL benchmark (0.480 / 0.490), demonstrating that explicit frequency and gradient statistics confer unprecedented robustness against non-semantic distortions.

Highlights & Insights

  • Decoupling Semantic Cognition from Physical Perception: Rather than forcing a single neural backbone to master high-level semantic understanding and low-level signal degradation simultaneously, IQA-T1 delegates physical distortion extraction to deterministic tools, freeing the MLLM to focus on cognitive reasoning and evidence synthesis.
  • Non-Invasive Loss Masking During SFT: By masking the prediction loss over tool evidence tokens during SFT, the framework avoids burdening the model with learning generative rendering of tool maps, drastically improving training stability and data efficiency.
  • Transferable Agentic Paradigm: The "perceptual tool library \(\to\) evidence-injected CoT \(\to\) RL policy tuning" design directly generalizes to high-precision domain-specific inspection tasks such as industrial surface defect analysis, medical imaging distortion appraisal, and camouflage detection.

Limitations & Future Work

  • Author-Acknowledged Limitations: The current toolset primarily relies on classic deterministic transforms (FFT, Sobel, residual subtraction), which may have limited expressiveness when dealing with subtle AI-generated deepfake artifacts or complex nonlinear geometric warping; additionally, dynamic multi-turn inference introduces slight latency overhead compared to single-pass regression models.
  • Independent Critique: The current decision pipeline follows a sequential forward trajectory without a backtracking or hypothesis-revision mechanism; if initial tool calls fail to detect the root degradation, the model lacks an explicit mechanism to self-correct and re-sample diagnostic tools.
  • Future Directions: Integrating lightweight neural visual probes into the tool library and exploring test-time search (e.g., Monte Carlo Tree Search over tool sequences) could further enhance diagnostic reliability on ambiguous corner cases.
  • vs Q-Insight / VisualQuality-R1: While prior works pioneered textual reasoning for IQA, their rationales rely purely on internal semantic priors, producing ungrounded explanations; IQA-T1 anchors every step of the reasoning trajectory to explicit visual evidence generated by external tools.
  • vs Zoom-IQA / Q-Probe: Patch cropping approaches introduce localized raw pixels that remain unstructured and susceptible to semantic bias; IQA-T1 transforms pixels into isolated perceptual dimensions (noise, frequency, coherence), providing mathematically grounded evidence that MLLMs cannot internally infer.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the tool-based visual evidence reasoning paradigm for MLLM-based image quality assessment]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-distribution evaluations across 7 diverse benchmarks, comprehensive module ablations, and backbone generalization checks]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional clarity, rigorous mathematical formulation of semantic bias, and well-structured empirical analysis]
  • Value: ⭐⭐⭐⭐⭐ [Provides an actionable and generalizable blueprint for overcoming semantic bias in multimodal foundation models]