content_hash: 764a73710760a6ee

ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking¶

Conference: ECCV 2026
Paper: ECCV 2026 Official
Area: Information Retrieval & RAG
Keywords: Image Quality Assessment, Vision-Language Models, Discrete Collapse, Test-Time Memory, Ordinal Re-Ranking

TL;DR¶

To resolve the severe discrete collapse issue in reasoning-induced vision-language models for image quality assessment, ME-IQA introduces a training-free, test-time memory-enhanced re-ranking framework that retrieves aligned neighbors from a hybrid memory bank and fuses pairwise VLM preference probabilities with the initial score under Thurstone's Case V model.

Background & Motivation¶

Vision-language models (VLMs) have catalyzed a major paradigm shift in no-reference image quality assessment (NR-IQA), transitioning from direct numerical score regression toward reasoning-induced frameworks. Prompting VLMs to generate explicit chain-of-thought rationale before predicting quality scores delivers superior cross-domain generalization and transparent interpretability. However, reasoning-induced VLMs universally suffer from severe discrete collapse: images with perceptually distinct visual qualities are frequently assigned nearly identical scalar scores that cluster tightly around a few prominent numbers (such as 3.0, 4.0, or 5.0). This flaw stems from an inherent objective mismatch, as VLMs are fundamentally pretrained for autoregressive discrete token prediction rather than continuous metric estimation, which forces the model to coarsely quantize visual quality and blunts its sensitivity to subtle physical degradations.

Existing attempts to mitigate this collapse predominantly rely on token probability averaging or full pairwise tournament rankings. Single-stimulus token probability weighting lacks an explicit comparative frame of reference, remaining largely blind to subtle distortions. Conversely, exhaustive pairwise ranking incurs quadratic computational complexity and fundamentally violates the online streaming testing requirement that mandates immediate query evaluation without peeking into future data. Static anchor-based comparative designs avoid quadratic costs by evaluating queries against a fixed exemplar set, but static anchors under-represent long-tail artifacts and degrade sharply under domain shifts. In contrast, human observers assess visual quality relative to dynamic perceptual memories of contextually retrieved stimuli rather than isolated static references.

To bridge this fundamental gap, this paper introduces dynamic retrieval and ordinal calibration during inference. Core idea: construct a dual-tier memory bank combining offline anchors and online contrast cases, retrieve semantically and perceptually aligned neighbors using reasoning summaries as keys, reframe the VLM as a probabilistic comparator to fuse pairwise ordinal evidence with the initial score under Thurstone's Case V model, and consolidate hard cases via gated reflection for test-time plug-and-play quality re-ranking.

Method¶

Overall Architecture¶

ME-IQA strictly adheres to the online streaming testing paradigm. For each sequentially arriving query image, the VLM first outputs a free-form reasoning chain along with an initial scalar score, which is aligned to the continuous ground-truth scale via a fitted five-parameter monotonic logistic mapping. Next, the reasoning chain is distilled into a concise quality description, embedded, and used to retrieve a localized neighborhood from a hybrid memory bank consisting of Anchor Memory (AM) and Contrast Memory (CM). The VLM is then repurposed as a probabilistic comparator to estimate pairwise preference probabilities between the query and the retrieved neighbors. These ordinal preferences are fused with the mapped initial score via a closed-form ridge formulation of Thurstone's Case V model. Finally, if the refined score deviates from the initial estimate beyond a pre-set threshold, a gated reflection step updates the descriptive summary and consolidates the case into CM to support subsequent queries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Query image $x_i$"] --> B["Initial Reasoning & 5-Parameter Mapping<br/>generate initial score $s_i$ & chain-of-thought"]
    B --> C["Reasoning-Aware Hybrid Retrieval<br/>produce compact summary & retrieve from AM and CM"]
    C --> D["VLM Probabilistic Pairwise Comparison<br/>extract preference probabilities $y_{ij}$"]
    D --> E["Thurstone's Case V Ordinal Fusion<br/>solve closed-form ridge objective for refined score $s_i^*$"]
    E --> F{"Gated Reflection & Consolidation<br/>$|s_i^* - s_i| > \epsilon$ ?"}
    F -->|Yes| G["Update reflected summary $r_i^*$ and append to Contrast Memory CM"]
    F -->|No| H["Output calibrated continuous quality score $s_i^*$"]
    G --> H

Key Designs¶

1. Hybrid memory bank: balancing global scaffold stability with online adaptability Static references fail on novel distortions, whereas unrestrained dynamic updates induce catastrophic scale drift over long evaluation streams. To resolve this trade-off, ME-IQA introduces a dual-tier memory bank where each entry is stored as a structured triple $(\mathbf{x}_j, \mathbf{r}_j, s_j^*)$. Anchor Memory (AM) is pre-constructed offline from an annotated dataset (such as KonIQ-10K) by partitioning the quality range $[1, 5]$ into $B=5$ score bins and stratifying exemplars uniformly, establishing a permanent, drift-free perceptual scaffold. Conversely, Contrast Memory (CM) expands dynamically during test time, capturing solely hard queries and edge cases that trigger significant score updates. When CM exceeds its storage capacity $C$, agglomerative hierarchical clustering iteratively merges the most similar memory pairs based on cosine similarity of text embeddings, preserving critical prototypes under a bounded memory footprint.

2. Reasoning-aware retrieval: bypassing superficial template noise for perceptual alignment Retrieving neighbors via raw visual embeddings causes the search to be overwhelmed by high-level semantic content (e.g., scene categories or salient foreground objects) rather than fine-grained blur, noise, or compression artifacts. Conversely, directly embedding raw chain-of-thought outputs introduces substantial noise from boilerplate phrasing (e.g., "let us analyze this step by step"). ME-IQA implements reasoning-aware retrieval by prompting the VLM to compress its verbose reasoning into a compact description $\mathbf{r}_i$ strictly focused on visual content, texture sharpness, degradation type, and perceived quality impact. Under a fixed retrieval budget $K=32$ split equally ($K_A=16, K_C=16$), the system retrieves exemplars from AM via score-stratified selection and from CM via top-cosine similarity, assembling a neighborhood that provides both broad quality scaffolding and specific distortion alignment.

3. Thurstone's Case V ordinal fusion: continuous calibration from pairwise preference confidence Simple neighbor score smoothing (re-scoring via kernel regression) only interpolates within the existing flawed scale and fails to repair discrete collapse. ME-IQA transforms the VLM into a soft binary comparator, querying whether image $i$ is superior to neighbor $j$ and taking the softmax probability of token "A" as soft preference $y_{ij} = P(s_i^* > s_j^*)$. Under Thurstone's Case V assumption with standard Gaussian noise, the theoretical probability satisfies $p_{ij} = \Phi(s_i^* - s_j^*)$. To leverage fine-grained pairwise constraints while anchoring against outlier comparisons, the method minimizes a regularized binary cross-entropy objective: $$ \min_{s_i^} \sum_{j \in \mathcal{N}} \mathrm{BCE}\big(\Phi(s_i^ - s_j^), y_{ij}\big) + \lambda \, (s_i^ - s_i)^2 $$ To eliminate iterative numerical optimization during high-throughput inference, probit linearization maps each soft probability into a pseudo-score $\mu_{ij} = s_j^* + \Phi^{-1}(y_{ij})$, yielding an exact closed-form ridge regression solution: $$ s_i^* = \frac{\sum_{j \in \mathcal{N}} \mu_{ij} + \lambda s_i}{K + \lambda} $$ This analytical formulation pulls the discrete initial prediction toward local ordinal evidence while penalizing unjustified deviations, expanding point-concentrated scores into a smooth, dense continuum.

4. Gated reflection and causal updates: absorbing high-value contrastive feedback in stream Streaming deployment demands strict temporal causality, prohibiting any access to future queries or ground-truth labels. To prevent memory pollution from trivial samples and maintain an informative contrast bank, ME-IQA introduces a thresholded reflection gate. Only when the refined score differs markedly from the mapped initial score ($|s_i^* - s_i| > \epsilon$, with default $\epsilon=0.75$) does the system trigger the VLM to reflect on its perceptual oversight, generating an updated description $r_i^*$ for CM storage. This maintains a lean update rate of roughly 31%, selectively curating distribution shifts and border cases without human intervention.

Key Experimental Results¶

Main Results¶

ME-IQA is evaluated across seven benchmark datasets covering authentic distortions (SPAQ, LIVEW), AI-generated distortions (AGIQA), and synthetic artifacts (KADID, PIPAL, TID2013, CSIQ). Open-source models are benchmarked using weights fine-tuned on KonIQ-10K with matching anchor memory. The table below presents Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank-Order Correlation Coefficient (SRCC) results alongside the weighted average (WAVG) across all datasets:

Method / Model	SPAQ (PLCC/SRCC)	KADID (PLCC/SRCC)	CSIQ (PLCC/SRCC)	WAVG PLCC	WAVG SRCC
NIQE (Non-reasoning Classic)	0.664 / 0.664	0.430 / 0.430	0.645 / 0.645	0.349	0.349
Q-Align (Non-reasoning VLM)	0.886 / 0.887	0.795 / 0.792	0.814 / 0.750	0.663	0.631
Compare2Score (Static Anchors)	0.873 / 0.856	0.720 / 0.731	0.761 / 0.771	0.686	0.688
Q-Insight Baseline	0.901 / 0.899	0.732 / 0.737	0.749 / 0.717	0.714	0.683
Q-Insight + ME-IQA	0.922 / 0.924	0.770 / 0.785	0.777 / 0.751	0.744	0.719
VisualQuality-R1 Baseline	0.895 / 0.887	0.709 / 0.703	0.750 / 0.697	0.698	0.661
VisualQuality-R1 + ME-IQA	0.912 / 0.899	0.741 / 0.753	0.786 / 0.737	0.726	0.696
EvoQuality Baseline	0.903 / 0.901	0.744 / 0.738	0.821 / 0.778	0.748	0.716
EvoQuality + ME-IQA	0.925 / 0.919	0.783 / 0.783	0.851 / 0.815	0.777	0.751
GPT-5 Baseline	0.879 / 0.873	0.715 / 0.701	0.701 / 0.678	0.676	0.644
GPT-5 + ME-IQA	0.891 / 0.887	0.739 / 0.744	0.760 / 0.736	0.706	0.683

Ablation Study¶

Using VisualQuality-R1 as the primary backbone, the authors ablated retrieval embedding strategies, memory composition ratios, scoring mechanisms, and test-time compute scaling:

Configuration / Variant	WAVG PLCC	WAVG SRCC	Note
ME-IQA Full Model (K=32)	0.726	0.696	Full reasoning retrieval + hybrid memory + closed-form fusion
Retrieval: Random Retrieval	0.703	0.670	Lacks semantic and perceptual alignment
Retrieval: Image Embeddings	0.706	0.677	Dominated by semantic scene content rather than distortions
Retrieval: Reasoning Embeddings	0.726	0.696	Captures explicit degradation semantics effectively
Memory: AM only ($K_A:K_C = 32:0$)	0.714	0.682	Lacks online adaptability to novel streaming artifacts
Memory: CM only ($K_A:K_C = 0:32$)	0.717	0.685	Suffers from cold-start scale drift without global anchors
Mechanism: Re-scoring (Kernel)	0.707	0.704	On KADID; neighbor smoothing cannot resolve collapse
Mechanism: Re-ranking (Thurstone)	0.741	0.753	On KADID; pairwise ordinal constraints restore rank fidelity
Prior Weight: No Prior ($\lambda=0$)	0.724	0.727	On KADID; pairwise noise causes over-correction

Key Findings¶

Defeating discrete collapse: Statistical analysis shows baseline predictions collapse into sharp spikes (effective bin count between 9 and 17, JS divergence 0.39 to 0.49 against MOS). ME-IQA disperses predictions across intermediate continuous scores, expanding effective bins to 58–84 and driving JS divergence down to 0.05–0.10, matching human subjective distributions.
Pronounced improvements on synthetic distortions: The largest correlation boosts emerge on synthetic benchmarks (e.g., KADID SRCC improves from 0.703 to 0.753, a +0.050 jump), highlighting that retrieved ordinal comparisons excel at discerning fine-grained degradation levels.
Superior efficiency over sampling-based test-time scaling: Compared to majority voting or mean aggregation over 64 reasoning trajectories (Maj@64 / Mean@64 taking 34.9s per image), ME-IQA@32 requires only 14.3s per image (2.4x faster) while achieving higher overall accuracy. Even a lightweight ME-IQA@8 setup (6.32s per image, WAVG PLCC 0.705) comfortably outperforms prolonged direct reasoning (DS-L=500 taking 13.23s, PLCC 0.689).

Highlights & Insights¶

Plug-and-play test-time calibration: ME-IQA enhances existing reasoning-induced VLMs without requiring gradient updates, weight fine-tuning, or architectural adjustments, offering immediate applicability to open-source and proprietary models.
Closed-form probit linearization for real-time throughput: By framing the probit link within a ridge regression objective, ME-IQA derives an exact analytical solution for Thurstone Case V fusion, avoiding costly iterative convex optimization.
Bi-level perceptual memory inspired by cognitive psychology: Drawing inspiration from human adaptation-level theory, the framework combines a fixed global scaffold with an active, event-driven contrast memory, establishing an effective mechanism for dynamic VLM self-evolution.

Limitations & Future Work¶

Reliance on backbone pairwise discriminability: The re-ranking stage assumes the underlying VLM can reliably discern pairwise quality; severe baseline perceptual blind spots (e.g., subtle impulse noise or extreme compression) can still propagate pairwise errors.
Slight temporal stream dependency: Because Contrast Memory updates incrementally based on processed queries, final predictions retain a minor sensitivity to arrival order, though random permutation ablations show std deviations bounded below $\pm 0.008$.
Generality of text embedding representations: While compressed reasoning summaries outperform raw images, off-the-shelf text encoders are not explicitly calibrated for nuanced physical distortion lexicons, leaving room for dedicated perceptual text embeddings.

vs Compare2Score: Compare2Score depends on a static set of reference anchors that degrade under unseen domain shifts; ME-IQA employs a dynamically growing hybrid memory bank with reasoning-aware retrieval and reflection-driven learning.
vs Q-Align / DeQA-Score: Prior non-reasoning methods attempt to prevent collapse by learning discrete token logits or probability distributions during supervised training; ME-IQA operates strictly at test time, tackling the discrete collapse specific to reasoning-induced VLMs via ordinal constraints.
vs Standard RAG & Agent Memory: Mainstream VLM memory mechanisms are designed for long-dialogue recall and factual reasoning; ME-IQA repurposes dynamic memory for continuous metric regression and perceptual quality alignment.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Introduces a novel test-time memory-enhanced framework with closed-form Thurstone re-ranking to tackle discrete collapse in reasoning VLMs.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain zero-shot evaluations across 7 datasets and 5 strong VLM backbones, supported by extensive collapse and efficiency analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical modeling, lucid problem motivation, and comprehensive empirical validation.
Value: ⭐⭐⭐⭐⭐ Provides a practical, highly generalizable blueprint for deploying reasoning-induced foundation models on continuous perceptual regression tasks.