Enhancing prompt-image alignment evaluations via cyclic mutual information maximization

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/Buka-Xing/CMIM-AIGIQA
Area: Multimodal VLM
Keywords: AIGIQA, prompt-image alignment, mutual information maximization, contrastive learning, vision-language model

TL;DR

To resolve the decoupling between visual quality and prompt alignment as well as the overwhelming of sparse text by dense visual features, this paper proposes CMIM-AIGIQA, a cyclic mutual information maximization framework that filters modality noise forward and preserves critical prompt constraints backward, achieving state-of-the-art alignment evaluation across three benchmarks.

Background & Motivation

The explosion of generative AI has lowered the threshold for digital content creation, yet deploying these systems in practice faces two persistent bottlenecks: synthetic artifacts that degrade visual quality, and insufficient controllability where generated images fail to satisfy textual constraints. Because manual evaluation is labor-intensive and costly, developing objective AI-generated image quality assessment (AIGIQA) methods that simultaneously measure visual quality and prompt-image alignment is imperative. However, prevailing AIGIQA frameworks focus predominantly on perceptual quality, leaving semantic alignment underexplored. Directly retraining existing quality-sensitive metrics for alignment tasks yields suboptimal performance, with SRCC values consistently falling below 0.80 across benchmarks.

This deficiency stems from two fundamental tensions. First, the relationship between visual quality and prompt alignment is complex and weakly coupled. High visual fidelity does not guarantee faithful semantic alignment (e.g., photorealistic images omitting key objects), while faithfully aligned images often suffer from local artifacts. Linear regression models fitting alignment from quality scores yield low coefficients of determination (\(R^2 < 0.7\)), demonstrating that quality scores cannot be linearly extrapolated to alignment metrics. Second, visual and textual features extracted from pretrained vision-language models (VLMs) inherently suffer from modality noise. Images contain rich, dense visual backgrounds not specified in sparse prompts; Grad-CAM visualizations on CLIP reveal frequent attention drift, where the network focuses on irrelevant visual regions rather than the intended prompt semantics—a phenomenon present in nearly 50% of benchmark samples. Existing methods confine themselves to input-level preprocessing or output-level score pooling, lacking deep feature domain refinement and balanced fusion.

This paper tackles the challenge by formulating multimodal fusion in AIGIQA as an information-theoretic optimization problem. Core idea: introduce a cyclic mutual information maximization framework (CMIM-AIGIQA) that utilizes forward variational mutual information lower-bound maximization to purge modality-specific visual noise, and backward contrastive predictive coding (CPC-NCE) to prevent dense visual features from overwhelming sparse textual constraints in the joint embedding.

Method

Overall Architecture

The CMIM-AIGIQA pipeline consists of three sequential stages: multimodal feature extraction, cyclic mutual information processing and fusion, and quality/alignment score regression. Given an AI-generated image \(I\) and prompt \(P\), visual features \(E_v(I)\) and textual features \(E_t(P)\) are extracted using a partially fine-tuned VLM backbone (BLIP). Next, a bidirectional cyclic mutual information framework refines the representations: the forward phase computes a variational lower bound on mutual information to extract cross-aware features and eliminate modality noise, the intermediate fusion network combines representations via cross-attention into a joint embedding \(Z\), and the backward phase enforces statistical dependency via InfoNCE loss to preserve essential individual modality cues. Finally, a linear regression head maps the joint embedding to subjective MOS predictions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Image I + Prompt P"] --> B["Forward Phase: Variational MIM<br/>Gaussian likelihood & entropy regularization"]
    B --> C["Cross-Attention Feature Fusion<br/>CSA multimodal aggregation to joint embedding Z"]
    C --> D["Backward Phase: Contrastive Predictive Coding<br/>CPC-NCE modal dependency preservation"]
    D --> E["Task Prediction Head & End-to-End Training<br/>Quality/Alignment MOS prediction"]

Key Designs

1. Forward Phase: Variational MIM: filtering modality noise and preventing feature collapse

Generated images typically contain abundant visual details absent from sparse prompts, introducing modality noise that corrupts alignment evaluation. The forward phase maximizes the mutual information \(I(X; Y)\) between visual features \(X = E_v(I)\) and textual features \(Y = E_t(P)\) to isolate shared semantic concepts. Because the true joint distribution \(p(x, y)\) is intractable, the model applies the Barber-Agakov variational lower bound with an auxiliary distribution \(q_\theta(y|x)\):

\[I(X; Y) \geq \mathbb{E}_{p(x,y)}[\log q_\theta(y|x)] + H(Y)\]

Because visual features possess higher information density than sparse text, predicting text conditioned on image features is well-posed. The variational posterior \(q_\theta(y|x)\) is modeled as a multivariate Gaussian \(\mathcal{N}(y | \mu_\theta(x), \sigma_\theta^2(x)\mathbf{I})\) where two separate MLPs output the mean and variance, making expected log-likelihood maximization equivalent to minimizing negative log-likelihood \(\mathcal{L}_{lm}\). Crucially, the marginal text distribution \(p(y) \sim \mathcal{N}(\mu_y, \Sigma_y)\) provides a closed-form differential entropy:

\[H(Y) = \frac{1}{2} \ln \left( (2\pi e)^k \det(\Sigma_y) \right)\]

This entropy term \(H(Y)\) serves as a vital geometric regularizer. Without it, the text representations would collapse into a trivial degenerate manifold to minimize prediction variance; maximizing entropy enforces uniform dispersion on a hypersphere to maintain semantic discriminative power. The total forward loss is \(\mathcal{L}_{forward} = -\mathcal{L}_{lm} - H(Y)\).

2. Cross-Attention Feature Fusion: semantic cross-attentive interaction and aggregation

Following forward mutual information regularized representation learning, visual and textual features interact through a cross-attention module (CSA) to synthesize a compact joint embedding \(Z\):

\[Z = \text{Concat}\left(E_v(I), \text{CSA}(E_v(I), E_v(I), E_t(P))\right)\]

Using visual features as Query and Key, and textual features as Value, visual patches query corresponding text semantics. The resulting text-guided visual representation is concatenated with global visual features, retaining perceptual fidelity while focusing on aligned visual semantics.

3. Backward Phase: Contrastive Predictive Coding: preventing visual modality dominance

In unconstrained multimodal fusion, dense high-dimensional visual features dominate gradient backpropagation, causing the joint embedding \(Z\) to neglect textual constraints. The backward phase enforces statistical dependency between \(Z\) and individual modalities \(M \in \{X, Y\}\), guaranteeing that \(Z\) serves as a sufficient statistic for both inputs. Rather than imposing Gaussian assumptions—which cause oversmoothing and numerical instability with high-dimensional visual covariance matrices—the model pairs Contrastive Predictive Coding (CPC) with Noise-Contrastive Estimation (NCE). A projection MLP \(G_\psi(\cdot)\) maps normalized \(Z\) to each modality space to compute cosine matching scores \(s(M, Z) = \exp(\tilde{M} \cdot \tilde{G}_\psi(Z))\). For sample \(M_i\) within mini-batch \(\mathbf{B}_M\) containing \(N-1\) negative samples, the InfoNCE objective is formulated as:

\[\mathcal{L}_{NCE}^M = -\mathbb{E}\left[\log \frac{s(Z_i, M_i)}{\sum_{M_j \in \mathbf{B}_M} s(Z_i, M_j)}\right]\]

Summing contrastive objectives over visual and textual modalities \(\mathcal{L}_{backward} = \mathcal{L}_{NCE}^X + \mathcal{L}_{NCE}^Y\) forces the joint embedding \(Z\) to discriminate its corresponding prompt and image against in-batch negatives, effectively safeguarding sparse textual constraints against visual dilution.

Loss & Training

The framework is optimized via a two-stage training strategy: - Stage 1 (Forward MIM Warmup): The VLM backbone and other components remain frozen while only the forward MLPs are trained to minimize negative log-likelihood \(\mathcal{L}_{lm}\), stabilizing posterior estimation. - Stage 2 (Global Joint Training): The deep half of VLM layers, forward MLPs, fusion network, backward projection heads, and score regression head are optimized jointly:

\[\mathcal{L} = \mathcal{L}_{task} + \alpha \mathcal{L}_{forward} + \beta \mathcal{L}_{backward}\]

Here \(\mathcal{L}_{task}\) is the Mean Squared Error (MSE) against ground-truth MOS. Grid search sets hyperparameters \(\alpha = 0.1, \beta = 0.1\). The model employs BLIP as the backbone, resizing images to \(224 \times 224\) and processing raw prompts. It is optimized using AdamW with an initial learning rate of \(1 \times 10^{-5}\), decayed by \(0.5 \times\) every 4 epochs over 20 epochs total.

Key Experimental Results

Main Results

CMIM-AIGIQA is evaluated on three benchmark datasets: AGIQA-3k, AIGCIQA2023, and the text-to-image partition of PKU-AIGIQA-4k across Quality and Alignment dimensions. Key comparisons with prior state-of-the-art methods are reported below (from Table 2 of the original paper):

Dataset	Metric	CMIM-AIGIQA (Ours)	Prev. SOTA	Gain
AGIQA-3K [Alignment]	SRCC	0.8546	0.8238 (MoE-AGIQA)	+3.74% (+0.0308)
AGIQA-3K [Alignment]	PLCC	0.9181	0.8916 (MoE-AGIQA)	+2.97% (+0.0265)
AGIQA-3K [Quality]	SRCC	0.8755	0.8841 (IPCE)	-0.97% (Top-3)
AIGCIQA2023 [Alignment]	SRCC	0.8263	0.7979 (IPCE)	+3.56% (+0.0284)
AIGCIQA2023 [Alignment]	PLCC	0.8158	0.7887 (IPCE)	+3.44% (+0.0271)
AIGCIQA2023 [Quality]	SRCC	0.8679	0.8751 (MoE-AGIQA)	-0.82% (Rank 2)
PKU-AIGIQA-4k [Alignment]	SRCC	0.8223	0.7978 (CIA-Net)	+3.07% (+0.0245)
PKU-AIGIQA-4k [Alignment]	PLCC	0.8930	0.8466 (CIA-Net)	+5.48% (+0.0464)
PKU-AIGIQA-4k [Quality]	SRCC	0.8586	0.8638 (CIA-Net)	-0.60% (Rank 2)

Ablation Study

Ablation experiments quantify the relative contributions of the mutual information components and fusion architectural choices on SRCC (from Table 4 of the original paper):

Config	AGIQA-3k [Quality / Alignment]	AIGCIQA2023 [Quality / Alignment]	PKU-AIGIQA-4k [Quality / Alignment]	Note
Baseline	0.8124 / 0.7731	0.8015 / 0.7428	0.7756 / 0.7204	Fusion network trained solely with MSE
w/o Forward MIM	0.8432 / 0.8415	0.8329 / 0.8110	0.8104 / 0.8058	Drops variational filtering of modality noise
w/o Backward MIM	0.8567 / 0.7984	0.8412 / 0.7655	0.8321 / 0.7512	Severe alignment collapse due to modality dominance
w/o \(H(Y)\)	0.8654 / 0.8502	0.8496 / 0.8187	0.8412 / 0.8094	Text space collapses without entropy regularizer
\(\mathcal{L}_{NCE} \to \mathcal{L}_{MSE}\)	0.8512 / 0.8156	0.8533 / 0.7892	0.8445 / 0.7936	MSE averages features and degrades fine-grained cues
w/o CSA	0.8610 / 0.8327	0.8422 / 0.8045	0.8207 / 0.7984	Naive concatenation degrades cross-modal alignment
w/ CLIP (ViT-L/14)	0.8931 / 0.8214	0.8498 / 0.8103	0.8683 / 0.8101	Enhances visual quality but lags behind BLIP on alignment
Full CMIM-AIGIQA	0.8755 / 0.8546	0.8679 / 0.8263	0.8586 / 0.8223	Complete cyclic mutual information maximization

Key Findings

• Backward MIM is critical for prompt alignment: Removing the backward MIM module (w/o Backward MIM) causes alignment SRCC to plunge across all three benchmarks by 0.0562, 0.0608, and 0.0711 on AGIQA-3k, AIGCIQA2023, and PKU-AIGIQA-4k respectively, while visual quality remains largely unaffected. This validates that backward contrastive retention is essential to prevent text cues from being discarded during fusion.
• InfoNCE outperforms reconstruction MSE: Replacing contrastive loss with MSE reconstruction (\(\mathcal{L}_{NCE} \to \mathcal{L}_{MSE}\)) incurs a 0.03+ drop in alignment SRCC. Contrastive negative sampling preserves fine-grained prompt discriminability, whereas MSE incurs oversmoothing.
• Superior cross-dataset generalization: In cross-dataset transfer tests (Table 3), when trained on PKU-AIGIQA-4k and evaluated zero-shot on AIGCIQA2023, CMIM-AIGIQA achieves an alignment SRCC of 0.6305, outperforming MoE-AGIQA (0.5646) by +11.67%, proving that information-theoretic constraints prevent overfitting to dataset-specific scoring artifacts.

Highlights & Insights

• Bidirectional information-theoretic formulation: Reframing multimodal fusion as forward variational denoising and backward contrastive retention elegantly overcomes the fundamental asymmetry between high-density imagery and sparse text.
• Closed-form entropy regularization: Introducing analytical differential entropy \(H(Y)\) prevents posterior collapse during variational inference, providing a computationally stable regularizer for text manifolds.
• Demystifying CLIP's alignment weakness: Empirical replacement with CLIP demonstrates that contrastively pre-trained models naturally bias toward visual perceptual features; nonetheless, integrating CMIM with CLIP still pushes alignment SRCC beyond 0.80 across benchmarks.

Limitations & Future Work

• Admitted limitations: Due to GPU memory constraints, the batch size is capped at 5, which limits the diversity of contrastive negative pairs in backward NCE; images are evaluated at fixed \(224 \times 224\) resolution, potentially missing high-frequency micro-artifacts in high-resolution generation models.
• Identified limitations: The forward phase assumes multivariate Gaussian distributions for both conditional and marginal text representations, which may oversimplify the multi-modal clustering of complex open-ended prompts.
• Improvement directions: Future investigations could explore non-parametric mutual information bounds or diffusion-based density estimators, as well as integrating MLLMs for structured hierarchical attribute parsing prior to CMIM evaluation.

Related Work & Insights

• vs MoE-AGIQA [38]: While MoE-AGIQA adopts BLIP with mixture-of-experts gating, its lack of explicit backward mutual information preservation results in severe evaluation imbalance (achieving 0.8751 on quality but dropping to 0.7899 on alignment in AIGCIQA2023). CMIM-AIGIQA establishes superior balance and outperforms it by 3.5%~11.6% in alignment.
• vs IPCE [22] / CIA-Net [43]: Prior methods rely on handcrafted text templates or heuristic multi-scale cross-attention. CMIM-AIGIQA provides a principled information-theoretic objective that yields significantly stronger cross-dataset transferability.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pioneers cyclic mutual information maximization in AIGIQA, solving asymmetric multimodal fusion bottlenecks.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across three major benchmarks, cross-dataset transfer scenarios, and detailed component-wise ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative, lucid mathematical derivations, and precise empirical interpretations.
• Value: ⭐⭐⭐⭐☆ Fills a critical gap in automated prompt-image alignment evaluation; the open-source implementation is highly practical for AIGC model auditing.

Related Papers

• [ECCV 2026] Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs
• [ECCV 2026] ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
• [ECCV 2026] Rank-Aware Hyperbolic Alignment for Vision–Language Dataset Distillation
• [ECCV 2026] MoVA: Modular Long Video-Text Alignment via Asymmetric Dual Projection
• [ECCV 2026] Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs