MIRROR: Aligning Semantic Relations from Language to Image via Gromov-Wasserstein¶
Conference: ECCV2026
arXiv: 2606.29462
Code: None
Area: Multimodal VLM
Keywords: Relation Alignment, Gromov-Wasserstein, Multimodal Large Language Models, Geometric Regularization, Relational Reasoning
TL;DR¶
MIRROR proposes the Semi-Inverse Gromov-Wasserstein (SI-GW) geometric regularization framework, which maps the relational structure between concepts in language space to visual space via a closed-form solution. Without introducing additional parameters or inference overhead, it significantly improves the relational reasoning capabilities of MLLMs.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) have made tremendous progress in foundational visual understanding, but still struggle with tasks requiring relational reasoning. For example, an LLM can correctly reason at the text level that "Huskies and Alaskans differ in ear shape and body size," but the same model may still confuse these two breeds when shown an image; it can reason in text that "sitting down requires the center of gravity to be above the support surface," yet fail to judge spatial support relations in vision. This disconnect reveals a deep-seated issue in current MLLMs: while the language model internally encodes rich relational knowledge between concepts, these relational priors are lost when transitioning to the visual modality.
Tracing the cause, mainstream MLLMs map visual tokens into the input space of the large language model via a projection adapter, and are then trained using the language modeling loss (next-token prediction). This paradigm ensures that each visual token carries the correct semantics—i.e., identity-level alignment (aligning "what" each concept is)—but never questions whether the relational structure between concepts is also aligned across modalities. When a model only knows what "cat" and "mat" individually are, but does not know their spatial relation pattern (such as the visual distance structure corresponding to "lying on top of"), relational reasoning naturally fails. Recent theoretical works like the Platonic Representation Hypothesis suggest that cross-modal representations actually tend to share statistical structures, and LLMs can spontaneously develop visual reasoning priors from structured text—implying that the relational geometry of text space is an excellent "teacher signal" that can be used to systematically reshape the structure of visual space.
Core Idea: Elevate cross-modal alignment from identity-level to relational-structure level—proposing the Semi-Inverse Gromov-Wasserstein (SI-GW) problem. Starting from the relational geometry of language space (measured by distance matrices derived from self-attention) and cross-modal interactions (using cross-attention as coupling), the ideal distance structure that visual space "should have" is solved in closed form. This is then used as a regularization target during training to align visual representations with language relational geometry.
Method¶
Overall Architecture¶
MIRROR adds a geometric regularization term ℒ_SI-GW on top of the standard language modeling loss, resulting in a total loss of ℒ_total = ℒ_LM + λ · ℒ_SI-GW. During training, it extracts three attention components from the Transformer architecture of the MLLM: text self-attention Aₜ (encoding concept-to-concept relations on the language side), visual self-attention Aᵥ (encoding concept-to-concept relations on the visual side), and text-to-visual cross-attention C (encoding cross-modal correspondences). By transforming Aₜ and Aᵥ into distance matrices Dᵗ and Dᵛ via a negative logarithm transformation, the Semi-Inverse GW problem is solved to obtain the ideal visual distance structure D̂ᵛ. Finally, the Frobenius distance between Dᵛ and D̂ᵛ is minimized. This entire process introduces no new parameters and zero inference overhead—all SI-GW computations are performed only during training using the attention/coupling matrices.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image + Text"] --> B["ViT encodes visual tokens<br/>LLM forward pass"]
B --> C["Decoupled Layer Extraction<br/>A_t (shallow LLM), A_v (final ViT), C (middle LLM)"]
C --> D["Selective Head Aggregation<br/>Average low-entropy heads → C"]
C --> E["Non-semantic Token Filtering<br/>Soft attenuation of A_t"]
D --> F["Distance Transformation<br/>Dᵗ=-log(Aₜ) Dᵛ=-log(Aᵥ)"]
E --> F
F --> G["Semi-Inverse GW Closed-form Solution<br/>D̂ᵛ=(CᵀDᵗC)⊘(bbᵀ)"]
G --> H["SI-GW Loss<br/>‖Dᵛ-D̂ᵛ‖²_F"]
H --> I["Backpropagation<br/>Update LLM/ViT parameters"]
Key Designs¶
1. Semi-Inverse GW Problem: Closed-form Solution Mapping Language Relations to Visual Space
The classic Gromov-Wasserstein distance simultaneously optimizes the coupling matrix and distance matrices to align two metric spaces. In MLLMs, however, the situation is different: the cross-attention mechanism already provides a natural data-driven coupling C. The problem is not finding correspondences, but "given the language distance structure Dᵗ and coupling C, what should the visual distance structure look like to make the relations between the two spaces consistent?" This is the core of the Semi-Inverse GW problem—an inverse geometric problem. MIRROR proves that this problem has a unique closed-form solution D̂ᵛ = (Cᵀ Dᵗ C) ⊘ (b bᵀ), where b = Cᵀ 𝟙 represents the coupling mass (column sums) over visual tokens. This solution has an intuitive expectation interpretation: for a pair of visual tokens (k, ℓ), their expected distance on the language side is the "conditional expectation of distances of text token pairs (i, j) weighted by C." The closed-form solution brings massive computational benefits: directly evaluating the original quadruple sum requires O(n²ₜ n²ᵥ), while matrix factorization operations reduce the complexity to O(n²ₜ nᵥ + n²ᵥ nₜ).
2. Decoupled Layer Extraction: Obtaining Stable Geometric Components from Different Transformer Layers
Directly extracting Aₜ and C from a single attention layer leads to gradient coupling and training instability. This is because in a decoder-only MLLM, they belong to different sub-blocks of the same causal attention map, and are jointly normalized by the same softmax. MIRROR extracts the three components from decoupled layers: Aₜ is extracted from shallow LLM layers (layer 1)—where text relational geometry is pure and not yet heavily mixed with multimodal information; C is extracted from middle layers (layer 16)—balancing information richness and attention focus; Aᵥ is extracted from the last layer of ViT—which most directly impacts downstream visual reasoning. Ablation studies confirm that removing decoupled layer extraction causes training divergence (SI-GW gradients disrupt the attention distribution), demonstrating this is an "existential" component of MIRROR.
3. Selective Head Aggregation and Non-semantic Token Filtering: Cleaning High-Frequency Noise Sources
The quality of different heads in multi-head attention varies greatly—many heads exhibit near-uniform cross-attention distributions, and directly averaging them dilutes important cross-modal correspondences. MIRROR computes the cross-attention entropy for each head and retains the k heads (experimentally k=8) with the lowest entropy for averaging, resulting in a more concentrated and effective coupling matrix C. On the text side, tokens like system prompts and image placeholders carry no visual semantics but still appear in the self-attention Aₜ. MIRROR designs a position-aware soft attenuation mechanism: based on the token position and its attention quality with visual tokens, the corresponding entries are multiplied by (1-α·wⱼ) using weights wⱼ∈[0,1] and then re-normalized. This retains potentially useful context while reducing the footprint of irrelevant tokens. Removing head selection leads to a drop of 2.1 points on GQA and 7.3 points on BLINK, indicating that coupling quality is crucial for geometric alignment signal propagation.
Loss & Training¶
Total loss: ℒ_total = ℒ_LM + λ · ℒ_SI-GW, where ℒ_SI-GW = ‖Dᵛ − D̂ᵛ‖²_F. The optimal weight is λ = 2×10⁻³, which remains stable within the [10⁻³, 5×10⁻³] range; when the weight increases to 10⁻², the language modeling target is overly suppressed, causing performance to drop. Fine-tuning is conducted for 1 epoch on the LLaVA-1.5 and LLaVA-NeXT series (7B/13B), completed on 8× RTX 6000 Ada GPUs.
Key Experimental Results¶
Main Results¶
| Dataset | Metric | LLaVA-1.5-7B +MIRROR | LLaVA-NeXT-7B +MIRROR |
|---|---|---|---|
| GQA | Overall | 61.93 (+0.77) | 65.56 (+0.82) |
| GQA | Global | +3.16 | +2.13 |
| GQA | Relation | +0.63 | +1.44 |
| BLINK | Average | 42.9 (+2.5) | 39.2 (+2.4) |
| BLINK | Spatial Relation | +4.9 | — |
| VQAv2 | — | 81.1 (+1.8) | 82.1 (-0.1) |
| POPE | — | 86.3 (+0.6) | 87.1 (+0.4) |
| RealWorldQA | — | 56.8 (+0.2) | 57.7 (-0.1) |
Ablation Study¶
| Configuration | GQA | BLINK | Notes |
|---|---|---|---|
| Baseline (w/o SI-GW) | 61.2 | 40.5 | Original LLaVA-1.5-7B model |
| + Decoupled Extraction + Head Selection | 61.7 | 41.9 | Step-by-step integration: step 1 |
| + Token Filtering (Full) | 61.9 | 42.9 | Full MIRROR |
| w/o Decoupled Extraction | diverged | diverged | Training directly diverges |
| w/o Head Selection | 59.8 (-2.1) | 35.6 (-7.3) | Severe degradation |
Key Findings¶
- Decoupled extraction is an "existential" component: without it, training directly diverges. Head selection, although highly impactful, does not cause divergence, placing it as secondary in importance.
- The largest improvement is seen in Global scenarios (+1.04% to +3.16%): such tasks (e.g., "how many red objects are on the table?") require integrating relationship structures between multiple objects; Object/Attribute tasks show minimal improvement, as they rely on local feature identification rather than relational geometry, aligning with the expected behavior of the method.
- Mid-to-high-level reasoning benefits on BLINK are concentrated: spatial relations (+4.9%), multi-view reasoning (+3.8%), localization (+4.9%), and counting (+4.2%) all show significant improvements. Low-level vision tasks (reflections, depth estimation) remain largely unchanged, confirming that gains are indeed derived from relational geometry alignment rather than generic feature enhancement.
- General VQA performance remains intact: performance on VQAv2, POPE, and RealWorldQA remains stable or slightly improves. POPE (hallucination detection) consistently improves (+0.3 to +0.6), indicating that better cross-modal distance structures help mitigate object hallucination.
Highlights & Insights¶
- Deep insight: Elevates cross-modal alignment from identity-level to relational-structure level, capturing the root cause of MLLM relational reasoning failure: "what" each concept is is aligned, but "how they relate" is not. This analysis has strong inspiration value in itself, serving as an analytical framework for diagnosing reasoning deficiencies in future work.
- Elegant theory: The Semi-Inverse GW problem features a unique closed-form solution with a clear probabilistic expectation interpretation, rather than acting as a black-box regularization. The speedup from O(n²ₜ n²ᵥ) to O(n²ₜ nᵥ + n²ᵥ nₜ) relies on mathematical structure rather than approximations.
- Engineering practicality: Introduces no extra parameters or inference overhead, serving purely as a regularization term integrated directly into existing MLLM fine-tuning pipelines during training. Designs like decoupled layer extraction and head selection resolve stability issues in practical deployment.
- Good transferability: Can be directly applied to any projection-adapter-based MLLM. The concept of relational alignment can also be extended to other cross-modal scenarios (such as video-language, 3D-language).
Limitations & Future Work¶
- Experiments are only validated on LLaVA-1.5/LLaVA-NeXT, and generalization has not yet been tested on larger-scale MLLMs (such as Qwen2-VL, InternVL2, etc.) or more diverse training data.
- SI-GW relies on cross-attention as the cross-modal coupling—if the model's own cross-attention quality is poor (such as in some sparsely-activated architectures), the coupling C may be unreliable, limiting the effectiveness of the method.
- The sensitivity window for hyperparameter λ is narrow (optimal at 2×10⁻³, begins to drop at 10⁻²); different models may require re-tuning. Future work could explore adaptive weight strategies.
Related Work & Insights¶
- vs Standard Projection Alignment (e.g., LLaVA): Projection adapters ensure each visual token is semantically correct but do not guarantee relational structures are consistent across modalities; MIRROR adds a first-order relational loss to address this blind spot.
- vs Gromov-Wasserstein Distance: Classic GW simultaneously optimizes both the coupling and distance matrices; MIRROR fixes the coupling and one distance matrix to optimize the other—this "semi-inverse" viewpoint is more natural and efficient than classic GW for scenarios with natural coupling (cross-attention).
- vs Platonic Representation Hypothesis (Huh 2024): This theory suggests that cross-modal representations tend to share statistical structures. MIRROR actually provides a concrete algorithm to achieve this alignment, rather than merely observing the phenomenon.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Addresses relational geometry alignment as a new dimension of cross-modal alignment; the formulation of the Semi-Inverse GW problem is novel and yields a closed-form solution]
- Experimental Thoroughness: ⭐⭐⭐⭐ [Validated across 4 configurations, 2 relational reasoning benchmarks, and multiple general VQA tasks with complete ablation; however, only covers the LLaVA series]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, tidy mathematical derivations, compact theorem proofs, and well-organized experimental results]
- Value: ⭐⭐⭐⭐⭐ [Directly diagnoses a key deficiency in MLLMs and offers a plug-and-play solution compatible with existing models, carrying high practical and inspirational value]