Mixture of Specialized Vision Experts: Unlocking Complementary Visual Insights for Faithful MLLM Reasoning

Conference: ECCV 2026
OpenReview: https://eccv.ecva.net/virtual/2026/poster/4465
Paper: ECCV 2026 Official PDF
Code: https://github.com/jerry19h/MoSVE
Area: Multimodal VLM
Keywords: Mixture of Vision Experts / Faithful Multimodal Reasoning / Hallucination Mitigation / Query-Guided Complementary Clustering / Complementary Rejection Fine-Tuning

TL;DR

Addressing perceptual blind spots in single-encoder MLLMs and feature redundancy from naive expert concatenation, MoSVE introduces query-guided patch-level adaptive complementary clustering to compress visual tokens, paired with Complementary Rejection Fine-Tuning (CRFT) that routes anchor failure samples to the most visually disparate auxiliary expert, achieving SOTA hallucination mitigation on POPE (86.9%), CHAIR_S (27.1%), and MMVP (57.0%) using only 576 visual tokens.

Background & Motivation

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, instruction following, and complex multimodal reasoning. However, when faced with fine-grained object attributes, subtle spatial relationships, and precise boundary structures, MLLMs frequently suffer from severe visual hallucinations where generated responses conflict with ground-truth visual evidence. This fundamental lack of reasoning faithfulness is primarily driven by perceptual shortcomings in the visual front-end: mainstream architectures almost exclusively rely on a single CLIP-based visual encoder. While CLIP provides strong cross-modal global semantic alignment, it exhibits inherent perceptual blind spots regarding fine-grained spatial configurations, object contours, and dense local details.

To surpass the perceptual ceiling of single-encoder architectures, recent studies have turned to incorporating multiple heterogeneous vision experts (such as combining CLIP with DINOv2 and SAM-L). Nevertheless, existing multi-expert frameworks remain constrained by two critical limitations. First, coarse-grained integration strategies—such as direct token concatenation or naive interleaving—dramatically inflate visual token sequences (e.g., LEO consumes 3584 tokens) without accounting for spatial feature overlaps across experts, resulting in massive computational overhead and representation redundancy. Second, unspecialized alignment paradigms that rely on joint fine-tuning across standard datasets frequently cause feature conflicts or pull diverse expert representations toward a homogeneous average, stripping auxiliary experts of their unique perceptual strengths.

Diagnostic analysis of self-attention heatmaps across visual backbones reveals distinct and complementary focal patterns: CLIP prioritizes high-level query-relevant semantic semantics, DINOv2 spontaneously captures local object properties without text guidance, and SAM-L delineates instance shapes and geometric edges. Unlocking this inter-expert complementarity requires selectively preserving informative details while actively cultivating differentiated expert skills. The core idea of this paper is to build MoSVE (Mixture of Specialized Vision Experts), which combines query-guided complementary clustering to adaptively merge consensus tokens during inference with Complementary Rejection Fine-Tuning (CRFT) to route anchor failure cases to the most visually disparate auxiliary expert during training, establishing a faithful and token-efficient multi-expert reasoning system.

Method

Overall Architecture

The MoSVE framework is composed of three interconnected modules: a multi-expert perceptual foundation (featuring CLIP-L/14-336 as the anchor expert, alongside DINOv2-L/14 and SAM-L/16 as auxiliary experts), a Query-Guided Complementary Clustering module for patch-level feature aggregation, and a Complementary Rejection Fine-Tuning (CRFT) strategy for discrepancy-driven expert specialization.

During inference, an input image is concurrently processed by the frozen vision experts, and their extracted patch representations are projected into a unified embedding dimension via lightweight projectors. For each spatial patch, expert tokens are evaluated against the input textual query to determine cross-modal relevance and semantic divergence entropy. These values dynamically define an adaptive distance threshold for hierarchical clustering. Redundant tokens within each cluster are merged into cluster centroids, producing a compact sequence of complementary visual tokens that are fed into the Vicuna LLM backbone for faithful autoregressive generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Textual Query Instruction"] --> B["Multi-Expert Visual Extraction & Projection<br/>CLIP Anchor + DINOv2 Local + SAM-L Shape"]
    B --> C["Query-Guided Complementary Clustering<br/>Adaptive Thresholding via Relevance & Divergence"]
    C --> D["Patch-Level Hierarchical Clustering & Centroid Fusion<br/>Compress Consensus Redundancy & Retain Insights"]
    D --> E["LLM Autoregressive Decoding<br/>Faithful Hallucination-Free Reasoning Output"]

Key Designs

1. Query-Guided Complementary Clustering: Context-Aware Adaptive Token Compression Naive concatenation of multi-expert tokens multiplies sequence length and imposes substantial attention costs. To achieve precise feature fusion while preserving local spatial geometry, MoSVE executes adaptive hierarchical clustering within each spatial image patch. For the \(i\)-th patch, projected visual tokens from \(N_e\) experts form the set \(V_i = \{v_i^{(k)} \in \mathbb{R}^d\}_{k=1}^{N_e}\). The pairwise cosine similarity \(S_i(m,n)\) and complementary distance \(D_i(m,n) = 1 - S_i(m,n)\) are defined as:

\[S_i(m,n) = \frac{\langle v_i^{(m)}, v_i^{(n)} \rangle}{\|v_i^{(m)}\|_2 \|v_i^{(n)}\|_2}\]

Small distances reflect redundant consensus across experts, whereas larger distances indicate distinct complementary evidence. To dynamically regulate clustering aggressiveness according to the reasoning context, the framework computes an expert centroid via mean pooling \(\bar{v}_i = \frac{1}{N_e} \sum_{k=1}^{N_e} v_i^{(k)}\), followed by calculating its cosine similarity with the \(N_t\) text query tokens to yield similarity vector \(s_i \in \mathbb{R}^{N_t}\). Cross-modal relevance \(\bar{s}_i\) and semantic divergence entropy \(e_i\) are then calculated as:

\[\bar{s}_i = \frac{1}{N_t} \sum_{t=1}^{N_t} s_i^{(t)}, \quad e_i = -\sum_{t=1}^{N_t} p(s_{i,t}) \log p(s_{i,t})\]

where \(p(s_i) = \text{Softmax}(s_i)\). Using these two metrics, a soft function adaptively derives the clustering distance cutoff \(\tau_i\):

\[\tau_i = 1 - \sigma(\alpha \bar{s}_i + \beta e_i)\]

with hyperparameters empirically set to \(\alpha = 0.45\) and \(\beta = 0.75\). In query-critical or multi-faceted semantic regions, \(\tau_i\) contracts to enforce conservative clustering, preserving individual expert nuances. In background or uninformative regions, \(\tau_i\) expands to merge consensus tokens aggressively. For each resulting cluster \(\mathcal{C}_{i,m}\), member tokens are averaged into a single centroid \(\hat{v}_{i,m} = \frac{1}{|\mathcal{C}_{i,m}|} \sum_{v \in \mathcal{C}_{i,m}} v\). Concatenating these centroids across patches yields a compact visual sequence that fits a standard 576-token budget without forfeiting crucial fine details.

2. Complementary Rejection Fine-Tuning: Discrepancy-Driven Expert Specialization Joint end-to-end training across all multi-expert projectors on standard datasets often homogenizes feature representations toward the anchor CLIP space. To counter this, Complementary Rejection Fine-Tuning (CRFT) is proposed to explicitly train auxiliary experts on the specific perceptual blind spots of the anchor model.

First, an anchor-based rejection sampling step constructs a negative hard-sample pool \(P_{\text{neg}}\). The anchor single-encoder MLLM is evaluated across training samples: exact matching filters errors in deterministic tasks, while spaCy parses free-form descriptions to detect omitted target objects or conflicting attributes. Next, for each failed instance in \(P_{\text{neg}}\), global average pooling generates an image-level representation \(\bar{v}^{(k)}\) for each expert. Cosine similarity between each auxiliary expert and the anchor representation \(\bar{v}^{(\text{anchor})}\) is evaluated, and the failure sample is routed exclusively to the auxiliary expert \(k^*\) exhibiting the greatest feature discrepancy (minimal cosine similarity):

\[k^* = \arg\min_{k \in \text{Aux}} S(\bar{v}^{(\text{anchor})}, \bar{v}^{(k)})\]

By routing anchor failure cases strictly to the most orthogonal auxiliary expert, each projector is guided to specialize in rectifying specific perceptual weaknesses, directly preventing representational collapse.

Loss & Training

CRFT is executed through a two-stage training curriculum: - Stage 1 (Expert Projector Specialization): Vision encoders and the LLM backbone are kept frozen. Each auxiliary expert's lightweight projector is trained independently using its exclusively assigned hard-sample subset \(\mathcal{D}_k\). Optimization uses AdamW with an initial learning rate of \(1\times 10^{-4}\) and batch size 64. - Stage 2 (Collaborative Multi-Expert Alignment): Full training data is integrated. With frozen vision encoders, all expert projectors and the LLM backbone undergo joint instruction tuning for 1 epoch using a learning rate of \(2\times 10^{-5}\) under a cosine decay schedule. Distributed training is performed on 8 NVIDIA A100 GPUs.

Key Experimental Results

Main Results

MoSVE is thoroughly benchmarked against single-stage base models, multi-stage scaled systems, and state-of-the-art multi-expert architectures on POPE, CHAIR, MMVP, and MMBench, as reported in Table 1.

Model	Expert Paradigm	LLM Backbone	Visual Tokens	POPE Acc ↑	POPE F1 ↑	CHAIR_S ↓	CHAIR_I ↓	MMVP Acc ↑	MMBench Acc ↑
LLaVA-1.5	Single-stage	Vicuna-7B	576	83.1	82.9	45.6	16.3	22.0	64.3
LLaVA-1.5	Single-stage	Vicuna-13B	576	83.9	83.6	41.3	14.2	22.0	67.7
mPLUG-Owl	Two-stage	LLaMA-7B	64	83.2	82.9	39.9	14.8	–	64.5
SPHINX	Multi-stage	Vicuna-13B	2880	81.1	80.1	42.7	15.6	23.5	65.9
LLaVA-NeXT	Multi-stage	Vicuna-v1.5-7B	2880	85.5	85.2	36.9	12.4	39.0	67.4
LLaVA-NeXT	Multi-stage	Hermes-Yi-34B	2880	86.1	86.0	35.8	12.0	44.0	–
MARINE	Guidance	Vicuna-v1.5-7B	–	84.3	83.8	37.5	12.7	42.0	65.7
MoF	Expert-fusion	Vicuna-v1.5-7B	512	84.3	84.2	39.8	15.2	28.0	66.2
LLaVA-HR	Multi-scale	Vicuna-v1.5-7B	1024	85.9	85.8	38.0	13.2	52.0	68.6
MoVE-KD	Distillation	Vicuna-v1.5-7B	577	86.3	–	–	–	37.0	67.6
LEO	Multi-stage	InternLM2-7B	3584	86.5	86.3	31.2	12.1	46.0	67.8
MoSVE (Ours)	Complementary	Vicuna-v1.5-7B	576	86.9	86.5	27.1	12.2	57.0	68.3

Ablation Study

The impact of Query-Guided Complementary Clustering is ablated against a static threshold baseline and an uncompressed upper bound (all variants trained with CRFT), detailed in Table 2.

Variant	Visual Tokens	POPE Acc ↑	CHAIR_S ↓	MMVP Acc ↑	Note
Fixed Threshold (Fixed + CRF)	600	85.1	34.0	48.0	Static cutoff \(\tau = 0.45\), ignoring query context
Query-Guided Clustering (QG + CRF, MoSVE)	576	86.9	27.1	57.0	Dynamic thresholding via relevance & divergence
Theoretical Upper Bound (All Tokens + CRF)	1536	87.2	25.1	59.0	Bypasses clustering, retaining all expert tokens

Table 3 evaluates the progressive integration of vision experts alongside the impact of CRFT (C: CLIP, D: DINOv2, S: SAM-L).

Configuration	With CRFT	POPE Acc ↑	CHAIR_S ↓	MMVP Acc ↑	Key Insight
CLIP (C)	✗	83.1	45.6	22.0	Single-encoder baseline
C + D	✗	84.5	38.8	38.0	Incorporates local object-level representations
C + S	✗	84.2	39.5	36.0	Adds instance boundary and shape priors
C + D + S	✗	85.3	36.8	42.0	Naive multi-expert combination suffers from homogenization
MoSVE (C + D + S)	✓	86.9	27.1	57.0	CRFT routing sparks a +15.0% leap on MMVP

Table 4 validates the necessity of discrepancy-driven sample routing within CRFT:

Routing Strategy	POPE Acc ↑	CHAIR_S ↓	MMVP Acc ↑	Analysis
Random Routing	85.7	30.8	50.0	Arbitrary assignment disrupts specialized skill learning
Standard SFT (No Routing)	86.5	29.2	54.0	Joint unspecialized training dilutes complementary gains
Complementary Routing (Minimal Cosine Sim.)	86.9	27.1	57.0	Explicitly routes failures to the most capable expert

Key Findings

• High Information Density under Compact Token Budgets: MoSVE retains 96.6% of the theoretical upper bound's performance on MMVP (57.0 vs. 59.0) and closely mirrors its sentence hallucination rate (27.1 vs. 25.1), while utilizing 2.67 times fewer visual tokens (576 vs. 1536).
• CRFT is the Linchpin of Multi-Expert Synergies: Adding experts without CRFT only raises MMVP from 22.0 to 42.0, whereas incorporating discrepancy-driven routing elevates MMVP by an additional 15.0 percentage points to 57.0 and slashes CHAIR_S from 36.8% to 27.1%.
• Balanced Inference Latency: On POPE, MoSVE achieves an inference throughput of 0.68 samples/s (1.49 s/sample), outperforming LEO (0.29 samples/s, 3.49 s/sample) by over \(2.3\times\) while delivering superior accuracy (86.9 vs. 86.5).

Highlights & Insights

• Relevance-and-Entropy Driven Adaptive Clustering: By integrating query cosine similarity and attention entropy into the distance cutoff function, MoSVE dynamically preserves multi-expert visual features in question-critical regions while consolidating background redundancy, resolving the dilemma between sequence expansion and fine detail retention.
• Counter-Intuitive Minimum Similarity Negative Routing: Rather than training all experts indiscriminately on hard samples, CRFT routes anchor failure cases specifically to the auxiliary expert with minimal cosine similarity. This discrepancy-driven mechanism systematically forces auxiliary models to master the exact visual representations that the anchor fails to perceive.
• Generalizable Architecture: The dual principles of query-guided token clustering and discrepancy-driven hard-sample routing can be directly adapted to other heterogeneous perceptual setups, including infrared, depth, or high-resolution encoders.

Limitations & Future Work

• Reliance on a Fixed Anchor Model: CRFT's routing pipeline assumes a designated base visual encoder (CLIP in this work). Extending this methodology to fully unified or omni-modal vision architectures requires rethinking anchor-auxiliary role assignment.
• Offline Negative Mining Overhead: Assembling the negative hard-sample pool requires running offline inference across the entire training corpus using the anchor model and parsing outputs with external rule-based NLP tools (spaCy), which introduces pre-training computation overhead.
• Extension to Dynamic Video Domains: Validation in this work centers on static images. Applying patch-level clustering and expert routing across temporal video frames poses challenges in maintaining cross-frame tracking consistency and centroid stability.

Related Work & Insights

• vs. LEO / SPHINX (Concatenation & Multi-Stage Expansion): Prior methods concatenate multi-expert visual tokens directly, blowing up token budgets to 2880~3584 tokens and imposing heavy compute burdens; MoSVE achieves superior hallucination mitigation within a lightweight 576-token budget via query-guided clustering.
• vs. MoF (Token Interleaving): MoF naively interleaves tokens without adaptive clustering or specialized routing, introducing sensory noise on subtle visual differences (scoring only 28.0 on MMVP); MoSVE achieves 57.0 on MMVP by combining centroid fusion with discrepancy-driven CRFT.
• vs. MoVE-KD (Knowledge Distillation): MoVE-KD compresses multi-expert representations into a single encoder, which remains bottlenecked by the parameter capacity of a single ViT; MoSVE maintains specialized parallel representations that are merged adaptively during inference.

Rating

• Novelty: ⭐⭐⭐⭐⭐ An elegant fusion of query-guided spatial clustering and discrepancy-driven failure routing that overcomes multi-expert redundancy and homogenization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across hallucination benchmarks (POPE, CHAIR), fine-grained visual discrimination (MMVP), general reasoning (MMBench), and inference latency.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, well-structured mathematical formulations, and rigorous experimental presentation.
• Value: ⭐⭐⭐⭐⭐ Offers a highly practical and computationally efficient architectural blueprint for building faithful, hallucination-resistant MLLMs.