MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: model routing, cost-aware evaluation, vision-language fusion, offline benchmark, cross-modality transfer
TL;DR¶
MMR-Bench provides reusable offline evaluation for selecting among heterogeneous multimodal large language models on image-text queries, comparing cost-performance curves across features and routing policies; the authors report an operating point that matches the strongest single model at approximately 33% of its cost, not a uniform saving across all methods and tasks.
Background & Motivation¶
Multimodal large language models (MLLMs) do not form a simple sequence in which capability increases monotonically with size. Their visual encoders, alignment strategies, and training data differ, producing different strengths in OCR, scene question answering, and mathematical reasoning. Sending every request to an expensive model charges demanding-reasoning prices for straightforward recognition, whereas always using a cheap model can fail on detailed charts, small text, or geometric relationships. Model routing therefore offers more than assigning easy questions to small models: it can exploit the partially complementary sets of questions that different models answer correctly.
Several benchmarks already evaluate text-only routing, but the difficulty of a vision-language request cannot be inferred from its question alone. The same question, such as asking what an image says, may accompany either clear large text or densely arranged, low-resolution characters. A text-only router cannot distinguish their resource requirements. Conversely, different questions about the same image can turn object recognition into multistep relational reasoning, so image features alone are also insufficient. When routing policies use different model pools or pricing assumptions, their frontier differences may reflect candidate models rather than routing quality.
The paper consequently fixes the candidates, samples, output scoring, and costs before comparing what information a router observes and how it uses that information. Its main contribution is an evaluation environment and empirical analysis, not a new large-model architecture. Core Idea: precompute every candidate model's utility and cost for each query, then compare image-text routing policies across budgets in a shared offline environment, separating the contributions of multimodal information, selection policies, and the candidate models themselves.
Method¶
Overall Architecture¶
The input is a textual question with an optional image. A router outputs either one candidate model or selection probabilities over models, optionally conditioned on a budget. The benchmark first runs and scores candidate models offline; evaluating a new router subsequently requires table lookups rather than rerunning the entire MLLM pool.
Two costs must therefore be distinguished. Benchmark construction requires collecting many instance-model outcomes, whereas routing evaluation computes a simulated bill from precomputed model-call costs. A low-cost routing curve does not imply that building the outcome table was cheap or that the authors measured every overhead of an online system.
The authors report 11,000 instances, 10 candidate models, and 3 scenarios, covering over 100k instance-model outcomes. The scenarios are OCR, general visual question answering, and multimodal reasoning. Although the text states that there are 8 datasets, its named datasets and main table contain only 7: OCRBench, SEED-Bench v2 Plus, MMStar, RealWorldQA, MathVerse, MathVista, and MathVision. This discrepancy is retained here without inventing an eighth dataset.
Key Designs¶
1. Offline outcome table: separating model capability from routing decisions
Each query has a record containing every candidate's raw answer, task-specific utility, and normalized inference cost. Utilities are mapped to the interval from 0 to 1, and costs follow a shared monetary pricing scheme. Models use fixed prompts and decoding settings within each scenario. The candidate pool includes commercial families such as GPT-5, Gemini 2.5, and Claude, together with open-weight families such as Gemma, InternVL3, and Qwen2.5-VL. The purpose is not merely to collect more models: observing their different successes, failures, and costs on the same queries makes their relative advantages learnable.
The table makes evaluation reusable. A deterministic policy retrieves the cell for its selected model; a probabilistic policy computes probability-weighted utility and cost. This weighting describes expected metrics, not an ensemble that runs several models and combines their answers. Sharing outcomes also prevents repeated-generation randomness from obscuring routing differences. The trade-off is that evaluation remains tied to the recorded model versions, prompts, sampled outputs, and pricing assumptions.
2. Information and call constraints: distinguishing deployable routing from hindsight selection
A real router may observe only inference-time information: the query, optional budget, lightweight embeddings, token counts or scenario tags, and static model metadata such as cost, context length, and supported modalities. Evaluation labels and per-model utilities are forbidden inputs. Multiple adaptive calls on the same instance are also disallowed. The benchmark thus evaluates selecting one model before answering, rather than a cascade that queries a cheap model, checks its answer, and escalates to another model.
Oracle may inspect actual outcomes and select the best model under the budget, but it serves only as an upper bound for the fixed candidate pool, not a deployable system. The motivation study also uses a shared lightweight clustering protocol: run k-means on training features, assign each cluster the model with the highest average validation reward, and route a test query through its nearest cluster. Reward subtracts a cost penalty from utility, while sweeping the cost weight and cluster count generates operating points. This is described verbally because the cached equations are damaged by text extraction; a guessed repair is not presented as the authors' exact formula.
3. Vision-language feature fusion: exposing interactions between question content and visual difficulty
The benchmark extracts offline embeddings with frozen visual and text encoders, using a ViT-based visual encoder. The simple baseline aligns embedding dimensions, replaces missing modalities with zero vectors, and averages both modalities equally. This controls variables outside the router but assumes equal reliability. In dense OCR, the question often reveals little about visual difficulty; in complex question answering, the image alone does not specify the reasoning requested by the user.
The adaptive alternative derives modality-confidence signals from cosine-to-mean relationships and a norm-based sigmoid, then converts them into softmax weights. Fusion includes a weighted sum, a product, and an absolute difference, followed by linear mixing and L2 normalization. Product and difference features can expose agreement or mismatch instead of only the mean direction of two representations. However, these confidence signals are heuristic proxies, not calibrated probabilities of answering correctly. Changing feature geometry may also impair nearest-neighbor retrieval, so the ablation should not be summarized as adaptive fusion always being better.
4. Cross-budget metrics: reporting peak quality separately from cost savings
Varying the budget or cost weight produces operating points with mean cost and mean utility. Their Pareto upper envelope forms the performance-cost curve. The shared comparison interval is determined by the average costs of always choosing the cheapest model and always choosing the most expensive model. Fixing both the pool and interval matters because areas computed over different budget ranges cannot be ranked directly.
The three metrics answer different questions. nAUC is the normalized area under the curve and summarizes performance across budgets. Ps is the highest utility reached on the curve. QNC is the relative cost required to match the most accurate standalone model. QNC below 1 indicates cheaper quality matching; above 1 indicates a higher required cost; positive infinity indicates that the evaluated curve never reaches the target quality. A higher Ps does not imply superiority at every budget, and a higher nAUC does not guarantee a lower QNC. The paper does not fully specify the nAUC normalization and gives Best Single a value of 1.0000 throughout Table 2. That reference row is therefore not interpreted here as an area with the same meaning as all router curves.
Loss & Training¶
The main experiments use frozen 2:8 train/test splits and compare multiple routers with shared fused features. Linear learns per-model linear scores. EmbedLLM uses compact model representations and a lightweight predictor. k-NNRouter aggregates model performance on similar training instances. GraphRouter predicts query-model compatibility on a graph. AvengersPro selects one model for a performance-efficiency objective. RM-Softmax learns a regret-minimizing policy from logged outcomes. Random is a random-selection lower bound, while Oracle is an analytical upper bound with access to actual outcomes.
The benchmark introduces no single new loss for all methods and does not provide every training hyperparameter needed to reproduce each baseline. The original baselines' objectives are therefore not filled in as though they were verified details of this implementation. The motivation study mentions training and validation features, while the main experiments explicitly specify only a 2:8 train/test split; these should not be merged into an assumed fully documented three-way protocol. The text states that metrics are computed per dataset and macro-averaged within and across scenarios. The overall numbers below follow the original Avg column rather than replacing it with a recomputation from visible columns.
For cross-modality transfer, the trained router is frozen and its image channel is masked to zero before zero-shot evaluation on GSM8K, MMLU, and ARC. This tests whether an existing selection policy remains useful without visual input, not whether it can accommodate arbitrary new candidate models without supervision.
Key Experimental Results¶
Main Results¶
Table 1 reproduces selected Full Dataset / Avg results from the paper's Table 2. Conditions are a fixed 10-model pool, a 2:8 split, offline model outputs, and shared normalized monetary costs. Ps is utility on a 0-to-1 scale, and higher nAUC is better. Random has no reported nAUC; the Best Single area reference is not included in router rankings.
| Method | nAUC | Ps |
|---|---|---|
| Random | Not reported | 0.6135 |
| Linear | 0.7042 | 0.7533 |
| EmbedLLM | 0.6913 | 0.7494 |
| k-NNRouter | 0.6950 | 0.7457 |
| AvengersPro | 0.6885 | 0.7496 |
| GraphRouter | 0.6754 | 0.7420 |
| RM-Softmax | 0.6927 | 0.7479 |
| Best Single | 1.0000 (original reference value) | 0.7412 |
| Oracle | 0.8897 | 0.9188 |
Linear exceeds Best Single in Ps by 0.0121, or 1.21 percentage points, but remains well below Oracle. Figure 4 and Section 5.3 report matching the strongest single model at approximately 33% of its cost and exceeding it at higher budgets. The paper does not provide a complete numerical breakdown of this operating point, so it cannot be assigned to every method in the table.
Ablation Study¶
Table 2 reproduces the paper's Table 3, comparing adaptive fusion against equal-weight fusion within each router family. QNC measures relative cost, not accuracy. These fusion-ablation results should not be conflated with a single operating point from the main table.
| Router | nAUC change | Ps change | QNC: equal → adaptive |
|---|---|---|---|
| AvengersPro | +0.3403 | +0.1275 | Positive infinity → 1.0585 |
| EmbedLLM | +0.0033 | +0.0020 | Positive infinity → 0.9947 |
| Linear | +0.0124 | +0.0036 | 0.9055 → 0.9701 |
| k-NNRouter | -0.0074 | -0.0014 | Positive infinity → positive infinity |
| GraphRouter | +0.0025 | +0.0020 | Positive infinity → 0.9998 |
| RM-Softmax | +0.0117 | +0.0125 | 1.1747 → 0.9814 |
AvengersPro gains the most, but its QNC remains above 1. Linear improves its aggregate curve metrics while requiring a higher relative cost to match standalone quality. Both quality metrics decrease for k-NNRouter, showing that feature fusion and the downstream routing algorithm must be chosen together.
Table 3 reproduces the paper's Table 5: the router is frozen after multimodal training, its image channel is masked, and peak scores are reported on text-only tasks. Values retain the original percentage scale. This is not a matched-budget accuracy comparison, and corresponding QNC values are not reported.
| Method | GSM8K Ps | MMLU Ps | ARC Ps |
|---|---|---|---|
| Best single model | 94.5 | 91.2 | 65.7 |
| MM→Text | 96.7 | 92.4 | 66.7 |
Key Findings¶
- Distribution transfer has positive evidence: the paper's Table 4 reports cross-dataset Shift scores of 0.7234/0.8012/0.7914 for OCR/VQA/Math, versus target-domain best-single scores of 0.7062/0.7936/0.7592. These are Ps values and do not establish that transfer preserves the same cost-saving ratio.
- The overall winner is not the winner on every task. Linear leads overall, while k-NNRouter reaches 0.7807/0.8075 nAUC/Ps on MathVista, showing that method rankings still depend on the data distribution.
- Text-only transfer gains are 2.2, 1.2, and 1.0 percentage points. Without corresponding costs and variance estimates, the conclusion should remain limited to the reported peak-quality advantage.
Highlights & Insights¶
- Comparing decisions over a fixed outcome table is the benchmark's most reusable element. It reduces repeated model-call costs and makes incorrect selections traceable to recorded candidate-model performance.
- Modality importance changes with the query rather than being permanently determined by a task label. Weighted sums, products, and differences offer a lightweight interaction interface, but the ablation cautions against equating more elaborate fusion with better routing.
- QNC exposes differences that Ps alone cannot reveal. A router can simultaneously offer better peak quality and worse quality-neutral cost, so deployment choices must begin with an explicit budget objective.
Limitations & Future Work¶
- The authors' explicit evaluation scope is image-text input, a fixed model pool, and offline single-model selection. Cascades, multiround verification, online feedback updates, and audio/video routing are outside the current protocol.
- Note author's assessment: the 8-versus-7 dataset discrepancy, interpretation of the nAUC reference row, exact pricing and encoder configurations, and training/validation details require clarification from an implementation or supplementary material. The paper's broad conclusion that matrix-factorization-based routers are more robust also cannot replace the actual result that Linear leads overall in Table 2.
- Note author's assessment: normalized model-call charges are not total service costs. Future evaluations should include routing-feature extraction, queueing latency, hardware and throughput, and recalibration after vendor pricing or model versions change.
- Note author's assessment: fixing one output per instance-model pair does not characterize generation randomness, and transfer tables lack confidence intervals and budget-by-budget results. Repeated sampling, independent cost matching, and more explicit held-out-domain protocols would strengthen the generalization evidence.
Related Work & Insights¶
- vs RouterBench / RouterEval: These benchmarks primarily standardize text-model routing comparisons. MMR-Bench incorporates visual inputs and modality availability, making image complexity observable rather than hidden from the evaluation.
- vs routing experts within a model: Mixture-of-experts methods select internal experts, whereas MMR-Bench selects among complete heterogeneous models. Its concern is model-level deployment, so internal expert sparsity cannot directly substitute for its monetary cost metrics.
- vs EmbedLLM / GraphRouter / RM-Softmax: The paper compares these different selection mechanisms using shared features and an outcome table. A transferable research direction is to test which image-text interactions improve cost calibration on unseen domains, rather than optimizing only the highest score on a fixed test set.
Rating¶
- Novelty: 4/5. Standardized evaluation for image-text model-level routing is useful, while the underlying selection mechanisms largely reuse existing methods.
- Experimental Thoroughness: 3/5. Main comparisons, fusion ablations, and transfer tests are included, but reproducibility details, variance estimates, and cost breakdowns are incomplete.
- Writing Quality: 3/5. The central question is clear, but several reporting conventions and the scope of conclusions need tighter explanations.
- Value: 4/5. Useful for studying budget allocation across heterogeneous multimodal models, although online gains still require end-to-end validation.