title: >- [Paper Note] MED-LCDS: Multi-Expert-Domain CLIP Classification via Logit Calibration description: >- [ECCV 2026][Multimodal VLM][Multi-Expert-Domain Classification] Restoring cross-domain logit comparability for merged LoRA experts via domain-supervised routing and learnable domain temperature scaling. tags: - ECCV 2026 - Multimodal VLM - Multi-Expert-Domain Classification - LoRA Merging - Logit Calibration date: 2026-09-19 content_hash: debc2d644494671a
MED-LCDS: Multi-Expert-Domain CLIP Classification via Logit Calibration¶
Conference: ECCV 2026
Paper: ECCV 2026
Area: Multimodal VLM
Keywords: Multi-Expert-Domain Classification, LoRA Merging, Logit Calibration, Domain-Supervised Routing, Zero-Shot Open-Set Recognition
TL;DR¶
To resolve cross-domain interference and artificial overconfidence caused by heterogeneous logit scales when merging independently trained LoRA experts into a unified vision-language model, MED-DSLC (MED-LCDS) combines domain-supervised mixture-of-experts routing with learnable domain-wise temperature calibration, effectively restoring global logit comparability across unified label spaces.
Background & Motivation¶
Vision-language foundation models like CLIP establish the cornerstone of zero-shot open-vocabulary image classification by projecting images and text prompts into a shared multimodal semantic embedding space. A foundational premise underlying this capability is the global comparability of cosine similarity logits across arbitrary candidate classes, allowing standard Softmax normalization over arbitrary label sets to yield meaningful posterior probabilities. However, off-the-shelf vision-language models often suffer substantial performance degradation on fine-grained or specialized domains (such as aircraft models, satellite remote sensing, or texture categorization) where public pretraining data is scarce. Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), has thus emerged as the standard recipe to specialize foundation encoders on domain-specific datasets. While this dramatically boosts in-domain accuracy, out-of-domain discrimination degrades severely, resulting in a fragmented ecosystem composed of thousands of isolated adapters.
Multi-Expert-Domain (MED) classification aims to consolidate these independently adapted experts into a single unified classifier capable of recognizing classes from the union of all candidate domains without requiring domain identifiers at test time. Existing Mixture-of-Experts (MoE) merging paradigms, such as MoLE or Phatgoose, predominantly adopt unsupervised gating networks coupled with auxiliary load-balancing objectives. Such approaches not only struggle with expert balance and routing collapse, but often misroute test samples into mismatched adapter branches. Even more critically, independently adapted experts reshape both the internal geometry and absolute magnitude of their output logits. Because the classification probability ranking remains strictly invariant under arbitrary positive scalar transformations within any isolated domain, independent training leaves the absolute logit scales across disparate domains completely uncalibrated. When classes from multiple domains compete simultaneously under a shared Softmax normalization, domains that produce higher-magnitude logits systematically dominate predictions, producing spurious high-confidence errors on out-of-domain inputs.
This paper tackles these twin bottlenecks by recognizing that in MED training, the domain identity of every training image is readily accessible via its ground-truth class label, rendering fragile unsupervised routing unnecessary. Furthermore, cross-domain scale disparities can be cleanly reconciled at the output layer with minimal parameter overhead. Core idea: formulate multi-expert adaptation as a dual architecture combining domain-supervised gating routing and domain-wise logit temperature calibration (MED-DSLC), where explicit domain supervision drives accurate expert selection and learnable domain-specific temperature scalars restore global logit comparability across heterogeneous experts.
Method¶
Overall Architecture¶
The input to MED-DSLC comprises an input image and a unified text prompt vocabulary drawn from the union of all expert domains, with the goal of generating accurate class probabilities without access to domain annotations at test time. The underlying pretrained vision-language model (e.g., CLIP vision and text encoders) and all independently fine-tuned domain LoRA adapters remain strictly frozen throughout training. MED-DSLC introduces only a lightweight gating network and one learnable temperature scalar per expert domain. As image features propagate through the frozen encoder layers, the gating network dynamically predicts mixing weights to form a composite multi-expert residual. At the output layer, raw text-image similarity logits are divided by their corresponding domain-specific temperature scalars before being passed to a unified Softmax over all candidate classes.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image x"] --> B["Frozen VLM Feature Extraction"]
B --> C["Domain-Supervised Gating Routing<br/>Linear projection + Softmax expert weights"]
C --> D["Frozen LoRA Expert Weighted Merging<br/>Layer-wise multi-domain residual synthesis"]
D --> E["Unified Multimodal Logit Computation<br/>Cross-domain text-image candidate similarity"]
E --> F["Domain-Wise Logit Scaling<br/>Domain-specific temperature normalization tau_d"]
F --> G["Joint Cross-Domain Softmax Decision<br/>Global competitive classification across all classes"]
Key Designs¶
1. Domain-Supervised Gating Routing: Eliminating Expert Collapse via Class-Domain Priors
Conventional MoE methods for adapter merging assume that domain labels are absent during training, relying on heuristic balancing objectives to mitigate expert collapse. In contrast, in the MED formulation, each training sample arrives with a ground-truth class label \(y\), and every class uniquely belongs to an identified domain label set \(\mathcal{Y}_d\). The true domain assignment can thus be deterministically obtained via an indicator lookup: \(d^* = \arg\max_d \mathbb{I}(y \in \mathcal{Y}_d)\). MED-DSLC replaces unsupervised balance regularization with an explicit domain-supervised routing loss \(\mathcal{L}_{\text{domain}} = -\log \alpha_{d^*}(x)\), where \(\alpha_{d^*}(x)\) represents the routing probability allocated to the correct expert. This strong supervisory signal ensures sharp discrimination across domain boundaries and guarantees natural expert load balancing whenever training datasets across domains are balanced in size.
2. Domain-Wise Logit Scaling: Learnable Temperature Rescaling for Cross-Domain Alignment
Because each domain LoRA expert is optimized independently on its respective dataset, variations in parameter norm and convergence curvature induce systematic discrepancies in raw logit scales. While in-domain class ranking is preserved under any uniform positive scaling \(\gamma_d > 0\), heterogeneous constants across domains destroy cross-domain comparability when candidate classes compete under a joint Softmax: $$ \exists \gamma_a \neq \gamma_b \quad \text{s.t.} \quad \arg\max (p(a \mid x), p(b \mid x)) \neq \arg\max (e^{\gamma_a z_a(x)}, e^{\gamma_b z_b(x)}) $$ To eliminate out-of-domain score inflation, MED-DSLC assigns a single learnable scalar temperature \(\tau_d > 0\) to each domain \(d \in \{1, \dots, D\}\). For every class \(c \in \mathcal{Y}_d\), the raw logit \(z_c(x)\) is rescaled as: $$ \tilde{z}_c(x) = \frac{z_c(x)}{\tau_d} $$ This operation strictly maintains within-domain discrimination while aligning the dynamic ranges across disparate domains. Crucially, this mechanism adds only \(D\) scalar parameters in total (e.g., 10 floating-point numbers across 10 domains), incurring virtually zero compute and storage overhead.
3. Joint Cross-Domain Softmax Decision: Preserving In-Domain Discrimination While Suppressing Interference
Once all candidate class logits have been calibrated by their respective domain temperatures, they are concatenated into a unified cross-domain logit vector and evaluated via a single Softmax distribution: $$ p(c \mid x) = \frac{\exp(\tilde{z}c(x))}{\sum $$ Because domain-supervised routing ensures that image representations are primarily shaped by the relevant domain expert, while domain-wise logit scaling prevents non-activated experts from projecting aberrant logit spikes, the model completely circumvents cross-domain interference. At inference time, no domain identifier is required, and the classifier operates robustly across hundreds of fine-grained categories spanning distinct domains.}} \exp(\tilde{z}_{c'}(x))
Loss & Training¶
During the merging stage, the foundational CLIP backbone and all pre-trained domain LoRA parameters remain strictly frozen. The training objective optimizes only the gating projection matrix \(W_g\) and the domain temperature vector \(\{\tau_d\}_{d=1}^D\) via a composite loss: $$ \mathcal{L} = \mathcal{L}{\text{cls}} + \lambda \mathcal{L} $$ where }\(\mathcal{L}_{\text{cls}} = -\log p(y \mid x)\) enforces accurate class prediction across the full joint vocabulary, \(\mathcal{L}_{\text{domain}} = -\log \alpha_{d^*}(x)\) provides direct routing supervision, and \(\lambda\) balances the two objectives. Optimization is performed using AdamW in the low-shot regime (1-shot to 16-shot per class) with rapid convergence.
Key Experimental Results¶
Main Results¶
The method is evaluated across 9 fine-grained recognition benchmarks (Caltech101, EuroSAT, Stanford Cars, Food101, Oxford Pets, Oxford Flowers, DTD, UCF101, FGVC Aircraft) spanning 10 domain experts (783 filtered classes). Evaluations follow two protocols: Cross-domain, where all classes compete in a single shared label space without domain guidance at test time; and In-domain, where datasets are evaluated separately. Results are reported on both Base (seen classes) and All (seen + unseen classes) splits.
The table below summarizes the core cross-domain evaluation over the unified label space (from Table 1 of the original paper):
| Method | Base Split Mean (%) | Base Split \(\Delta\) (%) | All Split Mean (%) | All Split \(\Delta\) (%) | EuroSAT (All) | Cars (All) | DTD (All) | FGVC (All) |
|---|---|---|---|---|---|---|---|---|
| Expert-LoRA (Oracle) | 85.48 | +15.36 | 71.60 | +6.63 | 67.58 | 70.10 | 56.50 | 32.34 |
| MED-DSLC (Ours) | 85.51 | +15.39 | 71.80 | +6.83 | 68.43 | 70.33 | 56.26 | 32.19 |
| KnOTS-DARE-TIES [19] | 78.17 | +8.05 | 68.66 | +3.69 | 58.28 | 70.51 | 49.88 | 27.51 |
| KnOTS-TIES [19] | 74.91 | +4.79 | 67.70 | +2.73 | 54.19 | 69.22 | 46.10 | 28.47 |
| PHATGOOSE [12] | 72.97 | +2.85 | 59.77 | -5.20 | 23.10 | 65.86 | 47.10 | 23.37 |
| MoLE [21] | 71.80 | +1.69 | 65.85 | +0.88 | 56.25 | 65.03 | 44.86 | 25.20 |
| LoRA-Mean (Parameter Average) | 70.12 | +0.00 | 64.97 | +0.00 | 45.81 | 66.98 | 42.67 | 26.13 |
| Single-LoRA (Joint Tuning) | 42.35 | -27.77 | 36.28 | -28.69 | 28.10 | 27.27 | 25.65 | 0.99 |
| Zero-Shot CLIP | 66.22 | -3.90 | 62.52 | -2.45 | 40.21 | 65.13 | 40.54 | 24.57 |
Under the unified cross-domain evaluation, MED-DSLC achieves 85.51% on Base Split and 71.80% on All Split, surpassing existing adapter-merging schemes (MoLE and PHATGOOSE) by substantial margins (5.9% to 13.7%) and matching or slightly outperforming the test-time domain oracle Expert-LoRA.
Ablation Study¶
The table below presents the ablation of individual components in the cross-domain setting, evaluating combinations of logit scaling, domain supervision, and gating activation choices (from Table 3 of the original paper):
| Logit Scaling | Domain Supervised | Gating Activation | Base Split Mean (%) | Base \(\Delta\) (%) | All Split Mean (%) | All \(\Delta\) (%) | EuroSAT (All) | DTD (All) |
|---|---|---|---|---|---|---|---|---|
| \(\checkmark\) | \(\checkmark\) | Softmax | 85.51 | +15.39 | 71.80 | +6.83 | 68.43 | 56.26 |
| \(\times\) | \(\checkmark\) | Softmax | 85.43 | +15.31 | 71.61 | +6.64 | 67.47 | 56.26 |
| \(\checkmark\) | \(\checkmark\) | Sigmoid | 79.21 | +9.09 | 69.97 | +5.00 | 69.04 | 49.76 |
| \(\times\) | \(\checkmark\) | Sigmoid | 78.55 | +8.43 | 69.67 | +4.71 | 68.94 | 49.35 |
| \(\times\) | \(\times\) | Softmax (MoLE) | 71.80 | +1.69 | 65.85 | +0.88 | 56.25 | 44.86 |
In the target-domain imbalance sweep experiment, restricting the target domain to \(k=2\) classes reveals that Logit Scaling delivers an additional +2.2% boost specifically on the minority domain. Moreover, under 1-shot routing adaptation (Table 7), MED-DSLC achieves 71.71% mean accuracy on the All Split using just a single labeled image per class, demonstrating exceptional sample efficiency compared to the 16-shot baseline (71.80%).
Key Findings¶
- Domain supervision is the primary driver of robust routing: Moving from unsupervised routing (MoLE at 71.80% Base / 65.85% All) to supervised routing elevates performance to 85.43% / 71.61%, proving that leveraging readily available class-domain mappings prevents severe misrouting.
- Logit calibration is vital under extreme imbalance and larger scales: While the overall mean gain of logit scaling on balanced benchmarks is modest (+0.19%), its impact expands sharply under severe class imbalance (+2.2% at \(k=2\) classes), resolving cross-domain histogram overlap and suppressing out-of-domain false positives.
- Softmax gating outperforms decoupled Sigmoid routing: Normalized Softmax routing enforces competitive expert selection across mutually exclusive domains, outperforming independent Sigmoid gates by over 6% on the Base Split.
Highlights & Insights¶
- Minimalist, zero-latency post-hoc calibration: Introducing just one learnable scalar temperature per domain aligns divergent logit dynamic ranges resulting from decentralized adapter tuning, incurring zero additional inference latency and negligible parameter cost.
- Surpassing the single-domain oracle: Conventional intuition presumes an adapter merger can at best match an oracle with ground-truth domain knowledge. MED-DSLC marginally exceeds Expert-LoRA (85.51% vs. 85.48% Base, 71.80% vs. 71.60% All), demonstrating that soft multi-expert blending acts as an effective regularizer against domain overfitting.
- Extreme sample efficiency: Because routing and calibration introduce only a linear layer and a handful of scalars, 1-shot supervision achieves 71.71% on All Split, almost matching the 16-shot performance (71.80%) and dramatically outperforming prior merging baselines.
Limitations & Future Work¶
- Reliance on disjoint and well-defined domain sets: The framework assumes each class strictly maps to an unambiguous expert domain (\(\mathcal{Y}_d\) partition), requiring manual filtering when domains share common overlapping categories (e.g., Caltech101 overlapping with Flowers or Cars).
- Domain-level rather than class-level temperature granularity: A single scalar \(\tau_d\) is shared across all classes within a domain, which may fail to capture intra-domain scale variations stemming from text prompt disparity or class-frequency imbalance.
- Future directions: Extending calibration to continuous or soft clustering regimes where domain boundaries are unannotated, and exploring application to multimodal generative tasks and cross-domain retrieval.
Related Work & Insights¶
- vs. MoLE / Phatgoose: While MoLE and Phatgoose import unsupervised MoE routing from language models, they suffer from expert imbalance and overlook cross-domain logit scale inflation. MED-DSLC utilizes the deterministic class-domain mapping to train supervised gates and introduces domain temperature scaling, outperforming MoLE by over 13% on Base Split cross-domain evaluation.
- vs. Weight-space merging (KnOTS / DARE / TIES / LoRA-Mean): Arithmetic and SVD-based parameter merging methods enforce a single compromised checkpoint, inevitably losing domain-specific nuances across heterogeneous tasks. MED-DSLC preserves full expert specialization in activation space while eliminating competition artifacts at the logit layer.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Pinpoints the crucial issue of cross-domain logit miscalibration in decentralized PEFT merging and resolves it with a clean temperature scaling formulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across 9 benchmarks, 10 expert domains, Base and All splits, including imbalance sweeps and low-shot data efficiency analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Methodological formulations and theoretical justifications are lucid, supported by insightful histogram and logit distribution visualizations.
- Value: ⭐⭐⭐⭐⭐ Provides an immediately applicable, lightweight blueprint for scalable decentralized adapter ecosystems in foundation vision-language models.