Skip to content

Adaptive Knowledge Routing for Complex Reasoning in LLMs (Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing)

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/mmai-laboratory/Colla_Q
Area: LLM Reasoning
Keywords: Mixture of Experts, Low-bit Quantization, Activation Entropy, Minimax Precision Balancing, Collaborative Reasoning

TL;DR

Addressing the bottleneck where heavily degraded weak experts harm the aggregated output in low-bit MoE models, this paper proposes Colla-Q, an ensemble-inspired framework that estimates continuous expert capacity and uncertainty via label-free activation entropy and recursively compensates the weakest expert through minimax precision balancing, yielding superior multi-task reasoning accuracy and domain-invariant calibration at 1.57–2.54 bits.

Background & Motivation

Mixture-of-Experts (MoE) architectures have emerged as an indispensable sparse design for scaling modern large language models, significantly amortizing inference computational overhead by conditionally dispatching input tokens to a small active subset of experts. However, despite their sparse per-token execution, serving MoE models requires retaining all expert parameters in GPU memory, which imposes severe memory and deployment bottlenecks on resource-constrained platforms. While post-training quantization (PTQ) offers an appealing avenue to compress model footprint without full fine-tuning, pushing MoE models to extreme low-bit regimes (such as 2-bit or lower) consistently results in severe performance degradation, particularly on complex multi-step reasoning benchmarks.

This pronounced failure stems from the structural fragility and distinct parameterization of MoE systems compared to dense models. Unlike dense networks where layers share redundant parameters across tokens, individual experts in MoE contain relatively few parameters and specialize in distinct, fine-grained semantic contexts. Consequently, individual experts exhibit divergent, non-uniform sensitivity to low-bit quantization, leading to stark performance disparities across experts. Viewed through the lens of ensemble learning theory, the output of an MoE block is dynamically synthesized from routed experts, meaning that even a single under-performing "weak learner" can disproportionately degrade the aggregated representation. Conventional MoE mixed-precision quantization frameworks (e.g., PMQ, MxMoE) attempt to allocate bit budgets based on routing statistics—such as routing frequency or gate activation magnitude on a calibration set. Under this paradigm, frequently routed experts receive higher precision while less frequent experts are pushed to aggressively low bit-widths, accelerating their collapse and corrupting ensemble consensus; furthermore, these routing statistics strongly overfit to the domain distribution of the calibration data.

To resolve this fundamental tension, mixed-precision allocation must move beyond simple routing frequency and instead enforce balanced collective efficacy across all participating experts, operating without discrete token-level labels in continuous activation space. By modeling the MoE ensemble error under load-balanced routing as a convex optimization problem, it can be mathematically shown that the global aggregated error attains its minimum when individual expert errors are strictly equalized. Core idea: formulate low-bit MoE quantization as a collaborative ensemble balancing task, quantify label-free expert representation capacity and uncertainty using continuous differential activation entropy, and deploy a recursive minimax precision balancing algorithm to progressively elevate the weakest bottleneck expert, achieving uniform robustness and calibration-domain invariance.

Method

Overall Architecture

The Colla-Q framework consists of two core stages: continuous activation-entropy evaluation and discrete minimax bit allocation. Given an unquantized full-precision MoE block and an unlabeled calibration sequence, the model first executes forward propagation to capture the continuous output activations of each expert FFN layer. For each expert, it computes the within-channel variance and total global activation variance, formulating the activation-entropy ratio \(\rho\) grounded in differential entropy theory. This proxy is combined with GPTQ reconstruction error to quantify overall expert degradation. Subsequently, the minimax precision balancing algorithm initializes all experts at a minimal bit budget (1-bit) and recursively allocates bit increments to the expert with the highest combined loss until the target total bit budget is exhausted, producing an optimal expert-level mixed-precision assignment \(b^*\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Calibration Input and MoE Forward Pass"] --> B["Activation-Entropy Metric<br/>Calculate ratio ρ of within-channel to total variance"]
    B --> C["Quantization Sensitivity & Loss Modeling<br/>Weight activation entropy with GPTQ reconstruction error"]
    C --> D["Minimax Precision Balancing Algorithm<br/>Initialize at 1-bit, iteratively allocate to worst-performing expert"]
    D --> E["Expert-Level Mixed-Precision Model<br/>Execute INT1/INT2/INT3 low-bit GEMM kernels"]

Key Designs

1. Activation-Entropy Metric: Quantifying Continuous Expert Representation Without Token Labels

Because individual intermediate experts emit continuous hidden vectors rather than discrete next-token probabilities, standard cross-entropy evaluation is inapplicable. Colla-Q grounds expert performance estimation in differential entropy theory under a Gaussian approximation \(H(x) = \frac{1}{2}\ln(2\pi e \sigma^2)\), expressing activation entropy directly via variance. Specifically, it constructs the proxy ratio \(\rho\) between within-channel entropy and total entropy:

\[\rho = \frac{\exp(2H_{\text{within}})}{\exp(2H_{\text{total}})} \propto \frac{\sigma^2_{\text{within}}}{\sigma^2_{\text{total}}}\]

Here, total variance \(\sigma^2_{\text{total}} = \frac{1}{|T_e||C|}\sum_{t \in T_e}\sum_{c \in C}(A_{t,c} - \bar{\mu})^2\) captures the expert's global dynamic range across neurons, indicating its expressive representation capacity over assigned tokens; conversely, within-channel variance \(\sigma^2_{\text{within}} = \frac{1}{|T_e||C|}\sum_{t \in T_e}\sum_{c \in C}(A_{t,c} - \mu_c)^2\) measures the dispersion of activations within dedicated feature channels across tokens, reflecting encoding uncertainty. A lower \(\rho\) indicates that an expert maintains consistent, predictable channel responses while spanning a diverse global representation space, serving as a reliable proxy for superior expert quality. Controlled pair-intervention experiments confirm a strong negative Spearman rank correlation of \(-0.56\) between \(\rho\) and empirical expert performance.

2. Minimax Precision Balancing Algorithm: Preventing Aggregated Bottlenecks via Worst-Case Optimization

Prior quantization frameworks formulate bit assignment as minimizing the average error across experts (\(\min_b \sum_{i} L(b_i)\)), which often sacrifices vulnerable experts to reduce global average error. From the ensemble perspective, the MoE global error can be approximated as \(E_{\text{ens}} \approx \sum_{e \in \hat{E}} (\bar{g}_e(x))^2 R_e\). Assuming roughly uniform expected routing weights under standard load balancing (\(\bar{g}_e \approx \frac{1}{|\hat{E}|}\)), this convex formulation proves that ensemble error is minimized when individual errors are identical (\(R_{e_i} \approx R_{e_j}\)). Colla-Q therefore establishes a minimax objective:

\[b^* = \arg\min_b \max_{i=1,\dots,N} L(b_{e_i}), \quad \text{where } L(b_{e_i}) = \rho_{e_i} \times \|e_i(x) - \tilde{e}_i(x; b_{e_i})\|_2^2\]

This objective jointly accounts for unquantized expert capability via \(\rho_{e_i}\) and low-bit quantization sensitivity via the GPTQ reconstruction error \(\|e_i(x) - \tilde{e}_i(x; b_{e_i})\|_2^2\). Algorithmically, all experts in an MoE block are initialized to 1-bit; during each iteration, the algorithm identifies the candidate expert below the maximum bit limit exhibiting the largest combined loss \(k^* = \arg\max_{c \in C}(L_c)\), increments its precision by 1 bit, and repeats until the total budget \(B_{\text{total}}\) is satisfied. By recursively reinforcing the worst-performing expert, the algorithm prevents any single component from collapsing and destabilizing the aggregated MoE output.

Benchmark & Evaluation Protocol

Extensive experiments are conducted across three representative MoE architectures: Mixtral 8×7B (46.7B parameters), DeepSeek-MoE-16B-Base (16B parameters, featuring fine-grained and shared experts), and Phi3.5-MoE (42B parameters) targeting average bit budgets of 2.54, 2.05, and 1.57 bits. All methods utilize GPTQ for weight quantization on 128 sequences from C4 (length 2048), keeping attention and routing modules at 4 bits. Zero-shot performance is evaluated via EleutherAI's LM Evaluation Harness across eight reasoning and knowledge benchmarks (MMLU, PIQA, ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, WinoGrande, MathQA) alongside 5-shot MMLU. Cross-dataset calibration stability is tested using C4, MathQA, BoolQ, and Lambada_mul_fr, while practical expert-level execution throughput (TFLOPS) is measured on an NVIDIA RTX 3090 GPU using specialized low-bit GEMM kernels.

Key Experimental Results

Main Results

The table below summarizes zero-shot task accuracy across eight benchmarks on Mixtral 8×7B under different average bit budgets (directly extracted from Table 1 of the paper):

Bits Method MMLU PIQA ARC-e ARC-c BoolQ HellaS. Wino. MathQA Avg. (%)
16 bit FP16 Baseline 67.8 83.6 84.2 56.5 85.0 84.0 76.2 41.7 72.4
2 bit Uniform_2 30.2 60.7 47.1 25.7 62.3 41.9 52.8 22.4 42.9
2.54 bit OA-GPTQ 30.3 69.6 57.0 34.7 58.1 59.9 61.3 26.6 49.7
2.54 bit BSP 25.1 62.8 49.4 29.8 53.8 51.5 56.0 23.9 44.0
2.54 bit MxMoE 54.4 79.7 74.8 50.0 79.1 78.1 71.2 33.1 65.1
2.54 bit PMQ 56.4 80.5 77.1 51.3 82.5 79.0 74.0 39.2 67.5
2.54 bit Colla-Q (Ours) 59.4 80.5 79.8 54.0 84.7 78.5 74.9 36.5 68.5 (+1.0)
2.05 bit MxMoE 42.3 75.4 68.6 41.5 71.4 71.7 66.8 29.9 58.4
2.05 bit PMQ 46.8 79.2 73.1 48.4 80.6 75.0 71.3 31.8 63.3
2.05 bit Colla-Q (Ours) 52.9 77.8 74.9 46.4 83.3 72.4 72.0 32.3 64.0 (+0.7)
1.57 bit MxMoE 28.5 61.7 49.5 29.8 64.1 46.2 60.0 24.5 45.5
1.57 bit PMQ 32.3 72.4 62.5 37.9 73.6 63.2 66.4 26.8 54.5
1.57 bit Colla-Q (Ours) 36.6 71.0 63.1 36.2 75.8 59.8 66.2 26.9 54.5 (+0.0)

Ablation Study

The table below presents a cross-ablation on Mixtral 8×7B isolating the effects of the importance metric (PMQ routing statistics vs. Colla-Q activation entropy) and the allocation algorithm (PMQ ILP vs. Colla-Q minimax precision balancing) (derived from Tables 5, 6, and 19 of the paper):

Bits Metric Allocation Algorithm MMLU ARC-e BoolQ MathQA 8-Task Avg. (%)
2.54 bit PMQ (Routing) PMQ (ILP) 56.4 77.1 82.5 39.2 67.5
2.54 bit PMQ (Routing) Colla-Q (Minimax) 58.6 78.0 82.2 35.7 67.7
2.54 bit Activation Entropy (Ours) PMQ (ILP) 57.6 78.4 83.5 36.4 67.3
2.54 bit Activation Entropy (Ours) Colla-Q (Minimax) 59.4 79.8 84.7 36.5 68.5
2.05 bit PMQ (Routing) PMQ (ILP) 46.8 73.1 80.6 31.8 63.3
2.05 bit PMQ (Routing) Colla-Q (Minimax) 45.5 74.9 82.8 32.0 63.3
2.05 bit Activation Entropy (Ours) PMQ (ILP) 49.3 72.7 81.7 33.9 63.1
2.05 bit Activation Entropy (Ours) Colla-Q (Minimax) 52.9 74.9 83.3 32.3 64.0
1.57 bit PMQ (Routing) PMQ (ILP) 32.3 62.5 73.6 26.8 54.5
1.57 bit PMQ (Routing) Colla-Q (Minimax) 33.1 63.9 71.4 26.5 53.6
1.57 bit Activation Entropy (Ours) PMQ (ILP) 34.2 59.1 72.3 27.2 53.2
1.57 bit Activation Entropy (Ours) Colla-Q (Minimax) 36.6 63.1 75.8 26.9 54.5

Key Findings

  • Balanced Precision Defends Multi-Step Reasoning: At 2.54 bits, Colla-Q achieves 68.5% average accuracy on Mixtral 8×7B (just 3.9% below FP16's 72.4%), significantly outperforming PMQ (67.5%) and MxMoE (65.1%). On MMLU, Colla-Q reaches 59.4% (+3.0% over PMQ) at 2.54 bits and 52.9% (+6.1% over PMQ) at 2.05 bits, confirming that suppressing weak-expert failure modes is critical for complex contextual reasoning.
  • Robust Invariance to Calibration Shifts: In cross-dataset calibration tests across C4, MathQA, BoolQ, and Lambada_mul_fr, PMQ's routing factor vectors exhibit severe domain sensitivity, dropping to an average cosine similarity of 72.0% on Mixtral and 46.3% on Phi3.5. In contrast, Colla-Q's activation entropy vectors maintain near-perfect consistency (98.2%–98.9% on Mixtral, 91.6%–94.8% on Phi3.5), preserving downstream accuracy even when calibration and evaluation domains diverge drastically.
  • Expert-Level Granularity Outperforms Linear-Layer Splitting: Applying minimax precision balancing at the expert level yields higher average scores than fine-grained linear-layer-level allocation (e.g., 64.0% vs. 63.1% at 2.05 bits). Furthermore, utilizing low-bit INT1/INT2/INT3 GEMM kernels provides a \(1.39\times\)–\(1.48\times\) expert computation speedup over FP16 on an NVIDIA RTX 3090.

Highlights & Insights

  • Ensemble Perspective Inverting Quantization Intuition: Challenges the pervasive assumption that frequently routed experts should receive preferential precision by demonstrating mathematically and empirically that unrouted or weakly performing experts act as the bottleneck in dynamic MoE aggregation.
  • Differential Entropy as a Label-Free Quality Metric: Elegantly bridges information theory and neural representation by decomposing continuous activation distributions into channel uncertainty (\(\sigma^2_{\text{within}}\)) and representation capacity (\(\sigma^2_{\text{total}}\)), bypassing the lack of token labels for intermediate experts.
  • Broadly Applicable Resource Balancing Heuristic: The greedy minimax precision balancing algorithm provides a general paradigm for discrete budget allocation across modular components, readily transferable to multi-agent task routing, multi-adapter LoRA balancing, and sparse memory caching.

Limitations & Future Work

  • Restricted to Weight-Only Quantization: The current framework is evaluated on weight-only quantization. Extending activation entropy balancing to joint weight-activation quantization (e.g., W4A4, W8A8) remains an open challenge.
  • Evaluation Limited to Mid-Sized MoE Models: Empirical tests cover models up to 46.7B parameters (Mixtral 8×7B, DeepSeek-16B, Phi3.5-MoE). Evaluating behavior on models exceeding 100B parameters (e.g., DeepSeek-V2/V3) is constrained by compute budgets.
  • Incomplete End-to-End System Integration: Computational efficiency measurements evaluate isolated expert GEMM execution without factoring in end-to-end runtime bottlenecks such as dynamic token dispatching, cache movement, and small-GEMM scheduling.
  • vs PMQ (Huang et al., 2025): PMQ determines expert precision via calibration routing frequency and weights; Colla-Q establishes that routing statistics overfit calibration domains, whereas activation entropy captures intrinsic capacity and protects weak learners to maintain higher aggregate accuracy.
  • vs MxMoE (Duanmu et al., 2025): MxMoE splits precision at the sub-expert linear layer level using joint sensitivity and latency modeling; Colla-Q demonstrates that expert-level precision balancing avoids kernel fragmentation while outperforming MxMoE by 3%–9% on average benchmarks.
  • vs AWQ / OWQ (Lin et al., 2024; Lee et al., 2024): Dense activation-aware quantization identifies salient channels to scale weights; Colla-Q abstracts activation statistics to evaluate module-level capacity across heterogeneous sub-networks in sparse architectures.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering ensemble-based framing of MoE quantization with theoretical proof of error equalization and differential activation entropy.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three representative MoE architectures, eight core benchmarks, four cross-domain calibration sets, and hardware GEMM profiling.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical justification, compelling motivation, well-crafted figures, and crisp empirical presentation.
  • Value: ⭐⭐⭐⭐⭐ Essential milestone for deploying ultra-low-bit MoE models reliably under domain shifts and extreme resource constraints.