Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding¶

Conference: ECCV 2026
arXiv: 2604.07753
Code: To be confirmed
Area: Image Generation
Keywords: Mixture of Experts, Multimodal Unification, Catastrophic Forgetting, Generation-Understanding Synergy, Routing Collapse

TL;DR¶

Symbiotic-MoE resolves the gradient conflict between generation and understanding tasks within a native MoE Transformer architecture through modality-aware expert decoupling and a progressive training strategy. With zero parameter overhead, it achieves synergistic improvement in both capabilities—yielding generation quality superior to MoT/Bagel, while understanding capabilities (MMLU +20.1%, OCRBench +13.8%) even surpass the understanding-only baseline.

Background & Motivation¶

Large Multimodal Models (LMMs) are evolving toward "any-to-any" fully multimodal foundation models: capable of both understanding and generating images. However, when a pre-trained visual understanding model is endowed with image generation capabilities, it almost invariably suffers from catastrophic forgetting, leading to a sharp decline in performance on understanding tasks. The root of this issue lies in the fundamental conflict between the optimization objectives of understanding and generation tasks: understanding is a "many-to-one" convergent mapping (requiring consistent and precise representation of variants within the same semantic space), whereas generation is a "one-to-many" divergent mapping (where a single text prompt corresponds to infinite viable images). When trained together, the high-variance, large-magnitude gradients from the generation objective drown out the converged optimization landscape of the understanding task, irreversibly pulling expert weights toward the generation distribution.

To address this challenge, existing methods predominantly adopt structural isolation, with the most representative being Mixture-of-Transformers (MoT). MoT physically splits the Transformer's FFN experts into separate understanding and generation parameters, avoiding mutual interference. While this "split-brain" approach suppresses forgetting, it comes at a high cost: the semantic connectivity between the two sets of experts is completely severed. Consequently, the fine-grained visual features learned by the generation task cannot benefit understanding representations, and understanding performance is limited to isolated fine-tuning rather than truly benefiting from generation. Meanwhile, structural isolation often introduces parameter overhead or inference latency and leads to capacity fragmentation.

This raises the question: is isolation truly the only way? Symbiotic-MoE answers in the negative. The core insight of this work is that task conflict is not an inherent flaw of unified architectures, but rather a failure of routing dynamics—standard MoE training suffers from routing collapse, where generation tokens monopolize expert resources and starve understanding tasks. Core Idea: Introduce modality-aware expert decoupling (logically grouping experts into understanding and generation groups) and a shared expert semantic bridge (forcing all tokens to pass through shared experts to achieve cross-modal alignment) within a native sparse MoE Transformer. Coupled with a progressive training strategy (differential learning rates + warmup gradient shielding), this transforms the generation signal from a disruptor into a positive regularizer for understanding capabilities with zero parameter overhead.

Method¶

Overall Architecture¶

Symbiotic-MoE leverages Hunyuan-VL-30B-A3B (30B total parameters, with 3B active) as the backbone, re-architecting the FFN portion of each MoE layer in the Transformer. The input side simultaneously receives three types of tokens: text (Text), Vision Encoder outputs (ViT), and VAE continuous visual latent variables; the outputs serve understanding tasks (language modeling and multimodal QA) and generation tasks (flow matching in the VAE latent space). The entire redesign consists of two core components: (1) the Symbiotic-MoE architecture, which performs logical grouping and shared bridging of experts within MoE layers without adding parameters; and (2) a progressive training strategy that coordinates the optimization pace of both tasks using differential learning rates and warmup gradient shielding.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text / ViT / VAE tokens"] --> B["Modality-Aware Expert Decoupling<br/>Understanding Group 96 + Generation Group 32"]
    A --> C["Shared Expert Semantic Bridge<br/>All tokens must pass through"]
    B --> D["Knowledge Inheritance Initialization<br/>Routing Weight Slicing"]
    C --> D
    D --> E["Differential Learning Rates<br/>Generation 1e-4 / Understanding 1e-6"]
    E --> F["Warmup Gradient Shielding<br/>stop-grad during warmup"]
    F --> G["Understanding Outputs + Generated Images"]

Key Designs¶

1. Modality-Aware Expert Decoupling: From Routing Collapse to Expert Specialization

During standard MoE training, the gradients generated by generation tokens are much larger in magnitude than those of understanding tokens, systematically hijacking expert resources. This phenomenon is known as routing collapse. It can be visualized as the routing probability map of text tokens degrading from distinct, sparse spots to a fuzzy, uniform distribution after 30k training steps. Symbiotic-MoE resolves this not by physically separating parameters, but through data-driven logical grouping. First, zero-shot inference is run on a pre-trained VLM for MMLU (text tokens) and OCRBench (ViT tokens), tracking the cumulative activation frequencies of 128 experts across 47 MoE layers. It is observed that Text and ViT tokens heavily share a set of "core experts" (e.g., in Layer 16, Experts 9, 18, and 92 are simultaneously top choices for both token types), while another set of experts remains underutilized. Based on this, the top 96 experts with the highest activation frequencies in each layer are designated as the understanding group (shared by Text and ViT to maintain stability), while the remaining 32 less-active experts are reassigned as the generation group (dedicated to VAE tokens to ensure plasticity). Forcing the separation of Text and ViT leads to catastrophic collapse (MMLU drops from 0.69 to 0.24) because the two are deeply coupled in the pre-trained VLM. Thus, a Bimodal Split is the only viable option to preserve the pre-trained structure.

2. Shared Expert Semantic Bridge: Letting Generation Nourish Understanding

Decoupling alone is insufficient — if the two groups of experts are completely isolated, the fine-grained visual semantics learned during generation will be wasted, degenerating into a "split-brain" scenario. The key difference in Symbiotic-MoE is that it retains and activates the pre-existing shared experts in each layer. These experts receive inputs from all tokens (Text, ViT, VAE), forcing cross-modal semantic alignment. Ablation studies reveal the indispensability of shared experts: removing them from the original VLM causes MMLU to immediately drop from 0.69 to 0.46. What exactly does the generation task transmit to the understanding side through the shared experts? This is verified by an experimental probe: "mask all modality-specific routing experts and rely solely on shared experts for zero-shot inference." While the performance of the shared experts in the understanding-only control group degrades over training, the performance of the shared experts in Symbiotic-MoE improves significantly. The generation objective acts as a dense semantic compressor, forcing shared parameters to encode pixel-level spatial relationships, which conversely constraints the representation space, prevents overfitting to textual patterns, and sharpens visual perception.

3. Knowledge Inheritance Initialization and Routing Weight Slicing: Zero Cold-Start Transition

When transitioning from a unified MoE to a decoupled architecture, performance collapses immediately if new modules are initialized randomly. In Symbiotic-MoE, the expert weights for the understanding and generation groups are copied directly from the corresponding indices of the original VLM (without modifications), preserving the pre-trained semantic structure. Even more critical is the initialization of the two new routers: the original router weight matrix \(\mathbf{W}_r \in \mathbb{R}^{d \times N}\) is directly column-sliced according to the expert index subset \(\mathcal{I}_g\) to obtain the sub-routers for each group, \(\mathbf{W}_r^g = \mathbf{W}_r[:, \mathcal{I}_g]\). This ensures that tokens that originally scored expert \(i\) highly will still score it highly in the new sub-router. The routing prior is fully preserved, achieving near-original understanding performance with zero iterations (as shown in Table 2: under the 96/32 Bimodal Split, MMLU reaches 0.60 and OCRBench reaches 807 at iteration 0).

4. Progressive Training Strategy: Differential Learning Rates + Warmup Gradient Shielding

Even with proper structural decoupling, simultaneously optimizing a converged VLM and a generation module starting from scratch is highly unstable. Their optimal learning rates are vastly different: updating the understanding group at \(10^{-4}\) triggers immediate collapse (even without connecting the generation task, training LM+MMU alone at \(10^{-4}\) causes immediate collapse), whereas the randomly initialized experts in the generation group require \(10^{-4}\) to quickly escape their initial state. Therefore, a large learning rate of \(10^{-4}\) is applied to generation experts and routers, while a conservative learning rate of \(10^{-6}\) is applied to understanding experts, routers, and shared experts. However, during the warmup phase, the highly unstable generation module produces high-variance gradients that contaminate the understanding representation through the shared experts. The solution is to perform a stop-gradient operation on the shared experts during the warmup phase: VAE tokens can propagate forward through the shared experts (utilizing pre-trained features), but gradients do not update the shared expert weights during backpropagation — shared experts only accept stable gradients from Text and ViT. Once the warmup phase ends, the shielding is removed to restore full bidirectional gradient flow, by which point the optimization trajectory of the generation module has stabilized. Additionally, a gradient scaling factor of 0.1 is applied to generation tokens entering the shared experts to prevent high-learning-rate generation signals from monopolizing them.

Loss & Training¶

The total loss consists of three components:

\[\mathcal{L}_{total} = \lambda_{disc} \cdot \mathcal{L}_{discrete} + \lambda_{aux} \cdot \mathcal{L}_{aux} + \lambda_{img} \cdot \mathcal{L}_{img}\]

where \(\lambda_{disc} = 1.0\), \(\lambda_{aux} = 0.01\), and \(\lambda_{img} = 1.0\). \(\mathcal{L}_{discrete}\) is the standard next-token prediction cross-entropy loss covering pure language modeling, multimodal understanding, and text conditions for generation; \(\mathcal{L}_{aux}\) is the expert load balancing loss calculated independently for the understanding and generation groups (rather than forcing global balance, which would dilute modal specialization); \(\mathcal{L}_{img}\) is the flow matching loss on VAE latents, responsible for aligning visual features with the generative manifold.

Key Experimental Results¶

Main Results¶

All experiments are compared under the same data ratio (T2I:T2I-Long:LM:MMU = 3:3:2:2) and the same number of training steps, using Hunyuan-VL-30B-A3B as the base model.

Method	T2I-Comp↑	FID↓	CLIP↑	HPSv2↑	MMLU↑	OCRBench↑
Only_LM_MMU (No Gen)	—	—	—	—	0.405	662
Standard MoE	0.36	26.27	0.27	0.19	0.308	571
MoT	0.43	19.87	0.28	0.21	0.392	583
Bagel	0.45	18.42	0.29	0.21	0.396	590
Symbiotic-MoE (Ours)	0.49	13.65	0.31	0.23	0.507	768

Symbiotic-MoE not only outperforms all baselines across all generation metrics but also boosts MMLU from 0.405 (Only_LM_MMU baseline) to 0.507 (+20.1%) and OCRBench from 662 to 768 (+13.8%), providing the first empirical proof that image generation training can positively enhance understanding capabilities.

Ablation Study¶

Splitting Strategy	Und/Gen Groups	Shared Experts	MMLU (iter 0)	OCRBench (iter 0)
Original VLM	128 (Entangled)	✓	0.697	845
Original VLM, w/o Shared	128	✗	0.460	498
Tripartite	32/32/64	✓	0.244	109
Bimodal 32/96	32/96	✓	0.285	532
Bimodal 64/64	64/64	✓	0.453	742
Bimodal 86/42	86/42	✓	0.561	786
Bimodal 96/32 (Ours)	96/32	✓	0.601	807

Key Findings¶

Shared experts are the most critical component: removing them causes MMLU to drop from 0.697 to 0.460, which is the single most influential factor.
Text and ViT cannot be separated: the Tripartite split (separate groups for Text, ViT, and VAE) causes MMLU to collapse to 0.244, because the two share core expert paths in the pre-trained VLM.
The 96/32 understanding/generation expert ratio is the optimal balance: retaining sufficient pre-trained capacity (96 understanding experts) while providing the generation task with adequate plasticity (32 generation experts).
Visual verification of routing collapse: under standard MoE joint training for 30k steps of T2I, the routing probability map of text tokens degrades from distinct and sparse to uniform and fuzzy, whereas Symbiotic-MoE consistently maintains an expert utilization rate of ~0.95.
In the progressive training strategy, removing the warmup gradient shielding (Ours_wo_wgs) leads to a sharp decline in MMLU and OCRBench within the first 500 steps, confirming that early stability during the warmup phase is crucial for final performance.

Highlights & Insights¶

Routing collapse is the essence of the problem: The paper precisely diagnoses the empirical phenomenon of "generation destroying understanding" as a failure of routing dynamics, presenting clear mechanistic evidence through visualization and capacity rate curves. This diagnosis provides a solid foundation for the subsequent remedy (decoupled grouping instead of physical isolation), offering deeper insights than simple "isolation layers."
Shared experts as a bidirectional semantic conduit: Through probe experiments relying solely on shared experts for inference, the paper directly proves that generation signals strengthen understanding representations via dense regularization. This is the first mechanistic evidence of synergy between generation and understanding, overturning the intuition that the two are inherently zero-sum.
Zero cold-start transition via routing weight slicing: Initializing sub-routers via column-slicing of the original routing matrix preserves the pre-trained routing prior with zero parameter overhead. This method can be directly applied to other decoupling scenarios for MoE architectures.

Limitations & Future Work¶

Experiments were only conducted during the PT (Pre-training) stage at \(256 \times 256\) resolution. The synergistic effects during CT (Co-training) and SFT (Supervised Fine-Tuning) stages remain unverified, and behavior at high resolutions is uncertain.
All experiments are based on a single backbone (Hunyuan-VL-30B-A3B). The transferability of these conclusions to other MoE architectures (e.g., DeepSeek-MoE, Qwen3-MoE) requires further validation.
Generation quality evaluation focuses on early training stages, lacking a systematic assessment of the full aesthetic alignment process. Whether the 96/32 expert grouping ratio remains optimal for MoE models of different sizes or total expert counts remains to be explored.
Sensitivity analysis regarding the duration of warmup gradient shielding (warmup steps) is limited, requiring additional hyperparameter tuning in actual engineering deployments.

vs MoT/Bagel: MoT-style methods achieve modal isolation by physically partitioning expert parameters, successfully avoiding forgetting but severing the cross-modal information flow. The logical grouping + shared bridging in this work opens semantic channels on top of the same structural foundation. While MoT can only "preserve" understanding capability, Symbiotic-MoE "enhances" it, which is a fundamental difference.
vs Adapter/LoRA methods (e.g., Janus): Methods that freeze the backbone and use bypass adapters avoid forgetting through parameter isolation, but bypass architectures naturally restrict deep cross-modal fusion. The zero-overhead logical decoupling of Symbiotic-MoE provides a tighter and more synergistic alternative path.
vs Pure Discretization Unification (Chameleon, Emu3): Modeling images as tokenized discrete variables mixed with text achieves formal unification, but information loss from discrete quantization and gradient conflicts in unified architectures persist. Symbiotic-MoE preserves continuous VAE latents and specifically addresses routing conflicts, offering distinct advantages in terms of precision and stability.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Precisely diagnoses routing collapse as the root cause of generation-understanding conflict and provides a parameter-free mechanistic solution with a novel approach.
Experimental Thoroughness: ⭐⭐⭐⭐ Solid ablation design and convincing probe experiments, though validated on only a single backbone and during the PT stage.
Writing Quality: ⭐⭐⭐⭐⭐ Extremely clear logical flow; the closed-loop of diagnosis \(\rightarrow\) design \(\rightarrow\) validation is rigorous, with figures and text highly complementary.
Value: ⭐⭐⭐⭐⭐ Empirically proves for the first time that generation training can positively enhance understanding, offering a concrete, viable new paradigm for fully multimodal foundation model design.