MANGO: Unleashing Image Generation Capability of Unified Multimodal Models

Conference: ECCV 2026
Paper: ECCV Full Text
Code: To be confirmed
Area: Image Generation
Keywords: Unified Multimodal Models, Chain of Thought, Image Generation, Compositional Generation, Rectified Flow

TL;DR

To tackle cross-functional interference and residual defects in unified multimodal models during complex compositional generation, MANGO presents Functionality-Oriented Transformers (FoT) to raise the performance floor by disentangling visual understanding and generation branches, alongside Paint-CoT and a multi-task joint training scheme to elevate the ceiling through planning, acting, reflection, and inpainting-based correction.

Background & Motivation

Unified multimodal models (UMMs) such as GPT-5, Gemini 3, and various open-source architectures seek to integrate multimodal comprehension and synthesis within a singular neural backbone. Compared to cumbersome pipelines assembled from multiple specialized expert models, unified multimodal systems offer a more elegant and scalable paradigm for cross-modal synergy, training, and deployment. While early attempts relied on next-token prediction over discrete image tokens, discrete vector quantization inherently constrains continuous visual signals such as high-resolution images. Recent hybrid generative frameworks (such as Transfusion, Show-o, and JanusFlow) combine discrete autoregressive text modeling with continuous diffusion or Rectified Flow processes. However, existing models either rely on a single shared Transformer branch to process all modalities—inducing acute cross-modal competition—or adopt modality-oriented designs (such as MoT and LlamaFusion) that isolate text and vision into separate branches while still forcing visual understanding and visual generation to share the same visual branch.

Sharing a single visual branch introduces deep, unresolved functional conflicts. Visual comprehension tasks are optimized via autoregressive next-token prediction to align visual tokens with high-level textual semantics. In contrast, image generation is trained by minimizing diffusion denoising objectives (matching Rectified Flow velocity fields), which forces visual representations to capture fine-grained pixel-level noise trajectories and spatial geometry. These opposing optimization objectives pull the shared latent representations in conflicting directions, substantially degrading the model's baseline image generation floor. Consequently, under complex compositional instructions common in practical applications—such as multi-object co-occurrence, precise attribute binding, strict spatial positioning, and intricate structural layouts—existing unified models frequently suffer from concept confusion, attribute leakage, spatial inversion, and object part defects. Prior efforts that introduce chain-of-thought (CoT) reasoning to image synthesis either retreat to cascaded systems coupling an MLLM with an external diffusion model (breaking single-model unity) or restrict CoT to upfront layout planning via reinforcement learning, leaving residual rendering defects unaddressed.

The core motivation of this paper is to eliminate the intrinsic functional tension between visual understanding and generation at the architectural root to secure a solid generation floor, while empowering the unified model to self-inspect and repair generation defects to push the performance ceiling. Core idea: introduce Functionality-Oriented Transformers (FoT) with three dedicated branches (linguistic, semantic visual, and generative visual) to eliminate cross-functional interference, and establish Paint-CoT alongside a multi-task joint training paradigm that enables planning, acting, reflection, and targeted inpainting correction without requiring expensive end-to-end multi-step aligned annotations.

Method

Overall Architecture

MANGO establishes an end-to-end unified multimodal framework capable of complex compositional image synthesis. At the input level, textual prompts are tokenized by the Qwen2 tokenizer (augmented with 6 functional special tokens) to yield text embeddings \(x_{\text{text}}\). Input images are mapped into continuous latent representations via SD3's Variational Autoencoder (VAE), flattened into \(2 \times 2\) patch grids, and linearly projected into visual tokens \(x_{\text{image}}\) where each token corresponds to a \(16 \times 16\) pixel region. Within the model backbone, MANGO deploys Functionality-Oriented Transformers (FoT) featuring three dedicated branches: a Linguistic Branch, a Semantic Visual Branch, and a Generative Visual Branch. Each branch maintains independent feed-forward networks (FFNs) and projection parameters while interacting through a global multimodal attention module. During inference, MANGO executes Paint-CoT through an explicit four-stage chain: planning dense descriptions and bounding boxes, acting via continuous generation, reflecting on visual artifacts, and executing inpainting correction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Input with Complex Prompt & Directives"] --> B["Functionality-Oriented Transformers (FoT)<br/>Linguistic + Semantic Visual + Generative Visual Branches"]
    B --> C["Paint Chain of Thought (Paint-CoT)<br/>Planning & Acting → Artifact Reflection → Inpainting Correction"]
    C --> D["Multi-Task Joint Training Paradigm<br/>Decoupled Subtasks Bypassing Multi-Step Supervised Data"]
    D --> E["High-Fidelity Defect-Free Generated Image"]

Key Designs

1. Functionality-Oriented Transformers (FoT): Disentangling Understanding and Generative Visual Representations

Mainstream modality-oriented architectures separate text from images, yet force a single visual branch to optimize both next-token prediction (for comprehension) and diffusion denoising (for image synthesis). This creates persistent representational and gradient interference. FoT resolves this issue by organizing computation branches according to functionality rather than modality alone: a Linguistic Branch (\(T\)) handles text understanding and autoregressive text generation; a Semantic Visual Branch (\(C\)) specializes in image feature extraction, visual question answering, and reflection; and a Generative Visual Branch (\(N\)) is dedicated exclusively to continuous diffusion denoising.

Formally, given the concatenated sequence \(x = x_T \circ x_C \circ x_N\) representing text tokens, clean image tokens, and noisy image tokens, each branch undergoes independent layer normalization and routing before projection into query, key, and value matrices: $\(\hat{x}_i = W_i(\text{Router}(\text{LN}(x))), \quad \hat{x}_{q,k,v}^i = W_{Q,K,V}^i(\hat{x}_i), \quad i \in \{T, C, N\}\)$ The projected tokens from all three branches are concatenated across the sequence dimension and interact via shared multimodal attention: $\(\hat{x} = \text{Attn}(\hat{x}_q, \hat{x}_k, \hat{x}_v) + x\)$ Following attention, each branch executes its own nonlinear mapping: \(\hat{x}_i = \text{FFN}_i(\text{Router}(\text{LN}(x)))\). Within this attention framework, language tokens apply causal masking, visual tokens (clean and noisy) maintain bidirectional attention, and overall cross-stage dependencies follow a strict causal sequence to prevent future information leakage. This design eliminates gradient conflicts between semantic abstraction and noise trajectory modeling while preserving inter-modal alignment.

2. Paint Chain of Thought (Paint-CoT): A Human-like Artistic Four-Stage Synthesis Loop

To overcome structural incompleteness, concept blending, and attribute misallocation in single-step generation, MANGO replicates the deliberate creative workflow of human artists via four sequential reasoning stages. In the Planning step, given an input prompt, the model autoregressively generates a detailed dense caption elaborating on background, lighting, and textures, alongside bounding-box coordinates (\([x_{\min}, y_{\min}, x_{\max}, y_{\max}]\)) establishing layout planning. In the subsequent Acting step, the model generates an initial image guided jointly by the detailed prompt and bounding-box spatial constraints, ensuring proper object grounding and reducing missing entities.

Recognizing that complex single-step synthesis inevitably leaves localized rendering flaws, MANGO introduces subsequent Reflection and Correction steps. In the reflection phase, the Semantic Visual Branch takes the initial generated image and user prompt to inspect four primary defect categories: structural incompleteness, object entanglement, object redundancy, and object distortion. The model outputs a continuous artifact heatmap scoring regional defect severity. In the correction phase, the artifact heatmap is converted into a localized repair mask. Guided by the original planning context and unmasked image patches, the Generative Visual Branch performs targeted inpainting to replace defective regions seamlessly, substantially lifting the quality and compositional ceiling.

3. Multi-Task Joint Training Paradigm: Decoupled Subtasks Bypassing Multi-Step Supervised Data

Training Paint-CoT end-to-end across full multi-step trajectories would require collecting aligned tuples of {prompt, dense caption, layout boxes, flawed initial image, artifact map, correct final image}. Collecting such datasets at scale is prohibitively expensive. Moreover, forcing an end-to-end model to generate a realistic "defective intermediate image" contradicts the core objective of generative optimization.

MANGO circumvents this limitation through a multi-task joint training paradigm that decomposes the pipeline into three decoupled subtasks sharing the FoT backbone: 1. Planning and Acting Task: Prompt-image pairs are enriched using advanced MLLMs (e.g., Qwen2.5-VL) for dense captioning and open-vocabulary detectors (e.g., Grounding DINO) for bounding boxes, yielding {prompt, dense caption, layout boxes, image} quadruples to supervise planning and synthesis. 2. Reflection Task: Sourced from public artifact datasets and focused defect annotations, the Semantic Visual Branch is trained to predict artifact heatmaps given an image and text prompt. 3. Correction Task: Formulated as standard inpainting, the Generative Visual Branch learns to reconstruct masked image regions conditioned on the unmasked context and planning rationale. This decoupled formulation leverages accessible single-task datasets, eliminates the paradox of training models to generate errors, and equips the model with robust self-reflection and self-correction capabilities at minimal annotation cost.

Loss & Training

The model is trained by jointly optimizing autoregressive cross-entropy and continuous flow velocity matching: - The Linguistic Branch and Semantic Visual Branch optimize next-token prediction negative log-likelihood: $\(\mathcal{L}_{\text{LLM}} = -\mathbb{E}_{z \sim \mathcal{D}} \left[ \sum_{i=1}^N \log P_\omega(z_i \mid z_{<i}, \varepsilon(x)) \right]\)$ - The Generative Visual Branch minimizes the Rectified Flow velocity matching loss: $\(\mathcal{L}_{\text{RF}} = \mathbb{E}_{t \in \mathcal{U}(0,1), x, \omega, c} \left[ \| (x - \omega) - v_\omega(x_t, t, c) \|_2^2 \right]\)$ where the deterministic trajectory between noise and data is parameterized as \(x_t = t x + (1-t)\omega\) with \(t \in [0, 1]\) and \(\omega \sim \mathcal{N}(0, I)\).

Training follows a two-stage protocol: initial pretraining on large-scale multimodal corpora to establish baseline T2I and I2T capabilities, followed by multi-task Paint-CoT fine-tuning. Optimization uses AdamW (\(\beta_1=0.9, \beta_2=0.999\), weight decay 0.02) with a constant learning rate of \(5 \times 10^{-5}\) after 10,000 warm-up steps, accelerated via DeepSpeed ZeRO-2.

Key Experimental Results

Main Results

MANGO is rigorously benchmarked on complex compositional image benchmarks, including GenEval and T2I-CompBench.

Table 1: Evaluation of text-to-image generation on GenEval

Category	Model	Params	Overall ↑	Single Obj.	Two Obj.	Counting	Colors	Position	Attr. Binding
T2I Diffusion Models	SD v1.5	1.0B	0.43	0.97	0.38	0.35	0.76	0.04	0.06
	SD v2.1	1.3B	0.50	0.98	0.51	0.44	0.85	0.07	0.17
	SD-XL	3.4B	0.55	0.98	0.74	0.39	0.85	0.15	0.23
	SD 3	12.7B	0.68	0.98	0.84	0.66	0.74	0.40	0.43
	DALL-E 2	4.5B	0.52	0.94	0.66	0.49	0.77	0.10	0.19
	DALL-E 3	–	0.67	0.96	0.87	0.47	0.83	0.43	0.45
	IF-XL	10.1B	0.61	0.97	0.74	0.66	0.81	0.13	0.35
Unified Multimodal Models	Chameleon	34B	0.39	–	–	–	–	–	–
	Transfusion	7.3B	0.63	–	–	–	–	–	–
	LWM	7B	0.47	0.93	0.41	0.46	0.79	0.09	0.15
	Janus-Pro	7B	0.80	0.99	0.89	0.59	0.90	0.79	0.66
	BAGEL	7B	0.82	0.99	0.94	0.81	0.88	0.64	0.63
	GoT (MLLM+Diff)	2.8+3B	0.64	0.99	0.69	0.67	0.85	0.34	0.27
	GoT-R1	7B	0.75	0.99	0.94	0.50	0.90	0.46	0.68
	Show-o	1.3B	0.53	0.95	0.52	0.49	0.82	0.11	0.28
	Janus	1.3B	0.61	0.97	0.68	0.30	0.84	0.46	0.42
	JanusFlow	1.3B	0.63	0.97	0.59	0.45	0.83	0.53	0.42
Ours	MANGO	1.3B	0.77	0.99	0.86	0.71	0.82	0.60	0.64
	MANGO-4B	4.2B	0.82	0.99	0.89	0.76	0.87	0.68	0.70

Table 2: Evaluation of text-to-image generation on T2I-CompBench

Category	Model	Params	Color ↑	Shape ↑	Texture ↑	Spatial ↑	Non-Spatial ↑	Complex ↑
T2I Diffusion Models	SD v1.5	1.0B	37.65	35.76	41.56	12.46	30.79	30.80
	SD-XL	3.4B	63.69	54.08	56.37	20.32	31.10	40.91
	SD 3	12.7B	81.32	58.85	73.34	32.00	31.40	37.71
	PixArt-\(\alpha\)	0.6B	68.86	55.82	70.44	20.82	31.79	41.17
	DALL-E 2	4.5B	57.50	54.64	63.74	12.83	30.43	36.96
Unified Multimodal Models	Emu 3	7B	75.44	57.06	71.64	–	–	–
	Janus-Pro	7B	63.59	35.28	49.36	20.61	30.85	35.59
	T2I-R1	7B	81.30	58.52	72.43	33.78	30.90	39.93
	GoT-R1	7B	81.39	55.49	73.39	33.06	31.69	39.44
	Show-o	1.3B	56.00	41.00	46.00	20.00	30.00	29.00
Ours	MANGO	1.3B	82.37	59.81	74.21	35.71	34.19	42.78
	MANGO-4B	4.2B	86.63	61.92	78.75	40.45	37.41	45.29

Ablation Study

Table 3: Ablation results of Paint-CoT components on GenEval and T2I-CompBench

Config	GenEval Overall ↑	Position	Attr. Binding	T2I-CompBench Overall ↑	Spatial	Complex	Note
T2I Gen. Twice (Best of 2)	0.67	0.40	0.47	42.26	13.42	32.72	Equal compute baseline
Paint-CoT Planning & Acting Only	0.73	0.55	0.58	50.62	28.36	39.55	Incorporates caption & layout
Paint-CoT Full Process	0.77	0.60	0.64	54.85	35.71	42.78	Complete 4-stage pipeline

Table 4: Ablation of FoT visual branch design on MS-COCO

Architecture Variant	CIDEr ↑	FID ↓	Note
Dense (Single shared Transformer branch)	116.2	11.30	Dual modality and functionality conflicts
Modality-Oriented (Separate text & vision branches)	121.1	9.56	Isolates modalities, but mixes visual tasks
MANGO (FoT Functionality-Oriented)	126.5	7.24	Disentangles linguistic, semantic, and generative branches

Key Findings

• FoT Lifts the Baseline Generation Floor: On MS-COCO pretraining ablations, isolating visual understanding and visual generation into separate functional branches reduces the FID from 9.56 (modality-oriented) to 7.24 and boosts CIDEr from 121.1 to 126.5. This verifies that unburdening generative visual features from next-token language alignment allows the diffusion branch to fit noise velocity fields accurately.
• Synergy of Planning and Reflection-Correction: Adding dense captioning and layout box planning elevates GenEval overall performance from 0.67 to 0.73, with positioning jumping from 0.40 to 0.55. Complementing planning with artifact reflection and inpainting correction further boosts overall performance to 0.77, attribute binding to 0.64, and spatial accuracy on T2I-CompBench by +7.35 points, demonstrating that reflection effectively eliminates residual rendering defects.
• Superior Parameter Efficiency and Scalability: At only 1.3B parameters, MANGO (0.77) outperforms the 12.7B SD 3 (0.68), 7B GoT-R1 (0.75), and peer-scale Show-o (0.53) on GenEval. When scaled to 4.2B parameters, MANGO-4B reaches an overall score of 0.82, matching the 7B BAGEL model and setting new state-of-the-art results across compositional categories.
• Convergence of Reflection-Correction Cycles: Exploring multiple reflection-correction iterations indicates that a single correction pass resolves the vast majority of visible defects. A second cycle yields negligible quantitative gains while incurring substantial latency overhead, identifying single-cycle reflection-correction as the optimal trade-off.

Highlights & Insights

• Functionality Disentanglement Transcends Modality Boundaries: While prior research focused on separating input modalities (text vs. visual), MANGO recognizes that conflicting task objectives (discrete semantic alignment vs. continuous noise trajectory regression) within the same modality represent the primary bottleneck. Dedicating specialized visual experts resolves this tension cleanly.
• Autonomous Self-Inspection and Self-Healing: Moving beyond feedforward generation or pure text prompt re-writing, Paint-CoT equips the unified model with genuine artist-like metacognition, actively localizing structural or semantic flaws and performing targeted inpainting repairs.
• Pragmatic Multi-Task Training: Decomposing end-to-end CoT into decoupled subtasks cleverly sidesteps the logical contradiction of training models to generate flawed intermediate outputs, while avoiding expensive multi-step dataset construction.

Limitations & Future Work

• Increased Inference Latency: Executing the full Paint-CoT sequence (planning autoregression, initial diffusion sampling, artifact reflection inference, and inpainting denoising) requires noticeably higher test-time compute than standard one-pass diffusion models.
• Planning Quality Bound by Language Capacity: In complex counterfactual or heavy physical reasoning scenarios, lightweight language backbones (0.5B/1.5B) can produce suboptimal layout plans, introducing downstream geometric bias.
• Granularity Limits of Heatmap Detection: Identifying extremely subtle fine-grained artifacts (e.g., distorted finger joints or micro-textures) remains challenging for heatmap regression, leaving room for future reinforcement learning or higher-resolution perceptual supervision.

Related Work & Insights

• vs. Janus / JanusFlow: Janus decouples visual encoding while maintaining separate heads, and JanusFlow co-trains autoregression and Rectified Flow in a single Transformer. MANGO structurally separates semantic visual and generative visual branches, eliminating optimization interference while preserving cross-modal attention.
• vs. GoT / GoT-R1: GoT stitches together a separate MLLM and an independent diffusion model, abandoning unified architecture. GoT-R1 utilizes reinforcement learning for upfront planning but remains an open-loop generator. MANGO realizes a closed-loop "plan-act-reflect-correct" cycle within an end-to-end unified model at lower parameter scale.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering functionality-oriented branch disentanglement for unified models alongside a complete artistic chain-of-thought self-correction loop.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across GenEval, T2I-CompBench, WISE, and MS-COCO, backed by rigorous ablations across architecture, stages, and visual quality.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined problem formulation, and structured architectural exposition.
• Value: ⭐⭐⭐⭐⭐ Offers an inspiring architectural and reasoning blueprint for high-fidelity compositional synthesis in lightweight unified multimodal foundation models.