Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/ChonghuinanWang/ILLUME-X
Area: Image Generation
Keywords: Interleaved Text-Image Generation, Multimodal Unified Model, Diffusion Transformer, Interleaved CFG, Data Curation

TL;DR

ILLUME-X presents an advanced unified multimodal paradigm supporting free-form, variable-length interleaved text-image generation via decoupled continuous visual encoding, an interleaved attention mask that eliminates cross-step diffusion noise leakage, a decoupled interleaved CFG mechanism, and an automated 100K causal data curation pipeline, matching complex multi-agent systems and substantially outperforming prior unified models on ISG-Bench with a score of 6.26.

Background & Motivation

Multimodal Large Language Models (MLLMs) and diffusion transformers have driven notable strides in visual comprehension and image generation. However, modern multimodal foundation models are predominantly engineered for single-direction conversion—either comprehending mixed inputs to answer in text, or translating textual prompts into standalone images. Advancing toward next-generation general intelligence demands autonomous free-form interleaved text-image generation (N-to-M multimodal generation), enabling a unified model to synthesize variable sequences of texts and images in arbitrary order to power rich interactions such as dynamic storytelling, multi-step tutorials, and fine-grained visual decomposition.

Despite initial attempts, existing multimodal architectures suffer from acute structural and optimization bottlenecks when scaling to flexible interleaved generation. Pure autoregressive discretization models serialize continuous visual patches into discrete tokens, leading to catastrophic sequence expansion, token truncation, and severe loss of high-frequency visual details. Meanwhile, autoregressive-diffusion hybrid models face severe error accumulation during sequential multi-image synthesis: residual Gaussian noise from diffusion denoising steps leaks into the shared autoregressive context, disrupting downstream multimodal predictions. Furthermore, conventional Classifier-Free Guidance (CFG) algorithms cannot adapt to composite conditions where text prompts and image contexts continually alternate, and open-source interleaved datasets suffer from poor aesthetic fidelity and ambiguous causal dynamics. Finally, widely used benchmarks such as ISG-Bench depend heavily on rigid output syntax templates, frequently failing to evaluate models that generate naturalistic multimodal responses.

The entry point of this work is that autonomous interleaved generation requires harmonizing architectural attention insulation, adaptive multimodal guidance, rigorous causal data synthesis, and syntax-agnostic multi-dimensional evaluation into a cohesive framework. Core idea: Build a unified transformer model ILLUME-X with modality-specific projections and shared attention, equipped with a noise-blocking interleaved attention mask and decoupled text-anchor CFG, trained on 100K curated causal video and CoT-guided synthetic samples to achieve stable and efficient free-form N-to-M interleaved text-image synthesis.

Method

Overall Architecture

ILLUME-X adopts an integrated decoder-only Transformer backbone to seamlessly process heterogeneous token sequences containing text tokens, continuous visual semantic tokens, latent diffusion tokens, and modality control tokens. On the input side, it employs decoupled continuous visual encoders: a ViT encoder extracts high-level semantic features for cross-modal context modeling, while a VAE encoder captures and reconstructs low-level continuous latent representations for high-fidelity diffusion generation. The Transformer backbone employs shared self-attention layers across all transformer blocks to maximize deep cross-modal reasoning, paired with modality-specific QKV projections and Feedforward Networks (FFNs) dedicated to multimodal understanding and image generation streams respectively.

During inference, special tokens (BOI for Begin-of-Image, EOI for End-of-Image, and EOS for End-of-Sequence) are introduced to explicitly demarcate modality transitions. Autoregressive causal attention governs text and special control tokens, while bidirectional full self-attention operates across visual latent tokens within each image segment to capture holistic spatial relationships. Crucially, during subsequent token generation, the noisy latent tokens from preceding image diffusion processes are explicitly masked out, allowing only clean reconstruction tokens to participate in context conditioning and preventing noise amplification across turns.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Arbitrary Interleaved Multimodal Input<br/>Interleaved Text Prompts and Multi-Image Context"] --> B["Dual Continuous Visual and Text Encoding<br/>ViT Semantic Features + VAE Latents + Text Tokenizer"]
    B --> C["Modality-Decoupled Unified Transformer<br/>Shared Self-Attention + Modality-Specific QKV & FFN"]
    C --> D["Interleaved Attention Mask with Noise Blocking<br/>Causal Text Flow + Bidirectional Image Blocks + Masking Prior Noisy Latents"]
    D --> E["Decoupled Interleaved Classifier-Free Guidance<br/>Text Condition as Anchor + Independent Image Scaling Coefficients"]
    E --> F["Free-form Alternating Multimodal Generation<br/>Autoregressive Text Tokens + Rectified Flow Diffusion Image Latents"]

Key Designs

1. Interleaved Attention Masking with Noise Blocking: Mitigating Modality Asymmetry and Error Accumulation

Directly applying standard causal attention to multi-image interleaved generation creates two core pathologies: it deprives visual tokens of essential 2D bidirectional spatial context, and it allows unmitigated diffusion Gaussian noise from intermediate denoising steps to contaminate future autoregressive context tokens. ILLUME-X resolves this via a segmented interleaved attention mask. Sequences are partitioned into modality-specific blocks bounded by BOI and EOI markers:

Text and special control tokens strictly follow causal attention to preserve the autoregressive language generation property, while continuous image latent tokens within each image block interact via bidirectional attention to support flow-based denoising over spatial layouts. Most critically, the framework explicitly separates noisy latent tokens and clean reconstruction tokens in the sequence; when generating subsequent multimodal tokens, the noisy latent tokens of all preceding images are forcefully masked out, completely blocking noise propagation across the generative trajectory.

2. Decoupled Interleaved Classifier-Free Guidance: Balancing Multi-Condition Generative Alignment

Standard T2I diffusion models apply CFG solely on textual conditions. In contrast, interleaved multimodal sequences provide composite conditions and targets containing multiple alternating texts and images: \(c = \{c_{txt}^0, c_{img}^0, c_{txt}^1, \dots\}\). Jointly dropping arbitrary modalities at random during training introduces combinatorial complexity and destabilizes textual reference chains. ILLUME-X decomposes conditions into a text condition set \(c_{txt}\) and an image context set \(c_{img}\), formulating a decoupled guidance scheme anchored by text guidance:

\[ \nabla_{x_{img}} \log p(x_{img} | c_{txt}, c_{img}) = \nabla_{x_{img}} \log p(x_{img}) + \gamma_{txt} (\nabla_{x_{img}} \log p(x_{img} | c_{txt}, c_{img}) - \nabla_{x_{img}} \log p(x_{img} | c_{img})) + \gamma_{img} (\nabla_{x_{img}} \log p(x_{img} | c_{img}) - \nabla_{x_{img}} \log p(x_{img})) \]

During training, three condition-sampling modes are randomly executed: preserving both text and image conditions, dropping text conditions only, and dropping both conditions simultaneously. Whenever any condition is dropped, losses on text and special tokens are explicitly omitted to preserve control token parsing. At test time, independent scaling factors \(\gamma_{txt}\) (set to 8.0) and \(\gamma_{img}\) (set to 1.0) allow fine-grained balance between prompt fidelity and cross-image consistency.

3. Three-Tier High-Fidelity Interleaved Data Curation: Resolving Data Scarcity and Causal Ambiguity

To overcome the limited scale and fidelity of existing public interleaved corpora, ILLUME-X constructs a comprehensive pipeline producing 100K high-fidelity training instances: - Video Data Extraction: Overcoming the information loss of uniform or naive keyframe sampling, the pipeline introduces multi-interval sliding-window sampling (2, 5, or 8 frame windows across 1, 3, or 10 second intervals). Frames are filtered for aesthetic quality via Laplacian variance and MANIQA scores alongside Qwen-3-VL-32B validation; dynamic redundancy is eliminated using RAFT optical flow (discarding static or uniform rigid translation) and DINOv2 feature difference analysis. Multi-level captions across global narrative, single-frame attributes, and cross-frame state transitions are extracted by Qwen-3-VL-32B. - Multi-Step In-Context Generation: An external MLLM (GPT-5) designs a chain-of-thought blueprint containing global semantics, per-step texts, and per-image prompts. An in-context image generator (Gemini 3 Pro) iteratively synthesizes subsequent frames guided by intermediate image-to-image semantic difference descriptions produced by Qwen3-VL-32B. - Multi-Round Self-Reflection Data: A critic MLLM inspects initial generated outputs against user specifications, diagnoses defects, and feeds corrective feedback back into the prompt over multiple iterative refinement turns (\(T_K, I_K\)), with suboptimal trajectories eliminated to enrich the corpus with reflective problem-solving traces.

4. Comprehensive Multi-Dimensional Metric ILScore: Mitigating Rigid Syntax-Dependent Evaluation Failure

To address the vulnerability of ISG-Bench to minor output formatting deviations, ILLUME-X introduces ILScore, a hierarchical metric evaluating interleaved generation across four complementary dimensions: - D1: Image-Text Accuracy, measuring holistic cross-modal semantic alignment, logical coherence, and narrative relevance; - D2: Single Image Accuracy, assessing single-frame concept fidelity, relationship matching, visual sharpness, and aesthetic quality; - D3: Image Sequence Accuracy, evaluating identity, visual style, and environment continuity across multiple generated images; - D4: Text Accuracy, assessing the fluency, richness, and grammatical precision of generated textual narratives.

Loss & Training

ILLUME-X is initialized from the public weights of BAGEL (7B LLM + 7B DiT) and trained on open datasets (SEED-Story, VINCIE, WEAVE) alongside the curated 100K dataset with a base learning rate of \(2 \times 10^{-5}\). The multi-task optimization objective combines language and flow losses:

\[ \mathcal{L}_{total} = \mathcal{L}_{CE}(y_{txt}, y_{special}) + \lambda_{flow} \mathcal{L}_{flow}(v_{\theta}, v_{RF}) \]

where cross-entropy loss \(\mathcal{L}_{CE}\) supervises text tokens and special control markers (BOI, EOI, EOS) to guarantee language fluency and structural boundaries, while Rectified Flow velocity matching loss \(\mathcal{L}_{flow}\) trains continuous image latents for precise generative synthesis. Joint training on in-context image generation data and heterogeneous interleaved tasks stabilizes optimization and prevents performance plateaus during multi-image generation.

Key Experimental Results

Main Results

On the comprehensive benchmark ISG-Bench, ILLUME-X is systematically benchmarked against leading unified multimodal models and non-unified agent pipelines.

Model	Unified?	Style Transfer	Progressive Gen	3D Scene	Image Decomp	Image-Text Comp	Temporal Pred	Storytelling	Visual VQA	Average (AVG)
Show-o	Yes	2.11	2.41	1.43	2.87	2.06	2.58	3.32	1.86	2.33
Anole	Yes	2.93	2.76	1.85	1.49	3.21	2.58	2.97	4.70	2.81
MiniGPT-5	Yes	2.15	3.15	1.79	2.54	2.72	2.73	2.91	4.29	2.79
CoMM-MiniGPT-5	Yes	2.96	2.60	3.09	2.24	3.09	2.52	2.72	2.87	2.96
Seed-LLaMA-14B	Yes	1.84	3.30	1.52	3.69	1.94	1.78	2.84	2.20	2.39
Gemini + SD3	No	4.89	6.59	2.68	7.26	6.37	5.26	5.68	7.89	5.83
ISG-AGENT	No	5.87	6.46	4.89	7.58	6.93	4.54	7.03	6.80	6.26
ILLUME-X (Ours)	Yes	6.02	5.75	3.96	6.97	7.06	6.43	7.25	6.67	6.26

On standard text-to-image benchmarks, ILLUME-X scores 0.85 Overall on GenEval (outperforming BAGEL's 0.82 and Janus-Pro's 0.80) and 86.38 Overall on DPG-Bench (surpassing BAGEL's 85.07 and Janus-Pro's 84.19), registering 94.08 in Attribute precision. In computational overhead, ILLUME-X (7B+7B) requires 81.33 seconds per image generation, operating roughly 5x faster than Emu 3.5 (34B at 409.50s per image).

Ablation Study

Ablations on Visual Storytelling and the "How-to" subset of Image-Text Complementation analyze text guidance scale \(\gamma_{txt}\), image guidance scale \(\gamma_{img}\), and Chain-of-Thought (CoT) reasoning.

\(\gamma_{txt}\)	\(\gamma_{img}\)	w/ CoT	Visual Storytelling	Image-Text Complementation	Note
2.0	1.0	No	7.02	5.35	Insufficient text guidance weakens prompt following
4.0	1.0	No	7.13	5.59	Steady improvements with stronger text guidance
8.0	1.0	No	7.12	5.69	Optimal balance point without CoT
8.0	1.25	No	7.00	5.52	Excessive image guidance causes rigid generation
8.0	1.5	No	6.85	5.60	Over-conditioning on images hurts narrative diversity
8.0	1.0	Yes	7.28	5.91	Full model: CoT substantially boosts sequential logic

Key Findings

• Generational Leap Among Unified Architectures: ILLUME-X achieves an average score of 6.26 on ISG-Bench, representing a >120% improvement over preceding unified baselines (Anole at 2.81, CoMM-MiniGPT-5 at 2.96), and effectively matches the multi-agent ensemble ISG-AGENT (6.26) within a single end-to-end network.
• Guidance Decoupling and CoT Synergies: Setting text guidance to an assertive anchor (\(\gamma_{txt}=8.0\)) while anchoring image conditioning moderately (\(\gamma_{img}=1.0\)) maximizes visual sharpness and context adherence. Adding CoT yields substantial gains (+0.16 in Visual Storytelling, +0.22 in Image-Text Complementation).
• Superior Sequence Consistency with Practical Efficiency: On ILScore, ILLUME-X achieves top image sequence accuracy (D3 = 5.62 vs. Emu 3.5's 4.87 and Gemini 3 Pro's 4.34) while delivering ~5x lower inference latency than Emu 3.5 (81.33s vs. 409.50s).

Highlights & Insights

• Noise-Blocked Autoregressive Masking: Strategically masking out intermediate noisy diffusion latents from the causal history in multi-turn sequences prevents compounding noise corruption across consecutive visual outputs.
• Decoupled Asymmetric Interleaved CFG: Formulating CFG around a dominant text anchor with independent image condition scaling resolves condition conflicts in dense interleaved sequences without combinatorial compute explosion.
• Causally Grounded Synthesis Engine: Combining temporal sliding-window optical flow filtering with multi-level MLLM narrations generates 100K high-fidelity training instances capturing genuine real-world physical transitions and reasoning steps.

Limitations & Future Work

• Native Resolution Constraints: Current training and evaluation are conducted at 512×512 resolution. Due to memory bandwidth and context length boundaries in diffusion transformers, scaling to native 1024×1024 generation remains challenging.
• Context Length Fatigue over Ultra-Long Sequences: Generating long documents with dozens of alternating images and text passages can saturate the transformer's KV cache, leading to subtle identity drift across distant turns.
• Disparity in Pure Knowledge Reasoning: In pure text evaluation (D4) on complex factual VQA tasks, the unified model exhibits a slight performance gap compared to pure text-focused LLMs of equivalent size.

Related Work & Insights

• vs. Emu 3.5 / Chameleon (Discrete Autoregressive Models): Emu 3.5 relies on discrete vision tokenization, requiring 34B parameters and heavy inference latency (409.5s/image) while frequently suffering codebook visual quantization loss. ILLUME-X pairs autoregressive language modeling with continuous Rectified Flow diffusion, outperforming Emu 3.5 in sequential consistency (D3: 5.62 vs 4.87) at 5x faster speed.
• vs. Show-o / Transfusion (Standard Hybrid Models): Earlier hybrid architectures primarily support predefined, one-directional text-to-image or image-to-text generation. ILLUME-X establishes full N-to-M free-form bidirectional interleaving supported by noise-insulated attention and decoupled CFG.
• vs. ISG-AGENT (Pipeline Multi-Agent Systems): ISG-AGENT chains disparate closed-source commercial APIs, incurring heavy API overhead and latency. ILLUME-X delivers matching top-tier quality (6.26 on ISG-Bench) within a single self-contained model.

Rating

• Novelty: ⭐⭐⭐⭐ [Systematic innovations in noise-blocking interleaved attention, decoupled CFG, and automated causal data synthesis]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across ISG-Bench, ILScore, GenEval, DPG-Bench, with exhaustive ablations and cost analysis]
• Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear architectural illustrations, and self-consistent empirical analysis]
• Value: ⭐⭐⭐⭐⭐ [Sets a benchmark and provides a clear technical roadmap for unified multimodal foundation models entering free-form interleaved generation]