Transferability Between Understanding and Generation in Unified Multimodal Models¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: unified multimodal models, cross-task transfer, object counting, spatial relation, text generation
TL;DR¶
Using single-task fine-tuning and cross-task evaluation, this paper finds the strongest bidirectional capability transfer in Lumina-DiMOO, which shares its backbone and visual encoder, and improves counting, spatial relations, and several text-generation metrics through understanding-only training with less distribution shift than direct generation fine-tuning.
Background & Motivation¶
A unified multimodal model (UMM) can answer questions about images and generate images from text, but supporting both tasks does not establish that they benefit each other. Prior analyses often use overall changes on MMMU, POPE, or GenEval as evidence of synergy. Such changes can also reflect additional data, regularization, or training budget, leaving the transferred knowledge unclear.
The paper asks a narrower question: if a model only learns to count objects in images, does it become better at generating a specified number of objects? Counting has both understanding and generation formulations and an exact correctness criterion, making it a useful architectural probe. There is also a practical motivation: directly fine-tuning generation on a narrow capability dataset may make the model imitate that dataset's appearance and disrupt a generation distribution established through high-quality data or preference optimization.
Core Idea: separate understanding and generation supervision for the same visual capability, measure transfer without explicit target-task supervision, and exploit shared representations to inject generative capabilities through understanding with less pressure to imitate the training images.
Method¶
Overall Architecture¶
This is a capability-level intervention study and post-training strategy, not a new visual module. The authors select four architectures in their 7B–8B parameter versions, train understanding and generation separately, and evaluate counting in the opposite task. The two directions are independent training runs, not a sequential understanding-then-generation curriculum.
Lumina-DiMOO, which shows the strongest bidirectional transfer in this probe, becomes the primary testbed for comparing understanding-to-generation transfer against direct generation training. The analysis extends from counting to diagonal spatial relations and text in images. Evaluation covers target capabilities, changes in the generation distribution, and general understanding, rather than treating higher specialist accuracy as an unqualified improvement.
The improvement occurs within the shared model. Inference does not require a separate understanding model to filter, score, or correct generated images. The contribution is the transfer relationship and its use in training, so an experimental description is more appropriate than presenting it as a new network pipeline.
Key Designs¶
1. Capability-level bidirectional probing: keep the knowledge fixed and change the supervised task
The authors filter PixMo-Points and PixMo-Count to images with object counts in 0–20. Understanding training formulates counting as visual question answering. Generation training uses the same images, captioned by Qwen3-VL with object-count information appended to construct text-to-image supervision. This controls the image source and target capability, rather than comparing aggregate scores after training on unrelated datasets.
Understanding evaluation uses the PixMo-Count test split and requires a numerical answer. Generation evaluation follows the GenEval counting protocol with prompts of the form A photo of [NUMBER] [OBJECT]s, using counts of 2–10. A detection model checks the number of objects in each generated image. The training count range and generation evaluation range therefore differ.
Accuracy requires an exact count match, while mean absolute deviation (MAD) measures how far an incorrect answer is from the target. The following notation restates the metric definition; it is not a new loss introduced by the paper:
Here \(c_i\) is the target count and \(\hat c_i\) is either the understanding answer or the count detected in a generated image. Both metrics matter: Janus-Pro slightly improves reverse-transfer accuracy while worsening MAD, so the isolated accuracy gain does not establish consistent bidirectional benefits.
2. Architectural comparison: distinguish backbone sharing from visual representation sharing
Lumina-DiMOO shares both its Transformer backbone and visual encoder. Janus-Pro shares the backbone but uses separate image encoders for understanding and generation. BAGEL uses a Mixture-of-Transformers design with branches coupled through joint multimodal self-attention. BLIP3-o separates the understanding backbone from the generation Transformer and passes conditioning signals between them.
This comparison decomposes architectural unification into concrete forms of parameter and representation sharing. A common visual representation and backbone may allow knowledge to be reused across tasks, but the findings do not imply that separate paths prevent all transfer. BAGEL and BLIP3-o both improve counting generation after understanding training; the clearest distinction concerns the magnitude and consistency of bidirectional effects.
The authors consequently use Lumina-DiMOO for the main capability-injection experiments. This remains an empirical comparison between differently pretrained models, not an encoder-only ablation with all other conditions fixed. It does not establish shared visual encoding as either a sufficient or necessary condition for transfer.
3. Understanding-side capability injection: learn the concept without directly fitting the generated appearance
Beyond counting, spatial experiments focus on top-left, top-right, bottom-right, and bottom-left, since pretrained models already handle basic horizontal and vertical relations reasonably well. The authors select two ImageNet class-image pairs and place them on a canvas according to a random diagonal relation, constructing 200K samples. The relation can be expressed as visual question answering or as a generation prompt. Evaluation checks the relative centers of detected object boxes.
These synthetic images have clear spatial labels but do not necessarily look like natural scenes. Direct generation training can learn both the relation and the pasted-object appearance. Understanding training only requires answering the relation, allowing spatial knowledge to transfer without equally strong pressure to imitate that appearance. Figure 5 illustrates this distinction.
For text, the authors render Markdown documents into images to construct a separate dataset of 200K image-text pairs. They consider small text in natural scenes and dense full-page documents too difficult for discrete image tokenizers, motivating moderately difficult rendered text. Understanding training performs text recognition, while the target generation capability requires readable characters, strokes, spacing, and glyph structure.
GLM-OCR extracts text from generated images for comparison with the prompt using WER, CER, edit distance, BLEU, METEOR, and F1. Table 4 contains six metrics despite the prose referring to five. These scores measure OCR recoverability and text matching, not direct human judgments of visual quality.
4. Transfer strength and fidelity evaluation: separate acquired capability from its costs
Table 6 measures transfer strength as the improvement from transfer divided by the improvement from direct training on the same dataset. Let \(\Delta\) denote improvement in the favorable direction of the metric; the definition can be written as:
For counting generation accuracy, both training routes improve by 9.0 percentage points, giving 100% transfer strength. MAD improvements of 0.28 and 0.20 give 140%. Although the paper calls direct training an assumed upper bound, 140% demonstrates that this is not a mathematical upper bound. Small or negative denominators also limit the ratio's usefulness as a stable, universal capability scale.
The authors additionally report Inception Score (IS) and FID to expose changes in generated appearance. Crucially, FID compares images generated by the baseline model with those generated by the fine-tuned model, not generated images with real data. Lower FID indicates greater proximity to the original model's distribution and cannot establish realism on its own. General understanding is separately evaluated on POPE, MMBench, MMMU, and MME-P.
Loss & Training¶
The architecture probes use LoRA-based supervised fine-tuning (SFT), with understanding and generation trained separately. The transfer direction receives no explicit target-task supervision. The practical strategy trains only understanding and then generates images with the updated model, without introducing an additional transfer loss or reward model.
Section 5.2 also jointly trains counting understanding and generation in Lumina-DiMOO. Figure 8 shows alternating improvements rather than simultaneous progress. This is an observation under a particular post-training dataset and objective configuration, not proof that multitask pretraining necessarily produces conflicting objectives.
The available cache ends after the main paper and references and does not include the repeatedly cited appendix. LoRA rank, learning rate, batch size, training epochs, detailed loss weights, and the appendix results for the additional model MMaDA cannot be verified from this cache and are therefore not supplied.
Key Experimental Results¶
Main Results¶
The following selection from Tables 2 and 3 compares Lumina-DiMOO training strategies. Accuracy is in %. Baseline FID is undefined and retained as a missing value; counting MAD does not apply to spatial relations.
| Capability | Train → Test | Accuracy ↑ | MAD ↓ | IS ↑ | FID to Baseline ↓ |
|---|---|---|---|---|---|
| Counting | Baseline | 48.0 | 1.11 | 16.86 | — |
| Counting | und → gen | 57.0 | 0.83 | 17.55 | 31.47 |
| Counting | gen → gen | 57.0 | 0.91 | 15.29 | 52.51 |
| Spatial relation | Baseline | 67.0 | — | 16.86 | — |
| Spatial relation | und → gen | 74.0 | — | 17.50 | 30.95 |
| Spatial relation | gen → gen | 80.0 | — | 17.41 | 32.28 |
Counting transfer and direct training both reach 57.0%, but transfer has lower MAD, higher IS, and lower FID. Spatial transfer improves accuracy by 7.0 percentage points, compared with 13.0 for direct training. It does not beat direct training on relation accuracy; it remains closer to the original generation distribution.
The text results below retain all six metrics from Table 4 to show their divergent behavior.
| Train → Test | WER ↓ | CER ↓ | Edit Distance ↓ | BLEU ↑ | METEOR ↑ | F1 ↑ |
|---|---|---|---|---|---|---|
| Baseline | 0.654 | 0.362 | 36.812 | 0.277 | 0.403 | 0.446 |
| und → gen | 0.641 | 0.351 | 35.636 | 0.274 | 0.410 | 0.452 |
| gen → gen | 0.641 | 0.367 | 37.312 | 0.354 | 0.533 | 0.525 |
Understanding transfer improves five metrics, but BLEU drops from 0.277 to 0.274. Direct training produces higher METEOR, F1, and BLEU while worsening CER and edit distance. Any claim that direct training provides better text accuracy must identify the metric rather than imply uniform superiority.
Ablation Study¶
The central analysis is a cross-architecture intervention rather than a module-removal ablation. This selection from Table 1 reports counting accuracy (%), with each arrow connecting baseline performance to performance after cross-task training.
| Model | Architectural Property | und → gen | gen → und |
|---|---|---|---|
| Lumina-DiMOO | Shared backbone and visual encoder | 48.0 → 57.0 | 22.0 → 30.0 |
| Janus-Pro | Shared backbone, separate visual encoders | 38.0 → 32.0 | 37.0 → 38.0 |
| BAGEL | Branch-specific backbones, joint attention | 42.0 → 47.0 | 63.0 → 64.0 |
| BLIP3-o | Separate backbones, conditioning signals | 31.0 → 37.0 | 63.0 → 62.0 |
Key Findings¶
- The preferred direction depends on the capability. Table 6 reports spatial accuracy transfer strengths of 53.8% for und → gen and 21.4% for gen → und, versus text METEOR values of 5.4% and 27.6%. However, text WER gives 100% and 28.9%, so reverse transfer is not stronger on every text metric.
- Fine-grained transfer exists. In Table 5, generation training reduces text-recognition CER from 0.073 to 0.061, while direct understanding training reaches 0.042. Transferability does not mean target-task supervision is replaceable.
- General ability is largely preserved, not entirely unchanged. Table 7 shows MMMU improving from 57.7 to 62.0 after counting understanding training, while MME-P falls from 1534 to 1492 and POPE from 86.7 to 86.4. These local regressions should not be omitted.
Highlights & Insights¶
- Formulating the same capability as two tasks makes transfer easier to interpret than aggregate benchmark gains. Independent training directions also prevent explicit target supervision from being mistaken for transfer.
- Synthetic data need not be equally suitable for teaching concepts and teaching appearance. The spatial collage experiment shows how understanding supervision can exploit useful concept labels while importing less of a low-fidelity visual style into generation.
- Separately measuring generation-distribution preservation and task correctness is broadly reusable. A practical post-training workflow can check both specialist performance and drift from the original model instead of selecting the checkpoint with the highest specialist score alone.
Limitations & Future Work¶
- Scope acknowledged by the authors: joint-training analysis concerns post-training of an established model using tightly counting-aligned paired data. The authors cannot disentangle optimization conflict, dataset configuration, and differences from pretraining dynamics.
- Limited causal evidence: the four models do not have fully matched pretraining corpora, objectives, or initialization. Table 1 alone cannot show that encoder sharing causes transfer; stronger evidence requires controlled sharing variants under matched training conditions.
- Evaluation constraints: detector and OCR errors can affect capability scores. Baseline-referenced FID measures drift and may reward retaining baseline defects. The reported main tables lack repeated-run error bars and human evaluation, leaving the stability of small text gains unresolved.
- Limited extrapolation: three capabilities and synthetic datasets do not cover all generative knowledge, and the appendix is absent from the available cache. Larger models, complex natural relations, and joint injection of multiple capabilities require further validation.
Related Work & Insights¶
- vs Lumina-DiMOO and Janus-Pro: the former supplies a shared visual representation for transfer, while the latter uses separate visual encoders for task specialization. This study identifies potentially different preferences for specialization and knowledge reuse rather than invalidating separate-path designs.
- vs BAGEL and BLIP3-o: selective sharing and conditional coupling can also produce understanding-to-generation gains. Architecture selection should consider original task performance, transfer direction, and the target capability, not maximize shared parameters alone.
- vs UniFork: UniFork examines different representation-alignment requirements for understanding and generation. The alternating joint-training behavior here is compatible with that perspective but does not directly verify the same internal mechanism.
- vs GenEval: the study borrows object-counting and spatial-detection evaluation ideas to investigate capability transfer; it does not introduce a new general-purpose generation leaderboard.
Rating¶
- Novelty: 4/5. Capability-level bidirectional interventions lead to a concrete post-training strategy with a clear research question.
- Experimental Thoroughness: 4/5. Four architectures, three capabilities, and quality checks provide breadth, but causal controls, error bars, and available training details remain limited.
- Writing Quality: 4/5. The main argument and comparisons are clear, although the metric count, upper-bound terminology, and no-degradation summary require qualification.
- Value: 4/5. A useful approach for UMM post-training when improving specialist generation capabilities without repeating generation alignment is desirable.