Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervision¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://segros-project.github.io/
Area: Multimodal VLM
Keywords: Unified Multimodal Models, Multimodal Alignment, Text-to-Image Generation, Visual Grounding, Adaptive Masking
TL;DR¶
Addressing the core granularity mismatch and supervisory redundancy in Unified Multimodal Models (UMMs), SeGroS filters discriminative text tokens via dual intra- and inter-modal affinities and builds a visual grounding map to provide semantic visual hints while restricting masked reconstruction loss exclusively to text-aligned core regions, substantially boosting cross-modal alignment and generation fidelity.
Background & Motivation¶
Unified Multimodal Models (UMMs) have emerged as a compelling paradigm in modern multimodal intelligence. Moving beyond earlier decoupled pipelines that loosely cascaded an MLLM reasoner with a dedicated diffusion generator, UMMs tokenize both visual signals and text sequences into a unified modeling space, handling understanding and image generation within a single autoregressive backbone. To overcome the low computational throughput and weak 2D structural modeling of naive unidirectional token-by-token autoregression, contemporary UMM architectures broadly embrace masked reconstruction and denoising objectives. Prominent examples include Show-o, which marries autoregressive modeling with discrete masked diffusion, and Harmon, which incorporates masked autoregressive (MAR) visual decoding. During training, these models ingest corrupted inputs where subsets of visual tokens are masked out and optimize the model to reconstruct those tokens conditioned on textual prompts.
However, this conventional generative fine-tuning paradigm is constrained by a fundamental bottleneck: the severe granularity mismatch between abstract, coarse textual prompts and dense, pixel-rich visual targets. Natural language descriptions typically convey sparse high-level entities and relations while leaving spatial layout, lighting, pose, and local textures underspecified. Because a single text prompt inherently corresponds to numerous visually distinct, semantically valid configurations, forcing the model to reconstruct a single arbitrary ground-truth image penalizes legitimate semantic variations. Consequently, the network expends immense parameter capacity fitting incidental instance-level artifacts and background clutter rather than internalizing robust cross-modal semantics. While recent approaches such as Reca attempt to supply visual context by appending image tokens as conditioning hints, incorporating all visual tokens indiscriminately introduces heavy supervisory redundancy that dilutes attention; meanwhile, their reliance on semantic-agnostic random masking continues to squander reconstruction loss on background regions irrelevant to the text.
The core insight of this paper is that rather than struggling against unconstrained noise across random spatial masks, generative fine-tuning should be explicitly structured around fine-grained cross-modal semantic correspondences. The core idea is to filter discriminative text tokens via dual intra- and inter-modal affinities to build a visual grounding map, leveraging it to supply high-groundedness regions as conditioning visual hints while retaining low-groundedness background as visible context, thereby concentrating masked reconstruction loss strictly on core text-aligned entities.
Method¶
Overall Architecture¶
SeGroS operates as an architecture-agnostic fine-tuning framework that upgrades cross-modal alignment without altering the inference graph or adding runtime latency to the underlying UMM. During training, paired text prompts and reference images enter the pipeline. First, Discriminative Text Token Filtering computes intra-modal linguistic centrality and inter-modal visual correspondence to eliminate non-grounded stopwords and isolate dominant concept tokens. Next, a fine-grained Visual Grounding Map is generated by aggregating cross-attention transfer probabilities from these filtered tokens to individual visual patches, augmented with light uniform noise to prevent spatial overfitting. Finally, two complementary signals are constructed: the most aligned visual patches are extracted as semantic Visual Hints to enrich prompt conditions, while the least aligned background patches are preserved as visible context in the Corrupted Input, forcing the UMM backbone to reconstruct the masked, highly grounded core semantic targets under text and visual guidance.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Text Prompt Z_T and Ground-Truth Image Z_I"] --> B["Discriminative Text Token Filtering<br/>Text Intra-modal + Image-Text Inter-modal Affinity"]
B --> C["Visual Grounding Map<br/>Weighted Attention Projection + Uniform Noise Injection"]
C --> D["Visual Hints & Grounded Corrupted Input<br/>Top-K Condition Hints + Bottom-K Visible Context"]
D --> E["Unified Multimodal Transformer Backbone<br/>Reconstruct Core Semantic Masked Regions"]
E --> F["Output: Highly Aligned Multimodal Representations"]
Key Designs¶
1. Discriminative Text Token Filtering: Joint Intra- and Inter-modal Affinity Selection
Evaluating text-image similarity uniformly across all prompt tokens suffers from severe score dilution caused by grammatical function words and attention-sink tokens (such as [BOS] and [EOS]). SeGroS resolves this by evaluating text tokens across two orthogonal dimensions before grounding. On the intra-modal linguistic side, normalized text embeddings \(\hat{Z}_T \in \mathbb{R}^{L_T \times D}\) form a self-affinity matrix \(S = \hat{Z}_T \hat{Z}_T^\top\). Excluding special tokens, a row-wise Softmax produces attention probability map \(P^{\text{self}}\), yielding the intra-modal centrality score \(s^{\text{intra}}_j = \sum_{k=1}^{L_T} P^{\text{self}}_{kj}\), which effectively preserves phrase-level syntactic couplings (e.g., "boys-lying" or "playing-tennis"). On the inter-modal grounding side, cross-affinity \(A = \hat{Z}_I \hat{Z}_T^\top\) between normalized visual tokens \(\hat{Z}_I\) and text tokens \(\hat{Z}_T\) determines how visual tokens attend to text tokens, aggregated across all image locations to yield the inter-modal relevance score \(s^{\text{inter}}_j = \sum_{i=1}^{N_I} P^{\text{cross}}_{ij}\). After Min-Max scaling, both metrics are unified multiplicatively into an overall importance score:
$\(\Omega_j = \tilde{s}_j^{\text{intra}} \cdot \tilde{s}_j^{\text{inter}}\)$
Using a text preservation ratio \(\rho = 0.4\), only the top \(K_T = \lfloor \rho L_T \rfloor\) scoring tokens activate the binary mask \(w \in \{0, 1\}^{L_T}\), while ungrounded syntactic filler is filtered out.
2. Visual Grounding Map: Weighted Cross-Attention Projection with Uniform Noise Perturbation
With the active text mask \(w\) established, the system measures the spatial grounding of each visual patch with respect to the key concepts. SeGroS computes the text-to-image attention probability \(P^{\text{cross}}(Z_T \to Z_I)\) and aggregates the probabilities across text positions weighted by \(w_j\), deriving patch grounding scores \(m_i = \sum_{j=1}^{L_T} w_j \cdot P^{\text{cross}}(Z_T \to Z_I)_{ji}\) for each patch \(i \in \{1, \dots, N_I\}\). These are normalized across the image to \(\bar{m}_i \in [0, 1]\). Because image patches exhibit spatial continuity and natural redundancy, deterministic score sorting would repeatedly single out identical grid boundaries across training epochs, leading to representation collapse and memorization. To preserve semantic rankings while infusing structural variability, SeGroS injects lightweight uniform noise into the score map: $\(\tilde{\mathbf{m}} = \bar{\mathbf{m}} + \boldsymbol{\xi}, \quad \boldsymbol{\xi} \sim \mathcal{U}([0, 0.5]^{N_I})\)$ This perturbation breaks deterministic patch lock-in and prevents the model from overfitting to static patch configurations.
3. Visual Hints and Grounded Corrupted Input: Dual-Target Supervision Allocation
Guided by the perturbed grounding map \(\tilde{\mathbf{m}}\), SeGroS restructures both the input conditioning and the masked reconstruction targets. First, to mitigate prompt ambiguity without inducing visual redundancy, the top-scoring visual tokens \(Z_I^{\text{hint}}\) are extracted according to a selection ratio \(\eta \in [0.3, 0.4]\) as explicit Visual Hints. Second, for constructing the Corrupted Input \(\tilde{Z}_I^{\text{SeGroS}}\), given a masking ratio schedule \(\gamma(t) \in [0.7, 1.0)\) with \(K_{\text{seen}} = N_I - \lfloor \gamma(t) N_I \rfloor\), the lowest-scoring patches \(S = \text{BottomK}(\tilde{\mathbf{m}}, K_{\text{seen}})\) are retained as the visible context \(Z_I^{\text{seen}}\), whereas all remaining high-groundedness patches are masked with a learnable [MASK] token. By deliberately keeping the background visible and masking the core semantic entities, the network is forced to dedicate its full reconstruction capacity toward synthesizing the text-aligned objects rather than learning incidental background shortcuts.
Loss & Training¶
The overall training objective combines text-to-image (T2I) masked reconstruction with an autoregressive image-to-text (I2T) objective to preserve multimodal understanding: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{t2i}} + \lambda \mathcal{L}_{\text{i2t}}\)$ where \(\mathcal{L}_{\text{t2i}} = \ell(f_\theta(Z_T, Z_I^{\text{hint}}, \tilde{Z}_I^{\text{SeGroS}}), Z_I)\) evaluates the reconstruction loss \(\ell(\cdot, \cdot)\) solely over the masked subset of \(\tilde{Z}_I^{\text{SeGroS}}\). For discrete token architectures like Show-o, \(\ell\) is the standard cross-entropy loss; for continuous latent spaces like Harmon, it is mean squared error (MSE). Hyperparameters are fixed at temperature \(\tau = 1.0\), text ratio \(\rho = 0.4\), multitask weight \(\lambda = 1.0\), and hint ratio \(\eta = 0.3\) (or \(0.4\) for Show-o at \(512 \times 512\)). During inference, the visual grounding pipeline and hints are discarded entirely: the model receives only the standard text prompt and generates images through its default decoding schedule with zero architectural modifications.
Key Experimental Results¶
Main Results¶
SeGroS was evaluated across three distinct UMM backbone families (Show-o discrete diffusion, Harmon continuous MAR, and OpenUni dual-stream hybrid) spanning model capacities from 0.5B to 3.6B parameters and multiple resolutions (\(256 \times 256\) and \(512 \times 512\)). Benchmarks encompass GenEval (object-level compositional alignment), DPGBench (dense prompt alignment), and CompBench (attribute binding and spatial composition).
| UMM Architecture | Fine-Tuning Method | GenEval Single | GenEval Two | GenEval Count | GenEval Color | GenEval Position | GenEval Attr. | GenEval Overall ↑ | DPGBench Overall ↑ | CompBench Overall ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Show-o-256 | w/o SFT | 97.4 | 63.3 | 52.1 | 82.3 | 14.2 | 30.3 | 56.60 | 70.65 | 73.58 |
| Show-o-256 | SFT* | 97.8 | 63.4 | 53.4 | 80.3 | 16.5 | 33.5 | 57.50 | 74.60 | 75.37 |
| Show-o-256 | Reca | 97.4 | 73.6 | 56.0 | 83.8 | 20.3 | 40.2 | 61.90 | 75.70 | 77.50* |
| Show-o-256 | Ours (SeGroS) | 98.1 | 72.5 | 54.4 | 86.4 | 21.0 | 41.0 | 62.22 | 76.52 | 78.30 |
| Show-o-512 | w/o SFT | 97.2 | 80.3 | 61.9 | 78.2 | 27.3 | 52.3 | 66.20 | 82.21 | 80.53 |
| Show-o-512 | SFT* | 97.2 | 84.1 | 66.6 | 80.9 | 26.8 | 46.8 | 67.03 | 81.80 | 80.02 |
| Show-o-512 | Reca | 98.1 | 93.4 | 64.7 | 79.8 | 38.0 | 55.8 | 71.63* | 84.94 | 84.47 |
| Show-o-512 | Ours (SeGroS) | 97.8 | 91.4 | 65.0 | 81.9 | 35.0 | 53.0 | 70.69 | 85.18 | 84.35 |
| OpenUni-1.6B | w/o SFT | 96.8 | 63.3 | 46.4 | 80.1 | 18.5 | 30.8 | 56.00 | 76.29 | 75.47 |
| OpenUni-1.6B | SFT* | 97.8 | 76.8 | 55.9 | 81.9 | 25.5 | 36.0 | 62.32 | 79.70 | 81.25 |
| OpenUni-1.6B | Reca | 96.6 | 85.4 | 52.5 | 84.3 | 46.5 | 50.8 | 69.33* | 80.45 | 83.10* |
| OpenUni-1.6B | Ours (SeGroS) | 98.8 | 84.6 | 56.9 | 84.0 | 38.8 | 54.0 | 69.50 | 81.33 | 83.83 |
| OpenUni-3.6B | w/o SFT | 99.1 | 71.8 | 51.9 | 83.9 | 23.3 | 41.6 | 61.90 | 79.02 | 78.84 |
| OpenUni-3.6B | SFT* | 98.8 | 81.3 | 55.9 | 85.9 | 25.0 | 48.8 | 65.94 | 80.45 | 82.60 |
| OpenUni-3.6B | Reca | 99.1 | 92.7 | 52.3 | 87.1 | 43.8 | 70.3 | 74.10 | 82.75 | 86.00* |
| OpenUni-3.6B | Ours (SeGroS) | 99.7 | 95.2 | 52.5 | 85.1 | 46.8 | 73.0 | 75.37 | 83.37 | 86.14 |
| Harmon-0.5B | w/o SFT | 99.7 | 80.5 | 55.8 | 86.7 | 32.2 | 49.7 | 67.60 | 80.12 | 80.34 |
| Harmon-0.5B | SFT* | 100.0 | 86.4 | 64.4 | 87.5 | 37.8 | 56.5 | 72.10 | 82.50 | 83.50 |
| Harmon-0.5B | Reca | 99.9 | 92.3 | 59.4 | 91.7 | 58.5 | 70.7 | 78.70 | 84.67 | 85.70* |
| Harmon-0.5B | Ours (SeGroS) | 100.0 | 90.2 | 65.3 | 91.5 | 66.0 | 75.8 | 81.50 | 85.40 | 86.90 |
| Harmon-1.5B | w/o SFT | 99.4 | 87.3 | 68.7 | 86.4 | 44.9 | 51.1 | 72.90 | 80.93 | 81.36 |
| Harmon-1.5B | SFT* | 99.1 | 94.7 | 77.8 | 86.4 | 78.3 | 74.0 | 85.00 | 85.60 | 87.32 |
| Harmon-1.5B | Reca | – | – | – | – | – | – | 85.20 | 86.50 | 87.20* |
| Harmon-1.5B | Ours (SeGroS) | 100.0 | 97.7 | 79.1 | 90.4 | 83.5 | 81.3 | 88.66 | 86.58 | 88.08 |
Ablation Study¶
1. Effect of Visual Hint Ratio \(\eta\) (Harmon-0.5B on MidjourneyV6)
| Visual Hint Ratio \(\eta\) | GenEval Overall ↑ | DPGBench Overall ↑ | Note |
|---|---|---|---|
| 0% (No visual hints) | 72.10 | 82.50 | Baseline text-only conditioning (standard SFT) |
| 10% | 80.40 | 84.50 | Hints too sparse to anchor complex layouts |
| 30% (Ours default) | 81.50 | 85.40 | Optimal trade-off; captures core entities without dilution |
| 50% | 81.20 | 85.10 | Slight degradation due to uninformative background patches |
| 80% | 79.80 | 84.90 | Redundancy begins diluting attention |
| 100% (Full image, Reca-style) | 79.20 | 84.50 | Dense redundancy drops GenEval by 2.3 points vs. 30% hint |
2. Component Ablation on Text Filtering and Supervision Partitioning (Harmon-0.5B)
| Experimental Aspect | Configuration | GenEval Overall ↑ | DPGBench Overall ↑ | Key Insight |
|---|---|---|---|---|
| Text Affinity Metrics | No filtering baseline (– / –) | 79.20 | 84.00 | Unfiltered tokens dilute grounding score |
| Intra-modal only \(\tilde{s}^{\text{intra}}\) (✓ / –) | 79.80 | 84.80 | Captures phrase structures but lacks visual proof | |
| Inter-modal only \(\tilde{s}^{\text{inter}}\) (– / ✓) | 81.50 | 84.80 | Direct visual grounding yields large GenEval gain | |
| Combined Affinity (✓ / ✓) | 81.50 | 85.40 | Synergistic best performance across both benchmarks | |
| Hint vs Context Allocation | Hint: Bot, \(Z_I^{\text{seen}}\): Top | 79.50 | 84.60 | Inverting logic forces model to reconstruct background |
| Hint: Bot, \(Z_I^{\text{seen}}\): Bot | 80.40 | 84.50 | Background hints provide ineffective prompt cues | |
| Hint: Top, \(Z_I^{\text{seen}}\): Top | 79.90 | 84.80 | Core entity visible; loss wasted on incidental background | |
| Hint: Top, \(Z_I^{\text{seen}}\): Bot (Ours) | 81.50 | 85.40 | Hints supply core entities; loss focuses on masked targets |
3. Validation of Supervision Allocation (Drop-Loss Analysis)
| Method Variant | Hint Ratio | Masking Schedule | Supervised Target Subset | GenEval Overall ↑ | DPGBench Overall ↑ |
|---|---|---|---|---|---|
| Standard SFT | 0% | \(\mathcal{U}[0.7, 1.0)\) Random | All masked regions | 72.10 | 82.50 |
| Ours (SeGroS) | 30% | \(\mathcal{U}[0.7, 1.0)\) Adaptive | All masked regions (grounded focus) | 81.50 | 85.40 |
| Ours (drop-loss) | 30% | \(\mathcal{U}[0.7, 1.0)\) Adaptive | Top-30% grounded masked targets only | 82.10 | 84.70 |
Key Findings¶
- The Visual Hint Redundancy Paradox: Tab. 3 directly challenges the assumption that more visual conditioning is unconditionally better. Increasing hint coverage from 30% to 100% (the full image prompt paradigm in Reca) causes a 2.3-point drop on GenEval (81.5 to 79.2). Pruning 70% of redundant visual tokens actually improves semantic alignment by sharpening attention on core entities.
- Criticality of Inverted Masking (Bottom Context, Top Masked): Tab. 5 demonstrates that swapping patch assignments (retaining Top grounded patches as visible context \(Z_I^{\text{seen}}\) and masking Bottom patches) degrades GenEval to 79.5. The UMM achieves peak generative alignment only when background is visible and the model is compelled to reconstruct text-aligned foreground concepts from contextual clues.
- Concentrated Backpropagation via Drop-Loss: In Tab. 6, confining the reconstruction loss solely to the Top-30% grounded masked targets yields an even higher GenEval score of 82.10 (outperforming SFT by 10 points), demonstrating that backpropagating through background tokens primarily injects optimization noise.
- Bi-directional Multimodal Synergy: Rather than suffering catastrophic forgetting in understanding tasks, fine-tuning with SeGroS boosts I2T comprehension benchmarks (Tab. 2), outperforming both un-finetuned baselines and SFT on MME (1217 vs. 1195), MMMU (36.0 vs. 34.7), and POPE (84.5% vs. 84.0%).
Highlights & Insights¶
- Decoupled Intra- and Inter-modal Text Filtering: Simple text-image cosine similarities frequently over-rank isolated generic tokens (e.g., "standing" or "photo"). By multiplying syntactic self-attention centrality with cross-modal visual coverage, SeGroS ensures selected tokens correspond both to key linguistic phrases and actual visual entities.
- Stochastic Noise Perturbation to Counter Spatial Overfitting: Introducing uniform noise \(\mathcal{U}[0, 0.5]\) to the normalized grounding map avoids rigid deterministic grid boundaries across epochs. This simple trick prevents token memorization and improves generalization on complex multi-object scenes.
- Zero-Overhead Inference Compatibility: The entire visual grounding mechanism, prompt extraction, and adaptive masking are strictly training-time constructs. At deployment, the model operates in vanilla text-to-image mode with zero auxiliary models, detectors, or added computational latency.
Limitations & Future Work¶
- Dependence on Base Tokenizer Quality: SeGroS derives grounding scores directly from the initial representations of the pretrained tokenizer. When handling extreme occlusions or tiny objects where early visual-text representations are noisy or misaligned, the grounding map can produce spatial drift, misidentifying background patches as core entities.
- Static Heuristic Thresholds: The text preservation ratio \(\rho = 0.4\) and hint ratio \(\eta = 0.3\) are globally static hyperparameters. For images with vastly disparate semantic densities (a single portrait vs. a crowded street scene), fixed ratios may under- or over-sample core regions. Developing dynamic, entropy-aware hint ratios represents a natural future extension.
Related Work & Insights¶
- vs Reca [46]: Reca relies on full-image visual prompts to mitigate text sparsity, yet incurs supervisory redundancy and retains random spatial masking. SeGroS proves that 30% filtered hints achieve superior alignment while introducing grounded adaptive masking to concentrate the reconstruction loss.
- vs Standard SFT in UMMs (Show-o [47], Harmon [42]): Existing UMM fine-tuning schedules apply random masks agnostic to semantics, wasting modeling capacity fitting background textures. SeGroS focuses gradient updates strictly onto text-aligned structures.
- vs Adaptive Masking in Self-Supervised Learning (SAM [10], OneRef [45]): While prior adaptive masking works focus on representation learning (MAE) or referring grounding, SeGroS pioneers adaptive semantic masking explicitly tailored for dual-branch generative fine-tuning and cross-modal alignment in unified multimodal architectures.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Identifies supervisory redundancy and granularity mismatch in UMMs; proposes an elegant dual-affinity filtering and inverted grounded masking scheme)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive validation across Show-o, Harmon, and OpenUni backbones, scales from 0.5B to 3.6B, 3 T2I benchmarks, 6 understanding benchmarks, and comprehensive ablations)
- Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, well-structured arguments, and logical progression from pilot studies to detailed ablations)
- Value: ⭐⭐⭐⭐☆ (Provides a practical, zero-inference-overhead post-training recipe for unified multimodal models)