UniReflect: Self-Reflection Tuning for Unified Multimodal Understanding and Generation

Conference: ECCV 2026
Paper: ECCV Official Link
Area: Multimodal VLM
Keywords: Unified Multimodal Model / Self-Reflection Tuning / In-Model Verification / Test-Time Scaling / Image Generation & Editing

TL;DR

UniReflect presents the first unified multimodal framework that integrates visual comprehension, generation, and in-model verification, utilizing a shared Semantic Perceiver and a decoupled verification strategy to realize autonomous test-time scaling (TTS) without external discriminators.

Background & Motivation

Multimodal generative models have advanced rapidly in recent years, evolving from early convolutional U-Net architectures to modern Diffusion Transformers (DiT) capable of synthesizing photorealistic images from textual prompts. Nevertheless, when presented with complex compositional prompts requiring precise spatial localization, attribute binding, or multi-object counting, single-pass forward generation models frequently exhibit hallucinations, omitted entities, or misplaced relations. To address these failures, recent efforts have explored test-time scaling (TTS) and self-reflection mechanisms inspired by large language models, where the generation process is augmented by verification steps that evaluate semantic alignment between textual prompts and generated visual content to guide iterative refinement.

However, existing visual verification and reflection frameworks almost universally treat verification as an auxiliary, post-generation process. Prevailing pipelines rely heavily on external Visual Large Language Models (VLLMs) acting as independent arbiters to critique the generated image, provide textual feedback, and forward rewritten editing prompts to a third-party image editing model. This cascading architecture suffers from three major drawbacks: first, invoking multiple large foundation models in sequence introduces prohibitive latency and memory overhead; second, distilling rich visual discrepancies into pure natural language inevitably discards granular spatial and geometric signals; and third, standard VLLMs lack specialized calibration for image verification, exhibiting brittle instruction-following and unstable thresholding when tasked with binary alignment classification.

Inherently, verification is a visual understanding task, while refinement relies on visual generation; thus, understanding and generation represent two deeply complementary facets of multimodal intelligence. The core idea of this work is to unify visual comprehension, in-model verification reflection, and visual generation into a single end-to-end framework, leveraging a lightweight shared Semantic Perceiver to bridge multimodal embeddings and employing a decoupled verification strategy to achieve an autonomous "generation-verification-reflection-refinement" closed loop without external verifiers or editing prompts.

Method

Overall Architecture

The UniReflect architecture comprises four principal components: a frozen Visual Large Language Model backbone (VLLM, utilizing Qwen3-VL-4B), a lightweight shared Semantic Perceiver trained from scratch, a Variational Autoencoder (VAE) for extracting low-level latent visual representations, and a Diffusion Transformer (DiT, 4B parameters, initialized from OmniGen2) operating under a flow-matching formulation. The model operates across three unified modes: general multimodal understanding and question-answering, prompt-image alignment verification and reflection, and condition-guided image synthesis and editing.

During inference, given an initial textual prompt, the DiT synthesizes a candidate image. The prompt and candidate image are then routed into the unified VLLM backbone. Dedicated tokens—<verifier>, <reflect>, and <img>—are extracted from the VLLM hidden states and mapped by the shared Semantic Perceiver into aligned feature embeddings (\(E_V\), \(E_R\), and \(E_S\)). The decoupled verifier evaluates the image; if a discrepancy is detected, the model autoregressively generates structured reflection text, projects it into a continuous reflection embedding \(E_R\), and directly conditions the DiT to perform localized error correction and content-preserving regeneration.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>User Prompt / Candidate Image / Edit Instruction"] --> B["Frozen VLLM Backbone<br/>Multimodal Feature Fusion & Task Token Extraction"]
    B --> C["Semantic Perceiver<br/>Multi-layer Transformer Mapping to EV / ER / ES"]
    C --> D["Decoupling Verification Strategy<br/>Dedicated Classifier Head & Conditional Reflection Text"]
    D -->|Alignment Verified / Score >= Threshold| E["Final Output Image"]
    D -->|Semantic Mismatch / Omission Detected| F["Sequential Test-Time Scaling<br/>Reflection Embedding ER Directs DiT Regeneration"]
    F --> C

Key Designs

1. Semantic Perceiver: Unifying Understanding and Generation Spaces

Conventional assembled unified models typically extract the final hidden representations from a frozen VLLM and directly feed them to a diffusion model as conditioning signals. However, because VLLMs are pretrained primarily on open-ended question-answering, their representations favor abstract lexical concepts, creating a substantial representational gap with the spatial layouts and high-frequency textures required by diffusion generators. UniReflect addresses this bottleneck by introducing a shared Semantic Perceiver structured as a multi-layer Transformer block.

The vocabulary is augmented with three specialized tokens: <verifier>, <reflect>, and <img>, while keeping the core VLLM parameters frozen to preserve visual reasoning capabilities. From the VLLM output states \(E = F_{vllm}(V, T)\), the Semantic Perceiver projects the corresponding hidden slices into generation-aligned continuous embeddings: verification features \(E_V\), reflection features \(E_R\), and semantic instruction features \(E_S\). By sharing parameters across verification and generation tasks, the perceiver enables multi-task synergy, allowing fine-grained spatial and structural feedback identified during verification to directly enrich generative representations.

2. Decoupling Verification Strategy: Separating Scoring from Reflection

Standard VLLM-based verifiers commonly attempt to output a binary "True" or "False" judgment alongside textual rationales in a single autoregressive sequence. In practice, this approach suffers from instruction-following degradation on ambiguous samples, yielding uncalibrated confidence scores and unstable decision boundaries that fail to provide dependable triggers for downstream editing.

UniReflect resolves this through a decoupled verification mechanism that separates quantitative scoring from qualitative critique. A dedicated classification head is placed directly on the post-perceiver verification token \(E_V\), producing a sharply calibrated semantic consistency score. Only when this classification score falls below a confidence threshold does the model trigger the autoregressive language head conditioned on <verifier> to produce explicit natural language critiques (such as specifying "the tabby cat is positioned on top of the sofa, whereas the prompt specifies under the sofa"). This decoupled paradigm guarantees high verification accuracy and deterministic thresholding while retaining rich, human-interpretable reflection text.

3. Sequential Test-Time Scaling: Autonomous In-Model Refinement Loop

In traditional TTS pipelines, the verifier's critique must be formulated into an external editing prompt and fed into a disjoint image editing model, a process prone to prompt drift and context loss. UniReflect-TTS internalizes test-time scaling entirely within the single model architecture.

Upon generating structured reflection text, the tokens are encoded via the Semantic Perceiver into a specialized reflection embedding \(E_R\). The flow-matching diffusion decoder \(F_D\) then synthesizes the refined image \(Y_{t+1}\) conditioned jointly on the original semantic instruction \(E_S\), the newly derived reflection embedding \(E_R\), the latent visual features \(X_v\) from the initial generation, and diffusion noise \(N_0\):

\[Y_{t+1} = F_D(E_S, E_R, X_v, N_0)\]

Because \(E_S\) preserves the correctly synthesized global composition while \(E_R\) supplies targeted correction vectors for the identified flaws, UniReflect executes surgical repairs on missing attributes or incorrect spatial relationships. This autonomous "generate-verify-reflect-refine" loop dynamically converts inference-time computation into substantial gains in image-text alignment fidelity.

A Worked Example

Consider a complex compositional prompt: "On a red wooden table, there are three green apples; to the right of the table sits a brown teddy bear wearing a blue hat." 1. Initial Generation: The UniReflect DiT outputs an initial candidate image \(Y_0\). The table and bear are synthesized accurately, but only two apples appear, and the teddy bear's hat is rendered red instead of blue. 2. In-Model Verification: \(Y_0\) and the prompt are fed into the unified VLLM. The classification head on \(E_V\) outputs False with a low alignment score of 0.21. 3. Structured Reflection Generation: Conditioned on the verification state, the model autoregressively generates: "Omission detected: only two green apples are present instead of three. Attribute binding error: the teddy bear's hat is red instead of blue." 4. Targeted Refinement: The reflection text is transformed into \(E_R\) by the Semantic Perceiver. The diffusion decoder, conditioned on both \(E_S\) and \(E_R\), adds the third green apple and repaints the hat blue while preserving the exact texture of the wooden table and teddy bear. On the subsequent pass, the verifier head scores 0.94 (True), outputting the verified final image.

Loss & Training

The framework is optimized end-to-end using a joint multi-task loss formulation that balances autoregressive text generation \(\mathcal{L}_{text}\), verification classification cross-entropy \(\mathcal{L}_{cls}\), and flow-matching visual reconstruction mean squared error \(\mathcal{L}_{visual}\):

\[\mathcal{L} = \mathcal{L}_{text} + \lambda_1 \mathcal{L}_{cls} + \lambda_2 \mathcal{L}_{visual}\]

The training schedule is executed across two distinct stages: - Stage 1 (Understanding–Generation Alignment): Focuses exclusively on text-to-image synthesis and image editing datasets at \(512 \times 512\) resolution. The model is trained for 40,000 steps with AdamW, a learning rate of \(1 \times 10^{-3}\), batch size 128, and weight decay 0.0, aligning the Semantic Perceiver with the DiT backbone. - Stage 2 (Joint Multi-Task Reinforcement): Incorporates the visual verification corpus alongside generation and editing data at \(1024 \times 1024\) resolution. The model is trained for 30,000 steps with a reduced learning rate of \(1 \times 10^{-6}\), batch size 64, cosine decay schedule, and weight decay 0.05, solidifying in-model verification and self-reflection capabilities.

Key Experimental Results

Main Results

UniReflect is comprehensively benchmarked across general multimodal understanding (MMBench, MMMU, MathVista), compositional text-to-image synthesis (GenEval, DPG-Bench), image editing (ImgEdit-Bench, GEdit-Bench-EN), and visual verification (ViVerBench).

Method	Parameters (MLLM + Gen)	GenEval (T2I) ↑	DPG-Bench (T2I) ↑	ImgEdit (Edit) ↑	GEdit-EN (Edit) ↑	ViVerBench (Veri) ↑
LLaVA-v1.5	13B + -	-	-	-	-	-
FLUX.1-dev	- + 12B	0.66	84.00	-	-	-
SD3-medium	- + 2B	0.62	84.08	-	-	-
Step1X-Edit	- + 8B	-	-	3.06	6.70	-
OmniVerifier	7B + -	-	-	-	-	0.559
Qwen 2.5-VL 7B	7B + -	-	-	-	-	0.523
Emu3	8B (Native)	0.54 / 0.66†	80.60	-	-	-
BAGEL	7B + 7B	0.82 / 0.88†	85.07	3.20	6.52	-
OmniGen2	3B + 4B	0.80 / 0.86†	83.57	3.44	6.41	-
UniReflect (Base)	4B + 4B	0.83 / 0.87†	86.69	3.28	5.14	0.579
UniReflect-TTS (Closed-Loop)	4B + 4B	0.89	-	-	6.07	-

(Note: † indicates prompt engineering. With in-model TTS, UniReflect reaches 0.89 on GenEval, outperforming BAGEL which is trained on 1.6B image-text pairs with 14B parameters.)

Ablation Study

The table below details the individual contributions of the shared Semantic Perceiver and the Decoupling Verification strategy on GenEval compositional alignment and ViVerBench verification accuracy:

Configuration	Semantic Perceiver	Decoupled Verification	GenEval (T2I Alignment) ↑	ViVerBench (Verification) ↑	Note
Baseline	✗	✗	0.77	0.525	Direct hidden state conditioning & lexical verification
+ Perceiver	✓	✗	0.81	0.534	Perceiver aligns multimodal and generative representations
+ Decoupling	✗	✓	0.78	0.569	Dedicated classification head stabilizes decision boundary
Full UniReflect	✓	✓	0.83 (+0.060)	0.579 (+0.054)	Combined synergy yielding optimal generation and verification

On the demanding GenEval++ benchmark, UniReflect achieves 0.654, rising to 0.680 under UniReflect-TTS. This significantly outperforms both FLUX.1-dev (0.314) and Bagel (0.371), demonstrating exceptional strength on fine-grained counting (0.798) and relative spatial positioning (Pos/Size: 0.733).

Key Findings

• High Data and Parameter Efficiency: Trained on only 3.9M text-to-image pairs and 20k verification samples with a 4B DiT generator, UniReflect outperforms models trained on orders of magnitude more data (e.g., BAGEL with 1.6B pairs and 14B parameters) across DPG-Bench (86.69 vs. 85.07) and GenEval (0.87 vs. 0.82).
• Superiority of Decoupled In-Model Verification: Achieving 0.579 on ViVerBench with a 4B backbone surpasses the 7B OmniVerifier (0.559) trained via reinforcement learning, proving that internal representation sharing is more effective than external post-hoc verifiers.
• Measurable Compute-to-Quality Conversion: Self-reflection TTS consistently boosts complex instruction adherence, raising GenEval from 0.83 to 0.89 and GEdit-EN from 5.14 to 6.07 without any external manual prompt modifications.

Highlights & Insights

• Unified In-Model Closed Loop: Eliminates the fragmented three-step pipeline of "generation \(\to\) external VLLM critique \(\to\) external image editing", establishing an autonomous self-correcting generative system within a single foundation model.
• Shared Representation Bridge: By updating only three task tokens (<verifier>, <reflect>, <img>) and training a lightweight Semantic Perceiver, the framework effectively unlocks rich multimodal knowledge from frozen foundation LLMs for generative diffusion guidance.
• Cross-Task Synergies: Verification and generation act as mutual regularizers; training on verification tasks directly improves the generator's sensitivity to spatial relations, object counts, and attribute bindings.

Limitations & Future Work

• Admitted Limitations: The out-of-domain image editing score on GEdit-EN (5.14 base) is slightly behind specialized editing systems, primarily due to the modest size (1.0M) and diversity of the image editing training split.
• Inference Search Breadth: The current TTS mechanism relies primarily on greedy sequential reflection; it does not yet explore parallel tree-search strategies (such as MCTS or Best-of-N sampling) during inference.
• Future Directions: Potential enhancements include incorporating direct preference optimization (e.g., DPO or GRPO) to train the reflection embedding end-to-end, and exploring multi-scale latent feature caching to accelerate localized diffusion refinement.

Related Work & Insights

• vs OmniGen2: While OmniGen2 pioneered assembled unified multimodal models by using VLLM hidden states for image synthesis, it functions strictly in a feed-forward manner without verification capabilities. UniReflect introduces bidirectional perception and self-reflection loops, enabling active self-correction.
• vs OmniVerifier / ReflectionFlow: These methods rely on independent 7B/13B models to serve as external critics and pass text prompts to secondary editing models, resulting in high latency and semantic bottlenecks. UniReflect unifies the verifier and generator within one 4B architecture, achieving superior performance with lower operational overhead.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering unified multimodal architecture integrating in-model verification and autonomous reflection tuning]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across understanding, compositional synthesis, editing, and verification covering 20+ baselines]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, rigorous motivation, and precise technical explanations]
• Value: ⭐⭐⭐⭐⭐ [Establishes an elegant and practical blueprint for test-time scaling in multimodal foundation models]