FlowInOne: Unifying Multimodal Generation as Image-in, Image-out Flow Matching

Conference: ECCV 2026
Paper: ECCV Official
Code: https://csu-jpg.github.io/FlowInOne.github.io
Area: Image Generation
Keywords: Image Generation, Flow Matching, Vision-centric, Visual Prompting, Instruction Following

TL;DR

By rendering heterogeneous instructions (text descriptions, bounding boxes, visual markers, and force vectors) directly onto an input image canvas, FlowInOne reformulates multimodal generation as a pure image-in, image-out continuous flow matching process within a shared isomorphic latent space.

Background & Motivation

Multimodal generation has long operated under a text-dominant paradigm: language dictates intent, and visual models passively execute conditional sampling. In conventional latent diffusion models (LDM) and diffusion transformers (DiT), text prompts are mapped by isolated text encoders (such as CLIP or Flan-T5) into discrete linguistic embeddings, which are subsequently injected into visual generative backbones via cross-attention mechanisms. This dual-tower design creates a fundamental structural asymmetry: language unilaterally guides visual synthesis, yet visual representations cannot reason or evolve autonomously within their native modality. When tasks expand from standard text-to-image synthesis to localized image editing, inpainting, and spatial control, traditional systems inevitably require auxiliary control branches (e.g., ControlNet), multi-stream condition heads, or handcrafted task interfaces, inducing severe feature alignment bottlenecks and architectural fragmentation.

Recent vision-centric explorations demonstrate that the pixel domain itself possesses sufficient representational capacity to encapsulate symbolic semantics; rendering text directly onto image canvases allows vision encoders to perform multimodal perception without auxiliary linguistic pathways. However, these investigations remain overwhelmingly confined to perceptual understanding and discrete classification, leaving the generative potential of pure visual inputs largely unexplored. Furthermore, traditional diffusion models rely on step-by-step stochastic Gaussian noise injection and denoising schedules, imposing strict Brownian motion assumptions that impair sampling efficiency and destroy fine-grained spatial and topological structures when processing directional arrows, spatial bounding boxes, or continuous physical force dynamics.

Flow matching provides a principled alternative by discarding stochastic noise schedules and directly learning a deterministic velocity field between an arbitrary source distribution and a target distribution, provided both endpoints are isomorphic. Core idea: reformulate multimodal generation into a vision-centric image-in, image-out flow matching paradigm by rendering all textual instructions, spatial boxes, directional markers, and sketches onto a unified visual prompt canvas, and learning a deterministic continuous transport from the instruction latent state to the target image latent state using a single flow matching network.

Method

Overall Architecture

FlowInOne eliminates separate text encoders and task-specific conditioning branches, establishing an end-to-end continuous flow matching pipeline within a single visual modality. The system takes a unified visual prompt canvas \(I_v\) containing all generative instructions as input and outputs the target image \(I^\star\). The architecture consists of three core components: unified visual prompt encoding into a source latent state, a flow matching transformer backbone featuring Dual-Path Spatially-Adaptive Modulation, and deterministic inference governed by ordinary differential equation (ODE) integration.

The input canvas \(I_v\) is first processed by the visual encoder of Janus-Pro-1B (based on a SigLIP Vision Transformer) to extract patch-level fused representations \(X_{\text{fuse}}\), which are mapped into a continuous source latent state \(z_0 = Z_{TI}\) by a lightweight Text-Image VAE. Symmetrically, a frozen image VAE encodes the target image \(I^\star\) into an isomorphic latent state \(z_1 = Z_I\). The flow matching backbone updates latent states via self-attention for global scene context, while an adaptive gating network modulates a cross-attention branch to selectively inject reference image structural priors. At inference time, starting from the source latent state \(z_0\), the model numerically solves the learned velocity field along the probability path to synthesize the target image latent state \(\hat{z}_1\), which is decoded by the frozen image VAE decoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input visual prompt canvas Iv<br/>text/boxes/arrows/doodles rendered"] --> B["Visual Prompt Input Unification<br/>SigLIP & Text-Image VAE yield source z0"]
    B --> C["Dual-Path Spatially-Adaptive Modulation<br/>task indicator controls gating & structure injection"]
    C --> D["Intra-Modal Isomorphic Flow Matching<br/>deterministic velocity field ODE integration"]
    D --> E["Output target image I_hat<br/>frozen image VAE decoder synthesizes pixels"]

Key Designs

1. Visual Prompt Input Unification: rendering heterogeneous control signals onto an image canvas

Conventional multimodal architectures rely on discrete token sequences or specialized adapters to handle spatial coordinates, bounding boxes, and editing descriptions, creating representational misalignment between symbolic tokens and continuous pixel grids. FlowInOne circumvents this issue by rendering textual instructions, category labels, spatial bounding boxes, directional arrows, and doodle sketches directly onto a 2D image canvas \(I_v\), ensuring all control signals natively carry spatial and geometric semantics. The framework utilizes the SigLIP vision transformer from Janus-Pro-1B followed by an MLP projector to extract patch-level fused representations: $\(X_{\text{fuse}} = \text{MLP}(\text{SigLIP}(I_v)) \in \mathbb{R}^{N \times D}\)$ A dedicated Text-Image VAE then parameterizes these visual tokens into a continuous distribution and samples the source latent state \(z_0 \sim \mathcal{N}(\bar{\mu}_{z_0}, \text{diag}(\bar{\sigma}_{z_0}^2)) \in \mathbb{R}^{H \times W \times C}\). This eliminates tokenizer-induced discretization errors and allows the network to process spatial semantics entirely within the visual domain.

2. Intra-Modal Isomorphic Flow Matching: deterministic transport in a shared continuous latent space

Standard diffusion models perturb clean images with isotropic Gaussian noise, destroying structural topological correspondence between input and target and necessitating stochastic sampling tricks during image editing. FlowInOne frames generative modeling as continuous probability transport between a non-Gaussian source distribution \(p_0\) and a target distribution \(p_1\). A frozen LDM image VAE maps the target ground-truth image \(I^\star\) into an isomorphic latent representation \(z_1 = \text{Enc}_{\text{img}}(I^\star) \in \mathbb{R}^{H \times W \times C}\). Over the continuous time interval \(t \in [0, 1]\), a deterministic linear probability path is defined: $\(z_t = t z_1 + \left(1 - (1 - \sigma_{\min})t\right)z_0\)$ The ground-truth instantaneous velocity field is \(v_t^\star = z_1 - (1 - \sigma_{\min})z_0\). A transformer network \(v_\theta(z_t, t)\) is trained to approximate this velocity vector field, converting generative inference into an initial value ODE problem \(\frac{\mathrm{d}z_t}{\mathrm{d}t} = v_\theta(z_t, t)\) solved from \(t=0\) to \(t=1\). Operating within a shared isomorphic latent space eliminates cross-modal alignment losses and noise decay schedules, stabilizing training and ensuring smooth generative trajectories.

3. Dual-Path Spatially-Adaptive Modulation: decoupling text generation and editing via dynamic gating

Due to intrinsic compression in visual encoding and variational sampling, the initial latent \(z_0\) alone can lose fine-grained high-frequency structural details of the source image \(I_{\text{src}}\). Unconditionally injecting source features, however, introduces irrelevant structural noise in pure text-to-image synthesis and causes over-preservation conflicts in localized editing. FlowInOne resolves this via a dual-path mechanism gated by a binary task indicator \(\mathbb{I}_{\text{edit}} \in \{0, 1\}\). For image editing tasks involving a source image, an Image VAE encodes \(I_{\text{src}}\) into latent features flattened as a reference sequence \(\mathbf{S} \in \mathbb{R}^{N \times C}\). With the self-attention updated state \(\tilde{\mathbf{H}}^{(l)}\) as Query, the structural increment is computed via cross-attention: $\(\Delta \mathbf{H}_{\text{struct}} = \text{Softmax}\left(\frac{(\tilde{\mathbf{H}}^{(l)}\mathbf{W}_Q)(\mathbf{S}\mathbf{W}_K)^\top}{\sqrt{d_k}}\right)(\mathbf{S}\mathbf{W}_V)\)$ To prevent source features from corrupting edited regions, a lightweight adaptive gating network concatenates the intermediate feature and structural increment to predict an anisotropic token-level weight vector \(\boldsymbol{\Lambda} = \sigma(\text{Linear}([\tilde{\mathbf{H}}^{(l)} \parallel \Delta \mathbf{H}_{\text{struct}}])) \in [0, 1]^N\). The final layer representation is formulated as: $\(\mathbf{H}_{\text{out}}^{(l)} = \tilde{\mathbf{H}}^{(l)} + \mathbb{I}_{\text{edit}} \cdot (\boldsymbol{\Lambda} \odot \Delta \mathbf{H}_{\text{struct}})\)$ For pure text-to-image inputs (\(\mathbb{I}_{\text{edit}} = 0\)), the structural branch is fully bypassed to allow autonomous semantic evolution. For image editing (\(\mathbb{I}_{\text{edit}} = 1\)), \(\boldsymbol{\Lambda}\) selectively suppresses source priors in modified areas while preserving the background manifold, dynamically balancing original fidelity with instruction compliance.

Loss & Training

FlowInOne contains 1.2B parameters and is initialized from the CrossFlow architecture. The model is trained jointly on the VisPrompt-5M dataset covering all 8 visual instruction tasks (text-to-image, class-to-image, text-in-image editing, bounding box editing, visual marker editing, doodles editing, force dynamics, and trajectory prediction). Training is conducted at \(256 \times 256\) resolution for 240k steps using balanced WebDataset loaders with a batch size of 512. The overall objective combines flow matching MSE, variational KL divergence, and CLIP contrastive regularizers: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{FM}} + \lambda_{\text{KL}}\mathcal{L}_{\text{KL}} + \lambda_{\text{CLIP}}\mathcal{L}_{\text{CLIP}}\)$ where the core flow matching loss is: $\(\mathcal{L}_{\text{FM}} = \mathbb{E}_{(z_0, z_1), t}\left[ \| v_\theta(z_t, t) - (z_1 - z_0) \|_2^2 \right]\)$ Joint training across diverse prompt types prevents catastrophic forgetting observed in multi-stage setups, forcing the model to learn semantic synthesis, geometric alignment, and physical rules within a unified continuous flow space.

Key Experimental Results

Main Results

Evaluation is performed on VP-Bench, a curated benchmark covering diverse visual instruction categories. Because traditional metrics like FID fail to evaluate fine-grained spatial and physical instruction adherence, evaluations employ advanced multimodal foundation models (Gemini 3, GPT 5.2, Qwen 3.5) alongside human evaluations. An output is deemed successful (Pass) only if it simultaneously satisfies four criteria: Instruction Faithfulness (IF), Content Consistency (CC), Visual Realism (VR), and Spatial Precision (SP). The success rates across evaluators and subcategories are summarized below.

Evaluator / Method	C2I	T2I	TIE	FU	TBE	TU	VME	DE	Total Pass Rate
Evaluator: Gemini 3
Nano Banana (Commercial)	0.810	0.980	0.521	0.500	0.600	0.020	0.537	0.740	0.589
OmniGen2	0.720	0.760	0.313	0.013	0.020	0.000	0.020	0.140	0.248
FLUX.1-Kontext-dev	0.620	0.700	0.363	0.027	0.163	0.020	0.096	0.180	0.271
Qwen-Image-Edit-2509	0.680	0.690	0.383	0.047	0.060	0.000	0.040	0.160	0.258
FlowInOne (Ours)	0.890	0.700	0.355	0.727	0.302	0.520	0.292	0.535	0.540
Evaluator: GPT 5.2
Nano Banana (Commercial)	0.760	0.960	0.402	0.163	0.100	0.020	0.227	0.495	0.391
OmniGen2	0.660	0.820	0.203	0.001	0.000	0.000	0.001	0.160	0.231
FLUX.1-Kontext-dev	0.620	0.690	0.266	0.013	0.093	0.000	0.056	0.160	0.237
Qwen-Image-Edit-2509	0.640	0.680	0.286	0.040	0.020	0.020	0.020	0.140	0.231
FlowInOne (Ours)	0.850	0.800	0.079	0.500	0.116	0.240	0.083	0.465	0.392
Evaluator: Human
Nano Banana (Commercial)	0.790	0.940	0.372	0.287	0.220	0.020	0.306	0.740	0.459
OmniGen2	0.720	0.710	0.268	0.013	0.020	0.000	0.010	0.120	0.233
FLUX.1-Kontext-dev	0.640	0.680	0.317	0.013	0.080	0.020	0.048	0.120	0.240
Qwen-Image-Edit-2509	0.700	0.665	0.331	0.047	0.020	0.000	0.023	0.160	0.243
FlowInOne (Ours)	0.800	0.645	0.242	0.705	0.255	0.280	0.255	0.400	0.449

In fine-grained spatial and physical structure evaluation using dense DINOv3 feature similarity, FlowInOne achieves an average DINOv3 Sim score of 0.487, surpassing Nano Banana (0.473) and significantly outperforming all open-source models (OmniGen2 0.125, Kontext 0.161, Qwen-IE-2509 0.201). The advantage is especially pronounced in force understanding (0.536 vs 0.474) and trajectory understanding (0.570 vs 0.486). Furthermore, FlowInOne achieves the highest Spatial Precision (SP) scores across all three VLM evaluators (3.42 on Gemini 3, 3.24 on GPT 5.2, and 3.30 on Qwen 3.5).

Ablation Study

Ablation studies analyze the impact of feature compression strategies, spatial modulation designs, and data training schedules. Metrics reflect the average pass rate across Gemini 3, GPT 5.2, and Qwen 3.5 evaluators.

Category	Ablation Setting	Gemini 3 ↑	GPT 5.2 ↑	Qwen 3.5 ↑	Avg Pass Rate ↑	Key Findings & Notes
Compression Strategy	MLP + truncation	0.179	0.153	0.176	0.169	Truncating token sequences discards critical spatial and edge information
	VAE expansion	0.169	0.147	0.155	0.157	Increasing VAE layers introduces excessive optimization difficulty
	MLP + MLP	0.192	0.170	0.185	0.182	Dual MLP projections preserve sequence length, channels, and spatial topology
Modulation Method	Without cross-attention (Wo CA)	0.192	0.170	0.185	0.182	Relying solely on self-attention fails to anchor source image structures
	With Dual-Path CA	0.227	0.185	0.229	0.214	Cross-attention injects structural priors but lacks spatial selectivity
	Dual-Path SAM	0.242	0.214	0.238	0.231	Token-level adaptive gating balances background preservation and editability
Training Strategy	Two-stage training	0.336	0.256	0.283	0.291	Sequential pretraining on generation followed by editing causes catastrophic forgetting
	Joint training	0.540	0.392	0.503	0.478	Jointly training on all 5M samples across 8 tasks yields an 18.7% performance leap

Key Findings

• Dominance in physical and trajectory control: Existing text-driven generative pipelines fail on force dynamics (FU) and motion trajectory (TU) tasks (OmniGen2 scores 0.000 on TU; Nano Banana reaches only 0.020). Discrete textual descriptions cannot convey continuous vector angles and magnitudes. By rendering arrows and force markers directly as visual prompt flows, FlowInOne achieves a 72.7% success rate on force understanding and 52.0% on trajectory understanding under Gemini 3.
• Adaptive gating prevents over-preservation artifacts: Ablation results confirm that unmodulated cross-attention (W Dual-Path CA) tends to copy source objects into edit zones; Dual-Path SAM uses spatial weight vector \(\boldsymbol{\Lambda}\) to isolate edited regions while maintaining background manifolds, boosting average score from 0.214 to 0.231.
• Unified joint training trumps stage-wise tuning: Mixing 5 million diverse multimodal samples from inception achieves a 47.8% pass rate, substantially outperforming two-stage training (29.1%). In a shared continuous visual space, generative priors and geometric manipulation tasks reinforce one another rather than competing.

Highlights & Insights

• Converting discrete control to continuous visual primitives: Eliminating complex task heads and adapters by drawing text prompts, boxes, arrows, and doodles directly onto an image canvas leverages 2D spatial inductive biases inherent in vision backbones, eliminating cross-modal alignment gaps.
• Noise-free transport via isomorphic endpoints: Unlike conventional diffusion models that map images to Gaussian white noise, FlowInOne adopts a non-Gaussian initial state \(z_0\) extracted from the visual canvas and a target state \(z_1\) from the target image. Straight ODE trajectories in isomorphic latent space drastically accelerate sampling and enhance structural consistency.
• Extensible paradigm for physical simulation: Representing multi-sensor controls, robotic end-effector forces, and autonomous driving trajectories as visual prompt drawings offers a clean blueprint for embodied AI simulation, world modeling, and interactive CAD manipulation.

Limitations & Future Work

• Author-admitted limitations: In scenarios with dense, lengthy text instructions, the SigLIP vision encoder occasionally experiences character occlusion or semantic misreading; additionally, trained at \(256 \times 256\) resolution, high-frequency photorealistic texture rendering slightly lags top commercial closed-source engines such as Nano Banana.
• Empirical scope: Although the model demonstrates strong physical understanding, the majority of force and trajectory training pairs are derived from synthetic Blender simulations; real-world robustness under extreme lighting, occlusions, and non-rigid physical collisions requires further validation.
• Future directions: Exploring native multi-resolution vision transformers, applying flow trajectory distillation for one-step inference, and expanding the single canvas into temporal video streams to construct interactive visual world models.

Related Work & Insights

• vs Conventional text-guided generation & editing (e.g., Stable Diffusion, InstructPix2Pix, Prompt-to-Prompt): Traditional methods depend heavily on separate text encoders and cross-attention map manipulations, providing weak spatial and physical constraints; FlowInOne visualizes all conditions on canvas and uses deterministic flow matching, significantly improving spatial precision (SP).
• vs Unified multimodal generation baselines (e.g., OmniGen2, FLUX.1-Kontext): Existing multi-task frameworks retain multimodal tokenizers and discrete language pathways; FlowInOne completely discards text encoders, realizing a pure image-in, image-out pipeline that simplifies architectures and stabilizes optimization.
• vs Cross-modal flow matching (e.g., CrossFlow): While CrossFlow explored continuous transport from text embeddings to image pixels across heterogeneous modalities, FlowInOne maps inputs into an isomorphic latent space via Text-Image VAE, minimizing transport path curvature and velocity estimation errors.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Resolves cross-modal asymmetry by reformulating multimodal generation into an intra-modal image-in, image-out flow matching process.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces the 5M-sample VisPrompt-5M dataset and VP-Bench, rigorously combining three frontier VLMs, human blind testing, and comprehensive ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, intuitive conceptual flow, and informative diagrams and tables.
• Value: ⭐⭐⭐⭐⭐ Establishes a foundational paradigm for vision-centric generative modeling, offering impactful implications for physical simulation and autonomous systems.