DiTailed: Ensuring Visual Object Consistency in Text-Image-to-Image Flow Matching Models

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Project Page: https://francescotaioli.github.io/DiTailed
Area: Image Generation
Keywords: Rectified Flow Models, Visual Object Consistency, Image Editing, Multimodal DiT, Multi-Scale Auxiliary Supervision

TL;DR

Addressing the frequent degradation of visual object consistency in text-guided image editing, this paper introduces ABO-Edit—a benchmark rendered from 3D assets with degree-level rotation control—and uncovers that rectified flow models inherently encode target predictions in the conditioning latent space at early denoising stages, leading to FlowMirror, a parameter-free multi-scale auxiliary loss that markedly enhances visual fidelity and fine-grained consistency.

Background & Motivation

With the rapid evolution of large-scale multimodal generative models, instruction-guided image editing has witnessed significant breakthroughs. However, in mission-critical applications such as e-commerce product visualization, digital 3D asset production, and novel view synthesis, state-of-the-art models persistently struggle to maintain visual object consistency. This property requires that an object's intrinsic attributes—such as geometry, surface textures, component details, and color palettes—remain strictly preserved throughout the editing trajectory, except where explicit modifications are mandated by the text prompt. When tasked with editing objects situated in cluttered real-world environments, existing foundation models frequently drop fine-grained components (such as chair cup-holders or rivets), distort structural geometry, or inadvertently overwrite genuine product textures with generic statistical priors.

This persistent failure stems from two fundamental dilemmas. The first lies in data-level supervision: existing instruction editing datasets rely primarily on synthetic diffusion pipelines or noisy web-crawled pairs, lacking paired instances that simultaneously feature complex real-world clutter and pixel-accurate ground-truth geometry under precise 3D transformations. Consequently, isolating viewpoint rotation errors from texture degradation has been infeasible. The second bottleneck involves architectural representation learning: standard multimodal Diffusion Transformers (MM-DiTs) concatenate noisy target tokens with conditioning image tokens across dozens of joint attention blocks, yet the final conditioning latent is discarded entirely during inference. The network relies solely on the noisy stream's velocity field to infer alignment, failing to provide explicit structural anchors for the conditioning representation.

This paper tackles the challenge by delving into the internal representational dynamics of rectified flow editing architectures. The authors unveil an intriguing emergent phenomenon: in specific model families, the conditioning embedding space spontaneously encodes a prediction of the clean target image even at the earliest, noise-dominated timesteps (\(t \approx 1\)), surpassing the instantaneous velocity field in early semantic similarity. Core idea: curate ABO-Edit, a 12K+ multi-view benchmark rendered from CAD 3D models with degree-level annotations, and propose FlowMirror, a parameter-free auxiliary loss that applies Laplacian-pyramid multi-scale supervision directly to the conditioning embedding space in VAE latent coordinates, enforcing early visual mirroring with negligible overhead.

Method

Overall Architecture

DiTailed targets the conversion of cluttered "lifestyle" photographs into standardized, studio-grade product presentations on a white backdrop under precise rotation and elevation angles. The backbone adopts a Multimodal Diffusion Transformer (MM-DiT) coupled with a Rectified Flow continuous-time formulation. During training, the pipeline processes noisy target latents \(z_{\text{noisy}}\), reference condition latents \(z_{\text{cond}}\), timestep \(t\), and text prompt embeddings \(z_{\text{txt}}\). Inside the transformer, image tokens and conditioning tokens undergo joint self-attention across \(L\) stacked blocks. At the output, the noisy branch \(h_{\text{noisy}}^L\) is projected to predict the velocity vector field \(v\), while the conditioning branch \(h_{\text{cond}}^L\) is supervised against the clean target latent \(z\) via the proposed FlowMirror multi-scale pyramid loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data<br/>Reference image x_cond + text instruction p + target latent x_0"] --> B["Latent Encoding & Patch Packing<br/>VAE encodes z_cond and z_0; linear noise flow z_t formed"]
    B --> C["ABO-Edit Precise Viewpoint & Annotation<br/>12K+ triplets provide degree-level rotation and clean 3D renders"]
    C --> D["MM-DiT Joint Sequence Forward Pass<br/>Concatenated noisy and condition tokens interact across L layers"]
    D --> E["Dual Representation Splitting<br/>Last hidden state splits into velocity features h_noisy and condition features h_cond"]
    E --> F["Standard Rectified Flow CFM Loss<br/>h_noisy projected to predict constant velocity field v"]
    E --> G["FlowMirror Multi-Scale Conditioning Supervision<br/>High-frequency L1 detail loss + low-frequency L2 structural loss"]
    F --> H["Joint Backpropagation & Inference Convergence<br/>Zero extra inference parameters; stabilizes target geometry early"]
    G --> H

Key Designs

1. ABO-Edit Benchmark Construction: Grounded Multi-View Triplets from Artist-Designed 3D Assets

Real-world scene editing requires robustly disentangling subject objects from severe background clutter. Existing benchmarks fail to offer clean 3D geometric ground truths. The authors construct ABO-Edit from the Amazon Berkeley Objects dataset, which contains over 7,900 artist-created 3D CAD assets with 4K textures. The creation pipeline begins with product-level filtering to exclude transparent materials and minimal-transform categories, followed by a CLIP-based classifier that filters out plain studio shots to isolate authentic lifestyle source images (achieving 90% precision and 82% recall). For each retained product, high-resolution (\(1024 \times 1024\)) targets are programmatically rendered using Blender across front, left, and right canonical views with physically realistic floor shadows. A vision-language model (Qwen3-VL-8B) generates fine-grained attribute prompts, and exact Blender camera orientations are converted into degree-level azimuth and elevation instructions. Following crowdsourced quality verification, the benchmark provides 12,319 training and 400 validation triplets across 20 common household categories.

2. Latent Space Predictive Discovery: Emergence of Ahead-of-Time Target Predictions

In standard MM-DiT formulations, the conditioning hidden state \(h_{\text{cond}}^L\) serves purely as an attention source and is discarded at inference, despite receiving backward gradients throughout training. By decoding \(h_{\text{cond}}^L\) across timesteps \(t \in [0, 1]\) using the pretrained VAE decoder, the authors discovered that models such as Qwen-Image-Edit naturally encode a coarse reconstruction of the final target image inside \(h_{\text{cond}}^L\) even under severe noise (\(t \approx 1\)). Furthermore, evaluating the single-step reconstruction derived from the rectified flow trajectory \(\hat{x}_{0,t} = x_t - t v_\theta(x_t, t)\) reveals that the CLIP similarity between \(h_{\text{cond}}^L\) and the final target reaches 0.823 at \(t = 1\), substantially outpacing the velocity field estimate (0.723). This indicates an inherent inclination to utilize the conditioning sequence as an internal canvas for target layout, though left unsupervised, this internal prediction remains geometrically unstable.

3. FlowMirror Auxiliary Loss: Multi-Scale Frequency-Decoupled Latent Supervision

Building on the predictive latent phenomenon, FlowMirror enforces direct auxiliary supervision on \(h_{\text{cond}}^L\) using the clean target's VAE latent code \(z\). Operating directly on unpacked spatial tensors \(\mathbb{R}^{c \times h \times w}\) avoids invoking the computationally heavy image-space decoder during training. To prevent naive mean squared error from biasing the network toward blurry low-frequency averages, FlowMirror deploys a Laplacian-pyramid-style decomposition over \(S\) scales (empirically optimal at \(S=2\)). At each scale \(s\), depthwise per-channel Gaussian filtering \(\mathcal{G}(\cdot)\) separates high-frequency residuals, which are penalized via an \(\ell_1\) norm to enforce fine edge crispness, micro-textures, and structural seams. Simultaneously, the smoothed representation is downsampled via average pooling \(\mathrm{Down}(\cdot)\) and penalized using an \(\ell_2\) norm to lock in global geometric topology and illumination balance.

Loss & Training

The overall training objective combines the standard Conditional Flow Matching (CFM) velocity loss with the FlowMirror auxiliary regularization:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CFM}} + \lambda_{\text{aux}} \mathcal{L}_{\text{aux}}\]

For scale level \(s\), the high-frequency detail loss \(\mathcal{L}_{\text{HF}}^{(s)}\) and low-frequency structural loss \(\mathcal{L}_{\text{LF}}^{(s)}\) are formulated as:

\[\mathcal{L}_{\text{HF}}^{(s)} = \left\| \bigl(h_{\text{cond}}^{L,(s)} - \mathcal{G}(h_{\text{cond}}^{L,(s)})\bigr) - \bigl(z^{(s)} - \mathcal{G}(z^{(s)})\bigr) \right\|_1\]

\[\mathcal{L}_{\text{LF}}^{(s)} = \left\| \mathrm{Down}\left(\mathcal{G}(h_{\text{cond}}^{L,(s)})\right) - \mathrm{Down}\left(\mathcal{G}(z^{(s)})\right) \right\|_2^2\]

The total auxiliary loss averages across all \(S\) pyramid scales with a balancing factor \(\lambda_{\text{HF}}\):

\[\mathcal{L}_{\text{aux}} = \frac{1}{S} \sum_{s=1}^S \left( \mathcal{L}_{\text{LF}}^{(s)} + \lambda_{\text{HF}} \cdot \mathcal{L}_{\text{HF}}^{(s)} \right)\]

Optimization is performed using Weight-Decomposed Low-Rank Adaptation (DoRA, rank \(r=64\), scaling factor \(\alpha=64\)) at a learning rate of \(10^{-4}\). Training spans 10,000 steps with an effective batch size of 32 on a single node with 8 NVIDIA H100 GPUs, processing images at \(1024 \times 1024\) resolution. Hyperparameter ablations establish that \(S=2\), \(\lambda_{\text{HF}}=0.2\), and \(\lambda_{\text{aux}}=0.01\) deliver optimal performance, introducing subtle yet potent structural anchoring without derailing the primary CFM velocity trajectory.

Key Experimental Results

Main Results

Evaluation on the 400-sample ABO-Edit validation set spans generative quality (FID), structural similarity (SSIM, PSNR), perceptual fidelity (LPIPS), and semantic alignment (CLIP, DINOv2). All fine-tuned models report the mean and standard deviation across 3 separate training runs.

Method	FID ↓	SSIM ↑	PSNR ↑	LPIPS ↓	CLIP ↑	DINOv2 ↑
Zero-shot Baselines
Kandinsky 5.0	88.60 ±1.15	0.644 ±.004	9.14 ±.08	0.579 ±.007	83.55 ±.38	56.74 ±1.04
FLUX.1 [dev]	59.32 ±0.39	0.789 ±.001	12.17 ±.06	0.345 ±.003	90.76 ±.10	78.14 ±0.25
FLUX.2 [dev]	53.67 ±0.30	0.803 ±.000	12.80 ±.06	0.320 ±.000	92.66 ±.00	83.67 ±0.25
Qwen-Image-Edit-2511	64.23 ±0.47	0.805 ±.001	12.73 ±.03	0.336 ±.000	89.17 ±.22	77.83 ±0.31
Qwen-Image-Edit-2509	57.66 ±0.02	0.818 ±.005	12.76 ±.14	0.312 ±.007	91.95 ±.07	82.30 ±0.06
Fine-tuned on ABO-Edit
FLUX.1 [dev] (DoRA)	31.68 ±0.18	0.883 ±.002	18.62 ±.05	0.149 ±.003	96.20 ±.10	90.99 ±0.37
FLUX.1 + FlowMirror (Ours)	31.26 ±0.18	0.883 ±.001	18.67 ±.05	0.148 ±.002	96.30 ±.10	91.25 ±0.13
Qwen-2509 (DoRA)	26.48 ±0.86	0.893 ±.001	19.64 ±.03	0.112 ±.002	97.21 ±.08	93.98 ±0.28
Qwen-2509 + FlowMirror (Ours)	25.91 ±0.20	0.894 ±.001	19.79 ±.08	0.111 ±.001	97.17 ±.05	93.94 ±0.22

Furthermore, spatial rotation accuracy is benchmarked using Orient-Anything-v2 to evaluate absolute azimuth angular error (in degrees) across viewpoint buckets:

Model	Left Azimuth Error (°) ↓	Front Azimuth Error (°) ↓	Right Azimuth Error (°) ↓
Kandinsky 5.0 (Zero-shot)	47.2 ±25.8	21.0 ±22.0	37.2 ±26.4
FLUX.1 [dev] (Zero-shot)	21.7 ±22.0	23.7 ±17.7	38.9 ±20.6
FLUX.2 [dev] (Zero-shot)	48.2 ±25.2	11.9 ±13.6	42.2 ±21.0
Qwen-2509 (Zero-shot)	39.1 ±28.0	18.5 ±15.1	31.7 ±17.2
Qwen-2509 + FlowMirror (Fine-tuned)	2.4 ±10.2	1.0 ±2.2	1.2 ±0.9

Ablation Study

Ablations on Qwen-Image-Edit-2509 evaluate specific design choices under a single fixed seed on the ABO-Edit validation split:

Configuration	FID ↓	Note
Default full model (\(S=2, \lambda_{\text{HF}}=0.2, \lambda_{\text{aux}}=0.01\))	25.55	Optimal balance of geometry and micro-texture
Higher auxiliary weight (\(\lambda_{\text{aux}} = 0.1\))	25.93	Excessive regularization slightly perturbs CFM velocity flow
w/o Scale normalization	26.52	Unscaled pyramid terms unbalance gradient magnitudes
Remove high-frequency term (\(\lambda_{\text{HF}} = 0\))	27.07	Largest performance drop; verifies criticality of detail preservation
Overweighted high-frequency term (\(\lambda_{\text{HF}} = 1.0\))	26.40	Excessive detail pressure conflicts with low-frequency structural learning
Channel mean & std matching	26.15	Statistical alignment destroys spatial correspondence
Deeper pyramid levels (\(S > 2\))	26.24	Excessive downsampling diminishes meaningful spatial layout

Key Findings

• High-frequency supervision is indispensable: Eliminating high-frequency regularization (\(\lambda_{\text{HF}}=0\)) yields the worst FID (27.07 vs. 25.55), establishing that standard MSE supervision disproportionately rewards smooth low-frequency convergence while failing to preserve subtle patterns, stitches, and joints.
• Cross-architecture generalization: FlowMirror yields consistent gains not only on Qwen (whose conditioning latents naturally anticipate the target), but also on FLUX.1, which natively preserves the reference conditioning image throughout denoising (FID improves from 31.68 to 31.26). This demonstrates FlowMirror's general utility as an architectural regularizer.
• Mastery of spatial orientation: Zero-shot foundation models exhibit bimodal azimuth error distributions with large deviations (\(20^\circ \sim 48^\circ\)), frequently inverting left and right orientations. Supervised task adaptation shrinks angular discrepancies down to \(1.0^\circ \sim 2.4^\circ\), with human evaluation preferring the proposed approach by 4.4 percentage points over the baseline.

Highlights & Insights

• Incisive representation diagnostics: Rather than treating conditioning outputs as discarded artifacts, the work pioneers intermediate latent decoding to expose internal memory and predictive behaviors in rectified flow architectures.
• Zero-overhead plug-and-play design: FlowMirror alters no architectural dimensions, incurs zero inference compute or parameter cost, and operates entirely in latent coordinates without decoding, achieving steady perceptual gains at minimal training cost.
• High-fidelity 3D-to-2D asset benchmark: ABO-Edit bridges the gap between uncontrolled web editing datasets and exact physical 3D ground truths, providing a valuable testbed for object consistency, novel view synthesis, and studio-to-lifestyle inverse tasks.

Limitations & Future Work

• Synthetic-to-real domain gap: Although rendered with realistic shadows and 4K textures from artist-built CAD assets, target renders still exhibit subtle material and lighting discrepancies compared to authentic studio photography.
• Parameter-efficient adaptation budget: Constrained by compute, fine-tuning relied on DoRA under a 10k-step ceiling; full-parameter adaptation on larger clusters may unlock even stronger latent representation convergence.
• Broader domain scope: Current benchmarks focus primarily on rigid and semi-rigid e-commerce household goods; scaling the approach to articulate subjects (humans, animals) undergoing non-rigid deformations remains an open exploration.

Related Work & Insights

• vs Attention-manipulation methods (MasaCtrl, Prompt-to-Prompt): Prior zero-shot approaches hijack cross- and self-attention keys and values, which struggle to accommodate large spatial rotations. DiTailed optimizes end-to-end flow matching dynamics with multi-scale frequency constraints, reconciling wide-angle reorientation with detail preservation.
• vs Instruction editing datasets (InstructPix2Pix, MagicBrush): Predecessors depend heavily on synthetic diffusion models or uncalibrated human web edits lacking precise 3D camera geometry. ABO-Edit delivers exact degree-level camera ground truths rendered from artist-grade CAD models.
• vs One-step flow distillation (TReFT): While works like TReFT target single-step inference through distillation, DiTailed diagnoses the internal temporal evolution of flow matching models and introduces a parameter-free auxiliary objective.

Rating

• Novelty: ⭐⭐⭐⭐ [First study uncovering internal target prediction within conditioning latents of rectified flow models and translating it into a frequency-decoupled auxiliary loss]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Introduces a 12K+ multi-view 3D-rendered benchmark, benchmarks multiple foundation architectures, and provides extensive ablation and angular control diagnostics]
• Writing Quality: ⭐⭐⭐⭐⭐ [Extremely lucid narrative, rigorous theoretical and empirical analysis, accompanied by clean formulations and insightful visualizations]
• Value: ⭐⭐⭐⭐ [Offers an effective, zero-overhead training technique and a benchmark directly applicable to generative image editing and e-commerce visual standardization]

Related Papers

• [ECCV 2026] FlowInOne: Unifying Multimodal Generation as Image-in, Image-out Flow Matching
• [ECCV 2026] ParaFlow: Parallel Sampling for Flow Matching Models
• [ECCV 2026] Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning
• [ECCV 2026] Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
• [ECCV 2026] RankT2I: A Submodular Framework for Discovering Interpretable and Diverse Semantics in Text-to-Image Models