InnoText: A Unified Model for Visual Text Generation and Editing

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: visual text generation / visual text editing / Diffusion Transformer / font size-aware modulation / bilingual text benchmark

TL;DR

Built on the Flux.1 Fill Diffusion Transformer backbone, InnoText unifies visual text generation and localized editing within a single model via a diptych inpainting paradigm, font size-aware modulation (FSAM), and small-character aware augmentation (SCAA).

Background & Motivation

Diffusion models have achieved remarkable success in photorealistic image synthesis, yet their application to visual text generation and visual text editing remains constrained by substantial fidelity bottlenecks. Visual text demands rigorous structural regularity, stroke topological geometry, inter-character spacing consistency, and semantic legibility. Conventional UNet-based models, such as AnyText and AnyText2, rely on limited convolutional receptive fields and shallow semantic binding, which frequently trigger localized artifacts, stroke distortions, and blurred glyph contours when handling complex textures, non-planar surfaces, or long text sequences. Meanwhile, emerging Diffusion Transformer (DiT) models such as TextFlux have demonstrated superior fine-grained modeling capacity, but existing architectures are almost exclusively engineered for a single task—either generation or editing—leading to fragmented training pipelines, inconsistent visual styles, and limited cross-task feature sharing.

A further compounding challenge arises from intricate glyph structures and extreme font-scale variations. Prevailing frameworks are predominantly tailored for Latin scripts with relatively simple alphabetic geometries, exhibiting severe vulnerability when exposed to non-Latin scripts such as Chinese. Chinese characters feature intricate logographic topologies, complex radical combinations, and high stroke densities. When characters shrink to micro-scales, spatial downsampling operations inevitably discard critical sub-pixel topological details, causing models to produce illegible stroke clumps. Compounding these algorithmic hurdles is the acute scarcity of high-resolution, aesthetically refined bilingual (Chinese-English) benchmark datasets, where existing corpora suffer from either skewed linguistic distributions, low aesthetic quality, or coarse spatial-textual alignments.

To dismantle these dual barriers of task fragmentation and multi-scale degradation, this paper introduces InnoText, the first unified DiT framework capable of executing visual text generation and editing within a shared backbone. The core insight lies in re-framing both tasks into an in-context learning inpainting formulation through a diptych spatial concatenation layout, while explicitly injecting continuous font-scale priors into the latent feature space. Core idea: unify visual text generation and editing into a single DiT inpainting paradigm via a diptych in-context formulation, coupled with continuous Font Size-Aware Modulation (FSAM) and foveal Small-Character Aware Augmentation (SCAA) to prevent topological collapse on micro-scale and complex Chinese glyphs.

Method

Overall Architecture

InnoText is built upon the open-weight Flux.1 Fill dev backbone, an expressive flow-matching diffusion transformer tailored for masked image inpainting. Taking advantage of its native conditioning capabilities, InnoText formulates both visual text generation and visual text editing into an in-context diptych spatial arrangement without requiring any dual-branch architectural surgery or external control networks.

The spatial inputs are decomposed into three primary components: the reference or masked image \(I_{\text{masked}}\), the rendered glyph condition image \(I_{\text{glyph}}\), and the spatial binary mask \(M\). The model concatenates the glyph image and canvas in the spatial dimension: - Editing Mode: The image input is structured as \(I_{\text{input}} = \text{Concat}([I_{\text{glyph}}, I_{\text{masked}}], \text{dim}=0)\) and the mask condition is set to \(I_{\text{mask}} = \text{Concat}([\mathbf{0}, M], \text{dim}=0)\). Here, the network performs localized synthesis strictly within the masked bounding contours while preserving the unaltered background context; - Generation Mode: The canvas is initialized as an empty black panel, yielding \(I_{\text{input}} = \text{Concat}([I_{\text{glyph}}, \mathbf{0}], \text{dim}=0)\), accompanied by an entirely open generation mask \(I_{\text{mask}} = \text{Concat}([\mathbf{0}, \mathbf{255}], \text{dim}=0)\). Under this setting, the model synthesizes a complete, coherent scene from scratch conditioned on the glyph prompts.

The overall pipeline is illustrated in the flowchart below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Diptych Input Construction<br/>Concatenate glyph and canvas/mask"] --> B["Font Size-Aware Modulation (FSAM)<br/>Derive continuous size map & calibrate latents"]
    B --> C["Small-Character Aware Augmentation (SCAA)<br/>Probabilistic geometric rescaling on micro glyphs"]
    C --> D["DiT Flow Matching Backbone<br/>Multimodal attention across visual-text latents"]
    D --> E["Task-Specific Region Weighted Loss<br/>Local precision for editing & global coherence for generation"]

Key Designs

1. Font Size-Aware Modulation (FSAM): Adaptive latent calibration via continuous grayscale size maps

A prominent bottleneck in visual text synthesis is the scale sensitivity imbalance between prominent headline typography and diminutive background text. Standard adaptive normalization layers apply spatially uniform affine parameters across the latent space, causing attention mechanisms to gravitate toward dominant large text regions and leaving small characters illegible. FSAM overcomes this limitation by abandoning crude binary masks in favor of a continuous grayscale size map. For every detected character instance, the approximate font height \(F_h\) is computed from its bounding box dimensions:

\[F_h = \max(\min(h_{\text{bbox}}, w_{\text{bbox}}), 1)\]

To explicitly prioritize small-scale characters, the inverse font height is assigned to the corresponding spatial bounding coordinates and normalized globally across the range \([0, 1]\). This continuous size map is patchified, flattened into a sequence \(S\), and projected via a linear layer into latent size embeddings \(S'\) matching the hidden state dimension of the DiT blocks. Two parallel MLP branches decode \(S'\) into dynamic scale factors \(\gamma(s)\) and shift vectors \(\beta(S')\):

\[\gamma(s) = 1 + \gamma_{\text{max}} \cdot \sigma(k(s - 0.5)), \quad \beta(S') = \lambda_{\text{shift}} \cdot \tanh(\text{MLP}_{\text{shift}}(S'))\]

The modulated hidden state is computed via element-wise transformation \(H' = H \odot \gamma(s) + \beta(S')\). This dynamic rescaling injects explicit geometric scale priors directly into the transformer blocks, ensuring scale-invariant legibility and sharp stroke delineation across heterogeneous font dimensions.

2. Small-Character Aware Augmentation (SCAA): Foveal spatial magnification for sub-pixel stroke distillation

For complex non-Latin characters possessing dense radical topologies, latent modulation alone cannot fully prevent stroke clumping under extreme spatial compression. SCAA introduces a biological foveal attention mechanism into the training pipeline. Whenever a character's estimated font height falls below a predefined geometric threshold \(\tau\), a stochastic spatial magnification operation is applied using a base scale factor \(\lambda\) perturbed by bounded noise \(\delta \sim \mathcal{U}(-\epsilon, \epsilon)\):

\[R' = \text{Resize}(R, \lambda + \delta)\]

This stochastic perturbation exposes the model to multi-resolution glyph representations during training, compelling the network to distill robust sub-pixel structural priors that resist stroke degradation. During inference, this augmentation is applied symmetrically in editing tasks to ensure strict train-test distribution alignment. For full-scene text generation, the magnification is bypassed to preserve the natural font size hierarchy and global typographic layout of the target reference.

3. Task-Specific Region Weighted Loss: Decoupling localized fidelity from global contextual realism

Unifying generation and editing within a single architecture introduces conflicting gradient dynamics: editing demands fine-grained, localized pixel precision over the modified area without leaking artifacts into unmasked regions, whereas generation requires holistic, scene-wide visual coherence and compositional harmony. To reconcile these divergent objectives, InnoText introduces an adaptive region-weighted loss tailored to the active task configuration:

\[\mathcal{L}_{\text{total}} = \begin{cases} \| v_t - u_t \|^2 \odot W_{\text{size}}, & \text{if Editing Mode} \\ \| v_t - u_t \|^2, & \text{if Generation Mode} \end{cases}\]

where \(v_t\) represents the predicted velocity field, \(u_t\) is the ground-truth flow-matching target, and \(W_{\text{size}}\) denotes a spatial weighting map derived from the continuous font size map. Under editing mode, higher penalty weights are allocated to edited bounding regions and small-font characters to enforce stroke fidelity; under generation mode, global uniform supervision ensures natural lighting, balanced perspective, and high aesthetic quality across the entire image canvas.

Loss & Training

The framework is optimized under the Rectified Flow matching formulation. During training, the pipeline alternates between editing and generation tasks with an equal probability of \(0.5\), mitigating multi-task catastrophic interference. Optimization is carried out using the Prodigy optimizer with a weight decay of \(0.01\) and a batch size of \(2\). Low-Rank Adaptation (LoRA) is integrated into the Flux.1 Fill dev backbone with rank \(r = 128\). For the SCAA module, the geometric threshold ratio \(\lambda\) and perturbation factor \(\epsilon\) are set to \(1.5\) and \(0.3\), respectively. All experiments are executed across 8 NVIDIA A100 GPUs for 30,000 steps, with images standardized to \(512 \times 512\) resolution.

To empower bilingual model capabilities, the authors constructed InnoText-30K, comprising 20K Chinese and 10K English image-text pairs. The Chinese subset extends Lex-10K prompts via InternVL-3 semantic expansion, generates candidate images using Seedream, and incorporates real-world Anyword-3M samples. The corpus undergoes rigorous multi-stage curation, including NIMA aesthetic score filtering, PP-OCRv4 bounding box re-annotation, and DeepSeek-VL2 dense captioning, establishing a high-aesthetic and semantically aligned foundation for text synthesis.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on the AnyText-Benchmark (for editing) and InnoText-Bench (for generation), using Sentence Accuracy (Sen. Acc), Normalized Edit Distance (NED), and Learned Perceptual Image Patch Similarity (LPIPS). Comparisons benchmark both UNet-based baselines (AnyText, AnyText2) and DiT-based models (Flux-Fill, TextFlux).

Tasks	Methods	English Sen. Acc ↑	English NED ↑	English LPIPS ↓	Chinese Sen. Acc ↑	Chinese NED ↑	Chinese LPIPS ↓
Edit	Flux-Fill	0.1342	0.2481	0.1650	0.0191	0.0443	0.1276
	AnyText	0.6839	0.8632	0.1234	0.6410	0.8209	0.1093
	AnyText2	0.7893	0.9082	0.1739	0.7079	0.8419	0.1509
	TextFlux	0.7732	0.8908	0.0838	0.7164	0.8498	0.0567
	InnoText (Ours)	0.7988	0.9016	0.0786	0.7257	0.8579	0.0591
Gen.	Flux-Fill	0.0126	0.0635	0.6232	0.0057	0.0200	0.6402
	AnyText	0.4707	0.7509	0.6463	0.4584	0.6440	0.6575
	AnyText2	0.5638	0.7712	0.6330	0.5262	0.6636	0.5979
	TextFlux	0.0071	0.0623	0.5968	0.0187	0.0470	0.6527
	InnoText (Ours)	0.6586	0.8066	0.4787	0.5907	0.6837	0.5311

Ablation Study

Ablation experiments analyze the individual contributions of each architectural module alongside baseline models fine-tuned on the InnoText-30K corpus (-FT).

Config	Edit Sen. Acc ↑	Edit NED ↑	Edit LPIPS ↓	Gen. Sen. Acc ↑	Gen. NED ↑	Gen. LPIPS ↓	Note
AnyText-FT	0.5794	0.7138	0.1009	0.4622	0.6513	0.6186	UNet baseline on identical data
AnyText2-FT	0.5882	0.7341	0.0532	0.5358	0.6801	0.5539	Enhanced UNet baseline on identical data
w/o FSAM	0.5672	0.7048	0.0604	0.5508	0.6479	0.5928	Removal of size-aware modulation
w/o SCAA	0.5963	0.7196	0.0497	0.5615	0.6418	0.5539	Removal of small-character augmentation
w/o \(\mathcal{L}_{\text{T-SRW}}\)	0.6180	0.7533	0.0509	0.5892	0.6592	0.5671	Removal of region-weighted loss
full model (w/ all)	0.6374	0.7528	0.0484	0.5907	0.6837	0.5311	Full unified framework

Key Findings

• FSAM is the primary driver of visual quality and multi-scale legibility: Omitting FSAM causes generation LPIPS to deteriorate sharply from 0.5311 to 0.5928, accompanied by a 7.02% plunge in editing accuracy (from 63.74% down to 56.72%), underscoring the critical necessity of continuous scale-aware feature recalibration.
• SCAA protects structural integrity in diminutive characters: Removing small-character augmentation triggers a 4.11% drop in editing accuracy and a 2.92% drop in generation accuracy, highlighting that foveal spatial oversampling is essential for encoding fine-grained sub-pixel stroke connectivity.
• Task-specific loss prevents multi-task gradient interference: Stripping the region-weighted loss reduces editing sentence accuracy by 1.94%, confirming that decoupling localized high-frequency supervision from global scene-level matching is key to dual-task performance.
• Human perceptual evaluation shows decisive preference: In a double-blind Side-by-Side (SbS) user study with 15 participants over 100 cases, InnoText garnered overwhelming preference rates of 78.5% in editing and 80.2% in generation against AnyText2, with participants praising its natural background blending and sharp Chinese character delineation.

Highlights & Insights

• In-Context Inpainting Unification: InnoText seamlessly transforms full-image visual text generation into an inpainting problem via spatial diptych conditioning, achieving unified dual-task mastery on top of a single DiT backbone without structural redesign.
• Continuous Inverse-Height Latent Modulation: Moving beyond blunt binary masks, the derivation of a normalized continuous inverse font-height map introduces fine-grained spatial scale priors into the transformer at minimal computational cost.
• Rigorous Multilingual Data Pipeline: By combining LLM prompt synthesis, foundation generative engines, aesthetic scoring, and MLLM alignment, InnoText-30K resolves the data scarcity bottleneck for non-Latin typographic generation.

Limitations & Future Work

• Overly Expansive Inpainting Masks: When editing requires covering broad regions containing surrounding context, the model occasionally introduces duplicated words due to relaxed spatial localization constraints.
• Ultra-Dense Logographic Characters: For rare or complex Chinese glyphs featuring over 20 intricate strokes, subtle stroke clumping or minor stroke omissions can still occur under extreme micro-scale conditions.
• Future Directions: Exploring explicit character skeleton conditioning or vector font constraint heads within DiTs could further eliminate topological inaccuracies in dense non-Latin typography.

Related Work & Insights

• vs AnyText / AnyText2: AnyText methods operate on UNet architectures with dual text/background branches, frequently suffering from blurred contours and localized distortions on non-flat surfaces. InnoText transitions to a unified DiT architecture, leveraging FSAM and SCAA to overcome micro-scale stroke collapse.
• vs TextFlux: TextFlux focuses strictly on DiT-based text editing, failing completely on open-canvas visual text generation (near 0% sentence accuracy in generation). InnoText establishes full unification across both editing and generation tasks while setting new state-of-the-art benchmarks in both.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Elegant architectural unification via diptych in-context learning and continuous font-scale modulation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative benchmarks across languages, exhaustive ablations, double-blind user studies, and failure analyses]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural flow, rigorous mathematical explanations, and well-contextualized empirical validation]
• Value: ⭐⭐⭐⭐⭐ [Provides a robust blueprint and high-quality benchmark for multi-scale, multilingual typographic synthesis in real-world creative workflows]

Related Papers

• [ECCV 2026] DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing
• [ECCV 2026] Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
• [ECCV 2026] EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation
• [CVPR 2026] Omni2Sound: Towards Unified Video-Text-to-Audio Generation
• [CVPR 2026] UniPercept: A Unified Diffusion Model for Generalizable Visual Perception