Skip to content

Histogram-constrained Image Generation

Conference: ECCV2026
arXiv: 2606.31683
Code: None (Project page https://maps-research.github.io/hig/)
Area: image_generation
Keywords: histogram constraint, optimal transport, diffusion models, controllable generation, information embedding

TL;DR

This paper proposes HIG, which leverages Optimal Transport (OT) during the diffusion sampling process to explicitly transform intermediate predictions into a distribution that precisely matches a target histogram. This achieves precise, interpretable, and training-free controllable generation at the histogram granularity, and is extended to high-capacity information embedding and latent space histogram matching.

Background & Motivation

Diffusion models have achieved clinical dominance in image generation, followed by a surge of work in controllable generation. Based on the granularity of control signals, existing methods are roughly distributed at opposite ends of a spectrum. At one end are high-level control methods like text prompts and LoRA-style fine-tuning, which provide abstract semantic or stylistic constraints, leaving the diffusion process with substantial creative freedom. At the other end are ControlNet and its variants, which utilize dense signals such as edge maps, depth maps, and pose skeletons to precisely constrain every local region of the generated structure. While these two granularities suit different scenarios, a natural gap exists between themโ€”distribution-level control. Distribution-level control refers to imposing precise constraints on global statistical properties of the generated image (e.g., color histograms, latent space token frequency distributions) without specifying pixel-by-pixel content or merely providing a semantic label. This granularity of control is directly required in scenarios such as style transfer, illumination normalization, color consistency across multiple images, and information hiding, yet a general solution that precisely satisfies distribution constraints without severely degrading image quality is lacking.

Existing methods face a fundamental contradiction when dealing with histogram-level constraints. Performing direct post-processing matching (e.g., histogram equalization/specification) on the final generated image ensures exact alignment with the target distribution but ignores the spatial continuity of pixels, often leading to local color banding and noticeable artifacts. Conversely, guiding the model through backpropagation of a differentiable loss during generation incurs high computational costs, as gradients must be computed at each step, and exact constraint satisfaction is still not guaranteed over dozens of diffusion sampling steps. More importantly, these approaches struggle to scale to the control of latent space token distributions, where differentiable approximations lose meaning when distributions are defined over a discrete codebook. Therefore, an intermediate-granularity control mechanism that is training-free, exactly satisfies distribution constraints, and remains orthogonally compatible with existing control schemes is a direction worthy of in-depth exploration in controllable generation.

The core insight of this paper is that Optimal Transport (OT) theory naturally aligns with distribution-constrained problems. Given a source and a target distribution, OT solves for the minimum-cost transport plan, guaranteeing exact matching of the target distribution with minimal transformation cost. Based on this idea, the paper proposes HIG (Histogram-constrained Image Generation). During the sampling process of diffusion models, at selected intermediate timesteps, the current predicted latent code is decoded to pixel (or token) space. An OT problem is then solved to find the minimum-cost transformation to the target histogram, which is subsequently encoded back to the latent space to resume sampling. Core Idea: Model the histogram constraint as an optimal transport problem and apply explicit minimum-cost transformations on intermediate predictions at critical diffusion sampling timesteps. This achieves precise constraint satisfaction in a training-free manner, while leveraging the subsequent refinement capability of the embedding diffusion model to eliminate artifacts caused by the rigid transformation.

Method

Overall Architecture

The core idea of HIG is straightforward: within the standard DDIM sampling loop, a set of guidance timesteps \(\mathcal{T}\) is selected. At these steps, an intervention pipeline of decoding, OT transformation, and encoding is performed on the currently predicted clean image \(\mathbf{z}_0^t\) to enforce exact matching with the target histogram, while non-guided steps follow the standard update rules. This paradigm does not change the network architecture of the diffusion model, nor does it introduce any extra parameters, requiring only the insertion of a configurable transformation module during inference.

The figure below illustrates the overall sampling pipeline of HIG. The area within the dashed box represents the core intervention point of HIGโ€”inserting an explicit distribution transformation step into the conventional denoising loop:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt + Noise z_T"] --> B["Noise Prediction Network ฮต_ฮธ<br/>Predict Current z_0^t"]
    B --> C{"t โˆˆ Guidance Steps ๐’ฏ?"}
    C -->|Yes| D["VAE Decode<br/>z_0^t โ†’ Pixel Space I"]
    D --> E["OT Histogram Matching<br/>ฯ†(I, h_tgt)"]
    E --> F["VAE Encode<br/>I' โ†’ z_0^t'"]
    F --> G["Update z_t-1"]
    C -->|No| G
    G --> H{"t > 1?"}
    H -->|Yes| B
    H -->|No| I["VAE Decode<br/>Final Image"]

Mathematically, at each timestep \(t\), the diffusion model first predicts the current estimation of the final image \(\mathbf{z}_0^t = \frac{1}{\sqrt{\bar{\alpha}_t}}\mathbf{z}_t - \frac{\sqrt{1-\bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}}\epsilon_\theta(\mathbf{z}_t,\mathbf{c},t)\). If \(t \in \mathcal{T}\), then \(\mathbf{z}_0^t\) is decoded to the pixel space, the OT transformation \(\varphi\) is applied, and the result is encoded back to the latent space to yield \(\mathbf{z}_0^{t'}\). Finally, this transformed value is used to calculate the latent variable for the next timestep:

\[\mathbf{z}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\,\mathbf{z}_0^{t'} + \sqrt{1-\bar{\alpha}_{t-1}}\,\epsilon_\theta(\mathbf{z}_t,\mathbf{c},t)\]

Non-guided steps directly use the original \(\mathbf{z}_0^t\). The elegance of this design lies in the fact that subsequent diffusion steps continue to refine and denoise the OT-intervened results. Thus, even if the OT transformation introduces unnatural hard boundaries or local artifacts, the model has the opportunity to smooth them out during the remaining steps.

Key Designs

1. Exact Histogram Matching via Optimal Transport: Transforming Distribution Constraints into Minimum-Cost Transportation Problems

The core of HIG is formulating the constraint "make the pixel value distribution of the image equal to the target histogram" as an optimal transport problem. Given an image, the source histogram \(\mathbf{h}^{src}\) is obtained by counting the number of pixels falling into each bin, and the target histogram \(\mathbf{h}^{tgt}\) is specified by the user (which can be extracted from a reference image or constructed otherwise). The OT problem solves for a transport plan \(\gamma\) to minimize \(\langle \gamma, \mathbf{M} \rangle_F\) under the constraints that the row sum of \(\gamma\) equals \(\mathbf{h}^{src}\) and the column sum of \(\gamma\) equals \(\mathbf{h}^{tgt}\), where \(\mathbf{M}\) is a precomputed distance-cost matrix between bins. The resulting \(\gamma\) specifies the proportion of pixels in each source bin that should be transported to each target bin. During execution, HIG randomly samples the corresponding number of pixels from each source bin and sets their color values to the representative colors of the target bins, thereby satisfying the target histogram exactly while minimizing the total transformation cost. The paper employs the classic Network Simplex algorithm to solve the OT problem. For a configuration with a \(1024^2\) image and \(d=4096\) bins, the solver takes about 0.2 seconds, which has a negligible impact on overall inference latency.

2. Multi-Candidate Binning: Preserving Perceptual Similarity Between Colors Under Bin-Level Constraints

Single-candidate binning maps each bin to a specific color value (e.g., a small cube in RGB space). When the target and source histograms differ significantly, this rigid mapping tends to output visual artifactsโ€”forcing pixels to be assigned to color blocks highly distinct from their original values, resulting in isolated, unnatural pixels. To address this, HIG introduces multi-candidate binning: each bin contains \(k\) candidate color values. The OT transport plan only constrains the total number of pixels in each bin, while the specific assignment to a candidate value within that bin remains flexible. Formally, the OT plan becomes a \(kd \times d\) matrix, with the cost matrix defined as \(\mathbf{M}_{ik+j,p} = \min_q \text{dist}(\mathbf{v}_{ik+j}, \mathbf{v}_{pk+q})\), i.e., the minimum distance from each source option to the target bin is used as the cost. Consequently, while satisfying the bin-level histogram constraints, each pixel can automatically choose the visually closest target candidate value, significantly reducing visual distortion caused by rigid matching. Experiments in the paper demonstrate that images generated using multi-candidate binning for information embedding are visually almost indistinguishable from unconstrained generations.

3. Inference-Time Progressive Guidance: Progressively Enforcing Distribution Constraints at Selected Timesteps

The timing of the OT transformation intervention is critical to the final quality. If OT is applied strictly as a post-processing step on the final image (the "direct OT" variant), although the HistKL can reach a perfect 0, the image displays noticeable color banding and local artifacts. The key improvement of HIG is inserting the OT transformation in the intermediate steps of diffusion sampling, rather than processing only the final output. Through ablation studies, the authors find that for color histogram constraints, applying 1-2 OT transformations in the mid-to-late stage of sampling (e.g., \(\mathcal{T}=\{20\}\) out of 50 steps) achieves the best balance between histogram alignment (HistKL=1.09) and image quality (CLIP=27.19, Aesthetics=6.78). For scenarios requiring exact constraints like information embedding, 3-4 OT transformations (e.g., \(\mathcal{T}=\{40,30,20,10\}\)) combined with a post-hoc OT at the end are necessary to ensure perfect alignment. The earlier the OT is applied, the more refinement steps remain, resulting in better image quality, though the histogram alignment accuracy may partially decay due to subsequent denoising. Conversely, applying it later yields more precise alignment but degrades quality. The cleverness of this progressive guidance is that it leverages the interpolation capability of the diffusion modelโ€”OT "nudges" the intermediate prediction, and the diffusion model naturally "repairs" it, collaborating alternately to maintain both constraints and naturalness.

A Complete Example: This Histogram-Based Information Embedding

Information embedding is one of the most impressive applications of HIG, fully demonstrating the entire pipeline of histogram constraints from construction to application. The goal is straightforward: generate a completely natural-looking image whose color histogram encodes a hidden text, allowing the original text to be precisely restored via histogram decoding.

The process consists of two steps. First, the target text is encoded into a soft prompt vector \(\mathbf{p} \in \mathbb{R}^{4096}\) using prompt tuning: all parameters of Llama-3.1-8B are frozen, and only \(\mathbf{p}\) is optimized so that when prepended to the input sequence, the model accurately outputs the target text (e.g., a 512-token README). The optimization converges in about 500 steps for text sequences of up to several hundred tokens. Second, the soft prompt vector is converted into a color histogram via a simple deterministic mapping \(\mathbf{h}^{tgt} = \texttt{softmax}(\mathbf{p})\). This step is crucial because the mapping is bijective; during decoding, the histogram can be recovered from the image, mapped back to \(\mathbf{p}\), and decoded back to the original text using the LLM without requiring any additional key storage. Subsequently, HIG performs constrained generation targeting \(\mathbf{h}^{tgt}\), applying 4 OT transformations during sampling (\(\mathcal{T}=\{40,30,20,10\}\)) along with a post-hoc OT to ensure exact alignment. To share a concrete figure: for embedding a 256-token text, soft prompt optimization takes only about 20 seconds with a 99.7% success rate, yielding a precise decoding rate of 97.3% for the generated images. Even when embedding ultra-long texts of 512 tokens, the decoding rate remains above 90%.

Loss & Training

HIG is completely training-free, involving no loss functions or model parameter updates. The OT transformation itself is based on closed-form optimization (Network Simplex) and does not require gradient propagation. The only "optimization" occurs in the information embedding scenarioโ€”the gradient ascent objective for the soft prompt embedding \(\mathbf{p}\) is to maximize the conditional log-likelihood of the target text (Equation 6), which is part of the information preprocessing stage rather than the generation stage.

Key Experimental Results

Main Results

The paper compares against multiple baselines on the color-constrained generation task, including GPT-4o-Image, LoRA+ControlNet+IP-Adapter stacking, PixelShuffler, MPGD, and various style transfer methods (InST, StyleID, StyleShot, InstantStyle++). HIG leads in histogram alignment, prompt compliance, and aesthetic quality:

Method HistKL โ†“ CLIP โ†‘ Aesthetic Score โ†‘ Training-Free?
Unconstrained 10.87 29.47 6.98 โ€”
LoRA+CN+IP 3.16 22.53 5.39 โœ—
PixelShuffler 5.90 23.56 4.87 โœ—
MPGD 4.24 24.46 4.94 โœ“
GPT-4o-Image 8.96 25.72 6.62 โœ“
StyleShot 1.90 26.39 5.83 โœ“
InstantStyle++ 2.72 26.23 6.54 โœ“
Ours (guidance) 1.09 27.19 6.78 โœ“
Ours (guidance + post-hoc) 0.00 26.91 6.66 โœ“

Ablation Study

Configuration HistKL โ†“ CLIP โ†‘ Aesthetic Score โ†‘ Description
Unconstrained (๐’ฏ={}) 10.87 29.47 6.98 Unconstrained baseline
Post-hoc OT only 0.00 26.94 6.52 Apply OT directly on the final image
๐’ฏ={30} 2.55 28.09 6.82 Single-step guidance, weaker alignment
๐’ฏ={20} 1.09 27.19 6.78 Optimal balance point
๐’ฏ={10} 0.43 26.88 6.71 Later guidance, better alignment but slight quality drop
๐’ฏ={20,30,40} 0.43 25.93 6.60 Multiple guidance steps further degrade quality

Key Findings

  • OT guidance timing is a critical hyperparameter: Applying a single OT transformation at step 20 of 50 sampling steps (\(\mathcal{T}=\{20\}\)) achieves the best quality-alignment balance in the color-constrained task. Earlier guidance results in higher quality but weaker alignment, while later guidance leads to more precise alignment but lower image quality. For scenarios requiring exact alignment like information embedding, 3-4 guidance steps combined with post-hoc OT are required.
  • Multi-candidate binning significantly mitigates visual artifacts: In the information embedding scenario, the direct OT variant with single-candidate binning yields coarse, noisy images, whereas the variant using multi-candidate binning (coupled with progressive guidance) generates embedded images that are visually almost indistinguishable from unconstrained images.
  • HIG is fully compatible with ControlNet / LoRA: The appendix demonstrates the effects of stacking HIG's color control with DreamBooth LoRA and ControlNet++. The three control mechanisms function independently (content, structure, and distribution) without mutual interference.
  • Robustness analysis of information embedding: It exhibits high tolerance to random scaling (0.5-2.0ร—), maintaining a decoding rate of around 90% for up to 128 tokens, but is highly sensitive to JPEG compression, where the decoding rate drops below 90% when exceeding 32 tokens.

Highlights & Insights

  • Training-free and near-zero overhead: HIG is active without any fine-tuning. The 0.2-second OT solver only adds about 2 seconds of total latency in a 50-step diffusion context, which is comparable to the overhead of loading a LoRA. This extremely low computational cost makes it easily integrable into any existing T2I pipeline.
  • Elegant design of intermediate-step guidance vs. post-processing: Applying the OT transformation at intermediate steps of the diffusion process rather than solely on the final output is the most critical design choice. The remaining denoising steps naturally act as a "smoother," eliminating local artifacts generated by the rigid mapping. This strategy is worth extending to other distribution-constrained tasksโ€”wherever an iterative denoising process exists, one can insert a transformation in the middle and let the subsequent steps repair it.
  • Histograms as information carriers: Encoding textual information into a color histogram is an elegant approach. The bijective softmax mapping ensures that encoding and decoding do not require an extra key, and the histogram is a natural statistical attribute of the image, keeping it unobtrusive. Extending steganography to texts of several hundred tokens (rather than fixed-bit payloads in traditional steganography) opens up new application spaces.
  • General control paradigm with "distribution-level" granularity: HIG fills the gap between text-level and pixel-level control. Moreover, OT is a domain-agnostic mathematical tool; as long as the target attribute can be represented as a histogram (such as color, edge orientation distribution, depth histogram, or attention distribution), it can be constrained using the same methodology. This provides a modular new dimension for mixed-granularity control (text + structure + distribution).

Limitations & Future Work

  • Potential improvements for random sampling in OT: The current implementation randomly samples pixels from each source bin and assigns them to target bin colors, ignoring spatial locality and structure preservation. Factoring in spatial neighborhood consistency during transportation (e.g., maintaining color continuity within superpixels) could further reduce visual artifacts.
  • Fundamental contradiction between explicit transformations and naturalness: The OT transformation is inherently rigidโ€”satisfying constraints inevitably perturbs content. Although intermediate-step guidance alleviates this issue, visual degradation remains visible when constraints are extremely strong (e.g., highly sparse or unnatural target histograms). Exploring softer designs, such as interpolating between representations before and after the transformation, might be a more promising direction.
  • Practical bottlenecks in information embedding: Although the decoding rate for 512 tokens exceeds 90%, the robustness against JPEG compression and post-encoding edits remains insufficient. For practical scenarios, incorporating error-correcting codes or more robust embedding schemes is required.
  • Systematic research on latent-space control: The current exploration of latent-space histogram matching is proof-of-concept. The control effect depends heavily on the semantic characteristics of the tokenizer's latent space, and different tokenizers (VQ-GAN, TokenFlow, TiTok) produce vastly different control effects (ranging from low-level color to high-level semantics). A systematic analysis of this phenomenon is still missing.
  • vs. Dense Controls like ControlNet: ControlNet finely controls local structures via an auxiliary conditioning network with pixel-level granularity. HIG operates at a distribution-level granularity, making the two approaches orthogonal. Experiments show that both can be stacked without mutual interference, suggesting distribution-level control can serve as a modular new dimension in existing control frameworks.
  • vs. Style Transfer Methods (StyleShot, InstantStyle++): Style transfer works focus on transferring the style (texture, brushstrokes, etc.) of a reference image to target content, rather than strictly satisfying a histogram. HIG aims for "exact equivalence" rather than "similarity," excelling in this regard, though it is less effective than specialized style transfer methods in tasks requiring high-level semantic understanding.
  • vs. Information Hiding / Steganography: Traditional generative steganography either encodes a fixed-bit payload or hides another image within a coverless container. HIG strikes a new balance between coding capacity (arbitrary text up to hundreds of tokens) and carrier naturalness, while utilizing a soft prompt mechanism on pretrained LLMs to achieve keyless bijective mapping for encoding/decoding, which is a significant distinction.
  • vs. Differentiable Guidance Methods (e.g., CFG, Classifier Guidance): Differentiable methods require computing gradients (or score functions) with respect to the target objective, rendering them powerless against non-differentiable constraints. HIG handles both continuous color histograms and discrete latent-space token histograms within a unified OT framework without relying on gradients, giving it broader applicability.

Rating

  • Novelty: โญโญโญโญโ˜† Introducing optimal transport into diffusion sampling for distribution-level control is a novel angle that fills the intermediate-granularity gap; the application to information embedding is also highly ingenious.
  • Experimental Thoroughness: โญโญโญโญโญ Detailed quantitative and qualitative experiments across three scenarios (color constraints, information embedding, latent space matching), with ablation studies covering guidance steps, binning strategies, resolution, and robustness.
  • Writing Quality: โญโญโญโญโญ Highly logical, progressing systematically from the intuition of the "control granularity spectrum." The mathematical formulation of OT is rigorous yet restricted to the necessary scope, avoiding reading fatigue.
  • Value: โญโญโญโญโ˜† As a training-free, plug-and-play distribution control module, it holds high practical value. The potential application in information embedding is especially noteworthy. The main drawback is the current lack of official open-source code.