Discrete Noise Inversion for Next-scale Autoregressive Text-based Image Editing

Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation
Keywords: Text-guided image editing, visual autoregressive models, discrete noise inversion, Gumbel sampling, background preservation

TL;DR

VARIN constructs inverse noise with a controllable preservation bias for next-scale visual autoregressive models, enabling editing without training or test-time optimization and reducing the HART Regeneration baseline's structure distance from 25.56 to 11.54 on PIE-Bench while improving background preservation and text alignment.

Background & Motivation

Text-guided image editing must produce content matching a new prompt while retaining regions that the user did not request to change. Diffusion editing often inverts the source image before generating under modified text conditions, but iterative denoising and some methods' test-time optimization add latency. Visual autoregressive models offer a different generation path: they predict entire token maps from coarse to fine instead of generating individual tokens in raster order. Next-scale models such as HART generate images with relatively few scale predictions, creating an opportunity for low-latency editing. However, fast generation does not automatically identify which source details should remain unchanged.

The straightforward Regeneration baseline preserves early source scales and generates the remaining scales again under the target text. Early scales encode overall layout, so the edited image may retain the broad composition, but background textures represented at later scales are also resampled. Changing a cat into a dog may consequently alter unrelated chair textures or floor details. Preserving these details through noise requires solving a discrete inversion problem: argmax retains only the winning class and discards the other scores. Even noise that reconstructs the original tokens need not permit object changes while keeping the background stable after a prompt change.

The paper focuses on this distinction between reconstructable and editable discrete noise representations. Simple one-hot inversion gives the original class an excessive advantage, making the editing strength difficult to control smoothly even when fresh noise is mixed in. VARIN instead uses the model's predicted logits to construct perturbed scores and adjusts the source-image bias through a preservation margin between classes. The objective is not to recover a unique historical random seed, but to construct inverse noise satisfying reconstruction constraints while remaining useful for editing. Core Idea: sample pseudo-inverse argmax scores around the model logits with a controllable margin, extract inverse noise at each scale, and mix it with fresh noise across scales so that target text changes content while source information constrains the background.

Method

Overall Architecture

The inputs are a source image, text describing that image, and target text describing the desired result; the output is an edited image. The source text conditions inversion and can be written by a person or supplied by an image captioning model; it should not be confused with the target text. The encoder first converts the source image into discrete token maps of increasing resolution, with each position taking a vocabulary class. Inversion predicts logits at each scale using the actual source-token prefix and source text, then performs Location-aware Argmax Inversion and Scale-wise Inverse Noise Extraction. Editing switches to the target text, preserves early tokens, and performs Scale-wise Noise-scheduled Editing before decoding the resulting image. Model weights remain fixed: inversion is not additional training, and the method does not require a user-drawn editing mask as input.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Source image + source text"] --> Encode["Multiscale tokens<br/>and source-conditioned logits"]
    Encode --> LAI["Location-aware<br/>Argmax Inversion"]
    LAI --> Noise["Scale-wise<br/>Inverse Noise Extraction"]
    Noise --> Edit["Scale-wise<br/>Noise-scheduled Editing"]
    Target["Target text + fresh noise"] --> Edit
    Edit --> Output["Decode the edited image"]

Key Designs

1. Location-aware Argmax Inversion: make the original class win without eliminating its competitors

Discrete generation can be understood through Gumbel-max sampling: add noise to the class logits and select the class with the largest score. During inversion, the source image specifies which class should win, but not the complete perturbed score vector. Argmax therefore has no unique inverse; the paper seeks a pseudo-inverse satisfying the winning-class constraint. Onehot Argmax Inversion (OAI) sets the desired class score to 0 and the other scores to very large negative values, then subtracts the predicted logits. This guarantees reconstruction but imposes an excessive advantage for the source class, and its noise magnitudes are difficult to combine with ordinary Gumbel noise. In Location-aware Argmax Inversion (LAI), location means the Gumbel distribution's location parameter, namely the predicted logit, not an additional spatial localization detector.

LAI first samples the original class's perturbed score \(q_{\max}\) using that class's predicted logit as the location parameter. It then samples every competing class from an upper-truncated Gumbel distribution centered at that class's predicted logit. The truncation threshold is \(q_{\max}-\tau\), where nonnegative \(\tau\) is the preservation margin; the original class entry is finally replaced with \(q_{\max}\). The source class still wins, but competing scores retain information about the model's preferences among alternative content. Algorithm 3 gives the following truncated sampling expression, where \(\phi\) is the location parameter, \(T\) the upper bound, and \(u\) a uniform random variable.

\[ q=\phi-\log\left(\exp(\phi-T)-\log u\right),\qquad u\sim\operatorname{Uniform}(0,1),\quad T=q_{\max}-\tau. \]

When \(\tau=0\), competing classes can come very close to the winner, allowing small noise perturbations to change background tokens. A positive preservation margin stabilizes these unrequested regions while still allowing sufficiently large target-conditioned logit changes to drive an edit. The paper describes the resulting inverse noise as closer to standard Gumbel noise; this should not be read as a proof of exact recovery of independent standard Gumbel samples. The noise is constrained by the source class and truncation, so reconstruction and distribution matching are distinct requirements.

2. Scale-wise Inverse Noise Extraction: encode source information as reusable sampling residuals

At scale \(t\), the model predicts \(p_t^{\mathrm{src}}\) from the source text and all preceding source-token maps. LAI produces \(q_t\) with the same shape as the logits: the number of positions at that scale multiplied by the vocabulary size. Algorithm 2 subtracts these tensors to obtain inverse noise for every position and class, saving the results across scales. The central constraint from Algorithms 1 and 2 can be written as:

\[ n_t=q_t-p_t^{\mathrm{src}},\qquad \operatorname{argmax}\left(p_t^{\mathrm{src}}+n_t\right)=r_t^{\mathrm{src}}. \]

Reusing this noise with the same source text and source prefix recovers the original discrete token map. This computation uses known source tokens, unlike editing, where later scales depend on the generated prefix. The saved representation is a set of residuals added to sampling logits, not a single global random seed or a newly learned image embedding. It prevents later scales from depending entirely on fresh randomness and carries source-detail information into regeneration.

Token-level and pixel-level reconstruction must be distinguished: recovering tokens exactly does not eliminate encoder quantization and decoder losses. HART's reconstruction PSNR of 26.00 in Table 1 makes clear that “perfect reconstruction” applies only to the discrete representation. Inverse noise extraction therefore makes the sampling path reconstructable; it does not turn lossy image compression into lossless compression. The foundation model's representation still limits background preservation, independently of the inversion algorithm.

3. Scale-wise Noise-scheduled Editing: preserve layout first, then release detail generation gradually

Editing fixes the early source scales and begins predicting subsequent tokens under the target text at starting scale \(s\). Under Algorithm 4's indexing, scales before \(s\) are retained; the default \(s=5\) therefore fixes scales 1 through 4. Each step computes target-conditioned logits from the current prefix and mixes fresh Gumbel noise with the corresponding source inverse noise. Later prefixes include edited tokens rather than repeatedly forcing the complete original source prefix. Algorithm 4 uses the following sampling rule, with \(g_t\) denoting fresh noise and \(\lambda_t\) the inverse-noise weight.

\[ \widetilde r_t=\operatorname{argmax}\left(p_t^{\mathrm{tgt}}+(1-\lambda_t)g_t+\lambda_t n_t\right). \]

The HART experiments use a logarithmic schedule that decreases \(\lambda_t\) from 1 at the starting scale to 0 at the final scale 14. Earlier steps retain more inverse noise to stabilize structure; later steps allow fresh noise to generate details rather than locking the edit to the source. Even when \(\lambda_t=1\), target-conditioned logits can differ from source-conditioned logits, so reproducing the source tokens is not guaranteed. Conversely, when \(\lambda_t=0\), the current scale no longer directly uses inverse noise but remains conditioned on previously generated scales. The noise weight is consequently a control over conditional sampling, not simply a fraction of the original image to copy. The provided main text does not specify the logarithmic schedule's complete analytical form, so an exact implementation cannot be inferred from it.

A Worked Example

Consider changing the cat on a wooden chair into a dog, as illustrated in the paper's Figure 4: the source text describes the cat, and the target text replaces the subject with a dog. The image is first encoded, and inverse noise is extracted under the source text at each scale, retaining representation information about the subject, chair, and background. With the default \(s=5\), scales 1 through 4 preserve the original composition before target-conditioned resampling begins. In the subject region, the target condition favors dog-related tokens, while inverse noise and the preservation margin continue to constrain unedited regions such as the chair. As \(\lambda_t\) decreases, later scales receive more freedom to generate random details; retained and newly generated tokens are finally decoded together. There is no explicit step that crops out the cat: regional selectivity emerges from the interaction of text conditioning, model predictions, and source inverse noise. The use of editing masks in evaluation therefore does not imply that the algorithm requires those masks as inputs.

Loss & Training

VARIN introduces no training loss, does not fine-tune the foundation model, and does not optimize per-image latents or null-text embeddings through gradient descent. Its adjustable quantities are the inversion preservation margin, editing starting scale, and inverse-noise schedule, all of which are inference-time controls. The main experiments use HART with default \(s=5\), \(\tau=18\), and final scale 14. The paper also provides qualitative fine-grained edits with Switti, but HART's quantitative results should not be transferred directly to Switti.

Key Experimental Results

Main Results

PIE-Bench contains 700 images covering 9 editing scenarios, with source prompts, target prompts, and region annotations used for evaluation. The following subset comes from Table 2 on page 14; lower structure distance and background LPIPS are better, while higher background PSNR and CLIP similarity are better. Background metrics are computed outside the editing mask; Whole CLIP evaluates the entire image, whereas Edited CLIP evaluates the masked region.

Method	Base Generative Model	Structure Distance ×10³ ↓	Background PSNR ↑	Background LPIPS ×10³ ↓	Whole CLIP ↑	Edited CLIP ↑
Regeneration	HART	25.56	20.45	106.50	24.65	21.13
VARIN	HART	11.54	26.47	54.62	25.59	22.35
DICE	Paella / MaskGiT	11.34	27.29	52.90	23.79	21.23
AREdit	Infinity	30.5	24.19	87.00	25.42	22.77
Null-text Inversion	diffusion	13.44	27.03	60.67	24.75	21.86
DDPM-Inversion	diffusion	22.12	22.66	67.66	26.22	23.02

The shared-HART comparison best isolates the method change: VARIN improves background PSNR by 6.02 and Edited CLIP by 1.22 while reducing structure distance by 14.02. Cross-model conclusions require caution: DICE has better structure distance and some background metrics, and AREdit has higher Edited CLIP than VARIN. DDPM-Inversion has higher scores on both CLIP measures, so VARIN does not dominate all text-alignment metrics. Page 13 reports 0.65 seconds per edited image on a single NVIDIA A100; this is the authors' reported timing, not a measurement reproduced for this note.

Ablation Study

The supplied main text provides no quantitative ablation table for \(\tau\), OAI versus LAI, or the noise schedule, so reconstruction analysis is used here instead of inventing component ablations. The following table comes from Table 1 on page 11 and evaluates full-image reconstruction after inversion of PIE-Bench source images, not the background regions measured above.

Inversion Method	Base Model	PSNR ↑	LPIPS ×10³ ↓	MSE ×10⁴ ↓	SSIM ×10² ↑
DICE	Paella	30.91	39.81	11.07	90.22
VARIN	HART	26.00	70.26	33.20	79.83

Both discrete inversion methods can recover source tokens, but the different lossy representations of Paella and HART leave a gap in pixel-level metrics. This table therefore does not establish that DICE's argmax inversion is more accurate than LAI; it compares complete representation and reconstruction pipelines. The Regeneration-versus-VARIN comparison in Table 2 supports the inverse-noise approach as a whole but does not separately identify the contributions of LAI, \(\tau\), and scheduling.

Key Findings

• With the same foundation model, background preservation and editing alignment improve together, indicating that the result is not merely better reconstruction similarity obtained by editing less.
• Against DICE, VARIN achieves Edited CLIP of 22.35 versus 21.23 without uniformly surpassing its background preservation metrics.
• Figure 5 shows localized Switti edits such as eye closure, head turns, and object replacement; the authors still identify substantial pose or structural changes as difficult.

Highlights & Insights

• The inversion target shifts from a unique original noise realization to a set of editable noises satisfying reconstruction constraints. This gives non-invertible argmax a practical inversion interface.
• LAI retains information from model logits rather than using only source-class labels. Competing classes can therefore remain responsive to new text.
• The preservation margin and scale schedule control different aspects: the former governs winning-class stability, while the latter governs how much source noise is used at each scale.
• The method requires neither cross-attention modification nor weight training, suggesting a useful add-on to existing next-scale samplers; combining it with other controls remains future work.

Limitations & Future Work

• The authors explicitly identify substantial pose or structural changes as difficult; fixing early scales also limits changes to the overall layout.
• HART's lossy encoding limits pixel fidelity, and exact token inversion cannot restore details already lost during encoding.
• The main text lacks numerical sensitivity analysis for the preservation margin and noise schedule, leaving the defaults' applicability across editing types uncertain.
• The main text mentions a supplementary user study, but the supplied 19-page cache contains neither its sample size nor its results, so no preference win rate can be reported.
• Equations (1) and (2), along with parts of the Gumbel explanation, have extraction damage; this note uses the readable Algorithms 1 through 4 and does not reproduce corrupted equations.
• Adapting the starting scale or preservation margin to editing difficulty is a possible research direction, not a result already validated in this paper.

Related Work & Insights

• vs DICE (paper reference 20): DICE targets discrete diffusion and masked generative models; VARIN extends inversion to next-scale autoregression and uses LAI to improve the controllability of one-hot pseudo-inversion.
• vs Regeneration and SDEdit (reference 33): starting from an intermediate generation state preserves coarse structure; VARIN additionally uses source inverse noise to constrain later details.
• vs HART (reference 51): HART supplies the efficient generation backbone, whereas VARIN supplies training-free inversion and editing; they are complementary rather than replacement models.
• vs Null-text Inversion (reference 34): the latter optimizes null-text embeddings to improve inversion, while VARIN avoids this test-time optimization through discrete sampling constructions.
• vs AREdit (reference 60): both address visual autoregressive editing, but VARIN primarily controls inverse noise rather than relying on complex cross-attention manipulation; metrics from different backbones are not a single-component comparison.

Rating

• Novelty: 4/5. Pseudo-inverse sampling with location parameters and a preservation margin directly addresses a specific obstacle in next-scale editing.
• Experimental Thoroughness: 3/5. PIE-Bench comparisons, reconstruction analysis, and qualitative examples on another backbone are included, but quantitative component ablations are absent from the main text.
• Writing Quality: 3/5. The algorithmic path is clear, although the level of “perfect reconstruction” and the Gumbel noise claims require careful distinctions, and cached equations are damaged.
• Value: 4/5. A practical interface for low-latency, training-free autoregressive image editing, still limited by the base representation and the scope of structural changes.