Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Authors: Xinyi Wang, Yuyang Huang, Yalin Su, Pengcheng Luan, Tao Zhang, Feiming Wei, Wenxian Yu
Area: Image Generation
Keywords: Text-image alignment, semantic anchoring, LP-SDS, reflective steering, compositional generation

TL;DR

Without training model weights, AnchorSteer uses CLIP priors and a frozen diffusion model to optimize initialization, then diagnoses, removes, and completes semantic content during denoising, improving SDXL GenEval from 55.933 to 63.393 at a substantial inference cost.

Background & Motivation

Text-to-image models can produce convincing individual objects yet still omit objects, miscount instances, or bind an attribute to the wrong object when prompts combine multiple constraints. Increasing denoising steps or classifier-free guidance (CFG) strength does not tell a model precisely what is currently wrong. Once an early layout is mistaken, subsequent sampling may simply make that mistake more visually polished.

Existing inference-time approaches intervene at two main locations. Initialization methods select a favorable noise sample from a pool or optimize noise directly; trajectory methods alter guidance during generation and sometimes re-noise an intermediate state to revisit an earlier point. The former face limitations from candidate pools, unreliable scoring, and distribution shift. The latter permit rollback without necessarily distinguishing content that should be removed from content that should be added. This paper treats initialization and online correction as complementary problems rather than expecting a single alignment score to resolve both.

Core idea: first turn textual semantics into a starting point compatible with the diffusion prior, then let a vision-language model distinguish incorrect or extra content from missing content, using directed local rollback and refinement to prevent semantic errors from accumulating.

Method

Overall Architecture

The input is a text prompt, and the output is an image that better satisfies its object, attribute, and relational constraints. Semantic Anchoring operates before sampling: CLIP synthesizes a pseudo-image, LP-SDS moves its latent representation toward the diffusion model's conditional distribution, and DDPM forward noising constructs the starting state. Reflective Steering operates inside denoising through Think diagnosis and Erase-Retouch correction.

Here, training-free means that pretrained model weights are not updated, not that optimization disappears. A pseudo-image and latent representation are optimized separately for each prompt, and a VLM is queried repeatedly. Correction acts through latents and textual guidance; the main paper does not specify an explicit spatial mask module. Consequently, targeted erasure should not be interpreted as a guarantee that edits remain spatially isolated.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Text prompt"] --> Anchor["Semantic Anchoring<br/>CLIP prior and LP-SDS"]
    Anchor --> Step["Start from forward-noised latent<br/>Run a normal denoising step"]
    Step -->|Inspection timestep| Think["Think Dual Diagnosis<br/>Decode a clean-image estimate"]
    Think --> Edit["Erase-Retouch Correction<br/>Negative rollback and positive completion"]
    Edit -->|Denoising steps remain| Step
    Step -->|No inspection and steps remain| Step
    Edit -->|Final step| Output["Decode the final image"]
    Step -->|Finished without inspection| Output

Key Designs

1. Semantic Anchoring: extract textual structure, then repair the gap to the diffusion prior

The first step does not retrieve an image from a database. Direct Ascent Synthesis (DAS) optimizes a learnable image to increase the cosine similarity between its CLIP image embedding and the prompt's text embedding. The resulting pseudo-image can encode target objects, color distributions, and spatial structure, but often contains high-frequency artifacts and unnatural colors. An image that CLIP considers text-aligned need not resemble the natural images familiar to the diffusion model. Using it directly would inject semantic information and distribution shift together.

The authors therefore encode the pseudo-image with the VAE and optimize its latent representation. Each iteration samples a noise level, corrupts the current latent, queries the frozen diffusion denoiser, and uses the difference between predicted and sampled noise as an update signal. LP-SDS follows SDS in omitting the U-Net Jacobian. The optimized variable is the latent for the current prompt, not the U-Net weights. With readable notation restored from the paper's Eq. (4), the central update direction is:

\[ \nabla_z\mathcal L_{\mathrm{LP\text{-}SDS}} \triangleq \mathbb E_{t,\epsilon}\left[w(t)\left(\hat\epsilon_\phi(z_t;c,t)-\epsilon\right)\right]. \]

Here, \(z_t\) is the noised current latent, \(\hat\epsilon_\phi\) is the text-guided noise prediction, and \(w(t)\) is a time-dependent weight. DAS supplies a structure representing what should be depicted; LP-SDS asks the frozen generator which latent configurations are more consistent with its own image prior. The paper interprets the update through a time-weighted conditional KL divergence, but that interpretation does not strictly guarantee preservation of every compositional constraint at every step.

The optimized latent \(z^*\) is converted into the sampling initialization using DDPM forward noising:

\[ z_T=\sqrt{\bar\alpha_T}\,z^*+\sqrt{1-\bar\alpha_T}\,\epsilon, \qquad \epsilon\sim\mathcal N(0,I). \]

This follows a known forward process rather than arbitrarily manipulating noise, but the paper's subsequent assertion that the result still follows \(\mathcal N(0,I)\) requires qualification. For fixed \(z^*\), the expression gives a Gaussian with mean \(\sqrt{\bar\alpha_T}z^*\) and covariance \((1-\bar\alpha_T)I\). It is only approximately standard Gaussian when the residual signal is sufficiently small. If that signal disappears entirely, the anchored semantics disappear as well. Distributional consistency is therefore better understood as an approximate motivation than an unconditional theorem.

2. Think Dual Diagnosis: turn vague misalignment into two actionable feedback types

At an inspection timestep, the method first performs a normal denoising update to obtain \(z'_{t-1}\). The VLM does not inspect the noisy latent directly, nor does it wait for the final output. The system predicts a clean latent using the DDIM prediction relationship, decodes it through the VAE, and asks the VLM to compare this current clean-image estimate with the original prompt.

Diagnosis separates missing content \(D_{\mathrm{pos}}\) from inconsistent content \(D_{\mathrm{neg}}\). Missing content covers required objects, attributes, or relationships that have not sufficiently materialized. Inconsistent content covers extra, hallucinatory, or contradictory elements. The purpose is not a more detailed ranking score but two distinct control targets: omitted content must be reinforced, whereas incorrect content must be suppressed. Simply strengthening the entire prompt does not explicitly separate those operations.

This makes diagnostic reliability a new bottleneck. Early clean-image predictions may remain ambiguous, and what the VLM sees may differ from the eventual image. The paper compares diagnostic VLMs of different sizes but does not report error-detection accuracy or confidence-based gating. Its results therefore do not establish that every intermediate diagnosis is trustworthy.

3. Erase-Retouch Correction: suppress errors on rollback and reinforce missing semantics on both legs

With the diagnosis available, a normal denoising step becomes \(z_t\rightarrow z'_{t-1}\rightarrow z_t^*\rightarrow z_{t-1}^*\). Erase replaces the unconditional input in CFG with \(D_{\mathrm{neg}}\) and uses negative-guided DDIM inversion to return to a noisier timestep. Unlike random re-noising, the rollback is directed by the semantics of the current error. Its purpose is to weaken inconsistent content before refinement, rather than merely give a mistaken image another random attempt.

Retouch incorporates the missing descriptions into the original prompt, forming the enhanced condition \(c^+=c\cup D_{\mathrm{pos}}\). The important detail is that this positive condition is applied during both re-noising and re-denoising. On rollback, the enhanced prompt supplies the positive condition and the incorrect descriptions supply the negative condition. On the return leg, standard CFG uses the enhanced prompt. Erase and Retouch therefore overlap in their influence on the rollback sub-step; they should not be implemented as unrelated operations with positive completion reserved only for the final leg.

The refined latent re-enters ordinary sampling, and subsequent inspections produce fresh diagnoses rather than permanently reusing the initial error list. Equations (6)-(11) contain missing symbols in the locally extracted text. This note preserves the explicit state path and condition-substitution logic without inventing missing guidance coefficients or inversion formulas; an exact implementation should consult the original PDF.

A Worked Example

Figure 7 uses the prompt "A horse and a giraffe" and shows clean-image predictions from the first 8 denoising steps. Standard SDXL confuses the concepts and generates two giraffes, whereas AnchorSteer progressively removes the incorrect instance and restores the missing horse.

Interpreting this through the algorithm, Semantic Anchoring supplies an initialization related to both objects; Think must then identify the missing horse and extra giraffe; Erase-Retouch applies negatively conditioned rollback and positively enhanced refinement. This is an explanation of the figure and mechanism, not a verbatim VLM transcript from the cache. It also does not establish that every similar prompt is corrected within a fixed number of steps.

Loss & Training

The main experiments use SDXL and HunyuanDiT-v1.2 with \(T=50\) DDIM steps and a normal CFG scale of 5. DAS uses OpenCLIP ViT-B/32, and LP-SDS runs for \(N_{\mathrm{lp}}=400\) optimization steps. Reflective Steering uses Qwen-VL-Chat, inspection stride \(\tau=1\), erasure guidance parameter \(\gamma_e=1.0\), and retouch parameter \(\gamma_r=5.0\). Experiments run on NVIDIA H100 GPUs.

The main implementation section does not specify the exact DAS iteration count, learning rate, or all optimizer settings; these should not be filled in from convention. The cache also does not contain the referenced supplementary experiments. Few-step distilled-model results and human preference studies are therefore only mentioned by the authors here, not available as quantitative evidence to reproduce in this note.

Key Experimental Results

Main Results

The following subset of the paper's Table 2 reports SDXL results on GenEval. Higher is better throughout, and the original score scale is preserved. Overall improvements are score-point differences, not relative percentages; AnchorSteer is not uniquely best in every category.

Method	Two Objects	Counting	Position	Attribute	Overall
Standard	81.06	30.31	11.00	21.25	55.933
InitNO	85.10	37.81	16.00	22.75	58.540
DNO	80.81	44.06	12.75	22.00	58.430
Zigzag	85.35	40.94	12.25	20.50	58.547
Diffusion DPO	92.42	46.25	15.00	18.50	60.744
AnchorSteer	91.16	52.50	17.75	24.00	63.393

SDXL improves by 7.460 Overall points and 22.19 Counting points, but its Two Objects score remains below Diffusion DPO's 92.42. Table 1 additionally reports that HunyuanDiT improves from 68.342 to 72.470 Overall, a gain of 4.128. This supports effectiveness on two distinct backbones, not a claim of universal compatibility with all diffusion architectures.

The following selected columns come from Table 3 and report SDXL on T2I-CompBench++. The labels 2D-S/3D-S denote spatial relations, Non-S denotes non-spatial relations, and Num denotes numerical constraints. Higher is better in every column.

Method	Color	2D-S	3D-S	Non-S	Num	Complex
Standard	0.5766	0.2015	0.3824	0.3139	0.5029	0.3382
Zigzag	0.6172	0.2011	0.4024	0.3206	0.5349	0.3504
Diffusion DPO	0.6351	0.2153	0.4017	0.3156	0.5223	0.3536
AnchorSteer	0.6712	0.2158	0.4135	0.3171	0.5796	0.3759

Complex compositions and numerical constraints improve, but Non-S remains below Zigzag. Table 4 further reports an ImageReward increase from 0.4891 to 0.8781, while the diversity measure \(1-\mathrm{LPIPS}\) decreases from 0.3860 to 0.3794 and DINO similarity decreases from 0.6638 to 0.6314. Lower similarity means greater sample variation in these two diversity measurements, not degraded diversity.

Ablation Study

The following values come from Table 5. Anchor-only and Steer-only each enable a single component; Steer-only must not be read as a cumulative configuration that already includes Anchor.

Backbone	Configuration	Two Objects	Counting	Attribute	Overall
SDXL	Standard	81.06	30.31	21.25	55.933
SDXL	Anchor-only	88.64	44.38	24.00	60.731
SDXL	Steer-only	91.67	42.50	20.25	59.868
SDXL	AnchorSteer	91.16	52.50	24.00	63.393
HunyuanDiT	Standard	93.18	64.69	39.00	68.342
HunyuanDiT	Anchor-only	93.43	66.88	46.75	70.395
HunyuanDiT	Steer-only	96.72	71.56	45.50	71.728
HunyuanDiT	AnchorSteer	95.71	72.50	48.25	72.470

Key Findings

• The components complement each other in Overall score, but category gains are not monotonic. SDXL Steer-only scores 20.25 on Attribute, below the baseline's 21.25; on both backbones, the full method also scores below Steer-only on Two Objects.
• In Table 6, replacing the diagnostic model with Qwen2-VL-2B or Qwen2.5-VL-3B gives Overall scores of 62.073 and 62.843, respectively, versus 63.393 for Qwen-VL-Chat. Model family and version change alongside size, so the difference cannot be attributed solely to parameter count.
• Table 7 reports per-image runtimes of 3.7 seconds for Standard, 8.3 for Zigzag, 68.5 for AnchorSteer, and 155.5 for DNO. AnchorSteer's breakdown is 18.0 seconds for Anchor, 46.7 for Steer, and 3.8 for other work. Repeated diagnosis and bidirectional correction dominate the overhead, not one additional ordinary sampling pass.

Highlights & Insights

• Initialization quality and semantic drift during sampling are treated separately, and the ablations show that neither component replaces the other. Inference-time optimization benefits from separately designing and measuring starting-state control and online feedback.
• A discriminative model's semantic prior is calibrated against the generator's distribution before it is used for initialization. The reusable idea is prior extraction followed by generative-prior calibration, not the assumption that any high-CLIP-score image can directly support natural-image synthesis.
• Missing and contradictory content receive different directional controls, giving VLM feedback an explicit operational meaning. Transferring the idea to other generation tasks should include tests of diagnostic accuracy and correction side effects, not just final alignment scores.

Limitations & Future Work

• The authors explicitly acknowledge inference overhead. The difference between 68.5 and 3.7 seconds favors applications where faithfulness matters more than latency. The main comparisons do not use equal wall-clock budgets, so they do not establish greater compute efficiency than generating and selecting among multiple samples.
• This note's analysis identifies a tension between approximately Gaussian terminal noise and retaining a semantic residual. Future work should measure noise statistics, alignment benefits, and terminal noise strength together rather than relying solely on the form of the forward process.
• The cached main paper does not provide fine-grained ablations removing LP-SDS, removing negative erasure, or applying positive guidance on only one leg, nor an inspection-stride sweep. Existing evidence supports the Anchor/Steer modules but does not isolate every internal design's contribution.
• Early image estimates can cause VLM misdiagnosis, and textual negative guidance does not guarantee changes confined to the intended region. Confidence-triggered inspection and spatial constraints are possible research directions, not implemented features established by this paper.
• The main paper does not show repeated-run error bars, confidence intervals, or broader backbone validation. Missing supplementary material cannot be filled in speculatively. Absolute spatial and attribute scores also indicate that compositional semantics remain far from solved.

Related Work & Insights

• Compared with DAS and SDS: DAS extracts text-related visual structure from CLIP, while SDS supplies distillation gradients from a frozen diffusion model. AnchorSteer joins them at initialization and optimizes a latent to connect discriminative and generative priors, rather than adding a trainable generator.
• Compared with InitNO, DNO, and NPNet: These methods improve starting states through noise selection or optimization. AnchorSteer additionally diagnoses errors during generation, but its own initialization also requires iterative computation and should not be described as a cheap noise replacement.
• Compared with Zigzag: Bidirectional sampling supplies the basis for rollback and refinement; AnchorSteer adds error-type diagnosis and positive/negative condition control. The important distinction is why rollback occurs and which semantic direction it follows, not simply whether a round trip exists.
• Compared with Diffusion DPO: Preference training changes model parameters, whereas AnchorSteer spends inference compute on a fixed model. These are different ways of allocating cost, and their score comparison should be interpreted alongside training and deployment budgets.

Rating

• Novelty: 4/5. The connection between two initialization priors and dual-diagnosis correction is clear, although the foundations come from DAS, SDS, CFG, and bidirectional sampling.
• Experimental Thoroughness: 4/5. Two backbones, two compositional benchmarks, quality/diversity measures, and runtime analysis are included; equal-budget comparisons and finer internal ablations remain absent.
• Writing Quality: 4/5. The main argument and module-level evidence are clear, but the standard-Gaussian claim needs qualification and some reproducibility details are deferred to unavailable supplementary material.
• Value: 4/5. Useful for inference-time research prioritizing faithfulness; latency and diagnostic reliability limit direct deployment.