SAEdit: Token-Level Control for Continuous Image Editing via Sparse Autoencoder¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation
Keywords: continuous image editing, sparse autoencoder, token-level control, semantic disentanglement, T5

TL;DR¶

SAEdit maps T5 token representations into a sparse semantic space, selects and aggregates scalable attribute directions, and schedules their injection to control image editing strength; its overall preference win rate against ConceptSlider (Flux) is 71% in a Flux user study, although disentanglement remains constrained by the generator's priors.

Background & Motivation¶

Image editing must specify not only what to change, but also how much and which subject to change. Gradually turning a neutral expression into laughter while preserving hair color, a necklace, and clothing texture is more demanding than generating an image of a happy person. Discrete prompts offer descriptions such as a slight smile or a broad grin, while the entire sentence jointly affects the image rather than exposing an independent visual control. Interpolating dense text embeddings produces continuous numerical values, but semantic changes can remain entangled with other attributes; Figure 2 illustrates insufficient edits and unwanted changes.

Existing continuous editors face two different costs: general methods without per-edit optimization can disturb identity and composition, whereas specialized approaches such as Concept Sliders require learning a controllable representation for a new attribute. The paper therefore seeks more separable control directions within existing text conditioning, rather than another improvement in text-to-image generation itself. It aims to reuse a semantic direction across people and scenes while applying it only to a selected subject token. Here, continuous editing means varying the strength of one attribute, not multi-turn conversational editing or a guarantee that arbitrary instructions induce linear pixel changes.

A Sparse Autoencoder (SAE) offers a possible entry point by expanding dense vectors into a high-dimensional dictionary representation with relatively few active features. The authors operate on the final T5 output rather than an internal layer of a particular diffusion model, separating the semantic controller from the image synthesis backbone. Once a direction is found, a generator compatible with the same text encoder can consume the edited embedding; this does not imply direct transfer between different text encoders. Core Idea: expose sparse attribute coordinates with an SAE, extract a shared direction from multiple minimally different prompt pairs, and inject it into a selected token according to edit strength and denoising stage.

Method¶

Overall Architecture¶

Inputs comprise a source image and its text condition, the subject token to edit, and source/target descriptions defining the desired attribute; the output changes that attribute while retaining the original content. The offline stage trains the SAE, direction construction selects coordinates and combines prompt pairs, and the editing stage applies control to the token embedding for a particular image. These stages should not be confused with per-image optimization: the SAE requires training, but a new attribute does not require retraining the diffusion backbone or a dedicated LoRA.

The pipeline consists of Sparse Representation Learning, Attribute Coordinate Selection, Multi-Prompt Direction Aggregation, and Token-Level Scheduled Injection. The first establishes an encodable and decodable sparse space, the middle stages produce a reusable direction, and the last combines the target token, initial noise, and edit strength to generate outputs. The denoiser network and update rule remain unchanged, but its text condition varies across denoising stages; leaving denoising untouched does not mean keeping its conditioning constant.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Corpus["Prompt corpus<br/>Frozen T5 outputs"] --> Sparse["Sparse Representation<br/>Learning"]
    Sparse -->|Frozen SAE| Select["Attribute Coordinate<br/>Selection"]
    Pairs["Multiple prompt pairs<br/>for one attribute"] --> Select
    Select --> Aggregate["Multi-Prompt Direction<br/>Aggregation"]
    Aggregate --> Inject["Token-Level<br/>Scheduled Injection"]
    Source["Source text, target token<br/>Same initial noise"] --> Inject
    Inject --> Render["Frozen generator<br/>Images at varying strengths"]

Key Designs¶

1. Sparse Representation Learning: build a semantic dictionary over the generator's actual text condition

The authors pass a prompt corpus through frozen T5 and collect the final output embeddings of all non-padding tokens. The SAE encoder maps these dense embeddings into a higher-dimensional sparse latent space, and its decoder reconstructs the original T5 representation. Training requires accurate reconstruction while keeping relatively few nonzero activations in each token's latent code. This sparsity makes attributes easier to identify through a small set of active coordinates rather than mixed changes across many dense dimensions. Using the final output matters because these are the conditions consumed by the diffusion transformer, so editing does not require identifying corresponding layers inside each backbone.

This is statistical disentanglement, not a claim that every coordinate has a unique, causally validated label. An attribute can occupy multiple sparse coordinates, and a coordinate can depend on context, motivating the subsequent selection and aggregation stages. The SAE is tied to the T5 representation space rather than to a particular person, editing attribute, or set of denoiser weights. The main generator is Flux, with results also demonstrating reuse of the same edit directions in Stable Diffusion 3.5.

2. Attribute Coordinate Selection: identify the concept through relative activation changes in prompt pairs

After defining an attribute with a pair such as a woman and a woman laughing, the method encodes all tokens in both sentences. Comparing only the added word laughing would be insufficient because contextual text encoding can distribute its semantics across other tokens. The authors therefore apply coordinate-wise max-pooling to each sentence's sparse token representations, retaining a strong feature wherever it appears in the sequence. This aggregation discovers the direction; choosing the subject token that receives it later is a separate decision.

Selection uses a coordinate-wise target-to-source activation ratio, with a small denominator constant for numerical stability. The ratio is divided by its maximum, and a threshold \(\rho\in[0,1]\) selects the coordinates showing the strongest relative changes. Direction components at selected coordinates take values from the target representation, while all remaining components are zero; this is not a full dense prompt-difference vector. The purpose is to emphasize the newly introduced attribute rather than shared concepts, such as a person, that are already strong in both prompts. Equations (2)–(4) are damaged in the text extraction, so this account follows Section 4.2 and Figure 4 without presenting a reconstructed layout as the authors' exact equations.

3. Multi-Prompt Direction Aggregation: extract attribute changes that persist across contexts

A direction from one prompt pair can encode incidental scene details or produce a sufficiently strong edit only in a particular context. The authors use an LLM to construct \(N\) prompt pairs expressing the same attribute relationship, such as adding happiness to a person at the beach or eating cake. Applying the same coordinate-selection procedure to every pair produces \(N\) candidate directions, which are stacked as rows of a direction matrix. Singular Value Decomposition (SVD) then supplies the principal direction in feature space associated with the largest singular value.

This searches for a change axis supported across contexts, rather than averaging every incidental detail or selecting one appealing example. The aggregate need not contain as few nonzero components as any individual candidate, because candidate directions may have different coordinate supports. Its role is to improve cross-context stability, not to offer a theoretical guarantee of removing all unrelated concepts. Section 5.2 reports that multiple prompts strengthen expression changes but can still modify necklaces, hair color, and clothing details, showing that direction discovery alone does not ensure preservation. The cache excludes the appendix, so the exact value of \(N\), prompt lists, and complete direction-normalization details cannot be confirmed.

4. Token-Level Scheduled Injection: control both the recipient and the timing of an edit

In the standard text-to-image setting, the method first selects the source-prompt token corresponding to the subject, such as woman. It modifies only this token's sparse code, decodes it into T5 space, and replaces the original embedding while retaining the other tokens. The core transformation in Equation (5) is readable, with \(\omega\) controlling edit strength:

\[ e'_{\mathrm{token}}=S_{\mathrm{dec}}\left(S_{\mathrm{enc}}(e_{\mathrm{tgt}})+\omega d_{\mathrm{edit}}\right). \]

For generated source images, editing reuses the original initial noise \(x_T\) so changes primarily arise from text conditioning rather than a different random starting point. For real images, the standard text-to-image editing route additionally requires inversion to obtain a starting point capable of reconstructing the source. Replacing a local token is not an explicit pixel mask and cannot guarantee a spatially isolated effect, particularly with multiple subjects or strongly associated concepts. The paper states that \(\omega=0\) recovers the original embedding, but the formula strictly returns the SAE reconstruction; finite reconstruction error prevents assuming exact mathematical equality.

To reduce structural disruption, the authors also increase injection strength over the denoising trajectory, keeping it weaker early and stronger later, with \(\tau\) providing an upper bound. This lets the original condition establish composition before the attribute direction affects later details, unlike applying a constant strength at every step. The schedule is exponential, but the exponent, products, and parentheses in Equation (6) are corrupted in the cache, so its exact expression is not guessed here. The strength parameter exposes a continuous control interface, while the visual smoothness and monotonicity of outputs remain empirical properties of the generator. Integration with Flux-Kontext introduces another important change: the modified token describes the instruction's action rather than its subject. For make the man laugh, the method subtracts the laughter direction from the laugh token to weaken the directive; the standard subject-token addition cannot simply be reused unchanged.

A Worked Example¶

Consider the angry attribute in Figure 10, where the source shows a person wearing a necklace and the goal is to intensify anger while retaining identity and accessories. The method first selects sparse coordinates from prompt pairs differing only in the anger attribute, obtaining an initial attribute direction. It then aggregates a shared direction from multiple such pairs, reducing the incidental influence of one scene description. These stages construct an attribute controller rather than optimize the image being edited with gradients. Editing reuses the source noise, adds the direction to the subject token's sparse code, and varies strength to generate a sequence of outputs. If the stronger aggregated direction also changes the necklace or hair, scheduled injection limits its influence during the early composition-forming stages. The qualitative comparison supports stronger expressions with better detail preservation from this sequence; the cache provides no numerical strength settings for these examples, so none are invented here.

Loss & Training¶

The SAE uses a reconstruction objective and sparsity constraints, with mean squared error between input and reconstructed embeddings given as a possible reconstruction loss. Another hyperparameter controls the desired number of nonzero latent activations per token, and padding tokens are explicitly excluded from training samples. T5 and the image generation backbone are frozen; a new attribute requires prompt construction, coordinate selection, and SVD rather than training a new attribute-specific generator. Consequently, zero-shot or no per-edit optimization describes usage after training, not a completely learning-free method. Equation (1) is also corrupted, and implementation details are deferred to the unavailable appendix; the exact loss combination, sparsity implementation, dictionary width, and training budget cannot be established from this cache.

Key Experimental Results¶

Main Results¶

Section 5.3 on page 12 constructs an evaluation set from 63 source images depicting people, with 6–8 edits per image and 432 editing scenarios in total. Each scenario is generated at 3–5 strengths, yielding at least 1,296 images per method; these are not 1,296 independent source scenes. Image preservation is measured with LPIPS, where lower values generally indicate greater similarity to the source, and identity preservation uses cosine similarity between ArcFace features. Prompt adherence is measured by the change in VQA score relative to the source rather than the edited image's absolute score alone. The following restates that definition in this note's notation, where \(I_{\mathrm{src}}\) and \(I_{\mathrm{edit}}\) are the source and edited images and \(P_{\mathrm{tgt}}\) is the same target prompt:

\[ \Delta\mathrm{VQA}=\mathrm{VQA}(I_{\mathrm{edit}},P_{\mathrm{tgt}})-\mathrm{VQA}(I_{\mathrm{src}},P_{\mathrm{tgt}}). \]

The following reproduces the results of Table 1 on page 13; every number is the percentage of responses preferring SAEdit over the specified opponent. The study compares only Flux-based methods, presents 3 strengths from each method per comparison, and collects 390 pairwise responses, not necessarily 390 independent participants.

Opponent method	Image preservation win rate	Prompt adherence win rate	Overall quality win rate
Flux Kontext (CFG)	73%	71%	70%
Flux Kontext (LLM)	60%	68%	70%
ConceptSlider (Flux)	71%	67%	71%
Flux Space	59%	92%	93%

These preference rates support an advantage in this comparison, but they are not success rates and do not disclose the response allocation for each opponent. Because Flux Kontext lacks native continuous strength control, the authors construct proxy baselines by varying CFG and by using LLM-written instructions of different intensities. Automatic-metric comparisons in Figure 11 are plotted as curves without reliable numerical values in this extraction, so exact LPIPS or VQA gains are not fabricated.

Ablation Study¶

The following is a qualitative analysis table compiled from Section 5.2 and Figures 9 and 10 on pages 10–11, not a second numerical table from the paper. Only Table 1 provides directly readable exact results in the cache; ablation curves and the missing appendix cannot be converted into invented numerical experiments.

Configuration	Observation supported by the paper	Evidence boundary
Single prompt-pair direction	Weak expression edit; Section 5.2 reports preserved identity	Figure 10's caption also mentions unintended changes, so this does not mean zero disturbance
Multiple prompt-pair aggregation	Stronger editing, with possible changes to hair color, necklace, and dress texture	Aggregation increases attribute strength but does not independently ensure detail preservation
Multiple prompt-pair aggregation plus exponential injection	Better preservation of details with a strong target expression	Qualitative Figure 10 example; no readable numerical difference
No SAE, operating directly on dense T5	Insufficient edits and entangled attributes; mean/max pooling variants are evaluated	Supports the sparse space's role, but the cache does not quantify the decline
Mean aggregation instead of SVD, or mean token pooling instead of max-pooling	Figure 9 includes these controls and reports the full configuration as superior	No readable exact metric values for individual variants

Key Findings¶

Attribute directions and injection scheduling address different problems: semantic specificity versus disruption of structure and details under stronger editing.
Multi-prompt aggregation does not replace temporal control; stronger editing and better image preservation are distinct outcomes.
Transfer to SD3.5 and extensions to real images and style editing are primarily qualitative evidence, not a basis for a uniform cross-model win rate.

Highlights & Insights¶

Control targets a subject token, while direction discovery still examines the whole sentence. This distinction combines contextual semantics with localized intervention rather than equating attribute localization with finding an added word.
The SAE serves as an actionable control space, not only an interpretability tool. The important interface is decoding selected directions back into conditioning that the generator already understands.
Multiple prompt pairs and SVD abstract a specific edit into a cross-context attribute direction. This shifts the main cost of a new attribute toward direction construction rather than a dedicated LoRA.
A transferable design lesson is to evaluate direction quality separately from injection timing. Reusing the strategy in other conditioning spaces would still require independent experiments.

Limitations & Future Work¶

The authors show prior-driven failures in Figure 12 on page 14: adding a beard to a woman changes perceived gender, while making a dog green produces cartoon-like texture. Sparse control does not automatically separate concepts entangled in the backbone.
The main quantitative set focuses on human edits and contains only 63 source images. Its results cannot directly establish performance across broad nonhuman attributes, complex multi-subject references, or real photographs.
This note's analysis: token selection is not a spatial mask, and a continuous parameter does not guarantee visual monotonicity. Future evaluation could separate spatial leakage, identity drift, and intensity monotonicity.
The cache lacks the appendix, preventing verification of training costs, exact hyperparameters, complete prompt lists, and real-image inversion settings; extraction problems are identified in the corresponding method paragraphs.
The readable main text does not provide participant counts, confidence intervals, or per-method response allocations. This note reports observed preference rates without adding claims of statistical significance.

Compared with Concept Sliders (reference 24): that method learns attribute-controllable LoRAs, whereas SAEdit learns a shared SAE and finds new directions through prompt pairs. The distinction concerns the controllable representation and new-attribute cost, not the elimination of training.
Compared with Prompt Sliders (reference 58): the former optimizes concept tokens, while SAEdit applies directions in a sparse representation of existing T5 embeddings. Different backbones and inversion conditions prevent treating all comparisons as single-variable ablations.
Compared with FluxSpace (reference 15): both target continuous semantic editing, while SAEdit emphasizes sparse coordinate selection and token localization. Its user-study advantage should be interpreted alongside the particular strength-control settings used for the baseline.
Compared with SAEuron and Concept-Steerers (references 14 and 38): those works use SAEs for concept unlearning or generation steering, whereas this paper targets continuous editing that preserves source content rather than establishing stronger general-purpose interpretability.
The primary source is the official ECCV paper page; the comparisons above follow the paper's bibliography without adding unverified project or code links.

Rating¶

These are the note author's subjective judgments on a 5-point scale, not conference review scores. - Novelty: 4/5. Combines a shared text SAE, attribute selection, and temporal control into a reusable continuous editing interface. - Experimental Thoroughness: 3/5. Includes several baseline families, ablations, and a user study, but the human-image set is limited and the cache lacks the appendix. - Writing Quality: 4/5. Clearly separates direction discovery from application, although reproduction requires the appendix and cached equations are damaged. - Value: 4/5. Useful for lightweight conditioning control and practical applications of interpretable representations, provided model priors and reconstruction errors remain explicit concerns.