Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Models

Conference: ECCV2026
Paper: Official page · PDF
Area: Image Generation
Keywords: Visual autoregression, attribute disentanglement, semantic directions, continuous control, cross-category editing

TL;DR

ATA learns and subtracts two reconstruction directions from one reference image and prompts differing only in a target attribute, enabling compositional strength control in frozen Infinity text-conditioning space; in a same-backbone comparison, mean ΔVQA rises from 0.532 with the Concept Slider strategy to 0.567, and mean I-LPIPS from 0.879 to 0.895.

Background & Motivation

Making someone slightly older is not the same as replacing them with an older person. Instruction-based editors can understand a requested attribute without reliably preserving identity, background, and unrelated attributes. Replacing slightly with moderately or extremely in a prompt is also not equivalent to a continuous numerical control that composes independently with other edits. Character design and product exploration need precisely this ability to make controlled adjustments while retaining existing content.

Visual autoregressive models represent images through multiscale discrete residuals and predict them from coarse to fine. Their generation efficiency is attractive, but discrete image tokens do not provide an obvious continuous editing space. ATA instead operates on the continuous text conditions after the encoder and alignment MLP, before cross-attention. Subtracting text tokens from prompts with and without an attribute already produces some localized changes. However, that difference is not calibrated to a reference image and does not explicitly distinguish the identity of a particular cat from the meaning of fatness.

The authors therefore make two prompts reconstruct the same reference image. A prompt without the attribute must supply both the attribute and instance details through its correction, whereas a prompt already containing the attribute mainly needs instance-specific information. Core idea: learn these two conditioning corrections with the generator frozen, then subtract their shared instance-related factors to obtain an attribute direction that can be scaled, combined, and reused across categories.

Method

Overall Architecture

ATA takes a reference image and a pair of prompts with a well-defined semantic difference. It produces semantic directions that can be stored in an offline library, rather than a retrained generator. Infinity supplies the VAR backbone: frozen Flan-T5 encodes text, an alignment MLP maps it into VAR conditioning space, and the image encoder supplies multiscale residuals. The generator predicts progressively through self-attention, cross-attention, and feed-forward layers. The reported setup uses 13 scales, from \(1\times1\) to \(64\times64\), to generate \(1024\times1024\) images.

During learning, a lightweight attention module called Delta-Mod consumes the reference residual features and predicts paired directions. Both corrected text conditions must reconstruct the reference. At inference, instance directions and attribute differences are retrieved from the library and injected at the appropriate text positions before frozen Infinity generates an image. The continuous arithmetic changes conditioning vectors, not the integer indices of discrete image tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Reference image and paired prompts"] --> Pair["Paired-direction reconstruction<br/>Delta-Mod learns two directions"]
    Pair --> Difference["Disentanglement by subtraction<br/>Build an attribute library"]
    Difference --> Injection["Semantic-position injection<br/>Compose and scale directions"]
    Prompt["New prompt and instance directions"] --> Injection
    Injection --> Output["Frozen Infinity<br/>Multiscale edited-image generation"]

Key Designs

1. Paired-direction reconstruction: make two text conditions reach the same reference image

For a photograph of a fat cat, the general prompt is “A cat on the ground.” and the specific prompt is “A fat cat on the ground.” They share the object and context, concentrating the difference on fat. Delta-Mod uses the reference image's multiscale residual features as attention keys and values, with two learnable query vectors, to output \(d_{\mathrm{general}}\) and \(d_{\mathrm{specific}}\). Each direction is added at the corresponding token positions of its prompt, and the same frozen backbone reconstructs the reference under either corrected condition.

There are no ground-truth direction labels and no requirement for paired edited images. Supervision comes from the original VAR reconstruction objective: a direction that fails to supply missing conditioning information cannot accurately predict the reference residuals. The general condition requires a larger semantic correction because it omits fatness. The specific condition already expresses fatness, so its learned correction is more instance-specific. One Delta-Mod can also learn directions for multiple concepts in an image. Crucially, avoiding backbone retraining does not mean avoiding optimization altogether.

2. Disentanglement by subtraction: isolate the attribute difference between two reconstructions

The paper explains the operation using conditional distributions. Cats form a broad distribution, fat cats form a subdistribution, and the reference image lies within the latter. The general direction points from the broad distribution's center toward that instance; the specific direction starts at the subdistribution's center and reaches the same instance. Their difference should retain the shift from the general concept to its attribute-conditioned subset. The algorithm does not explicitly estimate these distribution means: reconstruction optimization learns the directions instead. This geometry motivates the design but does not prove statistical independence.

\[ d_{\mathrm{attribute}}=d_{\mathrm{general}}-d_{\mathrm{specific}}. \]

The shared endpoint is essential to this argument. Directions learned toward different instances could leave identity, texture, or background differences after subtraction. ATA uses the same reference to encourage cancellation of those shared factors and stores the resulting attribute direction in an offline library. It can then preserve a different image's instance direction while introducing this attribute, such as transferring fatness learned from a cat to a chair. What transfers is intended to be a semantic property, not reference pixels or the entire cat. Whether the factors are actually disentangled must still be tested through edited outputs.

3. Semantic-position injection: connect the strength control to the correct text location

At inference, image-specific or general attribute directions are added to VAR-aligned text tokens. The basic control operation is shown below, where \(T_{\mathrm{attr}}\) is the selected attribute-conditioning token and \(\gamma\) controls strength. Multiple attributes are handled by combining their directions at the corresponding positions.

\[ \widehat{T}_{\mathrm{attr}}=T_{\mathrm{attr}}+\gamma d_{\mathrm{attribute}}. \]

Increasing or decreasing \(\gamma\) can strengthen or weaken an attribute, but perceptual change need not be linear in that value, and arbitrarily large strengths are not guaranteed to work. Position matters just as much: in the fat-cat example, the paper treats “A” and “A fat” as attribute-side tokens and “cat” as the identity token. Optimizing the fatness direction on “cat” changes the subject's identity; adding it to “ground” does not make the cat fatter. This separation concerns semantic roles in conditioning, not an explicit spatial mask.

Directions can also be injected at selected generation scales, since coarse scales establish global structure and finer scales supply local details. The main text mentions this scale-selection hyperparameter but refers its details to an appendix. The local cache contains the main paper and references, not that appendix, so it does not establish an exact scale range or default \(\gamma\). Cross-attention's response to the modified conditions and the backbone's existing semantic knowledge jointly determine the edit.

A Worked Example

Start with the fat-cat reference and the paired prompts above. Learn two directions that both reconstruct that cat, then subtract them to obtain a general fatness direction. The cat's instance information can be retained separately, or only the attribute direction can be kept for other objects. Reusing an already learned attribute does not require learning it again for every edit.

For chair generation, preserve the chair's object condition and inject the fatness direction at the semantically appropriate attribute position. Varying its strength should produce a fuller chair design rather than replace the chair with a cat. Another furry direction can be combined with it, allowing shape and furry appearance to change separately. Figure 1 demonstrates this type of composition; no unreported numerical strengths or success rates are assigned to these illustrative stages here.

Loss & Training

Only Delta-Mod is optimized; the VAR backbone remains frozen. The original model's cross-entropy objective supervises multiscale residual prediction. Each prompt receives its corresponding correction direction, and both conditions target the same reference image. The cache reports no additional orthogonality, contrastive, or explicit identity loss, so direction subtraction should not be described as disentanglement training with those constraints.

The reported environment is a single NVIDIA H100, PyTorch 2.5.1, and CUDA 12.2, with 500 optimization steps, 5 epochs, and batch size 1. The main text does not fully explain how steps and epochs are organized, nor provide a reproducible learning rate or detailed timing. These settings are retained as reported without inferring training duration.

Key Experimental Results

Main Results

The Control Dataset covers humans and common objects, with 15 prompts per category, 2 seeds per prompt, and 6 attribute sets. Each source image is edited using two attributes at three intensity levels. For GEdit, 52 attribute-editing samples are randomly selected. Gemini generates attribute prompt pairs sharing the same context to build the direction library, and Qwen2.5-VL-72B serves as the evaluation backbone. Instruction-based baselines approximate control with slight, moderate, and extreme wording rather than a numerical control equivalent to ATA's.

The following selection from Table 2 uses moderate intensity. The Concept-slider row contains absolute scores; every other row reports percentage improvement relative to that baseline. They must not be interpreted as absolute values or percentage-point gains. ΔVQA measures semantic shift toward the target attribute; I-LPIPS is an inverted LPIPS-based preservation metric. S.C., P.Q., and Disen. denote semantic consistency, perceptual quality, and disentanglement, respectively, with higher values preferred. Disen. monitors unintended changes in one attribute while another is edited, but the main text does not provide its full scoring formula. Exact computations for ΔVQA and I-LPIPS are also insufficiently specified, so no missing metric formulas are invented here.

Method / moderate	ΔVQA	I-LPIPS	S.C.	P.Q.	Disen.
Concept-slider, absolute scores	0.513	0.847	4.826	7.261	5.018
Qwen-Image, relative improvement	+5.31%	+5.44%	+6.20%	+5.11%	+3.86%
VAREdit, relative improvement	+6.30%	+4.75%	+5.72%	+5.75%	+4.52%
ATA, relative improvement	+12.72%	+5.17%	+16.33%	+3.16%	+9.68%

ATA's advantage concentrates on semantic change and disentanglement, not every metric: Qwen-Image has the larger I-LPIPS improvement at moderate intensity, and VAREdit has the larger P.Q. improvement. ATA's relative ΔVQA gains at slight, moderate, and extreme intensity are +11.44%, +12.72%, and +14.58%. Improvements at these three discrete evaluation levels do not prove strict monotonicity over the entire control curve.

Ablation Study

Table 4 compares unlearned raw text-token differences with ATA under the same GEdit protocol. All entries below are absolute mean scores.

Method	Semantic consistency	Perceptual quality	Disentanglement
Direct text-token difference, training-free	5.048	7.447	5.296
ATA, Infinity-2B	5.374	7.584	5.455

Table 5 additionally fixes the Infinity-2B backbone and tests prompt perturbations. These entries are also absolute means.

Control Dataset setting	Mean ΔVQA	Mean I-LPIPS
Infinity-2B, Concept Slider strategy	0.532	0.879
ATA, original prompts	0.567	0.895
ATA, perturbed prompts	0.541	0.864

Key Findings

• Learning directions instead of using raw differences increases semantic consistency by 0.326, perceptual quality by 0.137, and disentanglement by 0.159. This supports reference-image calibration, but does not independently establish that every component is indispensable.
• With the backbone fixed, ATA exceeds the Concept Slider strategy by 0.035 ΔVQA and 0.016 I-LPIPS, reducing the backbone confound in the main comparison. Prompt perturbations reduce the two metrics by 0.026 and 0.031, respectively, showing that template changes, lexical paraphrases, and context deletion still affect performance.
• Figure 6's identity-token and unrelated-token injection tests are qualitative ablations without numerical scores. In the 2AFC study with 30 participants, ATA's reported preference rates are 37% for image preservation and 43% for prompt adherence. These aggregate preferences should not be interpreted as identical pairwise win rates against every competing method.

Highlights & Insights

• The distinction between attribute information and reference-instance information is built into paired reconstruction conditions, instead of assuming that an arbitrary vector is transferable. Sharing a reference gives subtraction a concrete cancellation rationale, which is the most reusable part of the design.
• Continuous control does not require continuous image representations. A discrete generator can expose controllable semantic directions through its continuous text-conditioning interface without changing its image tokenizer or backbone parameters.
• An offline direction library separates attribute learning from subsequent editing and can compose instance, appearance, and environmental concepts. It avoids repeatedly learning a reused attribute, but the main text lacks timing evidence for a specific speedup factor.

Limitations & Future Work

• The authors explicitly show artifacts or unrelated changes at excessive strength, difficulty with global color transformations, and failures when the backbone does not understand a cross-domain mapping. Applying a cat concept to a woman does not yield the intended cat-styled person, so cross-category transfer is not unconditional generalization.
• Subtraction depends on approximately shared instance factors and usable semantic positions. Quantitative degradation under prompt changes and failures at incorrect token positions show that disentanglement is not an automatically guaranteed independence property.
• Evaluation includes a small controlled dataset, 52 GEdit samples, and 30 participants. Main baselines differ in size and backbone, while instruction-based editors approximate strength through wording. Same-backbone ablations help, but confidence intervals, broader backbone validation, and systematic strength-curve analysis remain missing.
• Reproducibility has clear boundaries: the local main-text cache lacks the referenced scale-selection appendix, some extracted equations are damaged, and metric details and timings are incomplete. Useful extensions would report joint scale-strength sweeps, monotonicity of perceptual change, library construction costs, and reuse costs rather than merely emphasizing generation speed.

Related Work & Insights

• Compared with Concept Sliders: Concept Sliders adjust the model through attribute-related LoRA modules, whereas ATA learns directions injected into conditioning tokens while freezing the backbone. The Infinity-2B comparison tests this parameterization choice more directly than the main comparison across SDXL and Infinity.
• Compared with AREdit and VAREdit: AREdit emphasizes training-free representation editing, while VAREdit relies on instruction-editing fine-tuning. ATA provides attribute directions obtained through lightweight optimization that can be stored and composed; it should not be classified as fully training-free.
• Compared with TokenVerse / XVerse: These approaches also address controllable composition of identities and semantic concepts. ATA's specific contribution concerns VAR-aligned text-conditioning space and paired-direction subtraction using a shared reference, not proof that all generators possess universal linear semantic coordinates.

Rating

• Novelty: 4/5. Paired reconstruction differences and VAR conditioning injection form a concise combination, although semantic vector arithmetic and attribute control have clear precedents.
• Experimental Thoroughness: 3/5. Raw text differences, same-backbone comparisons, and prompt sensitivity are tested, but scale, statistical uncertainty, and efficiency evidence remain limited.
• Writing Quality: 4/5. The shared-endpoint geometric explanation is clear; metric definitions, optimization details, and scale settings need fuller specification.
• Value: 4/5. Useful for reusable attribute control and lightweight personalization, subject to backbone priors, strength limits, and prompt-position constraints.