Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking¶

Conference: ECCV 2026
arXiv: 2509.12046
Code: None
Area: Image Generation
Keywords: Autoregressive Image Generation, Layout-Controlled Generation, Structured Masking, GRPO Post-training, Text-to-Image

TL;DR¶

SMARLI proposes a structured masking strategy to explicitly control the attention interaction among three types of tokens (text prompts, layout, and image) in an autoregressive Transformer, achieving layout-controlled image generation for AR models without modifying the model architecture. Furthermore, it introduces GRPO post-training to AR layout-to-image generation for the first time, designing a composite layout reward containing VQA, mIoU, and precision, and outperforms most diffusion SOTA models on LayoutSAM-Eval and OverLayBench with only 1.3B parameters.

Background & Motivation¶

Background: The success of autoregressive (AR) models in large language models has extended to the field of image generation. Works like LlamaGen, Show-o, Emu3, and VAR demonstrate that the AR paradigm is a strong competitor to diffusion models. Among them, Show-o adopts a next-set prediction paradigm (predicting a set of tokens in parallel at each step), achieving a good balance between generation efficiency and quality.

Limitations of Prior Work: Layout-to-image (L2I) generation is a core scenario in controllable image generation, but the exploration of AR models on L2I is extremely limited. The mainstream approach for diffusion models in L2I is to introduce extra attention modules or branches to inject layout conditions (e.g., GLIGEN adds gated cross-attention layers, SiamLayout adds independent layout encoding branches). These methods disrupt the simplicity of the AR Transformer architecture, introducing significant parameter and computational overhead. Another ControlNet-style route (ControlAR, ControlVAR) relies on dense visual conditions like edge maps and depth maps, which are naturally unsuitable for sparse bounding box layouts.

Key Challenge: PlanGen first adopted a unified input sequence paradigm for L2I in AR models—directly concatenating layout tokens into the input sequence, avoiding architectural modifications. However, the standard causal attention mask treats the entire sequence equally, leading to two severe issues: (1) Semantic Interference: Layout tokens of different objects are mutually visible, causing image tokens to attend to layout information of irrelevant regions, which leads to feature entanglement and attribute confusion. (2) Insufficient Image Quality: The visual fidelity of PlanGen is significantly lower than that of diffusion SOTA, producing obvious artifacts in complex scenes.

Goal: To solve the semantic interference problem in the unified sequence paradigm while improving generation quality and layout accuracy, all while keeping the AR Transformer architecture unchanged.

Core Idea: Use a structured attention mask to explicitly define "who can see whom" among different token types—layout tokens are isolated by objects, and image tokens only attend to the layout of their corresponding regions. This injects the inductive bias of layout control without changing the model structure. This is further complemented by GRPO post-training, utilizing a composite layout reward to eliminate the exposure bias of AR models and further enhance quality and controllability.

Method¶

Overall Architecture¶

SMARLI uses Show-o (next-set prediction AR model) as the backbone, and the entire framework consists of three stages: tokenizing global text prompts, layout conditions, and images, then concatenating them into a unified sequence; introducing a structured masking strategy in the attention calculation of the AR Transformer to control the interaction mode of the three types of tokens; after SFT training, entering the GRPO post-training stage to jointly optimize the policy model using layout rewards and image quality rewards. The overall pipeline is shown in the figure below.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Global Text Prompt<br/>+ Bounding Boxes & Regional Descriptions<br/>+ Image"] --> B["Unified Layout Tokenization<br/>Text Encoding + Fourier Box Encoding<br/>→ Sequence Concatenation → MLP + Residual"]
    B --> C["Concatenate Unified Sequence<br/>[Prompt] [Layout] [Image/MASK]"]
    C --> D["Structured Masking Strategy<br/>Global Awareness / Intra-Object Causal<br/>Inter-Object Isolation / Regional Separation"]
    D --> E["SFT Training: next-set prediction"]
    E --> F["Layout GRPO Post-Training<br/>VQA + mIoU + Precision<br/>Composite Layout Reward × HPS Quality Reward"]
    F --> G["Layout-Controlled Image Generation"]

The input consists of a global text description, a set of object layouts composed of bounding box coordinates and regional text descriptions, and partially masked image tokens. The output is a complete image satisfying the layout constraints. The intermediate process goes through four key stages: unified sequence construction, structured mask attention, SFT, and GRPO post-training. Among them, "unified layout tokenization", "structured masking strategy", and "layout GRPO post-training" are the three core contribution nodes of this paper.

Key Designs¶

1. Structured Masking Strategy: Eliminating Cross-Object Semantic Interference

The core issue of the unified sequence paradigm is that the standard causal attention mask indiscriminately makes all tokens visible to each other. Information crosstalk exists between layout tokens of different objects, and image tokens may absorb noise from the layouts of irrelevant objects, causing attribute mismatch and spatial shift. SMARLI uses a set of structured masking rules to explicitly regulate the attention interaction patterns among the three types of tokens. Its design follows four principles.

First, Global Context Awareness: All layout tokens can attend to global text prompt tokens. This allows each region to acquire the semantic context of the entire image and understand its own role in the global scene. Second, Intra-Object Causal Consistency: A standard causal mask is adopted among layout tokens of the same object, enabling them to accumulate fine-grained spatial and semantic information of that object sequentially. Third, Inter-Object Isolation: Attention is completely blocked between layout tokens of different objects. This ensures that the layout representation of each object is not interfered with by other objects, avoiding attribute leakage (e.g., a "red shirt" leaking into a "blue skirt" region). Fourth, Regional Separation: Image tokens attend to all other image tokens and global prompt tokens, but only attend to layout tokens associated with their spatial location. The association is determined by checking whether the spatial position corresponding to the image token falls within the bounding box of an object. If a pixel falls inside multiple overlapping bounding boxes, the image token attends to all layout tokens of those objects simultaneously, where the global prompt provides context to disambiguate semantic relationships between objects.

These four rules constitute an inductive bias matching the sequence organization (Prompt-Layout-Image), injecting layout control signals directly at the attention computation level without introducing extra parameters or modifying the Transformer architecture. Ablation studies show that removing inter-object isolation and regional separation (w/o Local Causal) causes the spatial localization metric mIoU to drop from 80.08 to 78.39, and precision to drop from 80.69 to 77.04; removing global context awareness (w/o LP) severely harms attribute binding capability, with color dropping from 86.17 to 79.83.

2. Unified Layout Tokenization: Fusing Space and Semantics along the Sequence Dimension

A layout condition consists of two parts: bounding box coordinates \([x_0, y_0, x_1, y_1]\) (normalized to \([0,1]\)) and a regional text description (e.g., "a girl in a blue dress"). Diffusion models typically concatenate text tokens with Fourier-encoded box coordinates along the channel dimension and input them to an extra module. SMARLI's design principle is to fully reuse the pre-trained text representation capability of Show-o. Therefore, it makes two key decisions: (1) Use Show-o's text tokenizer \(E\) and codebook \(C\) to encode the regional description to obtain text tokens, while encoding bounding box coordinates into Fourier embeddings of the same dimension. (2) Concatenate text tokens and Fourier embeddings along the sequence dimension, rather than the channel dimension. The concatenated sequence then undergoes a transformation via a zero-initialized MLP with a residual connection:

\[x_i = \mathrm{Concat}\left(C(E(obj_i^{\text{text}})),\; \mathrm{Fourier}(obj_i^{\text{box}})\right)\]

\[obj_i^{\text{layout}} = x_i + \mathrm{MLP}(x_i)\]

The zero-initialized MLP ensures that layout tokens do not destroy the pre-trained text representation in the early stages of training, gradually incorporating spatial information as training progresses. Finally, the layout token sequence of each object, the global prompt tokens, and the image tokens are concatenated in the order of [Prompt][Layout][Image] and fed into the AR Transformer. This design naturally embeds the layout condition into the unified sequence framework of AR, eliminating the need for an extra condition encoder.

3. Layout GRPO Post-training: Composite Rewards Driving Quality and Controllability

AR models suffer from exposure bias caused by train-test discrepancy (using ground-truth tokens during training but using model-generated tokens during inference), and the single next-token prediction objective in the SFT stage cannot directly optimize image quality and layout accuracy. SMARLI introduces GRPO to AR layout-to-image generation for the first time, which requires solving two challenges.

The first challenge is defining actions under the next-set paradigm. In traditional next-token AR models under GRPO, each token represents an action, and log-probabilities across all timesteps can be computed in parallel with a single forward pass using causal masks. However, Show-o predicts and retains a set of high-confidence tokens at each step using the complete sequence as context; different timesteps may have different token values at the same spatial position, which cannot be reproduced with a fixed mask. SMARLI's solution is to explicitly record the set of retained tokens and their spatial locations at each step as a "composite action" and the current input sequence as the "state" during the rollout phase; during training, \(\pi_{\theta}\) and \(\pi_{\text{ref}}\) replay the same state-action trajectory step-by-step to compute the probability ratio \(r_{i,t} = \frac{\pi_{\theta}(o_{i,t} \mid c, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid c, o_{i,<t})}\) under the same context.

The second challenge is designing a reward function that improves layout accuracy without sacrificing image quality. SMARLI proposes a composite layout reward consisting of three components:

\[R_{\text{layout}} = \lambda_{\text{VQA}} \cdot S_{\text{VQA}} + \lambda_{\text{mIoU}} \cdot S_{\text{mIoU}} + \lambda_{\text{prec}} \cdot S_{\text{prec}}\]

\(S_{\text{VQA}}\) utilizes Qwen3-VL to determine whether the area inside each bounding box contains the specified attributes (color, shape, texture) through visual question answering (VQA), yielding a binary score. The limitation of this score is that it evaluates as positive as long as the subject appears within the region, without penalizing imprecise boundaries or false positives. \(S_{\text{mIoU}}\) and \(S_{\text{prec}}\) use Grounding DINO to detect objects in the generated image and compute mIoU and precision against the ground-truth layout, addressing spatial localization accuracy and false positives, respectively. The final advantage function is the weighted sum of the HPS image quality advantage and the layout advantage, and the weight \(\omega^{\text{layout}}\) is scaled by a factor of 1.2 for image tokens falling within the bounding boxes specified by the layout, making policy optimization more focused on layout-related regions.

A Concrete Example¶

Suppose the input is 'A photo of a living room', with a layout of two objects: (1) a red sofa on the left [0.05, 0.4, 0.45, 0.9], 'a red sofa'; (2) a green potted plant on the right [0.55, 0.3, 0.95, 0.85], 'a green potted plant'. During the tokenization phase, the text description of the red sofa is converted into text tokens by the Show-o text tokenizer, and its bounding box coordinates are encoded into vectors via Fourier encoding. The two sequences are concatenated and passed through the MLP plus residual connection to obtain the sofa layout tokens; the green potted plant is processed similarly. The global prompt tokens, two sets of layout tokens, and masked image tokens are concatenated in sequence to form a unified sequence.

Upon entering the AR Transformer, the structured mask takes effect: the sofa layout tokens can attend to the global prompt tokens (understanding the 'living room' scene) and the sofa's own layout tokens (accumulating the 'red' and 'sofa' semantics sequentially), but are completely blind to the potted plant layout tokens; the same applies to the potted plant layout tokens. During the image generation phase, image tokens falling within the sofa bounding box [0.05, 0.4, 0.45, 0.9] attend to the sofa layout tokens + all image tokens + global prompts, thereby generating the red sofa texture; tokens falling in the overlapping region attend to both sets of layout tokens, with the global semantic prompt 'living room' coordinating their relationship.

After SFT output, the GRPO phase uses Qwen3-VL to verify if the sofa region is indeed red and the potted plant region is green, uses Grounding DINO to detect generated object locations and computes mIoU/precision against the ground-truth layout, and updates the strategy using the combined HPS quality score. Ablation experiments show that when using HPS alone, the color metric drops from 86.17 to 83.96 (the model sacrifices attribute accuracy for higher aesthetic quality), while the composite reward recovers color to 87.35 and increases IR from 60.88 to 74.81.

Loss & Training¶

The SFT stage adopts Show-o's native next-set prediction loss. During training, image tokens are randomly masked according to a ratio, and the model learns to predict the masked tokens. The optimizer is AdamW with a fixed learning rate of \(2 \times 10^{-5}\), trained for 30,000 iterations with a global batch size of 128, at a resolution of \(512 \times 512\), on 8 AMD MI300X GPUs for 3 days. Trainable parameters include the entire AR Transformer and the MLP in the layout tokenizer.

In the GRPO post-training stage, the group size is \(G=4\), the number of generation steps is 10, the batch size is 28, and the learning rate is \(1 \times 10^{-4}\) for 100 iterations. Each Transformer block is fine-tuned using LoRA (rank=256). The rollout temperature is set to 1, and the classifier-free guidance scale is 5. \(\beta\) is set to 0 (consistent with DanceGRPO), \(\omega^{\text{hps}}=\omega^{\text{layout}}=1\), and the \(\omega^{\text{layout}}\) for image tokens within the layout boxes is increased to 1.2. The layout reward weights are \(\lambda_{\text{VQA}}=0.5\), \(\lambda_{\text{mIoU}}=0.5\), and \(\lambda_{\text{prec}}=0.1\). The GRPO phase requires only 4 hours on 8 AMD MI250 GPUs, making the training cost much lower than the SFT stage.

Key Experimental Results¶

Main Results¶

LayoutSAM-Eval Results: With 1.3B parameters, SMARLI outperforms diffusion model methods that are several times larger (SiamLay-Flux 22B, InstsAssemb 2B) on most metrics, showing significant advantages particularly in fine-grained attribute control such as color (87.35 vs. SiamLay-Flux 80.71), texture (90.90 vs. 83.53), and shape (89.82 vs. 82.80). Compared to PlanGen (1.5B), which also uses the AR unified sequence paradigm, SMARLI leads across the board in layout control, and its image quality is substantially improved (IR 74.81 vs. 39.58). Its FID of 18.16 outperforms most diffusion methods and is close to SiamLay-Flux (16.12).

Method	Parameters	Spatial	Color	Texture	Shape	mIoU	Precision	IR	FID
GLIGEN	1.2B	77.53	49.41	55.29	52.72	69.62	70.99	-10.31	21.92
InstanceDiffusion	1.4B	87.99	69.16	72.78	71.08	81.29	72.34	9.14	19.67
SiamLay-Flux	22B	95.67	80.71	83.53	82.80	78.01	78.33	80.48	16.12
InstsAssemb	2B	94.97	77.53	80.72	80.11	73.33	74.19	70.14	20.24
PlanGen	1.5B	92.21	82.69	86.53	85.36	78.04	81.15	39.58	13.91
SMARLI	1.3B	95.55	87.35	90.90	89.82	81.31	81.84	74.81	18.16

OverLayBench-Complex Results: In complex layout scenarios with a large number of overlapping bounding boxes, SMARLI achieves the best scores across mIoU of 56.42 (vs. SiamLay-Flux 54.50), O-mIoU of 29.59 (mIoU of overlapping regions, vs. SiamLay-Flux 28.97), and entity success rate SRE of 80.52 (vs. SiamLay-Flux 69.72). This demonstrates that the inter-object isolation design of the structured mask is particularly effective in handling semantic confusion for overlapping objects.

T2I-CompBench Results: After introducing layout conditions generated by GPT-5.2, SMARLI's spatial reasoning score increases from 39.6 (compared to Show-o without layout) to 49.7, with noticeable gains in color, texture, and numeracy, indicating that layout conditions not only improve spatial control but also exhibit positive transfer to attribute binding and counting capabilities.

Ablation Study¶

GRPO Reward Combination Ablation (Based on SFT baseline SMARLI-SFT, LayoutSAM-Eval):

Configuration	Spatial	Color	Texture	Shape	mIoU	Precision	IR
SFT only (Baseline)	95.26	86.17	89.79	88.56	80.08	80.69	60.88
Only HPS	93.82	83.96	87.72	86.78	77.79	77.89	76.62
Only Layout (VQA+DET)	95.41	87.04	90.15	89.32	80.76	80.81	58.34
HPS + VQA	95.48	87.30	90.01	88.97	77.73	74.50	73.36
HPS + DET	95.13	85.61	89.45	88.29	80.70	80.88	73.05
HPS + VQA + DET (Full)	95.55	87.35	90.90	89.82	81.31	81.84	74.81

Key Findings¶

HPS and Layout Rewards are Complementary: Although using the HPS reward alone significantly improves image quality (IR 60.88 \(\to\) 76.62), it severely degrades layout control—color drops from 86.17 to 83.96, and mIoU drops from 80.08 to 77.79. The model sacrifices attribute accuracy in pursuit of "aesthetic appeal". Conversely, using the layout reward alone slightly improves layout accuracy but degrades image quality. Combining the two leads to an IR improvement to 74.81 while layout control improves comprehensively, achieving a Pareto improvement.
VQA and Detection Rewards are Complementary: The VQA reward (HPS+VQA) excels at attribute alignment (Color 87.30), but its mIoU (77.73) and precision (74.50) are actually lower than the SFT baseline—because VQA awards positive scores as long as the object is inside the box, without penalizing low IoU or false positives. The detection reward (HPS+DET) effectively restores mIoU and precision but reduces attribute consistency. Combining both achieves the optimal balance.
Inter-Object Isolation in Structured Mask is Key: Removing inter-object isolation and regional separation (w/o Local Causal) causes a drop of 1.69 percentage points in mIoU and 3.65 percentage points in precision, far exceeding the impact of removing global context awareness (which drops color by 6.34 percentage points), showing that cross-object semantic interference is the main bottleneck of the unified sequence paradigm.
AR Models Can Outperform Much Larger Diffusion Models: With only 1.3B parameters, SMARLI outperforms the 22B SiamLay-Flux across all layout control metrics on LayoutSAM-Eval, showing a pronounced advantage in fine-grained attributes (color, texture, shape). This indicates that the potential of AR models in controllable generation has been underestimated.

Highlights & Insights¶

Refined Design of the Four Principles for Structured Masking: Global context awareness, intra-object causal consistency, inter-object isolation, and regional separation—these four rules cover all meaningful interaction combinations among the three token types (prompt, layout, image) without omission or redundancy. This explicit modeling of "who should see whom" is elegant and simple, and is implemented entirely at the attention level with zero-parameter injection, allowing seamless transfer to any AR Transformer.
Clear Division of Labor in Composite Layout Rewards: VQA manages semantic attributes (correctness of color/shape/texture), mIoU manages spatial accuracy (alignment of bounding boxes), and precision manages false positives (whether objects are generated in unintended areas). The three metrics perform their respective duties and mutually correct each other. This decomposed design can be extended to other generation tasks requiring fine-grained spatial and semantic joint control.
Ingenious Validation of the Exposure Bias Issue: The phenomenon in the ablation study where using HPS alone leads to a comprehensive decline in layout control is essentially exposure bias of the SFT model. The generated images during inference deviate from the training distribution; the HPS reward guides the model toward a "beautiful but incorrect" distribution, and the layout reward serves as an anchor. This provides empirical support for the necessity of RL post-training in AR image generation.
Spillover Benefits of Layout Conditions to T2I: After introducing layouts on T2I-CompBench, numeracy (counting capability) also improved (62.3 \(\to\) 68.4), suggesting that spatial constraints, as an extra structural signal, can help the model better perform compositional text understanding—an inspiring phenomenon for pure text-prompt T2I models.

Limitations & Future Work¶

Relatively Simple MLP Design in the Layout Tokenizer: Currently, only a zero-initialized MLP with residual connections is used to fuse text tokens and Fourier embeddings, which is essentially a shallow linear combination of two modalities. More complex layout conditions (e.g., free-form masks, keypoints, 3D bounding boxes) might require a stronger conditional encoder.
Data Filtering in GRPO Post-training Relies on Grounding DINO Pre-screening: The training data needs to filter out samples where Grounding DINO detection mIoU/precision does not reach 0.9, meaning the reliability of the reward signal is limited by detector performance. If the detector itself is inaccurate on specific classes, layout rewards might mislead the training.
Unexplored Open-Vocabulary Layout Control: Current layout descriptions use fixed regional text, and whether the model can generalize to object category and description combinations unseen during training has not been tested.
Only Validated on Show-o and Janus-Pro-1B Backbones: The paper shows transfer results on Janus-Pro-1B in the supplementary material, but has not validated the universality of structured masking on more AR architectures (such as VAR, LlamaGen).
Future Directions: (1) Extend the concept of structured masking to spatio-temporal layout control in video generation; (2) Replace Qwen3-VL with a stronger VLM for VQA evaluation to improve attribute detection accuracy; (3) Explore online GRPO (i.e., alternating rollout and training) to further alleviate exposure bias.

vs GLIGEN / InstanceDiffusion: The layout control schemes of diffusion models are implemented by adding trainable gated cross-attention layers or instance-level condition injections, where the core cost is architectural modifications and extra parameters. SMARLI replaces modules with masks, maintaining the AR Transformer structure. Conceptually, it is closer to "teaching the model to look at what it should look at" rather than "adding a new organ to the model".
vs PlanGen: As both belong to the AR unified sequence paradigm, PlanGen relies on a standard causal mask that leads to semantic interference. SMARLI's structured masking serves as a key patch to this paradigm. Although PlanGen has a lower FID (13.91), its IR is only 39.58 (vs. SMARLI's 74.81), indicating that PlanGen's images suffer from insufficient perceptual quality, and its FID advantage might arise from overfitting to the training distribution.
vs ControlAR / ControlVAR: Both follow the ControlNet approach by injecting encoded visual conditional feature maps into AR models. This paradigm is suitable for dense alignment conditions (edge maps, depth maps) but unsuitable for sparse layouts. SMARLI's sequence concatenation + masking scheme is a more natural way of layout injection, eliminating the need to train a conditional encoder.
Insight: The "type-based controlled attention" idea of structured masking can be abstracted into a general AR condition injection paradigm—applicable not only to layouts, but any condition that can be decomposed into "conditions associated with specific regions/segments" (e.g., reference images, sketches, human poses) can be integrated by defining new token types and masking rules, without altering the model structure.

Rating¶

Novelty: ⭐⭐⭐⭐ Structured masking translates layout control into attention rule design, a concept proposed for the first time in AR image generation; introducing GRPO post-training to L2I is also a first, though the backbone model Show-o is not self-developed.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers two L2I benchmarks + T2I-CompBench + rich ablation studies (individual components of masking strategies, individual sub-items of GRPO rewards, layout tokenizer design, weight hyperparameters), showing high completeness.
Writing Quality: ⭐⭐⭐⭐ Clear chain of motivation, concise presentation of the four principles of structured masking, and adequate explanation of how GRPO adapts to next-set prediction; some experimental details are relegated to the supplementary material, but the main text's ablation studies are well-organized despite their abundance.
Value: ⭐⭐⭐⭐ Opens a simple and effective entry point for AR models in controllable image generation. The design logic of structured masking and composite layout rewards features strong transferability; the lack of open-source code is currently the primary drawback.