Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning¶
Conference: ECCV 2026
Paper: ECCV paper page
Area: Image Generation
Keywords: Interleaved reasoning, process supervision, unified multimodal models, scene graphs, self-critique
TL;DR¶
STIR constructs progressive drawing trajectories with scene graphs and supervises both textual conflicts and image-instruction misalignment, teaching BAGEL-7B to alternate planning, sketching, inspection, and refinement while improving GenEval from 0.79 to 0.83 and WISE from 0.70 to 0.76.
Background & Motivation¶
Text-to-image models can render realistic objects without correctly rendering their relationships: a prompt requesting a bear hovering above a spoon may produce a bear beside it. Single-pass generation must resolve layout, object count, and attribute binding together. Even a preceding textual chain-of-thought cannot inspect the image that will subsequently be generated. External feedback on a finished image can help, but it does not establish that the generator can decide what to do while a scene is still incomplete.
The central difficulty is that an intermediate image should not be judged directly against every requirement of the final prompt. For a cat standing on a bench with a mouse to the left of the bench, drawing only the bench first can be valid progress; drawing the mouse on the left end of the seat may already violate the intended spatial relation. The model must track both unfinished content and mistakes in content already rendered. This calls for process supervision with local objectives, global constraints, and visual feedback, rather than simply more completed images.
The paper anchors this supervision in semantically recognizable states rather than blurry latent states along a denoising trajectory. Core Idea: first teach a model to assemble scenes incrementally through objects and relations, then use its own generated trajectories to distinguish unfinished work from genuine errors, internalizing textual reasoning and visual correction within one model.
Method¶
Overall Architecture¶
STIR builds on BAGEL-7B, which supports both understanding and generation. Its input is a text prompt, optionally accompanied by an image to edit; its output includes interleaved plans, drafts, critiques, corrections, and the final image. It does not predict brush-stroke coordinates or produce a vector drawing program: “strokes” refers to semantically incremental image construction.
On the training side, Scene-Graph Trajectory Construction is followed by Dual-Stream Critique Supervision, and both train Unified Interleaved Generation. At inference time, a single model describes the intended modification and resulting scene, draws a draft, checks consistency, corrects it when necessary, and decides whether to continue. GPT and Flux-Kontext construct training data; they are not required online inference components.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Training prompts"] --> B["Scene-Graph<br/>Trajectory Construction"]
B --> C["Dual-Stream<br/>Critique Supervision"]
B -.->|Training trajectories| D["Unified<br/>Interleaved Generation"]
C -.->|Training critiques| D
E["Inference prompt<br/>Optional input image"] --> D
D -->|Plan, sketch, inspect, refine| D
D -->|Completed| F["Final image"]
Key Designs¶
1. Scene-Graph Trajectory Construction: give incomplete images valid intermediate objectives
Data construction first represents the complete prompt as a scene graph, with object nodes, attribute nodes, and relation edges specifying the target composition. Subgraph sampling then yields progressively expanding local objectives. A scene containing only a bench no longer necessarily indicates a missing cat: it corresponds to a defined intermediate subgraph. Subsequent additions should preserve the spatial and semantic consistency of the existing scene. Flux-Kontext synthesizes target images for the steps, and GPT filters the trajectories, providing paired supervision for what to do and what the resulting state should look like.
Expanding subgraphs alone tends to restrict generation to adding content, whereas correction often requires attribute changes, swaps, or removals. GPT therefore rewrites a subset of step instructions to introduce these operations while preserving the intended semantics. This expands the state transitions the model can execute, rather than merely diversifying wording. The multi-turn generation subset in Table 1 contains 32,012 samples, averaging 3.51 images per sample with a maximum of 5. These describe training trajectories, not a fixed inference length.
2. Dual-Stream Critique Supervision: distinguish a drifting plan from failed visual execution
The first stream checks whether textual intermediate states conflict with the complete prompt. After fine-tuning on multi-turn generation data, the authors sample actual model trajectories and ask GPT to judge whether intermediate scene descriptions remain consistent with the original prompt. Conflicts receive an analysis and a corrective instruction. Self-sampling exposes the model's own failure modes while teaching it to preserve valid existing information instead of treating every unfinished object as a defect. This subset contains 15,201 samples: 6,905 positive and 8,296 negative.
The second stream checks whether the image implements the current drawing instruction. The authors extend and curate Gen-Ref, attaching alignment explanations to correct image-text pairs and error analyses plus refinement instructions to mismatched pairs. A correct plan can still yield the wrong object count, position, or attribute, so textual consistency cannot replace visual inspection. This subset contains 15,000 samples: 5,000 positive and 10,000 negative. Although the prose calls it balanced, Table 1 explicitly gives a 1:2 ratio; this note preserves the table counts.
3. Unified Interleaved Generation: make inspection change the subsequent image
Each Plan stage produces two kinds of text: <ins>...</ins> specifies the incremental addition or modification, while <des>...</des> describes the intended overall scene. Sketch generates an intermediate image from this text and the previous multimodal history. Inspect compares the stage plan with the original prompt and the draft with its current instruction. If a mismatch exists, the model emits corrective information within <refine>...</refine>, and Refine generates an improved image. A “sketch” here is an interpretable partial scene, not necessarily a low-resolution image or a blurry denoising state.
Text and images share an interleaved context, allowing later decisions to access previous plans, drafts, and corrections. Special visual boundary tokens control modality transitions: text is autoregressive, while image generation follows BAGEL's Rectified Flow mechanism. Thus, an “autoregressive interleaved sequence” does not mean that every image pixel is predicted as a discrete token. Trajectory length can vary with the task, and the model terminates after completing the final image rather than applying an identical number of edits to every prompt.
A Worked Example¶
Consider the Figure 1 prompt about a cat standing on a bench and looking down at a black computer mouse to the left of the bench. Constructing the bench before adding the mouse and cat grounds objects and relations progressively. The temporary absence of the cat should not immediately trigger a missing-object penalty when its generation step has not yet occurred.
Suppose adding the mouse yields an intermediate description placing it on the left end of the seat. Inspect compares that description with the complete requirement, “to the left of the bench,” and identifies a relational conflict. The correction removes the mouse from the seat and places it on the ground to the left of the bench before generation continues from the corrected scene. This also illustrates why additive-only operations cannot support every correction.
A different failure occurs when the textual plan is correct but the image contains two birds instead of one. The user objective does not need reinterpretation; image-instruction alignment inspection must identify the count error and remove the extra object. These examples illustrate the responsibilities of the two supervision streams, not a fixed execution order or number of rounds prescribed for every prompt.
Loss & Training¶
Textual segments use next-token cross-entropy, with supervision also applied to visual boundary tokens such as <|vision_start|> and <|vision_end|> so that the model learns when to switch from reasoning to drawing. Visual segments use the Rectified Flow mean-squared-error objective, conditioned on the original prompt and preceding interleaved trajectory. The total objective combines textual and visual losses with a weighting coefficient. This is supervised fine-tuning, not a newly introduced reinforcement learning objective.
Equations (1) through (6) are visibly corrupted in the text extraction, particularly the latent interpolation, expectation notation, and total-loss subscripts. This note therefore does not reconstruct the authors' exact equations. The supported mechanism is that textual supervision trains reasoning content and modality transitions, while visual supervision trains intermediate and final image generation conditioned on context. The available main text does not clearly specify the loss-weight value.
Training uses 8 NVIDIA H100 GPUs, fine-tuning all parameters end-to-end for 10,000 steps with packed sequences of 33,000 tokens, a learning rate of \(2\times10^{-5}\), and cosine decay. The three subsets sum to 62,213 samples, rounded to 62K in the process-baseline comparison. Sample count and the number of intermediate images are different quantities.
The termination description also requires caution: Section 3.3 says an end-of-sequence token marks completion after the final image, while Section 4.1 describes stopping when a visual end marker is not followed by a visual start marker. The model autonomously terminates the interleaved trajectory, but the complete implementation-level token protocol cannot be established from this cache alone.
Key Experimental Results¶
Main Results¶
The following results are selected from the paper's Table 2 on GenEval; higher is better. GenEval evaluates objects, counts, colors, spatial positions, and color-attribute binding. Overall scores are copied from the source rather than recomputed from the displayed columns.
| Model | Single object | Two objects | Counting | Colors | Position | Color attribute | Overall |
|---|---|---|---|---|---|---|---|
| FLUX.1-dev (12B) | 0.98 | 0.93 | 0.75 | 0.93 | 0.68 | 0.65 | 0.82 |
| Janus-Pro-7B | 0.99 | 0.89 | 0.59 | 0.90 | 0.79 | 0.66 | 0.80 |
| BAGEL-7B, authors' reproduction | 0.99 | 0.95 | 0.76 | 0.87 | 0.51 | 0.60 | 0.79 |
| STIR | 0.99 | 0.95 | 0.75 | 0.87 | 0.72 | 0.69 | 0.83 |
Relative to BAGEL, STIR gains 0.04 overall, with the largest changes in Position (+0.21) and Color Attribute (+0.09). Counting decreases by 0.01, while Single Object, Two Objects, and Colors remain unchanged. STIR has the highest overall score among the unified multimodal models in this table, but its Position score remains below Janus-Pro-7B's 0.79. This is not a lead on every subtask.
The next table selects results from Table 3 on WISE, which tests whether world knowledge is expressed correctly in generated images.
| Model | Culture | Time | Space | Biology | Physics | Chemistry | Overall |
|---|---|---|---|---|---|---|---|
| FLUX.1-dev | 0.48 | 0.58 | 0.62 | 0.42 | 0.51 | 0.35 | 0.50 |
| BAGEL | 0.76 | 0.69 | 0.75 | 0.64 | 0.75 | 0.58 | 0.70 |
| STIR | 0.74 | 0.82 | 0.73 | 0.70 | 0.76 | 0.78 | 0.76 |
Time improves by 0.13 and Chemistry by 0.20, whereas Culture and Space each decrease by 0.02. The source's “nearly 15%” summary should therefore not be applied uniformly to the two improving categories; the absolute table differences are more precise. The overall improvement supports the usefulness of process supervision, but does not establish a comprehensive increase in world knowledge itself.
Ablation Study¶
Table 7 separately introduces the two consistency-supervision streams, showing that they address different failures. Multi-turn SFT below is the baseline for this particular ablation and should not be conflated with configurations in other ablations.
| Config | Counting | Colors | Position | Color attribute |
|---|---|---|---|---|
| Multi-turn SFT | 0.61 | 0.84 | 0.66 | 0.62 |
| Add instruction-intermediate conflict supervision | 0.62 | 0.85 | 0.71 | 0.65 |
| Add image-instruction alignment supervision | 0.73 | 0.86 | 0.69 | 0.65 |
| Add both supervision streams | 0.75 | 0.87 | 0.72 | 0.69 |
Textual conflict supervision improves Position from 0.66 to 0.71; image-instruction alignment improves Counting from 0.61 to 0.73. Combining the streams outperforms either alone on every displayed metric. This supports treating plan validity and visual execution as complementary checks.
Key Findings¶
- In Table 5, diverse editing instructions alone improve Position from 0.58 to 0.67 and Color Attribute from 0.50 to 0.62. Adding self-critique then reaches 0.72 and 0.69. Modification, swapping, and removal expand the action space required for correction rather than serving as merely cosmetic augmentation.
- In Table 6, scene-graph corrections and self-sampled corrections score 0.70 versus 0.72 on Position and 0.67 versus 0.69 on Color Attribute. Self-sampling is better, but these displayed columns do not support a uniform claim of “6% better than symbolic corrections.”
- In Table 4, an external GPT Planner scores 0.60 on GenEval, a GPT Inspector scores 0.80, and STIR scores 0.83. Against BAGEL's 0.79, the Planner decreases by 0.19; the prose's “23% collapse” should not be interpreted as a 23-percentage-point drop.
- The same table reports 688K training samples, 1,000 cumulative sampling steps, and a score of 0.77 for PARM (RL+TTS), versus 62K, 131 steps, and 0.83 for STIR. These costs are approximately 1/11.1 and 1/7.6 of PARM's, not two identical 8-fold reductions. Sampling steps are also not measured wall-clock latency on matched hardware.
Highlights & Insights¶
- Intermediate supervision should ask whether a scene satisfies its current valid subtask, not simply whether it resembles the final image more closely. This provides a semantic reference for process evaluation and reduces misclassification of unfinished content.
- Separating planning errors from execution errors enables more targeted corrections. The former require changing a stage objective, while the latter may only require a local visual update; Table 7 supports their complementarity.
- Sampling the model's own errors better matches inference needs than constructing only easily controlled symbolic failures. The principle could inform other generators, although transfer would require separate validation.
Limitations & Future Work¶
- Data construction depends on Flux-Kontext, GPT filtering, and GPT critiques, without a complete quantification of their cost or annotation reliability in the main text. Independence from external models at inference does not imply independence throughout training.
- Process-driven generation adds computation: 131 denotes cumulative image-sampling steps and does not establish real-time interaction. The paper reports an average of 2.62 reasoning steps, but provides insufficient analysis of wall-clock latency, long-trajectory stability, or retries after failure.
- Quantitative evidence mainly comes from GenEval and WISE. It does not establish universal benefits for aesthetics, complex open-ended scenes, or editing tasks, and the ablation tables do not report variance across repeated runs.
- Corrupted extracted equations and inconsistent descriptions of sample balance, percentages, and termination tokens limit precise reproduction from the available text. This note prioritizes tabulated numbers and does not silently reconcile those conflicts.
- The authors propose extensions to video, 3D, and real-time human-in-the-loop control. More immediate tests would measure critique false positives, whether local corrections damage existing objects, and performance under explicitly bounded reasoning budgets.
Related Work & Insights¶
- vs BAGEL: STIR retains the unified understanding-generation backbone and its visual generation mechanism, adding process data and targeted supervised fine-tuning. Its gains should not be attributed simply to replacing the generator with a larger one.
- vs PARM: PARM evaluates intermediate denoising states for search or reinforcement learning; STIR supervises semantically readable scene evolution. Their process granularity differs, so efficiency comparisons must retain differences in sampling budgets and training methods.
- vs external Planner / Inspector: The original generator may not reliably execute external instructions. STIR jointly trains execution, critique, and correction within the model. The comparison motivates specialized training but does not prove that every external planning approach must fail.
- Insight from Gen-Ref: Reflection data can inform scene construction itself, rather than only optimization after a complete image. Critiques must be grounded in the current step objective to avoid mistaking intentional intermediate omissions for errors.
Rating¶
- Novelty: 4/5. Combines semantic intermediate states, dual-stream critique, and unified generation into a supervised process, with the main contribution in training and data design.
- Experimental Thoroughness: 4/5. Includes two benchmarks, process baselines, and several ablations, but lacks sufficient latency, variance, and open-ended evaluation.
- Writing Quality: 3/5. The central process is intuitive, but several prose claims conflict with tables, and corrupted cached equations hinder precise reading.
- Value: 4/5. Offers a concrete route toward interpretable, correctable unified image generation, without yet establishing a general real-time drawing system.