PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Design

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM / Layout Reasoning / Graphic Design
Keywords: perturbed supervised fine-tuning, geometric rewards, aesthetic feedback, multi-layer posters, controllable editing

TL;DR

PosterCopilot trains a 7B vision-language model to output layout JSON through perturbed supervision, geometric reinforcement learning, and aesthetic reinforcement learning, then enables professional poster iteration through a generative agent restricted to target layers, reaching 54.31% IoU on zero-shot Crello evaluation.

Background & Motivation

Professional poster design requires more than an attractive final image: brand images, people, text, and decorative assets must remain recognizable, traceable, and editable. Diffusion models that generate an entire image can coordinate visual style, but changing a background or a person may also alter unrelated regions. Predicting asset positions, sizes, and stacking order before compositing the original layers offers an alternative closer to conventional design software. LayoutNUWA, PosterLLaVA, and LaDeCo pursue this direction, yet structured output does not guarantee correct geometry: text may overlap, images may stretch, and bounding boxes may drift away from suitable locations.

The authors attribute part of this problem to textual coordinate representation: numerical proximity does not imply proximity between token representations, especially across tokenization boundaries. Ordinary supervised fine-tuning rewards a particular answer string without directly expressing the spatial difference between a small displacement and a large one. Even successfully copying a reference layout does not guarantee aesthetic understanding, since the same assets often admit several valid compositions. A single ground truth is therefore both a starting point for learning geometry and a potential constraint on aesthetic diversity.

The paper first makes coordinate supervision tolerant of local perturbations, then uses verifiable geometric rewards to correct drift and distortion, and finally introduces aesthetic feedback on rendered results. Here, feedback primarily means scores entering the training reward, not a requirement that the deployed model repeatedly evaluate and revise itself. Actual editing uses a workflow with explicit layers, avoiding regeneration of every asset. Core Idea: learn stable asset placement before optimizing visual appeal, and control asset-content editing separately from layout planning.

Method

Overall Architecture

Inputs comprise canvas dimensions, user instructions, and image, text, and shape assets; text is rasterized into image layers for a unified input representation. The design model outputs each asset's bounding box and stacking order as JSON, which a renderer composites into a poster. It predicts neither the pixels of the entire image nor a full set of vector typography parameters; fidelity primarily comes from reusing the supplied assets instead of asking a generator to redraw them. Training proceeds through perturbed supervised fine-tuning, visual-reality alignment, and aesthetic feedback alignment; these signals are not interchangeable training modules. Layer-wise generation and editing enter at deployment: complete assets go directly to layout planning, missing assets are completed first, and local instructions update only designated layers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Training["Training assets and reference layouts"] --> PSFT["Perturbed Supervised<br/>Fine-Tuning"]
    PSFT --> VRA["Visual-Reality<br/>Alignment"]
    VRA --> RLAF["Aesthetic Feedback<br/>Alignment"]
    RLAF -.->|Trained parameters| Model["Deployed design model"]
    Request["Assets, canvas, and edit requests"] --> Agent["Layer-wise Generation<br/>and Editing"]
    Agent -->|Completed or updated assets| Model
    Model -->|Layout JSON and layer rendering| Poster["Poster available for further edits"]
    Poster -->|New target-layer instructions| Agent

The deployment loop represents user-driven iterative editing; the dotted edge transfers trained parameters and does not place the aesthetic judge inside every deployment round. The deployed design-model node reuses the trained model rather than introducing another separately trained network. The four key designs cover the three training stages and the editing mechanism that connects their capabilities to practical asset workflows.

Key Designs

1. Perturbed Supervised Fine-Tuning: construct a supervision neighborhood around reference coordinates

PSFT retains supervision on the original JSON and adds Gaussian noise to each bounding-box numerical value to produce several nearby target layouts. It perturbs coordinate labels, not input images, and does not randomly move assets at inference time. Perturbed values are rounded to integers and clamped to the valid canvas range to avoid clearly out-of-bounds supervision targets. The paper explicitly describes perturbing bounding-box values, not randomly changing asset identities or stacking order. The training loss combines cross-entropy on the original target with the average cross-entropy across perturbed targets, controlled by a separate weight. The implementation uses \(\sigma=2.5\), \(n=5\), and \(\lambda_{\mathrm{pert}}=0.5\). The original target provides an anchor, while nearby targets reduce excessive dependence on one discrete coordinate-token sequence.

Learning continuous space here should be understood as smoothing the supervision distribution, not acquiring a strictly continuous coordinate representation. Outputs remain discrete text; PSFT does not replace the tokenizer or introduce a dedicated continuous coordinate-regression head. Its benefit is to give nearby reasonable layouts learning opportunities, establishing a stronger spatial prior for subsequent reinforcement learning. It still depends on reference layouts, so it cannot independently recognize every attractive composition that departs from the reference.

2. Visual-Reality Alignment: correct textual predictions with computable layout quality

RL-VRA treats complete layout generation as a single-step decision: the state contains assets and requirements, and the action is the entire layout JSON. The model samples a group of layouts for the same input and uses GRPO to update the policy according to relative within-group rewards, without explicitly training a value network. A frozen reference policy and KL regularization constrain updates so that improving a geometric score does not erase existing generation capabilities. Rewards do not require an additional subjective judge; they check spatial relationships between predicted and reference layouts as well as JSON validity. The spatial component uses DIoU to consider both box overlap and center displacement, providing positional feedback even when two boxes barely intersect. Element-level components constrain aspect ratio and size separately; the size component uses Huber loss, making penalties grow linearly rather than quadratically for large errors so severely distorted samples dominate updates less.

The reward composition confirmed by the prose in Section 3.3 and Eq. (9) can be expressed as:

\[ r_{\mathrm{VRA}}=r_{\mathrm{DIoU}}+\lambda_{\mathrm{size}}r_{\mathrm{size}}+\lambda_{\mathrm{AR}}r_{\mathrm{AR}}+r_{\mathrm{format}}. \]

Failed JSON validation receives a large penalty, while a valid structure receives the full format score; the main text does not specify the numerical penalty scale. The implementation uses \(\lambda_{\mathrm{size}}=0.6\), \(\lambda_{\mathrm{AR}}=0.4\), and the Huber transition parameter \(\delta=0.6\). These rewards constrain displacement and flattening separately, preventing an improvement in box overlap from concealing damaged asset proportions. Visual-reality alignment here concerns verifiable graphic-design geometry, not interaction with real-world physics. Its limitation remains clear: spatial and element rewards reference a single target layout and cannot enumerate all good designs for the same assets.

3. Aesthetic Feedback Alignment: evaluate rendered designs instead of only matching reference boxes

RLAF renders predicted layouts with their assets and uses VisualQuality-R1 to provide aesthetic rewards on the resulting posters. The training signal can therefore capture overall composition and stylistic coherence that coordinate distances alone cannot express, rather than merely measuring proximity to reference boxes. This stage combines the format reward with the aesthetic score, using an aesthetic weight of \(\lambda_{\mathrm{aes}}=2\). It does not simply retain and sum every geometric reward; switching stages deliberately relaxes dependence on a single ground truth. KL regularization and structural-validity rewards remain, and training is limited to 1 epoch on 1K curated samples to reduce reward-hacking risk. These measures constrain policy drift but do not guarantee that the aesthetic reward cannot be exploited.

The geometry-first order has a concrete purpose: optimizing subjective image scores before the model can preserve asset proportions may encourage unreliable compositional shortcuts. Sequential training establishes a reliable layout foundation before allowing departures from the reference composition. A small decrease in final IoU need not indicate failed training; aspect ratio, stacking order, and human preferences must also be considered. VisualQuality-R1 is the training reward model, whereas GPT-5 is an external experimental judge; they are distinct evaluation roles.

4. Layer-wise Generation and Editing: restrict changes to user-designated assets

The generative agent operates at inference time and introduces no additional trainable components. When users provide only partial assets, a reception model uses the poster-level description and a structured summary of existing layers to write prompts for missing layers. Those prompts and style references from existing assets enter the image-generation model; generated backgrounds or foregrounds then pass to the design model for layout planning. The paper uses Qwen-2.5-7B as the reception model and Qwen-Image-Edit-2509 as the generation component, with fixed prompt templates and decoding settings. Global context helps new assets match existing ones rather than treating each layer as an unrelated image-generation task.

During subsequent editing, the agent operates only on user-designated target layers and keeps all other layers fixed. Freezing unedited layers makes the editing scope a structural constraint, rather than relying solely on an instruction not to change unrelated content. A request to change canvas dimensions instead triggers layout regeneration; this global layout task should be distinguished from editing the contents of one layer. The paper illustrates multi-round local editing and canvas reframing but does not report large-scale multi-round editing success rates or per-round drift statistics. The supported conclusion is therefore that the workflow enables these operations, not that it executes arbitrary complex instructions without error.

A Worked Example

In the portrait-poster example in Figure 5, the initial canvas is \(2480\times3508\), and the user requests a modern design. When assets are incomplete, the reception model plans prompts for missing layers such as the background and portrait, and the generation component supplies them. The design model then outputs positions, sizes, and stacking order, and the renderer combines the layers into an editable draft. A request to remove the beard or change hair color targets the portrait layer without requiring regeneration of the text or background. If the next request changes the canvas to landscape orientation, the appropriate operation is layout regeneration rather than stretching the entire poster. This example separates content editing from spatial rearrangement; the figure gives no quantitative accuracy for individual edits and does not support an inferred success rate.

Loss & Training

The layout backbone is Qwen-2.5-VL-7B-Instruct, trained using the authors' MLP-169K dataset. Dataset construction uses fine-grained OCR boxes to merge over-segmented layers and filter redundant ones, preventing a meaningful visual element from remaining fragmented across many pieces. The dataset contains 169K posters and 2.6M layers, comprising 1.2M text layers and 1.4M image or decorative layers, with stacking order retained. The training and test splits contain approximately 160K and 9K samples; PSFT uses 140K samples for 3 epochs, and RL-VRA uses 20K complex-layout samples. RLAF uses 1K expert-validated samples, but the main text does not establish their overlap with earlier stages, so these numbers should not be summed into a distinct-sample training total. All experiments run on \(8\times\) RTX H20 GPUs; the main paper does not report complete training time or per-edit latency. Equations (1) through (8) and Eq. (10) are visibly corrupted in the cached extraction; this note follows the readable prose and retains only the confirmed reward composition, without inventing full equations, normalization details, or GRPO hyperparameters.

Key Experimental Results

Main Results

The following table excerpts the paper's Table 1 on page 13. Crello and WebUI are zero-shot evaluations, while MLP-169K uses the authors' test split. IoU measures overlap between predicted and reference boxes and is better when higher; ARD measures aspect-ratio distortion and IoPR the ratio of inverted layer-order pairs, both better when lower. The main text does not fully specify the aggregation implementations for ARD and IoPR; neither should be read as percentage accuracy.

Dataset	Method	IoU (%), higher is better	ARD, lower is better	IoPR, lower is better
Crello	LaDeCo	41.09	0.51	0.34
Crello	AesthetiQ	42.83	Not reported	Not reported
Crello	PosterCopilot	54.31	0.12	0.03
WebUI	LaDeCo	46.36	0.93	0.39
WebUI	AesthetiQ	48.29	Not reported	Not reported
WebUI	PosterCopilot	57.19	0.19	0.02
MLP-169K	LaDeCo	22.14	0.78	0.35
MLP-169K	PosterCopilot	34.23	0.04	0.04

AesthetiQ numbers are quoted from its original paper rather than obtained through a unified reproduction; the Crello gap is 11.48 percentage points, not an 11.48% relative improvement. The human evaluation in the paper's Table 2 on page 13 reports competitors' win rates against PosterCopilot, not PosterCopilot's win rates. The next excerpt includes average win rate, AWR, and GPT-5 scores for Text Legibility, TL, and Visual Appeal, VA, to avoid treating a single aesthetic score as overall quality.

Method	Competitor AWR (%)	GPT-5 TL	GPT-5 VA
Nano-Banana Pro	28.6	5.49	5.88
Nano-Banana	17.7	6.08	5.46
CreatiPoster	25.7	5.38	5.23
LaDeCo	16.2	4.96	4.00
PosterCopilot	Not applicable	5.96	6.16

Human evaluation uses identical prompts and complete assets, with 30 examples per competitor, 10 from each dataset, and 5 judgments per example. The text reports 750 responses from 40 professional designers and 40 general users, but Table 2 lists 6 competitors; applying the same protocol to all would yield 900 responses, so the coverage remains unclear. GPT-5 uses in-context scoring examples and evaluates each result 10 times before averaging, with temperature 0.7 and Top-p 0.9. Element Preservation, EP, and Instruction Following, IF, are assessed only by humans, not GPT-5; CreatiPoster additionally receives ground-truth foreground layouts, so input conditions are not fully identical.

Ablation Study

The following table reproduces the training-stage comparison from Table 3a on page 15, evaluated on the MLP-169K test set; Joint trains the two RL stages simultaneously.

Config	IoU (%), higher is better	ARD, lower is better	IoPR, lower is better
SFT	28.53	0.851	0.42
PSFT	31.14	0.699	0.38
PSFT + RLAF	27.12	0.531	0.33
PSFT + Joint	24.71	0.712	0.57
PSFT + RL-VRA	34.72	0.061	0.18
Full sequential training	34.23	0.045	0.04

Key Findings

• PSFT improves IoU over SFT by 2.61 percentage points; directly adding aesthetic RL or training jointly does not replace the geometry-first ordering.
• Adding RLAF after RL-VRA decreases IoU from 34.72 to 34.23, while ARD improves from 0.061 to 0.045 and IoPR from 0.18 to 0.04; not every metric improves monotonically.
• In Table 3b on page 15, format plus DIoU rewards yield ARD 0.734; adding size and aspect-ratio rewards reduces it to 0.061, supporting explicit element-fidelity constraints.
• Table 4 on page 15 gives PSFT + RL-VRA an AWR of 36.4% against the full model, showing that reference-box IoU alone cannot capture the value of aesthetic training.
• Section 6.4 validates the reward model on 200 poster pairs, with VisualQuality-R1 agreeing with human preferences on 82.5%; this is preference agreement on a limited sample, not editing accuracy.

Highlights & Insights

• Sequentially applying label neighborhoods, geometric rules, and rendered-image aesthetics targets layout errors more directly than simply increasing model size. The ordering ablation provides the strongest evidence for this training strategy.
• Explicit layers both provide an editable representation and restrict the generator's scope. Fidelity does not rely solely on perceptual similarity constraints: assets that should remain unchanged never enter regeneration.
• A transferable idea is to stabilize numerical predictions in VLMs producing structured spatial parameters, then add verifiable constraints, and finally optimize subjective quality. This is a reader interpretation, not a result already demonstrated on other tasks.

Limitations & Future Work

• The authors acknowledge the absence of a community-recognized benchmark and a poster-specific aesthetic reward model; VisualQuality-R1 preferences do not represent every designer or brand.
• Layer-wise editing is primarily illustrated through application figures, without systematic evaluation of target-selection errors, cross-layer dependencies, long-conversation drift, or latency; professional-workflow claims need this qualification.
• Text is rasterized, so layer controllability should not be equated with native vector control over fonts or character spacing; the main text does not establish this level of editability.
• The main table reports full-model ARD as 0.04, while the ablation table reports 0.045; both are preserved as published instead of silently harmonizing precision.
• Unclear user-study accounting and corrupted equations limit reproducibility details supported by this cache; the paper promises dataset and construction-code release but provides no confirmed repository link in the main text.

Related Work & Insights

• Compared with LaDeCo: both retain layered design representations, while this paper emphasizes coordinate perturbation and staged geometric and aesthetic optimization; its contribution is more than an asset-generation interface.
• Compared with AesthetiQ: both prioritize aesthetic alignment, but this paper stresses establishing geometry before optimizing aesthetics; quoted results omit ARD and IoPR and do not constitute a unified reproduction of every metric.
• Compared with CreatiPoster: both address editable multi-layer posters, while PosterCopilot combines geometric training with targeted layer editing; foreground-layout input conditions differ in the experiment and should remain explicit.
• Compared with holistic poster generation: Nano-Banana scores higher on TL, but its human-evaluated Element Preservation win rate is only 6.4%. Clearer text alone does not establish suitability for workflows requiring all supplied assets to be retained.

Rating

• Novelty: 4/5. The training order and layer-wise workflow form a clear contribution, although the underlying optimization tools are not entirely new.
• Experimental Thoroughness: 4/5. Cross-dataset tests, stage and reward ablations, and human evaluation are included, but quantitative multi-round editing and reporting consistency remain gaps.
• Writing Quality: 3/5. The main narrative is clear, but sample accounting and numerical precision require clarification, while corrupted cached equations also impede precise reading.
• Value: 4/5. The work offers practical ideas for layout systems where assets must remain faithful and designs must remain editable.