Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models¶
Conference: ECCV2026
Paper: Official page · PDF
Project: Beyond Atomic Layouts
Area: Multimodal VLM
Keywords: Compositional layout understanding, element grounding, multimodal alignment, layer-aware spatial relations, design intent
TL;DR¶
The paper introduces CoDeLayout and the MASON post-training approach, combining element grounding supervision with explicit layer-aware spatial metadata to improve Qwen2.5-VL 7B from 88.80% to 91.66% weighted compositional pairing accuracy over full-data direct finetuning.
Background & Motivation¶
A graphic design is not necessarily a collection of independent text boxes and images. A person can appear both in a base photograph and in a foreground cutout, while a word can be jointly formed by text, shadows, and a graphic replacing one of its letters. Editing only one member of such a composition can break the intended effect. Document parsing and interface understanding generally emphasize element identity, location, reading order, or interaction targets; they provide less direct supervision for identifying which source elements jointly create a visual effect.
This requires more than detecting overlapping boxes. A model must connect an element ID in metadata to the correct visual content and then interpret the relationship between layers. Cropping, occlusion, and blending make this connection ambiguous, producing what the authors call semantic drift. Separately, image-plane proximity does not determine stacking order or compositional purpose, creating structural ambiguity. An analysis of GPT-4o associates stronger element-description grounding and spatial-relation question answering with better pairing accuracy. These plots establish correlations; the subsequent module ablations provide stronger evidence about the proposed interventions.
The contribution therefore combines a task and dataset with a targeted adaptation strategy, rather than a new visual backbone. Core idea: teach the model what each metadata-defined element actually looks like through auxiliary grounding questions, and explicitly supply its query-relative spatial and layer relationships so that pairing can use both visual identity and structure.
Method¶
Overall Architecture¶
The input consists of a rendered design image, JSON metadata describing its elements, and a question naming one element ID. The output identifies the other member of its compositional pair and explains the design intent. This is not an image-only layer reconstruction task, nor does the model directly execute an edit. CoDeLayout supplies compositional questions, while MASON adds multimodal alignment (MA), structural perception (SP), and mixed QA training to adapt an existing VLM.
MA creates auxiliary element-grounding answers offline. SP derives spatial relationships relative to the element specified by the current question and appends them to the metadata. The model learns from both compositional and grounding questions. At inference, SP augmentation remains active, but the grounding teacher is no longer required and evaluation concerns only the compositional task.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Rendered image<br/>and layer metadata"] --> B["Compositional QA formulation"]
B --> C["Multimodal alignment MA<br/>Offline grounding QA"]
B --> D["Structural perception SP<br/>Query-relative relations"]
C --> E["Mixed QA training"]
D --> E
E --> F["Paired element ID<br/>and design intent"]
D -->|Retained at inference| F
Key Designs¶
1. Compositional QA formulation: identify jointly functioning elements, not merely nearby objects
CoDeLayout covers four composition types. Overlaying combines an element with its outline or shadow. Clipping extracts part of a base image into a separate layer so that other content can be inserted between the base and foreground. Blending organizes different crops of the same image into coherent stylized shapes. Morphing replaces a component of text with graphics. Their shared property is that the visible effect depends on relationships that cannot be fully described by isolated element boxes.
Each sample contains a rendered image, element metadata, and a question-answer annotation naming a compositional pair and its purpose. Metadata includes element IDs, types, opacity, placement, and stacking order. Asking for both the partner ID and an explanation separates selection accuracy from explanation quality and makes geometric heuristics useful controls. There are 20,009 training samples and 387 test samples, with an average of 17.4 layers. The training distribution is naturally imbalanced: Blending accounts for 56.78%, whereas Morphing accounts for only 2.63%. It should not be interpreted as a balanced four-class benchmark.
2. Multimodal alignment MA: connect metadata identities to actual visual content
Training only on compositional questions can encourage shortcuts based on element type or position without requiring reliable recognition of a layer's content. MA samples 1K training layouts and randomly selects 20% of the elements in each. GPT-4o then generates answers describing what each selected element is and where it appears. The annotation prompt provides the complete design, the element crop, and metadata: the crop exposes the individual layer, while the full image supplies its global context.
The generated descriptions become auxiliary supervision for the student VLM. In the 30% experiment, the authors report approximately 2K such grounding QA pairs. This is additional training data, not a teacher call for every element at deployment. It also introduces a potential failure mode: a distorted letter may be accurately described as a graphic while its typographic role is missed. Consequently, improved local descriptions alone do not guarantee improved compositional reasoning, which helps explain why MA does not benefit every category in isolation.
3. Structural perception SP: expose query-relative geometry instead of leaving it implicit in coordinates
Raw metadata already contains boxes and zIndex values, but the VLM would otherwise need to infer overlap, containment, direction, and stacking relationships from those values. For the query element, SP computes its relationship to every other element and adds a spatialRelationship field to each corresponding record. The six properties are hasOverlap, overlapPercentage, containment, centerDistance, relativePosition, and layerOrder.
Image-plane above/below and layer-order above_given/below_given encode different facts: an element can lie lower on the canvas while still covering another layer. Making this distinction explicit reduces the burden of extracting hierarchy from a long metadata sequence. SP does not directly select the answer, however. Similar geometry can describe decoration, a shadow, or a genuine compositional partner, so appearance and purpose remain necessary. The readable cache does not fully specify the overlap denominator, distance normalization, or associated implementation thresholds; this note does not invent those details.
4. Mixed QA training: connect visual identity to compositional purpose within one model
MASON mixes compositional and grounding QA examples and optimizes answer cross-entropy on Qwen2.5-VL 7B, using question-dependent SP-enriched metadata. Direct Finetune uses compositional QA alone, making it the relevant same-backbone comparison. Although the framework figure names grounding and compositional losses, the readable text does not specify independent weighting coefficients, so there is no basis for constructing a weighted-loss equation here.
The interaction is the important part: MA supplies a visual identity, SP supplies the query-relative structure, and compositional supervision teaches how the two support a partner selection and intent explanation. No new detector or separate inference network is introduced. The approach nevertheless assumes access to usable source-design metadata and requires SP computation at inference; it is not equivalent to sending an arbitrary flattened screenshot to an unmodified VLM.
A Worked Example¶
In the paper's Morphing case, a flower covers the letter O in a word. Given the flower's element ID, grounding should identify the flower and its location within the text region. Spatial and layer information then establishes its overlap with the text and its foreground position. The compositional answer should point to the related text element and explain that the flower replaces a letter visually, rather than select a similarly colored decoration aligned with the word.
MA alone may only establish that the object is a flower; SP alone may only establish that it overlaps the text. The compositional question asks why those two elements jointly create a particular effect. This example follows the qualitative case in the paper without inventing element IDs or numerical geometry that cannot be recovered from the cached figure.
Loss & Training¶
Post-training uses AdamW with learning rate \(3\times10^{-5}\) and LoRA on the attention q/v projections, with rank 4 and scaling parameter 8. The context length is 4096 tokens, training lasts 3 epochs, and the global batch size is 64 on 8 A100 GPUs.
Evaluation resizes images to \(336\times336\) and pairs them with structured JSON metadata. Decoding is greedy, with a maximum output of 256 tokens. The reasoning-enabled proprietary models use the authors' specified default medium reasoning setting. Several equations are corrupted in the text extraction, but these settings and the answer-level cross-entropy objective are readable in prose; incomplete equations are deliberately not reproduced.
Key Experimental Results¶
Main Results¶
The following values come from the paper's Table 4 and are pairing accuracies in percent, not intent-explanation scores. Weighted accuracy uses test-category proportions, while Average accuracy is the unweighted mean of the four category accuracies. Comparisons with zero-shot systems include the benefit of task-specific training; comparisons against Direct Finetune better isolate the additional MASON recipe.
| Model / setting | Overlaying | Clipping | Blending | Morphing | Weighted | Average |
|---|---|---|---|---|---|---|
| Max-Overlap | 61.74 | 69.70 | 23.08 | 36.17 | 44.27 | 47.67 |
| Qwen2.5-VL, zero-shot | 66.09 | 37.88 | 53.85 | 36.17 | 52.60 | 48.49 |
| Qwen3-VL, zero-shot | 86.09 | 71.21 | 79.49 | 55.32 | 77.08 | 73.02 |
| GPT-o3, zero-shot | 93.91 | 63.64 | 79.49 | 68.09 | 79.68 | 76.28 |
| Direct Finetune, full data | 93.04 | 83.33 | 94.87 | 65.96 | 88.80 | 84.30 |
| MASON, 30% compositional data | 92.17 | 86.36 | 93.59 | 72.34 | 89.32 | 86.12 |
| MASON, full data | 95.65 | 90.91 | 95.51 | 70.21 | 91.66 | 88.07 |
Full-data MASON improves Weighted accuracy over full-data Direct Finetune by 2.86 percentage points, including a 7.58-point Clipping gain. The gap over GPT-o3 is 11.98 Weighted points, but includes task-specific adaptation. The 30% setting uses approximately 6K compositional pairs plus 2K grounding pairs, so its total supervision is not simply 30% of the baseline's. Type-balanced subsampling also changes the representation of rare categories.
Ablation Study¶
The following values are from Table 5. This experiment uses type-balanced subsampling with equal sample counts across categories; it is not the full-data protocol above. All values are accuracies in percent.
| Configuration | Overlaying | Clipping | Blending | Morphing | Weighted | Average |
|---|---|---|---|---|---|---|
| DF | 86.09 | 74.24 | 80.77 | 72.34 | 80.21 | 78.36 |
| DF + MA | 87.83 | 81.82 | 81.41 | 70.21 | 82.03 | 80.32 |
| DF + SP | 89.57 | 83.33 | 82.69 | 74.47 | 83.85 | 82.51 |
| MASON, MA + SP | 86.96 | 81.82 | 83.97 | 80.85 | 84.11 | 83.40 |
Key Findings¶
- SP alone adds 3.64 Weighted points, versus 1.82 for MA alone. The combined gain over DF is 3.90 points, but only 0.26 points over SP alone, so the modules' effects are not additive.
- The combined model reaches 80.85% on Morphing, compared with 70.21% for MA alone, but underperforms SP alone on Overlaying and Clipping. The authors attribute some category-specific interference to noisy grounding under occlusion and semantic distortion.
- Explanation quality is evaluated separately using GPT-Score on a 0-10 scale for semantic agreement and BLEU/ROUGE for lexical agreement with reference answers. High pairing accuracy should not be treated as proof of faithful explanations.
- Full-data MASON reaches 70.21% on Morphing, below the 30% variant's 72.34%. More imbalanced data can improve the aggregate result without improving the rare category.
Highlights & Insights¶
- Editing dependencies become explicit evaluation targets. Asking which elements jointly produce an effect is closer to a design assistant's needs than recognizing each element independently. Separating partner selection from explanation also makes different failure types observable.
- Available geometry can be computed rather than relearned. When editable source metadata exists, deterministic relative relations can free model capacity for semantic interpretation. This benefit depends on actual source structure, not on assuming that layers are recoverable from any screenshot.
- Local appearance is not the same as functional meaning. A flower is both an image object and potentially a replacement for a letter. The category-level ablations show why grounding needs relational context rather than being treated as an independently sufficient capability.
Limitations & Future Work¶
- Strong input assumptions. Inference requires element metadata and SP augmentation. The experiments do not establish robustness to flattened images, missing layers, corrupted IDs, or inaccurate geometry.
- Limited test size and coverage. The test split contains 387 samples across four graphic-design relations. Without reported seed variance or confidence intervals here, a 0.26-point difference should not be overinterpreted as a stable advantage.
- Model-generated supervision and evaluation. Grounding descriptions and GPT-Score can introduce annotation or judging biases. The paper's grounding-teacher replacement study reduces concern about exclusive dependence on GPT-4o but does not eliminate these biases.
- Understanding is not yet successful editing. Qualitative examples motivate coordinated modifications, but the reported task does not establish end-to-end editing reliability. Useful extensions include consistency after edits and compositions involving more than two interacting elements.
- Some reproducibility details remain unresolved in the cache. Spatial normalization and parts of the mixing setup are not fully specified in readable text, and equation extraction is damaged. These are evidence boundaries rather than an invitation to fill in plausible settings.
Related Work & Insights¶
- Versus LayoutLM and DocLayNet: document understanding emphasizes semantics, position, and reading organization; this work targets layer-aware composition under visual entanglement. Its contribution is a more specific graphic-design relationship task, not a claim that prior methods cannot model any hierarchy.
- Versus ScreenQA and CogAgent: interface QA and navigation typically focus on actionable controls, whereas this task asks how multiple source elements jointly implement an effect. The natural application is a layer-aware design editor rather than general GUI navigation.
- Versus Crello and MLTD: multilayer generation data does not automatically supply explicit compositional-purpose labels. CoDeLayout adds paired-element and intent supervision, and MASON trains understanding rather than synthesizing a new layout.
Rating¶
- Novelty: 4/5. The task and targeted dataset are distinctive; the adaptation mechanism is deliberately lightweight.
- Experimental Thoroughness: 4/5. Broad model comparisons and module, data-scale, teacher, and visual-dependency studies are useful, but the test set remains small.
- Writing Quality: 4/5. The diagnosis-to-design connection is clear; reproduction would benefit from more explicit attribute and training specifications.
- Value: 4/5. Directly relevant to assistants with access to editable layer metadata, with limited evidence for arbitrary screenshot-based understanding.