COMPASS: Grounding Composition-Intent Guidance in Unified Multimodal Models

Conference: ECCV2026
Paper: ECCV Official Page
PDF: Full Paper
Area: Multimodal VLM
Keywords: composition recognition, mixture of experts, expert token, structural bottleneck, reference-guided generation

TL;DR

COMPASS adds a composition-expert branch and an expert token to Ming-Lite-Uni, then guides generation with grayscale pixelated references and restricted information pathways, reporting composition-recognition mAP of 90.6 and generation composition consistency of 0.74 on Comp-11, although its mAP aggregation and self-evaluated composition metrics require caution.

Background & Motivation

Photographic composition concerns not what objects appear, but how subjects occupy the frame, how lines direct attention, and whether the overall arrangement follows patterns such as the rule of thirds, symmetry, or leading lines. Aesthetic models such as AesExpert can discuss composition, but typically treat it as one attribute alongside color and lighting; unified multimodal models do not necessarily acquire fine-grained composition recognition and control merely by supporting both understanding and generation. Existing edge, depth, or box conditions constrain specific geometry, but do not directly transfer a reference photograph's high-level compositional intent to new content.

The input considered here is a reference image supplying composition together with text specifying new content; the goal is neither editing the existing subject nor copying the layout pixel by pixel. Composition is intertwined with object contours and positions: preserving too much reference information transfers its objects as well, whereas excessive compression loses spatial organization. Large-scale paired supervision with identical composition but different content is also difficult to obtain, while fine-tuning an entire unified model may interfere with its existing multimodal capabilities.

The paper therefore first makes composition an explicitly supervised specialty, then restricts how that specialty conditions the generator. Core idea: use a composition-supervised expert token to carry layout intent, combining an input-side structural bottleneck with model-side information isolation so that the reference supplies composition while text determines content.

Method

Overall Architecture

COMPASS builds on the MoE-based unified multimodal model Ming-Lite-Uni and uses two training stages: first recognizing and explaining composition from ordinary images, then guiding diffusion generation with perturbed references. The understanding stage produces a natural-language response and multi-label composition predictions; the generation stage takes a reference and a new-content prompt and produces an image retaining the reference's compositional intent. Its four designs are Comp-11 Construction for supervision, C-MoE and Expert Anchor for representation learning, Structural Bottleneck for filtering references, and Controlled Conditioning Pathways for separately delivering layout and text to the generator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Aesthetic and composition images"] --> B["Comp-11 Construction"]
    B --> C["C-MoE and Expert Anchor"]
    D["Reference image"] --> E["Structural Bottleneck"]
    C --> F["Controlled Conditioning Pathways"]
    E --> F
    G["New-content text"] --> F
    F --> H["Diffusion denoising and output image"]

The expert anchor supplies learned composition capability to generation; the dataset and experts are not rebuilt or retrained for each generated image. The structural bottleneck applies only to references on the generation side, while understanding learns composition from ordinary images.

Key Designs

1. Comp-11 Construction: turning broad aesthetic commentary into supervised composition decisions

Comp-11 aggregates 389,031 images from AVA, TAD66K, CADB, and KU-PCP into 11 composition categories: rule of thirds, center, horizontal, symmetric, diagonal, curved, vertical, triangle, pattern, leading lines, and fill the frame. An image may exhibit multiple techniques, so labels are multi-label rather than mutually exclusive; cluttered images whose composition experts cannot identify may receive no label, avoiding forced supervision for ambiguous cases.

Photography and fine-arts students first verify inherited labels and then annotate new images; multiple annotators review each image independently, resolving disagreements through consensus voting. Expert-verified labels subsequently constrain Qwen2.5-VL augmentation of composition-rationale questions, multiple-choice discrimination, and judgment tasks, yielding approximately 1.2M VQA pairs. The important step is not merely increasing image count, but connecting category decisions and explanations to the unified model's instruction interface; however, the natural-language rationales include model-generated content and should not be treated as entirely human-reviewed wording.

2. C-MoE and Expert Anchor: allocating separate capacity and a directly supervised carrier for composition

Alongside every sparse FFN layer in the backbone, the method adds a composition MoE branch with its own router and expert MLPs. A task flag selects either the original branch or the composition branch at runtime; their outputs are not always added together, nor are the new experts simply appended to the original pool under a shared router. Within the composition branch, routing scores still select and weight Top-K experts; the original backbone is frozen so that domain-specific updates avoid modifying general-purpose parameters. This isolation reduces interference by design, but the main tables do not measure general-capability retention, so freezing should not be mistaken for experimental proof of no forgetting.

A model that merely produces a composition critique can still rely on language priors to deliver a plausible but incorrect answer. COMPASS therefore inserts a learnable expert token \(\tau_c\) immediately after the assistant's answer prefix and before the first response token, using its final hidden state with a lightweight projection and classifier to predict the 11 categories. Classification uses binary cross-entropy with class-specific positive and negative weights to address multi-label supervision and the long tail; training it alongside next-token prediction subjects the dedicated representation directly to composition labels rather than relying only on the verbal answer.

3. Structural Bottleneck: preserving spatial massing while deliberately discarding reference appearance

Generation does not consume the full reference directly: grayscaling and pixelization first produce a structural proxy retaining coarse regional distributions and dominant layout cues. Grayscaling weakens color information, while pixelization disrupts recognizable detail and coherent contours; both aim to reduce reference-semantic leakage rather than improve the reference's visual quality. A single image can consequently provide its own structural proxy for training, without collecting a large set of same-composition, different-content image pairs.

The rationale for this treatment instead of edges or ordinary blur is that edges preserve fine object-identifying contours, while blur can retain recognizable silhouettes. Pixelization more strongly disrupts contour continuity while leaving information about the locations of major spatial masses. This explanation agrees with the qualitative comparison in the main paper, but the available cache contains neither a complete numerical ablation of alternative bottlenecks nor reproducible pixelization-strength settings, so it does not establish superiority for every layout.

4. Controlled Conditioning Pathways: routing layout through the expert token and content through text into denoising

Weakening the reference alone is insufficient because the model may still copy its content from residual shapes. A customized causal attention mask therefore prevents post-reference text tokens and learnable queries from directly reading reference-image tokens; the expert token is the only subsequent token allowed to read them. This preserves causal ordering while funneling reference information into a composition representation constrained by classification supervision. It restricts direct attention pathways, not mathematically guaranteeing that all reference semantics disappear.

The text pathway receives complementary reinforcement: query hidden states produced by the unified model serve as queries, and prompt hidden states supply keys and values for lightweight cross-attention refinement. The resulting queries delivered to the diffusion generator are more explicitly anchored to the new-content prompt, rather than relying on reference blocking alone to improve text faithfulness. The mask removes a shortcut while cross-attention strengthens the desired content signal, implementing a common division of informational roles.

For layout, the expert-token hidden state is projected and added to the diffusion timestep embedding, then broadcast to denoising blocks through AdaLN-style modulation:

\[ \tilde e_t = e_t + \psi(h_c). \]

Here \(h_c\) is the expert-token hidden state, \(\psi\) projects it into the conditioning space, and \(e_t\) is the original timestep embedding. Rather than appending a composition-category name to text, this makes a continuous layout representation a global signal influencing the denoising process. Generation training retains composition-classification supervision on the expert token so that the input-domain shift caused by grayscale pixelization does not detach the anchor from its composition meaning.

Loss & Training

Stage I jointly optimizes next-token prediction and weighted multi-label classification, freezing the backbone and updating only the C-MoE branch, expert token, and lightweight classification head. Stage II jointly optimizes diffusion and the same type of composition-classification loss, freezing the composition experts learned in Stage I while training generation-specific parameters and retraining the expert token and its lightweight head for perturbed references. Thus, shared experts means reuse of the composition-expertization module across tasks; the authors also state that expert tokens can be packaged as task-specific lightweight parameters, rather than requiring identical frozen token parameters throughout both stages.

The main paper delegates loss details, implementation hyperparameters, and some evaluation definitions to supplementary material, whereas the current local cache contains only the main paper and references. This note therefore retains the objectives and freezing scope without inventing loss weights, learning rates, expert counts, or training time; several cached equations also lose symbols during text extraction, and the equation above is reconstructed from the explicit prose description of timestep modulation.

Key Experimental Results

Main Results

The understanding evaluation reserves 20% of Comp-11 for testing; generation samples 10,000 evaluation groups from that split with uniform coverage over composition categories, each containing a layout reference and a separate content-reference image whose caption supplies the new-content prompt. Understanding evaluates AP separately for each composition category, with higher values preferred. The mAP column below preserves the values reported in the original table.

Model	Type	Reported mAP ↑
AesExpert	Understanding-only	49.5
Qwen3-VL (8B)	Understanding-only	66.4
InternVL3 (8B)	Understanding-only	60.0
Janus-Pro (7B)	Unified model	54.2
BAGEL (7B)	Unified model	63.2
Ming-Lite-Uni (8B)	Unified model, backbone	58.2
COMPASS	Unified model	90.6

Using these reported values, COMPASS exceeds Qwen3-VL by 24.2 percentage points and its backbone by 32.4 percentage points. However, the original mAP values do not equal the simple arithmetic mean of the 11 displayed class AP values, and the main text does not explain an alternative aggregation; they are therefore labeled reported mAP here, without correcting the source table or presenting these differences as independently verified under a clarified aggregation protocol.

For generation, lower FID is better; CLIPScore measures alignment between generated images and content text, with higher values preferred. Comp-Cons. measures agreement between composition labels predicted by COMPASS for reference and generated images; expert similarity is the cosine similarity between their expert-token hidden representations, with higher values preferred for both. The main text does not provide the precise multi-label aggregation formula or threshold for Comp-Cons., so it cannot safely be interpreted as exact label-match accuracy or intersection-over-union; both composition metrics depend on COMPASS itself rather than an independent evaluator.

Model	FID ↓	CLIPScore ↑	Comp-Cons. ↑	Expert similarity ↑
Step1X-Edit	7.5	0.72	0.60	0.51
BAGEL	7.8	0.76	0.59	0.48
Ming-Lite-Uni	9.6	0.64	0.53	0.44
COMPASS	8.2	0.83	0.74	0.62

COMPASS leads in text matching and both composition metrics, but its FID trails Step1X-Edit and BAGEL, so it does not win across every generation-quality measure. Baselines that do not support grayscale pixelated references receive the original reference with an instruction to inherit its composition; this is a capability-adapted comparison, not a strictly controlled experiment with identical inputs for every model.

Ablation Study

P denotes the understanding stage and G the generation stage; generation ablations follow the original table's cumulative removals rather than independently removing one component per row. An em dash means the original table does not report that metric, not a score of zero.

Config	Reported mAP ↑	FID ↓	CLIPScore ↑	Comp-Cons. ↑	Expert similarity ↑
Full model	90.6	8.2	0.83	0.74	0.62
P: without C-MoE	78.2	—	—	—	—
P: without expert token	71.1	—	—	—	—
G: without cross-attention	—	8.0	0.79	0.64	0.56
G: previous row plus mask removal	—	8.6	0.72	0.59	0.48
G: previous row plus expert-token conditioning removal	—	8.3	0.75	0.53	0.42

Key Findings

• Removing C-MoE or the expert token in understanding reduces reported mAP by 12.4 or 19.5 percentage points, respectively, supporting contributions from both dedicated capacity and a directly supervised anchor.
• Removing only cross-attention in generation lowers Comp-Cons. from 0.74 to 0.64; subsequently removing the mask and token conditioning lowers it to 0.59 and 0.53, consistent with the intended restricted layout pathway.
• In the final row, CLIPScore recovers from 0.72 to 0.75 while composition metrics deteriorate further. These cumulative ablations cannot establish independent, monotonic benefits from every component for every metric.

Highlights & Insights

• Composition recognition becomes a generation interface. The expert token receives multi-label supervision and globally modulates diffusion, connecting layout understanding and generation through an explicit representation rather than merely a shared model name.
• Both information entry and exit are managed. Grayscale pixelization reduces reference appearance information, while the attention mask restricts shortcuts by which residual information reaches generation queries, going beyond input preprocessing alone.
• Blocking is paired with reinforcement. Cross-attention strengthens the content prompt while reference access is restricted, clearly distinguishing reduced copying from improved text faithfulness.

Limitations & Future Work

• Evaluation independence is limited. Composition consistency and representation similarity use COMPASS itself as the measuring instrument and may favor its representations; these results do not replace independent annotations or human composition judgments.
• Aggregation and reproduction details remain incomplete. Reported mAP differs from the simple mean of class AP values, while Comp-Cons. aggregation and important training hyperparameters are absent from this cache, requiring clarification from supplementary material or implementation.
• Layout preservation does not imply exact geometric reproduction. The 11 labels summarize composition styles, and grayscale pixelization discards detail, so the experiments do not directly establish preservation of precise subject coordinates or fine lines.
• Quality and capability retention have boundaries. FID is not the best, and freezing the backbone lacks an independent general-capability retention evaluation in the main tables. Further failure cases are deferred to supplementary material not included here.

Related Work & Insights

• vs AesExpert and UNIAA: These works primarily provide holistic aesthetic evaluation; COMPASS makes composition a separate multi-label supervision target and generation condition, shifting the emphasis from critique to actionable control.
• vs Ming-Lite-Uni and BAGEL: Unified understanding and generation provide the foundation; COMPASS adds composition-specific capacity and restricted reference pathways. Its advantages mainly concern particular composition metrics and text matching, not unconditional superiority in general generation.
• vs ControlNet-style low-level conditioning: Edge conditions directly preserve specific geometry, whereas COMPASS deliberately weakens contours to transfer high-level layout intent. Their control objectives differ, so this experiment alone cannot establish that one replaces the other.

Rating

• Novelty: 4/5 — The combination of a composition-expert anchor and reference-information isolation is well targeted, while the underlying modules build on established techniques.
• Experimental Thoroughness: 3/5 — Understanding, generation, and stage-specific ablations are covered, but self-evaluation, aggregation ambiguity, and missing implementation details constrain the conclusions.
• Writing Quality: 3/5 — Information flow and design motivations are clear, while some statistical definitions and cross-task parameter-sharing claims require careful interpretation.
• Value: 4/5 — The work offers an understandable unified-model approach to high-level composition control, but its benefits should be judged alongside image quality and evaluation bias.