Panoptic Scene Program Diffusion Transformer

Conference: NeurIPS2026
arXiv: 2609.31780
Area: Image Generation
Keywords: text-to-image, panoptic scene program, joint denoising, attribute binding, compositional faithfulness

TL;DR

PSP-DiT turns a panoptic scene program containing instances, attributes, relations, and counts into a latent variable jointly denoised with the image, uses ownership and cycle-consistency supervision to constrain its visual realization, and improves GenEval 2 from 32.8 to 38.2 over a matched internal baseline while reporting approximately 1.12 times the inference latency.

Background & Motivation

Text-to-image models can produce realistic materials and lighting, but visual plausibility does not imply faithful composition. Prompts involving different colors for repeated object categories, exact counts, or directional subject–object relations still cause attribute swaps, missing instances, and role reversals. Stronger text encoders improve semantic understanding without automatically resolving which object owns which visual evidence during generation.

GLIGEN, ControlNet, and BoxDiff inject structure through boxes, masks, or attention constraints, and scene-graph methods explicitly represent objects and relations. However, external structure commonly remains a fixed condition: the image state evolves during denoising without the structure evolving alongside it. Serializing a scene graph into additional prompt tokens can also leave repeated instances conflated. The paper asks not merely whether structured input helps, but whether making structure part of the generative state is more effective than adding conditioning information.

The proposed instance-indexed panoptic scene program records objects while requiring them to correspond to actual image regions. It is not a perfect parse that must be executed literally; it is a latent variable that receives feedback from the current visual state and continues to change during sampling. Core idea: jointly generate an image and its grounded scene explanation, preserve instance identity, attribute ownership, and ordered relations throughout denoising, and use visual regions and structure-recovery supervision to check whether those constraints are actually rendered.

Method

Overall Architecture

The input is a text prompt; outputs include the final image, scene program, instance ownership maps, and coarse depth/occlusion ordering. The image stream uses image latents from a representation autoencoder (RAE), while the program stream contains instance, relation, cardinality-group, and background/context tokens. Each stream first updates its own tokens, then exchanges information through bidirectional cross-attention and predicts its own denoising target.

The pipeline combines instance-indexed scene programs, coupled-stream joint denoising, panoptic grounding constraints, and cycle-consistency supervision. During training, real images and structural targets provide supervision. At inference time, a frozen text parser supplies only the initial scene prior, after which both streams are sampled jointly. The frozen image-to-scene recognizer is used only during training, not for inference-time search or reranking.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    P["Text prompt"] --> A["Instance-indexed<br/>scene programs"]
    A -->|Noisy prior| B["Coupled-stream<br/>joint denoising"]
    N["Image noise"] --> B
    B --> C["Panoptic grounding<br/>constraints"]
    C --> O["Image and scene explanation"]
    C -.->|Training: provisional decoded image| D["Cycle-consistency<br/>supervision"]
    T["Training images and structural targets"] -.-> A
    T -.-> C
    D -.->|Training loss| B

Key Designs

1. Instance-indexed scene programs: preserve separate identities even within the same category

The scene program comprises object instances, instance-attached attributes, ordered relations, cardinalities, and context. Rather than sharing a single category token for “mug,” objects occupy separate instance slots; attributes such as color attach to a particular instance, and relations preserve their endpoint order. Consequently, “A behind B” and “B behind A” are different structures, and two mugs remain distinguishable despite having the same category.

The implementation uses fixed slots: up to 16 object instances, 32 relations, 8 cardinality groups, and 8 scene-context tokens. Attributes attach to instances rather than occupying a separately specified bank of attribute slots in the appendix. Unused slots are masked out of both program and grounding losses. This organization explicitly represents repeated categories and counts, but also places a capacity limit on complex scenes.

Training programs are not supplied directly by the text parser alone. The authors first parse text, obtain image support from a frozen panoptic segmentation/open-vocabulary grounding stack, and then remove unsupported objects, merge duplicate mentions, and retain only relations with grounded endpoints. Examples with no foreground instance or no grounding support are discarded. This reduces supervision conflicts between captions and images, but a shared image pool does not mean that the matched models receive identical supervision.

2. Coupled-stream joint denoising: structure influences images and receives visual feedback

Whereas a conventional model learns an image distribution conditioned on text, PSP-DiT learns the joint conditional distribution of images and programs:

\[ p_{\theta}(z,s\mid y). \]

Here \(z\) is the image latent, \(s\) is the scene-program latent, and \(y\) is the text. Each coupled block processes tokens within each stream, applies program-to-image and image-to-program cross-attention, and predicts denoising targets through stream-specific projection heads. The program stream is therefore neither an auxiliary prompt encoder nor a layout generated once and held fixed during sampling.

Feedback allows scene descriptions and visual realizations to continually inform one another: the current instance state can constrain attribute allocation in the image, while current image evidence can affect the next program state. A mutable program does not itself guarantee adherence to the original prompt. Region support, alignment, and cycle supervision are therefore needed to discourage both streams from converging on a mutually consistent but incorrect scene.

Inference begins with a coarse prior from a frozen text-to-program parser. Image latents start from Gaussian noise, whereas program latents start from a noisy distribution centered on the prior. Each step updates both latents and predicts ownership and ordering, before the RAE decodes the final image. The paper leaves corruption distributions and sampling update operators abstract, without fully specifying continuous program encoding, noise covariance, or the complete sampler implementation; it does not establish a particular precise diffusion parameterization.

3. Panoptic grounding constraints: program instances must own visual support

Attention exchange alone can still yield three mugs in the program but only two in the image. The grounding head predicts low-resolution soft ownership fields over the RAE latent grid, indicating which image tokens support each instance; the ordering head predicts coarse inter-instance depth or occlusion ordering. “Panoptic” here emphasizes binding instances and scene content to spatial support, not turning image generation into a conventional panoptic segmentation evaluation task.

Ownership and depth/ordering losses compare predictions against targets from the frozen grounding stack. A separate instance-level alignment term matches an instance latent representation to image evidence pooled over its owned region. Thus, assigning red to the left mug is not merely a whole-image text-similarity problem: the corresponding instance must also find support within its region. The paper does not detail the component distances, ownership normalization, or matching algorithm, so the explanation remains at the mechanism level.

4. Cycle-consistency supervision: recover the intended structure from provisional images

Region support primarily checks local grounding; cycle supervision additionally checks whether a generated image supports recovery of the overall scene. During training, a provisional image is decoded from the predicted clean latent and passed to a frozen image-to-scene recognizer. Recovered objects, attributes, relations, and coarse ownership are compared with the target program. This addresses images that look natural while realizing the wrong roles or attribute structure.

Freezing the recognizer means its parameters are not updated; it does not by itself establish how the entire recognition pipeline is differentiable. The appendix does not specify gradient handling for discrete parsing, matching, and the cycle loss, leaving an implementation gap for reproduction. The external recognizer is absent at inference time and is not used to select generated samples. The final scene explanation comes from the model's internal program and ownership/ordering outputs.

A Worked Example

Consider the paper's three-mug scene: a red mug on the left, a yellow mug on the right, a blue mug in the center, a spoon only inside the blue mug, and a green apple in front of the red mug. The program first assigns separate instances to the three mugs, binds their colors, records a mug count of three, and attaches the spoon and apple relations to their respective endpoints.

During sampling, the program stream continually informs the image stream that the mugs are distinct, while the image stream feeds current region evidence back to the program stream. The grounding head must associate the spoon's support with the appropriate instance, and the ordering head supplies coarse spatial/occlusion evidence. During training, the cycle recognizer checks whether the decoded image recovers the counts, colors, and relations. This illustrates the mechanism, not a measured intermediate trajectory reported by the paper or a hard guarantee of correct generation.

Loss & Training

The total objective in the paper's Eq. (6) is:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{img}}+\lambda_{\mathrm{prog}}\mathcal{L}_{\mathrm{prog}}+\lambda_{\mathrm{pan}}\mathcal{L}_{\mathrm{pan}}+\lambda_{\mathrm{align}}\mathcal{L}_{\mathrm{align}}+\lambda_{\mathrm{cycle}}\mathcal{L}_{\mathrm{cycle}}. \]

The five terms respectively cover image denoising, program denoising, ownership and ordering, instance visual alignment, and cycle-based structure recovery. Program, panoptic, alignment, and cycle weights are 1.0, 1.0, 0.25, and 0.5, respectively; the depth weight within the panoptic term is 0.1. These weights are shared across model scales rather than tuned per benchmark.

The 1B configuration uses width 2048, 24 Transformer blocks, 16 attention heads, and an MLP expansion ratio of 4. Training uses AdamW, bf16, gradient clipping at 1.0, and EMA decay 0.9999; the learning rate is \(1.0\times10^{-4}\) with a 10k-step warmup followed by cosine decay. The 1B model trains for 1.2M steps at 512px and uses 50 inference steps for the main evaluations; the authors report 64 H200 GPUs for this training run. Training-data sources and counts, exact parser/recognizer models, and batch size are not specified concretely, so these settings do not constitute a complete reproduction recipe.

Key Experimental Results

Main Results

The main table uses a unified reevaluation pipeline rather than combining scores from different papers. GenEval 2 uses Rewritten prompts with four fixed seeds and four images per prompt, aggregating all single-sample results without best-of-N selection. GE2 and SANEval-Simple are scaled to 0–100, PSG-Score retains its 0–1 scale, and DM averages accuracies over DetailMaster's eight tasks.

The table below selects results from the paper's Table 1(a). Ours-FlatText is the most relevant comparison for attribution: it matches PSP-DiT in backbone, image/text data, optimization budget, and inference settings, but lacks the program latent and its associated structural supervision. External models provide references under the same evaluation protocol, not matched training budgets.

Model	GE2 ↑	SANE ↑	PSG ↑	DM ↑
FLUX.1-dev	29.5	61.8	0.62	63.6
SD3.5-Large	19.0	55.4	0.60	59.6
SG-Adapter + SD3.5-Large	24.7	59.8	0.66	62.1
Ours-FlatText	32.8	65.9	0.65	66.8
PSP-DiT	38.2	73.6	0.73	72.4

Absolute gains over the matched baseline are 5.4, 7.7, 0.08, and 5.6. Table 1(b) reports FID-30K of 8.1 → 7.9, HPS v2.1 of 31.9 → 32.6, and ImageReward of 1.35 → 1.42, supporting compositional gains without an obvious sacrifice in the reported quality metrics.

Table 1(e) reports latency of 21.4 → 23.9 seconds/image and memory of 29.6 → 32.1 GB. Measurements use one H200, batch size 1, 1024×1024 resolution, 50 steps, bf16, FlashAttention, and torch.compile, including decoding; 100 runs are averaged after 10 warmup runs. The approximately 1.12 times latency applies only to this configuration and cannot be directly compared with external FLUX timing on RTX A6000. The 12.8B → 13.4B parameter counts describe the full deployed pipeline, not the 1B trainable backbone in Table 1(d).

Ablation Study

The following results come from Table 2(a). Each row cumulatively adds components to the preceding row, rather than independently removing one component from the full model.

Config	GE2 ↑	SANE ↑	PSG ↑	DM ↑
Ours-FlatText	32.8	65.9	0.65	66.8
Add scene-program tokens	33.7	66.8	0.66	67.4
Then add instance indexing	35.2	68.9	0.68	68.9
Then add joint program denoising	36.8	71.1	0.70	70.4
Then add panoptic grounding	37.6	72.6	0.72	71.6
Then add cycle consistency: PSP-DiT	38.2	73.6	0.73	72.4

Adding program tokens alone raises GE2 by 0.9. Subsequent adjacent gains from instance indexing, joint denoising, panoptic grounding, and cycle consistency are 1.5, 1.6, 0.8, and 0.6. Joint denoising contributes the largest increment in this addition order, but component dependencies prevent concluding that it contributes the most under every combination.

The robustness experiments in Table 2(b) perturb programs only at inference time, without retraining. GE2 is 37.6 after paraphrasing and reparsing, 36.9/35.8 after 10%/20% node or edge dropout, and 35.4 after 10% relation-label corruption, all above FlatText's 32.8. Dropping nodes also removes incident attributes and relations to preserve valid syntax; this does not establish correction of arbitrary parser errors.

Key Findings

• Table 1(c) reports counting of 62.7 → 74.1, attribute binding of 67.4 → 75.3, and role relations of 58.9 → 69.7. Counting and role-relation gains are 11.4 and 10.8, respectively, matching the intended instance and ordered-relation modeling goals.
• In Table 1(d), GE2 is 28.9 → 33.6 at 0.5B/512px and 34.7 → 41.5 at 3B/1024px. Larger models still benefit, but the 3B configuration changes both scale and resolution, so the two effects cannot be independently attributed.
• The source contains conflicting skill-aggregation descriptions: §4.4 describes prompt-count-weighted aggregation of multiple native submetrics, whereas Appendix A.11 assigns each skill to one subset with no nontrivial weighting and defines the five-skill Avg. as an unweighted mean. This note preserves reported values rather than repairing the definition.
• The human-preference table reports 48.7% for FlatText and 56.8% for PSP-DiT. Appendix A.6 specifies 200 prompts and prompt-level wins after majority votes from three raters, which would require 0.5-percentage-point increments. These values are incompatible with that protocol, and comparison opponents are not clearly identified. The appendix describes bootstrapping, but the readable main table supplies no numerical confidence intervals; statistical significance cannot be claimed from these results.

Highlights & Insights

• A generated scene explanation becomes internal state rather than a post-hoc description. The transferable idea is to maintain a separate structural stream for generation tasks requiring persistent entity identity, with feedback between visual and structural streams.
• Instance-level ownership grounds attribute binding in specific regions instead of whole-image text similarity. It offers more direct error localization than global rewards, although its usefulness still depends on pseudo-label and grounding quality.
• Cumulative ablations distinguish additional structural information from structure participating in generation. They support the full approach over token augmentation, but parameter-matched and supervision-matched experiments are still needed to separate the sources of improvement.

Limitations & Future Work

• The authors explicitly study only text-to-image generation. Video extensions would require cross-frame identity and scene-state consistency, which are not evaluated here.
• Fixed slots, including 16 foreground instances, discarded empty-program examples, and frozen grounding systems may limit dense scenes and weakly visible objects. Further evaluation should cover slot overflow, occlusion, small objects, and rare relations.
• Code and checkpoints are unavailable, while training-pool details, parser and recognizer models, continuous program encoding, and cycle-gradient implementation remain incomplete. Together with the reported approximately 1.1M H200 GPU-hours for the project, this creates a high independent-reproduction barrier.
• There is no parameter-matched widened FlatText baseline or isolated test of additional structural supervision, and main benchmarks lack error bars across repeated training runs. Matching reduces the backbone-difference explanation but does not strictly attribute all gains to the latent variable alone.
• Skill aggregation and human-preference accounting need clarification. Releasing subset lists, actual prompt counts, comparison opponents, and prompt-level outcomes would enable more verifiable statistical comparisons.

Related Work & Insights

• vs GLIGEN/ControlNet/BoxDiff: These methods use position, masks, or attention constraints as external controls, whereas PSP-DiT also denoises the program. The distinction is not simply richer conditioning but whether the structural condition can be updated as a random state.
• vs scene-graph/layout generation: Relational representations have extensive precedents; this paper emphasizes their combination with instance identity, visual ownership, and joint generation. It also cites Xu et al.'s joint diffusion modeling of grounded scene graphs and images, so joint generation of structure and images itself should not be presented as unprecedented.
• vs DiT+RAE scaling: The backbone and representation space provide image-generation capacity, while this paper changes internal scene representation. Useful follow-up questions concern propagating program uncertainty into the image stream and distinguishing valid correction of parsing errors from deviation from the original prompt.

Rating

• Novelty: 4/5, a coherent combination of instance-indexed programs, joint denoising, and visual ownership supervision, with precedents for joint structural generation.
• Experimental Thoroughness: 3/5, broad benchmark, ablation, and noise-test coverage, but incomplete supervision/capacity attribution and statistical accounting.
• Writing Quality: 3/5, a clear main narrative with implementation gaps and two evaluation definitions requiring clarification.
• Value: 4/5, an interpretable structural state for complex compositional prompts, with practical reuse dependent on code and data disclosure.