InstaPano: Zero-shot Instance Layout Controlled Panorama Generation Via Global Attention Fusion¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection / Image Generation
Keywords: Multi-Instance Generation, Controllable Panorama Generation, Joint Diffusion, Attention Fusion

TL;DR¶

InstaPano introduces a training-free framework for zero-shot instance layout-controlled panorama generation, leveraging a Sync-Fuse-Dispatch workflow for global attention fusion and a Conditional Position Mask to eliminate object repetition in large bounding boxes.

Background & Motivation¶

In wide-aspect-ratio (e.g., 1:2, 1:3, 1:4) panorama synthesis, direct training or fine-tuning of diffusion models on extended canvases is hindered by prohibitive GPU memory requirements and the severe scarcity of panoramic datasets annotated with fine-grained multi-instance bounding boxes. Under zero-shot and training-free settings, joint diffusion paradigms such as MultiDiffusion decompose the wide canvas into overlapping square local views, independently denoise each view, and merge them via weighted pixel averaging at each sampling step.

While this post-hoc local sample fusion strategy succeeds in generating seamless generic backgrounds, it introduces fundamental limitations when applied to fine-grained Instance Layout Controlled Panorama Generation (ILCPG). Because self-attention and cross-attention operations remain entirely confined within each isolated local view during denoising, the network lacks a holistic spatial representation of all bounding boxes and semantic entities across the global canvas. When an object straddles view boundaries, or when multiple instances share complex spatial relationships across views, isolated local denoising leads to fragmented attention maps, severed objects, context inconsistencies, and severe boundary artifacts. Subsequent methods such as SyncDiffusion and MAD introduce perceptual losses or partial cross-view attention fusion, yet they still generate object regions separately and lack global spatial reasoning over multi-instance layouts.

To break through this bottleneck, this paper proposes extending pre-trained Layout-to-Image (L2I) models to panoramic generation without retraining, facilitating global semantic planning across all instance bounding boxes on a unified canvas. Core idea: introduce the training-free InstaPano framework, integrating a Sync-Fuse-Dispatch (SFD) workflow into the U-Net attention layers to periodically synchronize local view latents into a unified global canvas, perform multi-level global attention fusion and layout-guided cross-attention, and dispatch enriched features back to local views, fundamentally shifting panorama synthesis from post-hoc sample stitching to holistic layout-aware generation.

Method¶

Overall Architecture¶

The input to InstaPano comprises a global scene description $P$, a set of bounding box coordinates $B = \{b_1, \dots, b_N\}$, and corresponding local textual descriptions $D = \{d_1, \dots, d_N\}$; the output is a high-resolution, wide-aspect-ratio panorama strictly adhering to the spatial layout with global semantic coherence.

During diffusion inference, the global canvas $J_t$ at timestep $t$ is divided into $I$ overlapping standard-resolution local views $\{v_1, \dots, v_I\}$. Within the U-Net attention layers, InstaPano periodically executes the three-stage Sync-Fuse-Dispatch workflow: ① Cross-View Latent Synchronization (Sync) aggregates latents from all local views into a unified global representation; ② Global Multi-Level Attention Fusion (Fuse) performs Global Self-Attention, Global Text Cross-Attention, and Layout-Guided Cross-Attention modulated by a Conditional Position Mask (CPM); ③ Global Context Dispatch (Dispatch) splits the globally planned features back into respective local views to continue standard convolution and normalization layers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Global Prompt P<br/>Bounding Boxes B and Local Prompts D"] --> B["Extract Latents from Overlapping Views<br/>{z_t^(i)}"]
    B --> C["Cross-View Latent Synchronization Sync<br/>Average aggregation to global canvas Z_t"]
    C --> D["Global Self-Attention Global SA<br/>Model long-range cross-view spatial dependencies"]
    D --> E["Combined Cross-Attention Combined CA<br/>Global Text GCA + Layout-Guided LCA"]
    E --> F["Conditional Position Masking CPM<br/>Center-focused Gaussian decay suppresses repetition"]
    F --> G["Global Context Dispatch Dispatch<br/>Split and inject global features into local views"]
    G --> H["Standard U-Net Denoising Blocks<br/>Yield coherent multi-instance panorama"]

Key Designs¶

1. Cross-View Latent Synchronization and Dispatch: Constructing a Bidirectional Global Bus To overcome the isolation of local views, cross-view feature communication must occur before spatial features are decoupled. At a given denoising timestep $t$ and U-Net attention layer $k$, the Sync operation collects local latent representations from all $I$ views $\{z_{t,k}^{(i)}\}_{i=1}^I$ and projects them onto a single continuous global canvas $Z_{t,k}$. For overlapping pixel coordinates $p$, an averaging aggregation strategy is applied: $$Z_{t,k}(p) = \frac{1}{|\mathcal{I}_p|}\sum_{i \in \mathcal{I}_p} z_{t,k}^{(i)}(p)$$ where $\mathcal{I}_p$ denotes the set of local view indices covering location $p$. After holistic multi-level attention updates, the Dispatch operation performs the inverse Split operator, slicing the global canvas $Z_{t,k}$ back into local view latents $z_{t,k}^{(i)}$ based on view coverage boundaries. This design bypasses the massive memory cost of executing the entire U-Net backbone over high-resolution panoramas, focusing global communication exclusively within key attention layers.

2. Global Multi-Level Attention Fusion: Unifying Structural Coherence and Instance Grounding On the synchronized global feature $Z_{t,k}$, InstaPano executes a two-stage attention block inherited from a pre-trained L2I model. The first stage applies Global Self-Attention (SA) without any mask, where queries, keys, and values are all projected from $Z_{t,k}$, encouraging unconstrained cross-view feature interactions to anchor continuous global backgrounds (such as skies, coastlines, and mountains) during early denoising steps. The second stage applies Combined Cross-Attention (CA), which marries Global Text Cross-Attention (GCA) for background conditioning with Layout-Guided Cross-Attention (LCA) for precise object placement. For each instance $i$, a guidance embedding $G_i = [E(d_i), \text{MLP}(\text{Fourier}(b_i))]$ is formed by concatenating the prompt embedding with Fourier positional features of bounding box $b_i$, enforcing strict spatial boundaries via a hard attention mask.

3. Conditional Position Mask (CPM): Eliminating Repetition Artifacts in Large Bounding Boxes When dealing with large bounding boxes, pre-trained L2I models exhibit a feature tiling tendency under uniform attention, frequently duplicating single objects (e.g., generating multiple miniature cats within a single large box specified for "a cat"). Conversely, bounding boxes meant for collective categories or natural textures (e.g., "pine trees" or "wildflowers") require uniform spatial attention. InstaPano addresses this with Conditional Position Masking (CPM). An LLM agent classifies local prompts into single-object ($S_{\text{single}}$) and multi-object ($S_{\text{multi}}$) sets. For single-object boxes, CPM applies a center-focused Gaussian decay over normalized coordinates $(u_p, v_p)$: $$\operatorname{CPM}_i(p) = \begin{cases} \exp\left(-\frac{u_p^2 + v_p^2}{2\sigma^2}\right), & \text{if } p \in b_i \text{ and } i \in S_{\text{single}} \\ 1, & \text{if } p \in b_i \text{ and } i \in S_{\text{multi}} \\ 0, & \text{if } p \notin b_i \end{cases}$$ CPM acts as a spatial weighting factor during cross-attention feature aggregation: $\text{Attn}_{\text{cross}} = \text{GCA}_P(Z) + \sum_{i=1}^N \operatorname{CPM}_i \odot \text{LCA}_{d_i}(Z)$. By anchoring dominant object semantics at the center and smoothly attenuating toward the box perimeter, peripheral object repetition is effectively eliminated without breaking context transitions.

Loss & Training¶

InstaPano is entirely training-free and operates purely during zero-shot sampling. Built upon Stable Diffusion XL and IFAdapter, it utilizes 30 sampling steps. Hyper-parameter $\sigma$ in CPM is set to 0.15. Global Self-Attention (SA) fusion is activated during the first 10 steps to anchor global geometry, whereas Cross-Attention (CA) fusion and CPM remain active across all 30 steps.

Key Experimental Results¶

Main Results¶

The authors introduce Pano-Layout-Bench, consisting of 1,341 layout-prompt pairs across 1:2, 1:3, and 1:4 aspect ratios with an average of 4.86 bounding boxes per image. Evaluations measure layout fidelity (mIoU, AP, AP50, AR via GroundingDINO), text-image consistency (CLIP, Local CLIP), perceptual transition smoothness across views (Intra-LPIPS), and aesthetic quality. Under the fair background-only setting, InstaPano significantly outperforms prior joint diffusion baselines, with further gains demonstrated under the holistic prompt setting (InstaPano*).

Method	mIoU ↑	AP ↑	AP50 ↑	AR ↑	CLIP ↑	Local CLIP ↑	Intra-LPIPS ↓	Aesthetic Score ↑
MultiDiffusion	0.57	0.17	0.29	0.37	30.07	25.91	0.6865	5.89
SyncDiffusion	0.52	0.13	0.25	0.31	29.10	25.06	0.6625	6.07
MAD	0.57	0.21	0.35	0.38	29.44	25.96	0.6007	5.79
InstaPano (Ours)	0.63	0.25	0.44	0.44	30.59	26.74	0.5665	5.81
InstaPano (Holistic Prompt)*	0.71	0.25	0.45	0.49	32.37	27.45	0.6038	6.12

Ablation Study¶

To verify that performance gains stem from holistic attention fusion rather than the underlying L2I backbone, Table 3 compares InstaPano with a controlled baseline (MAD+IFAdapter) employing the identical IFAdapter backbone and holistic prompt. Table 4 ablates the duration of Global Self-Attention (SA) fusion steps $t$.

Config / Variant	mIoU ↑	AP50 ↑	AR ↑	CLIP ↑	Local CLIP ↑	Intra-LPIPS ↓	Note
MAD+IFAdapter* (Post-hoc)	0.56	0.28	0.36	30.02	26.96	-	Standard post-hoc sample stitching baseline
*InstaPano (Holistic Fusion)**	0.71	0.45	0.49	32.37	27.45	-	Holistic layout-aware attention fusion
SA fusion duration $t=0$	0.68	0.37	0.45	32.34	27.71	0.6140	Lacks cross-view self-attention; higher boundary LPIPS
SA fusion duration $t=10$ (Default)	0.68	0.42	0.45	32.19	27.62	0.5752	Optimal trade-off between coherence and fidelity
SA fusion duration $t=20$	0.68	0.43	0.46	31.83	27.55	0.5613	Slightly smoother transitions; minor CLIP decrease
SA fusion duration $t=30$	0.67	0.46	0.43	31.53	27.36	0.5478	Over-constrained text alignment across full schedule

Key Findings¶

Holistic Fusion Outperforms Post-hoc Stitching: Under the exact same IFAdapter backbone, InstaPano outperforms MAD+IFAdapter by +0.15 mIoU (0.71 vs 0.56) and +0.17 AP50 (0.45 vs 0.28). This confirms that without global attention communication, post-hoc fusion undermines pre-trained layout grounding models.
Timing Trade-off in Global Self-Attention: Enabling global SA fusion during the first 10 steps suffices to establish macroeconomic perspective and background continuity; executing it for the full 30 steps over-constrains feature flexibility, decreasing global CLIP score from 32.34 to 31.53.
Manageable Memory Footprint: Profiled on an NVIDIA 48GB vGPU, the Fusion stage incurs minimal overhead (40–80 MB); peak overhead occurs in the Sync stage at high-resolution layers (~660 MB) for global tensor allocation, remaining well within manageable limits.

Highlights & Insights¶

In-Attention Layer Feature Routing: Shifting cross-view communication into U-Net attention layers before local decoding resolves the root cause of object fragmentation in joint diffusion without full-resolution retraining.
Semantic-Aware Attention Modulation via CPM: The introduction of Gaussian center-weighting coupled with LLM semantic classification provides an elegant mathematical remedy to the classic feature-tiling dilemma in large bounding boxes.
Zero-Shot Extension of Existing L2I Models: Seamlessly adapts off-the-shelf layout-to-image priors (e.g., IFAdapter) to arbitrary panoramic aspect ratios without costly panorama collection and fine-tuning.

Limitations & Future Work¶

Author-Acknowledged Limitations: In extremely cluttered scenes with heavily overlapping bounding boxes, attention interference between adjacent entities can still cause local semantic bleeding. Furthermore, global synchronization at ultra-high resolutions remains bounded by GPU memory allocations.
Additional Observations: Evaluation reliance on GroundingDINO introduces false-positive penalties when the model generates semantically harmonious background instances outside labeled boxes. Additionally, the method is designed for planar panoramas and does not model 360° equirectangular rotational continuity.
Future Directions: Adapting InstaPano's attention fusion pipeline to spherical coordinates and cylindrical projections to facilitate 360° VR scene synthesis and consistent video walkthroughs.

vs MultiDiffusion / SyncDiffusion / MAD: Prior joint diffusion methods treat local views independently and rely on post-denoising averaging or perceptual losses; InstaPano integrates cross-view communication directly into the attention blocks, achieving holistic instance-level spatial planning.
vs InstanceDiffusion / IFAdapter / CreatiLayout (Direct Generation): Direct inference of square-trained models on extended canvases suffers from catastrophic OOD collapse; InstaPano preserves high local fidelity while guaranteeing global spatial coherence.

Rating¶

Novelty: ⭐⭐⭐⭐☆ (Ingenious Sync-Fuse-Dispatch workflow and conditional position masking for zero-shot panoramas)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive 1,341-sample benchmark, detailed ablations, and layer-wise memory profiling)
Writing Quality: ⭐⭐⭐⭐⭐ (Clean narrative arc, clear mathematical formulations, and excellent problem grounding)
Value: ⭐⭐⭐⭐☆ (Highly practical, training-free approach for panoramic and wide-aspect-ratio content creation)