Semantic Browsing: Controllable Diversity for Image Generation¶

Conference: ECCV 2026
arXiv: 2606.23679
Project Page: https://saradorfman1.github.io/SemanticBrowsing-webpage/
Code: None
Area: Image Generation
Keywords: Text-to-Image Generation, Semantic Diversity, Controllable Generation, Multi-Agent Systems, Hierarchical Exploration

TL;DR¶

This paper proposes the Semantic Browsing framework, which utilizes a multi-agent VLM pipeline to automatically expand user prompts into structured JSON scene representations. It then systematically mutates semantic dimensions to generate hierarchical diverse image galleries, achieving navigable, interpretable, and controllable diversity generation.

Background & Motivation¶

Modern text-to-image (T2I) models (e.g., FIBO, FLUX) have achieved outstanding visual quality and prompt alignment, thanks to training paradigms that emphasize strict compliance with detailed descriptive text. However, this "strictness" introduces a side effect: images generated with different random seeds under the same prompt often converge to nearly identical visual interpretations. For example, a prompt like "a dog, a cat, and a parrot" repeatedly generates animals of the same breed, pose, and layout. This diversity collapse is not accidental, but rather a joint effect of the training objectives and sampling mechanisms: highly aligned training causes models to "over-commit" to a single semantic interpretation, and guidance techniques like CFG further suppress diversity.

Prior work has attempted to alleviate this issue from various angles. CADS and Guidance Interval perturb conditioning signals during the denoising process, Particle Guidance applies repulsive forces in the latent space, and SGI first generates a large pool of candidates and then filters out a diverse subset. While these approaches indeed increase variance among samples, the variations stem from random effects rather than meaningful semantic choices—what changes are superficial dimensions like texture details and lighting angles, while the character identities, interaction modes, and stylistic tones of the scene remain fixed. Users cannot instruct the system: "I don't want to see a bowl from different angles; I want to see two scenes of the bowl in a kitchen vs. on a picnic."

The core insight of this paper is that the strong prompt alignment of modern text-to-image models is not a limitation but rather a key enabling condition for fine-grained semantic control—if you can precisely manipulate text descriptions, the model will accurately reflect your modifications. The problem then shifts to "how to systematically generate structured semantic variants at the text level." Core Idea: Expand user prompts into structured JSON scene representations, employ a multi-agent pipeline to automatically identify mutable semantic dimensions and generate mutually exclusive yet plausible alternatives, and organize them into a hierarchical tree structure. Each branch of the tree corresponds to a meaningful semantic choice with constraint inheritance across levels, ultimately rendering a navigable semantic diversity gallery.

Method¶

Overall Architecture¶

The core pipeline of Semantic Browsing consists of two phases. The first phase is scene representation: the user inputs a natural language prompt \(p\), which is expanded by a VLM into a structured JSON scene \(s_0\) containing specific object lists, attributes, spatial layouts, environmental styles, etc., forming a fully-specified scene interpretation. The second phase is hierarchical diversity exploration: taking \(s_0\) as the root node, a four-agent pipeline (Context Analyst \(\rightarrow\) Brainstormer \(\rightarrow\) Decision Maker \(\rightarrow\) Critic) automatically analyzes the current scene, selects a semantic dimension to generate several mutually exclusive mutation instructions, and applies them to the parent node via a JSON Refiner to produce child nodes, progressively constructing an interpretation tree with scene JSONs as nodes and semantic constraints as edges. Once the tree is constructed, each leaf node is rendered into an image by a strongly aligned T2I model, forming a structured semantic gallery.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Prompt p"] --> B["VLM Scene Parsing<br/>→ Initial JSON s₀"]
    B --> C["Context Analyst<br/>Identify Mutable Details {dᵢ}"]
    C --> D["Brainstormer<br/>Categorize High-Level Semantic Dimensions {aᵢ}"]
    D --> E["Decision Maker<br/>Select Dimension a* → Generate Constraint Set {cᵢ}"]
    E --> F["Critic<br/>Validate Plausibility & Consistency"]
    F --> G["JSON Refiner<br/>Apply Constraints → Child Nodes"]
    G --> H{"Continue Expansion?"}
    H -->|Yes| C
    H -->|No| I["T2I Generator<br/>Render Leaf Nodes to Images"]

Key Designs¶

1. Structured Scene JSON Representation: Making Implicit Semantics Explicit into Actionable Blueprints

To achieve precise control over semantic variation, the semantics themselves must first be addressable—one cannot locate "animal breed" or "scene mood" within a latent vector. This paper represents each scene as a fully-specified JSON object, containing object lists (categories, attributes, materials, colors), spatial relations (positions, sizes, orientations), and global attributes (styles, lighting, weather, emotions). This representation is expanded and generated from the user prompt by a VLM, automatically filling in unspecified details of the prompt. For example, the prompt "a dog, a cat, and a parrot" might be expanded into a complete JSON containing specific breeds (Golden Retriever, Siamese cat, Budgerigar), environment (living room), and lighting (natural light). Crucially, it clearly distinguishes between attributes constrained by the user prompt (fixed and immutable) and those filled in by the VLM (open and mutable)—this distinction forms the operational basis for the subsequent Context Analyst.

2. Four-Agent Collaborative Reasoning: Precise Division of Labor in Semantic Exploration

Constructing a diversity tree requires simultaneously satisfying three sometimes conflicting goals: semantic uniformity of variation dimensions (Semantic Structuring), genuine differences between branches (Heterogeneity), and compatibility of all variations with the original prompt and accumulated constraints (Plausibility). This paper decomposes this task into four specialized agents:

Context Analyst receives the current scene JSON and the historical constraints accumulated along the path \(C_s\), reviewing item-by-item which attributes are locked by the prompt or historical constraints (e.g., "dog, cat, parrot must exist") and which are open details filled by the VLM (e.g., breed, coat color, pose), and outputs a set of mutable details \(\{d_i\}\). It is directly responsible for Plausibility—ensuring subsequent variations do not alter fixed elements.

Brainstormer abstracts the low-level details in \(\{d_i\}\) into high-level semantic dimensions \(\{a_i\}\). For example, instead of individually mutating "dog breed," "cat breed," and "bird breed," it aggregates them into a unified dimension "animal breeds and species." It also evaluates the potential visual impact magnitude (high/medium/low) of each dimension, ensuring that the tree grows toward dimensions with substantial semantic changes. It is responsible for Semantic Structuring—guaranteeing that branches at the same level revolve around the same meaningful semantic dimension, rather than mixing noise from different levels into a single dimension.

Decision Maker selects the most impactful candidate dimension \(a^*\), generating several concrete and mutually exclusive constraints \(\{c_i\}\) for that dimension, such as "playful chasing," "peaceful coexistence," or "cat in a dominant pose" under the "interaction mode" dimension. It actively reasons about "what the plausible realizations of this dimension are," ensuring that differences between sibling nodes represent structured variations rather than minor tweaks. It is responsible for Heterogeneity—originating the actual semantic increment of the diversity tree.

Critic serves as the final quality gatekeeper, reviewing whether each constraint conflicts with the original prompt and historical constraints. It refines ambiguous or contradictory constraints, ensuring that the final output consists of precisely executable instructions. It maintains Plausibility alongside the Context Analyst but from a different angle: the latter starts from "which areas can change," while the former tackles "whether the generated instructions conflict." Ablation studies prove their complementarity—removing the Context Analyst drops contextual consistency from 0.87 to 0.82, while removing the Critic reduces the VQAScore from 0.90 to 0.87.

3. Hierarchical Interpretation Tree and Constraint Inheritance: A Navigable Semantic Space

All nodes constitute a rooted tree with the original scene \(s_0\) as the root. Each edge from a parent node to a child node corresponds to an atomic semantic constraint—changing only one specific instantiation of an selected semantic dimension while keeping all other fixed attributes unchanged. This means the tree structure naturally encodes semantic distance: the number of hops between two nodes on the tree diagram is highly correlated with their semantic variance in the visual space (experiments validated a monotonic positive correlation from 1-hop DINO distance of 0.168 to 5-hops of 0.452). The tree construction follows three requirements: (i) all child nodes of the same parent must come from the same semantic dimension—making it clear to the user that all variants at this level represent different options along the "interaction mode" axis; (ii) each constraint under the same dimension must yield distinct instantiations; (iii) each constraint must be logically compatible with the root prompt and historical constraints along the path.

4. Preserve Branches: Maintaining Structural Integrity and Interactive Flexibility

A hidden but crucial engineering detail: during recursive expansion, in addition to generating \(k\) mutated child nodes, each node also generates an "identity-mapping" child node that preserves the current state. This ensures that intermediate nodes are also propagated to the leaf level, so that the final leaf collection captures the complete semantic combinations along all paths. More importantly, in interactive browsing, users can select any intermediate node to continue expanding—the preserve branch ensures that even if one chooses to keep a dimension unchanged, they can still explore along other dimensions later, creating a truly navigable semantic space rather than merely a predefined pool of specific paths. Furthermore, because tree construction is triggered node-by-node, users can manually select nodes of interest at any time to guide the system for progressive, interactive creative exploration.

A Complete Example: Dog, Cat, and Parrot¶

User prompt: "a dog, a cat, and a parrot". First, the VLM expands it into an initial JSON: Living room environment, a Golden Retriever sitting, a Siamese cat laying on the windowsill, a Budgerigar standing on a stand, natural light. Context Analyst identifies mutable details: breeds (replaceable by Husky / Himalayan cat / Sulphur-crested cockatoo), interaction mode (the VLM's default "minding their own business" can be changed to "chasing/snuggling"), layout (default living room can be changed to garden/study).

Brainstormer abstracts these low-level details into two high-level dimensions: "interactions between animals" and "animal breeds and species", evaluating the former as having a larger visual impact magnitude. Decision Maker selects "interaction mode" and generates three constraints: "playful chasing," "peaceful coexistence," and "cat showing dominant posture." Critic checks if "cat dominant" conflicts with previously set constraints—for instance, if the parent scene already specifies "coexistence," this constraint must be realized as a non-aggressive dominant posture to avoid self-contradiction. Once passed, the JSON Refiner applies each constraint to generate three child nodes (one of which is a preserve node).

For a child node of interest (e.g., "playful chasing"), the next round is executed recursively. At this stage, the accumulated constraints include "interaction = play," based on which the Context Analyst identifies the next available dimension (e.g., "animal breeds"—since interaction is fixed and breeds are not yet constrained). Ultimately, a tree of \(3^3 = 27\) leaf nodes is generated, mutating along a different semantic axis at each level, with all branches maintaining the coherence of historical choices.

Loss & Training¶

This method is completely training-free and does not modify the parameters or loss functions of the T2I model. All agents utilize existing commercial VLMs (defaulting to Gemini 2.5 Flash, with experiments validating the equivalence of ChatGPT-5.5), controlling their behaviors purely through meticulously designed system prompts. The core overhead lies in VLM API calls: generating a full tree of 27 images takes approximately 10.2 seconds and 15.9K tokens per result, falling within a similar practical and reasonable cost range compared to simple random seed generation (8.5 seconds, 3.3K tokens) and Post-Hoc optimization generation (11.4 seconds, 9.7K tokens).

Key Experimental Results¶

Main Results¶

Method	Vendi ↑	DINO Sim. ↓	Aesthetic ↑	VQAScore ↑
Semantic Browsing (Ours)	3.34	0.61	6.52	0.90
Stochastic VLM Seeding	2.60	0.76	6.53	0.93
Post-Hoc Diversity Opt.	2.79	0.67	6.51	0.92
High-Temp. Post-Hoc Opt.	2.85	0.66	6.56	0.92
CADS	3.29	0.67	6.30	0.89
Guidance Interval	2.96	0.71	6.42	0.92
Power-Law CFG	2.75	0.74	6.28	0.93

The proposed method comprehensively leads in all diversity metrics (highest Vendi of 3.34, lowest DINO Sim of 0.61) while maintaining a highly competitive Aesthetic Score. Though the VQAScore is slightly lower than the top baseline (0.90 vs 0.93), it remains at a high level—the authors attribute this to the trade-off where VQA evaluators naturally favor conservative compositions (low variance = high confidence) rather than reflecting the true alignment quality.

Ablation Study¶

Configuration	VQAScore	Hierarchical Consistency	Key Description
Full workflow	0.90	0.87	Full pipeline
w/o Context Analyst	0.90	0.82	Contextual consistency drops post-removal
w/o Critic	0.87	0.87	VQAScore drops, prompt alignment begins to drift
Unified (Brainstormer+DM)	—	—	After merging, DINO distance drops from 0.389 to 0.362 (-7.2%)

Key Findings¶

Value of Agent Division of Labor: Merging Brainstormer and Decision Maker into a single agent reduces the global DINO distance by 7.2% (from 0.389 to 0.362), showing degradation across all topological distance levels (1-5 hops). This validates that "abstracting semantic dimensions" and "generating concrete constraints" indeed require separated reasoning roles.
Complementarity of Context Analyst and Critic: The two maintain Plausibility from different levels—the former ensures "local boundaries are not crossed" (immutable regions remain unchanged), and the latter ensures "global contradiction-free" state (new constraints do not conflict with history). Removing either leads to a significant drop in either Hierarchical Consistency or VQAScore.
Positive Correlation between Semantics and Topology: Pairwise DINO distance increases monotonically with the tree chart distance (1 hop: 0.168 \(\rightarrow\) 5 hops: 0.452), proving that the tree structure genuinely encodes semantic distance. Neighboring nodes share abundant visual features, while distant nodes generate substantial semantic variance, satisfying the design goal of Semantic Structuring.
Verification of Model Agnosticism: Replacing the rendering backend from FIBO to FLUX.2 still produces consistent structured diversity results, confirming that the semantic control pipeline is completely decoupled from pixel rendering.
User Evaluation Sweep: In head-to-head comparisons among 25 participants, the proposed method overwhelmingly outperformed all baselines (CADS, Guidance Interval, Power-Law CFG, Post-Hoc Opt.) in the Diversity dimension, with a majority of participants simultaneously ranking it as their top choice in Overall Preference.

Highlights & Insights¶

Turning "Limitations" into "Enablers": The cleverest conceptual shift of this paper is treating the strong prompt alignment of T2I models—originally the culprit of diversity collapse—as the prerequisite for precise semantic control. This perspective of transforming a constraint into a feature is highly inspiring for deep learning methodology.
Text-Level Diversity > Latent-Level Diversity: When one can directly manipulate every element of a scene at the semantic level, random perturbations and repulsive forces in the latent space become redundant and clumsy. This insight is not limited to image generation but is equally applicable to video, 3D, and multimodal generation.
Exquisite Engineering Design of the Preserve Branch: Automatically reserving the current state as a branch at each node prevents users from being "forced to diverge" along a dimension. In interactive scenarios, users can explore several layers along one dimension and then proceed along others, greatly boosting flexibility without increasing system complexity.
Paradigm Shift Away from the "Optimal Solution": Unlike multi-agent systems like Maestro or PromptSculptor that pursue a single optimal alignment, this work conversely expands uncertainty to explore various plausible possibilities. This marks a shift from "verification" to "discovery," perfectly suited for the ambiguous early stages of creative exploration.

Limitations & Future Work¶

Dependence on the Rendering Precision of the Underlying T2I: The final image quality is bounded by whether the generative model can accurately render each instruction in the JSON. While FIBO excels in this, it may still fail on highly complex spatial relationships (such as meticulous layouts or character crossings). When a JSON specifies "the cat stands 30cm to the left of the dog, with its front paws resting on the parrot cage," current models struggle to execute this with precision.
Ceiling of VLM Creativity: While 43.7% of the variation dimensions are unique to the prompts, indicating that agents generate creative variants tailored to specific scenes, the mutable space shrinks rapidly when the prompt is already highly concrete and detail-dense. The tree's overall diversity is still bounded by the VLM's own imaginative limits.
Accumulative Degradation in Deep Trees: Experiments in Appendix F demonstrate that during pure depth-first expansion (BF=1, depth 5 \(\rightarrow\) 20), the VQAScore drops from 0.81 to 0.75—an excessively long chain of constraints gradually drifts from the original prompt's intent. Automatically detecting "when to prune" or introducing backtracking mechanisms remains an open question.
Computational Cost Structure: Although training-free, each tree requires multiple sequential VLM calls (number of calls = nodes \(\times\) 4 agents). Applications sensitive to latency or bound by tight budgets may require trade-offs in branching factors and tree depth.

vs CADS / Guidance Interval / Particle Guidance: These methods apply noise/repulsive forces in the latent or conditioning space to increase diversity, making variations uncontrollable and uninterpretable. This work operates at the text level, where each variation bears clear semantic labels and traceable constraint lineages.
vs SGI (Scaling Group Inference): SGI first generates a massive pool of candidates and then filters out the most diverse subset via QIP, which is still restricted by the base model's inherent diversity ceiling. This paper expands diversity boundaries by actively generating semantic variants rather than filtering from a pre-existing pool.
vs PAG (GFlowNet): PAG relies on specific training datasets to learn diverse sampling, whereas the proposed method is training-free and maintains global consistency via a tree structure.
vs Maestro / PromptSculptor / Twin-Co: These multi-agent systems aim to eliminate uncertainty to generate a single image that "best matches user intent," whereas this work takes the opposite path—expanding and structuring uncertainty to map out multiple plausible possibilities.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Defining "controllable semantic diversity" as a new task, this paradigm shift of replacing latent-space perturbations with text-level mutations is highly creative.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Includes quantitative, qualitative, ablation, user evaluation, structural validation, VLM sensitivity, and scaling studies, presenting a complete evaluation system.
Writing Quality: ⭐⭐⭐⭐⭐ The problem definition is clear, progressing logically from motivation to design to validation. The figures are beautifully designed and complement the text effectively.
Value: ⭐⭐⭐⭐⭐ The general methodology adapts easily to denoising diffusion and autoregressive T2Is. It holds great promise for interactive creative tools and can scale to multimodal generation scenes like video and 3D.