Visual Prompt Discovery via Semantic Exploration

Conference: ECCV 2026
arXiv: 2603.16250
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Visual Prompting, Large Vision-Language Models (LVLMs), Automated Prompt Engineering (APE), Semantic Exploration, Perception Failures

TL;DR

Addressing the systemic perception failures of Large Vision-Language Models (LVLMs) and the excessive token overhead of per-sample tool-use pipelines, this paper introduces SEVEX—an automated semantic tree search framework that discovers task-wise visual prompt scripts within an abstract idea space, achieving superior accuracy while cutting inference token consumption by 91.2%.

Background & Motivation

Large Vision-Language Models (LVLMs) demonstrate exceptional proficiency in high-level multi-step reasoning and open-ended dialogue. Nevertheless, extensive empirical benchmarks show that these models routinely stumble on fundamental perceptual operations, such as fine-grained attribute identification, counting line intersections, or resolving overlapping geometric contours. These perceptual failures corrupt grounding fidelity at the input stage, triggering cascaded hallucinations throughout subsequent reasoning chains. Visual prompting has surfaced as a compelling remedy, steering model attention and disambiguating spatial visual tokens by programmatically modifying input images through geometric overlays, bounding markers, or external perception tools.

However, existing visual prompting approaches encounter two fundamental bottlenecks. First, inference-time tool-use frameworks such as SketchPad dynamically synthesize Python programs on a per-sample basis. This setup treats the LVLM strictly as an external controller without diagnosing its intrinsic perceptual blind spots, incurring prohibitive token overhead (frequently exceeding tens of thousands of tokens per query) and compounding errors whenever initial tool selection misfires. Second, designing human-crafted visual scaffolding to mend internal perception requires grueling, non-intuitive trial-and-error. More critically, optimal visual prompts are deeply model-specific and exhibit virtually zero transferability across distinct architectures, rendering manual prompt design unscalable across evolving foundation models.

Automating this discovery via agentic search in raw code space presents another hurdle: the endless combinations of pixel-level manipulations create an intractable search space, while verbose, low-level Python code distracts LLM optimizers from high-level visual grounding strategies. This paper tackles the challenge by decoupling semantic intention from implementation details. Core idea: formulate visual prompt discovery as an agent-driven search over a high-level abstract idea space (SEVEX), integrating novelty-guided tree search with sample-wise semantic backpropagation to autonomously uncover counter-intuitive, highly effective task-wise visual prompts on minimal development sets.

Method

Overall Architecture

SEVEX establishes a closed-loop search pipeline over a dynamic tree structure \(\mathcal{T}\), where each node represents an abstract visual prompting concept rather than a fragile raw script. The discovery loop iterates through four sequential stages: first, a novelty-guided selection algorithm prioritizes nodes balancing empirical historical ceiling gains and conceptual novelty; next, an Engineer agent translates the chosen abstract idea into executable Python code leveraging a pre-configured vision toolbox; third, the generated prompt runs across a compact development set (30 samples) to harvest empirical rewards and intermediate diagnostic renderings; finally, an Analyst agent conducts sample-wise error diagnosis to extract actionable high-level insights, which are backpropagated to ancestor nodes to steer subsequent idea generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Definition & Dev Set"] --> B["Novelty-guided UCT (NUCT)<br/>Balances max empirical gain with novelty"]
    B --> C["Abstract Idea Space & Code Instantiation<br/>Decouples intent and invokes vision tools"]
    C --> D["Empirical Evaluation on Dev Set<br/>Collects quantitative rewards & output images"]
    D --> E["Sample-wise Semantic Backpropagation<br/>Diagnoses failure causes into actionable insights"]
    E --> F["Insight-driven Idea Generation<br/>Dynamically spawns candidate siblings & child nodes"]
    F -->|Iterative Loop| B
    B -->|Convergence| G["Optimal Task-wise Visual Prompt"]

Key Designs

1. Abstract Idea Space & Intent Decoupling: Mitigating Low-Level Code Distraction

Searching directly over executable Python code forces LLMs to expend context on trivial syntactical nuances and hyperparameter tweaks, triggering long-context distraction and search saturation. SEVEX represents each search node \(N\) as a structured tuple: an abstract idea \(I\) described in natural language, an implementation \(P\) combining Python script and textual prompt, self-evaluation estimates \(S = \{s_{\text{gain}}, s_{\text{novel}}\}\), and an empirical experiment history \(H\). By separating conceptual strategy (e.g., "draw typographic reference lines to distinguish letter case" or "composite candidate tiles and inspect seam continuity via depth maps") from programmatic implementation using standardized primitives (crop, GroundingDINO, DepthAnything, SemanticSAM, line drawing), the agent concentrates on diagnosing perceptual failures rather than debugging boilerplate code.

2. Novelty-Guided UCT (NUCT): Preventing Over-Branching and Local Saturation

Standard Monte Carlo Tree Search assumes every child node can be sampled, which breaks down in open-ended LLM ideation where child spaces are infinite and prone to superficial redundancy. NUCT bifurcates priority scoring based on node execution status. For executed nodes (\(n_i > 0\)), it prioritizes branches delivering the largest relative empirical gains:

\[P_i = (R_i^{\max} - R_{p_i}) + \lambda_{\text{expl}} \sqrt{\frac{n_{p_i}}{n_i}}\]

where \(R_i^{\max}\) denotes the maximum reward achieved across node \(i\) and its descendants, \(R_{p_i}\) is the parent's reward, and \(n_{p_i}\) is the parent visitation count. For unexecuted nodes (\(n_i = 0\)), NUCT predicts exploratory value by combining the agent's estimated gain, sibling-relative novelty, and a penalty on parent branch saturation:

\[P_i = s_{\text{gain}, i} + \lambda_{\text{novel}} s_{\text{novel}, i} - \lambda_{\text{sat}} \sqrt{c_{\text{exec}, p_i}}\]

where \(c_{\text{exec}, p_i}\) counts executed siblings under parent \(p_i\). This saturation penalty disincentivizes endlessly generating minor variations under an already well-explored hypothesis, driving the search toward unexplored strategic frontiers.

3. Sample-Wise Semantic Backpropagation: Distilling Qualitative Diagnostic Lessons

Propagating scalar rewards alone fails to inform subsequent ideation why a particular visual transformation failed. SEVEX deploys a dedicated Analyst agent to scrutinize model outputs against ground truth on the development set, examining representative successful and failed visual outputs. The Analyst pinpoints explicit visual mechanisms behind errors (e.g., "drawn bounding boxes occluded intersection endpoints" or "downsampling blurred critical thin lines") and synthesizes them into concise, Actionable Insights. These textual lessons are backpropagated into the experiment histories \(H\) of all ancestor nodes, equipping future prompt generators with persistent domain memory.

4. Insight-Driven Closed-Loop Ideation: Balancing Breadth and Specialization

Following evaluation and backpropagation, the tree dynamically expands. Guided by the updated history \(H\), the generator spawns sibling nodes at the same abstraction level to explore alternative conceptual hypotheses, alongside child nodes designed to specialize and refine promising attributes. Every prospective node undergoes an upfront Feasibility Check against the available toolset before entering the tree; ideas demanding non-existent tools are pruned immediately, preserving search efficiency.

A Worked Example

Consider the Jigsaw task from the BLINK benchmark, which requires identifying which of two candidate image patches correctly fills a missing region in a primary image. - Initial Conceptualization: Conventional approaches attempt direct edge-matching or color histogram comparison, which fails because LVLMs struggle to judge subtle seam alignment in raw RGB space. - Counter-Intuitive Discovery: The agent formulates a hypothesis: composite each candidate patch into the target image, and pass the composite through an external monocular depth estimation model (DepthAnything). - Physical Grounding Mechanism: An incorrect patch produces sharp, unnatural depth discontinuities and jarring boundary steps along the seam, whereas a correct patch yields smooth, physically plausible depth gradients across the reconstructed scene. - Prompt Formulation & Execution: The discovered prompt executes the composite depth generation and directs the LVLM: "Compare the depth maps of image a and image b; determine which depth map looks natural and lacks unnatural step discontinuities." This visual strategy lifts Jigsaw task accuracy from a Naive baseline of 75.8% to 95.8%.

Loss & Training

SEVEX is entirely training-free and model-agnostic, operating at test-time without updating LVLM backbone weights. Search iterations are budgeted to 50 rounds per task over a development split of 30 randomly sampled images. Hyperparameters are fixed at \(\lambda_{\text{expl}} = 0.5\), \(\lambda_{\text{novel}} = 0.15\), \(\lambda_{\text{sat}} = 0.5\), with \(k = 3\) unexecuted child nodes maintained per parent node.

Key Experimental Results

Main Results

Evaluation spans nine perception-intensive tasks across the BlindTest and BLINK benchmarks with Gemini-2.5-flash as the primary backbone, benchmarked against unprompted inference (Naive), dynamic per-sample code generation (SketchPad), and text-optimized SketchPad (SketchPad+APE).

Benchmark & Task	Naive Acc (%)	Naive Inf. Cost (tokens)	SketchPad Acc (%)	SketchPad Inf. Cost (tokens)	SEVEX (Ours) Acc (%)	SEVEX (Ours) Inf. Cost (tokens)
LineIntersections	73.0	981	33.3	16,730	90.5	991
CircledLetter	78.8	1,323	82.0	13,902	83.7	1,337
SubwayMap	60.0	1,333	58.1	16,482	62.8	1,585
OverlappingShapes	50.4	2,429	16.3	24,833	52.4	2,612
BlindTest Average	65.6	1,517	47.4	13,987	72.4	1,631
Jigsaw	75.8	869	70.8	14,896	95.8	682
Depth	83.0	1,349	85.1	14,158	85.1	1,457
Spatial	81.4	1,525	85.8	15,818	86.7	1,555
SemanticCorr.	55.0	712	58.7	9,477	63.3	1,358
VisualCorr.	87.3	641	90.9	14,291	89.4	801
BLINK Average	76.5	1,019	78.3	13,728	84.1	1,171
Overall Average	71.6	1,240	64.6	15,621	78.9	1,375

SEVEX also demonstrates superior exploration stability and search efficiency compared to iterative prompt engineering baselines:

Evaluation Dimension	SketchPad+APE (Overall Avg)	SEVEX (Ours) (Overall Avg)	Relative Comparison
Dev Accuracy (%)	67.8	83.3	+15.5% absolute gain
Test Accuracy (%)	57.7	78.9	+21.2% absolute gain
Generalization Gap (Dev - Test, %)	10.1	4.4	Overfitting sharply reduced
Exploration Cost per Iteration (tokens)	738,000	85,000	88.5% cost reduction
Inference Cost per Sample (tokens)	16,800	1,375	91.8% cost reduction

Non-Transferability Across LVLM Architectures

The paper verifies prompt transferability across diverse LVLM backbones on the LineIntersections task, revealing that optimal visual cues are intimately coupled to a specific backbone's internal perceptual biases:

Evaluated Prompt Strategy	Gemini-2.5-flash Acc (%)	Claude Sonnet 4.0 Acc (%)	GPT-4o Acc (%)
Naive (unprompted)	73.0	66.3	8.2
Prompt 1 (Discovered on Gemini: few-shot visual intersection cues)	90.5	82.9	59.1
Prompt 2 (Discovered on Claude: 3-way overlap partition & inclusion-exclusion)	87.9	87.8	31.0
Prompt 3 (Discovered on GPT-4o: region detection & crop zooming)	62.3	35.0	57.1

Key Findings

• Catastrophic Failure of Per-Sample Zero-Shot Tools: SketchPad degrades performance below unprompted Naive on BlindTest (47.4% vs. 65.6%), collapsing to 16.3% on OverlappingShapes. Without offline empirical feedback, misapplied segmentation tools inject severe visual noise that misguides the LVLM. SEVEX eliminates this vulnerability through empirical validation.
• Substantial Inference Amortization: Because SEVEX discovers a static, task-wise Python script during offline search, runtime inference token consumption is only 10.9% above Naive (1,375 vs. 1,240 tokens) while slashing SketchPad's footprint by 91.2%. When task queries exceed 273 samples, SEVEX achieves lower cumulative computing cost than SketchPad.
• Architectural Specificity of Visual Strategies: Visual prompts do not transfer cleanly across backbones. Prompt 2 achieves 87.8% on Claude but plunges to 31.0% on GPT-4o; Prompt 3 aids GPT-4o but degrades Gemini from 73.0% down to 62.3%. Automated, model-specific discovery is therefore essential for practical deployment.

Highlights & Insights

• Decoupling Semantic Strategy from Code Implementation: Transitioning the search space from concrete syntax trees to high-level conceptual descriptions shields LLM agents from low-level distraction, enabling systematic reasoning about visual perception.
• Creative Tool Repurposing Beyond Human Intuition: The framework autonomously devises unexpected visual strategies, such as repurposing monocular depth estimators to evaluate composite seam continuity in jigsaw matching, proving that automated exploration can break human cognitive bias.
• Dual-Guarded Tree Navigation: Combining saturation penalties on sibling branches with semantic lesson backpropagation across ancestor nodes prevents both shallow over-clustering and the repetition of invalid visual operations.

Limitations & Future Work

• Constrained by Fixed External Primitives: Code realization depends on an upfront suite of vision tools (GroundingDINO, SemanticSAM, DepthAnything). When tasks demand transformations beyond the toolbox's expressive capacity, the framework cannot synthesize novel low-level computer vision kernels from scratch.
• Development Set Representativeness: Although 30 samples proved sufficient on academic benchmarks, severe distribution shifts or heavy-tailed real-world data could induce mild overfitting during the 50-step search.
• Future Directions: Extending the agent to dynamically search and install external open-source vision tools from repositories on the fly, and exploring distillation of discovered visual transformations directly into LVLM visual token encoders.

Related Work & Insights

• vs SketchPad / Visual Programming: Earlier frameworks dynamically synthesize code per query at test time, resulting in astronomical compute bills and zero error correction; SEVEX amortizes discovery offline to yield efficient, deterministic, task-wise visual scaffolding.
• vs Automated Prompt Engineering (APE): Conventional APE optimizes discrete natural language phrases via paraphrasing; SEVEX bridges programmatic computer vision tools with discrete linguistic prompts, managing the complex combinatorial search of multimodal interventions.
• vs Continuous Visual Prompt Tuning: Feature-level prompt tuning requires gradient backpropagation through white-box model parameters; SEVEX treats the LVLM as a black-box API, operating purely on input images and instructions without model access constraints.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates visual prompt discovery as an abstract semantic tree search, pioneering model-specific task-wise visual prompt optimization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across nine perception tasks on two benchmarks, complete with ablation studies, search efficiency breakdowns, and multi-backbone transferability audits.
• Writing Quality: ⭐⭐⭐⭐⭐ Crisp problem framing, mathematically rigorous formulations of tree selection metrics, and insightful qualitative analyses.
• Value: ⭐⭐⭐⭐⭐ Provides a cost-effective, scalable blueprint to overcome intrinsic perception failures across modern commercial vision-language models.