ATP-Bench: Towards the Agentic Tool Planning for MLLM Interleaved Generation¶

Conference: ECCV2026
Paper: Official page · PDF
Area: Multimodal VLM
Keywords: tool planning, interleaved generation, visual-critical queries, multi-agent evaluation, visual omissions

TL;DR¶

ATP-Bench unifies five visual tools for interleaved text-image planning and evaluates 7,702 human-verified QA pairs with three MAM judge agents that separately inspect valid calls, missing visuals, and overall composition; the strongest tested model, Gemini 3 Pro, reaches an average FS of 79.88, with substantial gaps remaining in complex travel and renovation responses.

Background & Motivation¶

Producing an effective interleaved response requires more than generating an attractive image: the model must decide which passage needs which kind of visual evidence. Explaining an experiment calls for a real plot, describing a destination may require a photograph found through search, and previewing a renovation should modify the user's room image. These needs can coexist in one answer. Better image synthesis alone does not fix incorrect citations, unsuitable tool choices, or images placed where they disrupt the explanation.

Previous work largely follows two paths. Benchmarks such as OpenING emphasize open-ended image generation, whereas RAG-IGBench and MRAMG-Bench emphasize retrieving and referencing external images. Generation struggles to guarantee factual depictions of real entities and precise diagrams; retrieval cannot fully support user-specific modifications or creative visuals. Meanwhile, evaluating only the final rendered answer mixes the planner's decisions with search availability, image-backend quality, and API reliability. A sound plan can receive different scores simply because the execution backend changes.

The paper therefore isolates an interleaved tool plan: the model still writes a complete answer, but structured tool tags specify how each visual should be obtained and where it belongs. Human-verified reference answers remain available without forcing every valid response to reproduce one reference trajectory. Core idea: place reference and generation tools in a shared decision space, then assess both whether existing calls are justified and whether required calls are missing, instead of treating the number of rendered images as evidence of good planning.

Method¶

Overall Architecture¶

The input consists of a visual-critical query and a set of multimodal documents supplied by the user or retrieved externally; the query itself may be text-only or image-grounded. The tested MLLM receives a common toolkit specification and produces an ordered sequence of text and tool calls. MAM does not execute those calls: a Precision Inspector and a Recall Inspector produce separate reports, which a Chief Judge integrates into an overall score.

This is a benchmark and evaluation framework, not a new image-generation backbone or tool-learning objective. The diagram connects offline data construction, model planning, and the three-agent evaluation procedure. Reference answers support an additional tool-set comparison but are not required inputs to MAM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal sources<br/>and intents"] --> B["Visual-Critical<br/>Data Construction"]
    B --> C["Unified Tool Planning"]
    B --> G["Human-Verified<br/>Reference Answers"]
    C --> D["Complementary Call Inspection<br/>Precision Inspector"]
    C --> E["Complementary Call Inspection<br/>Recall Inspector"]
    D --> F["Tier-Anchored<br/>Holistic Judgment"]
    E --> F
    C --> H["Additional Tool-Set<br/>Comparison"]
    G --> H
    F --> I["FS, SR, MI"]

Key Designs¶

1. Visual-Critical Data Construction: require images to contribute information rather than decorate arbitrary answers

The benchmark starts with eight categories of high visual demand: Academic, Manual, Recipe, Fashion, Renovation, Product, Travel, and Encyclopedia, subdivided into 25 intents. Source documents come from resources including MRAMG-Bench, RAG-IGBench, and OVEN, covering Wikipedia, Xiaohongshu, and arXiv. Gemini 2.5 Pro first generates natural text queries from the documents and intent taxonomy. It then converts suitable queries into VQA by replacing key textual evidence with images: a portrait, for instance, supplies the skin tone and existing hairstyle that a text question would otherwise describe. Additional images come from web search, nano-banana synthesis, and public benchmarks. The second stage is therefore intended to make the question depend on its image, not merely attach one.

Ten professional annotators reject ambiguous or unnatural queries, VQA examples ungrounded in their images, and questions that are not visual-critical; approximately 95% survive. The resulting benchmark contains 7,702 QA pairs, including 1,592 VQA pairs. Each query averages 2.15 documents and 8.87 images, while each reference answer averages 4.32 calls. Reference answers undergo a separate three-stage process: write a grounded text answer, insert tool tags at visual gaps, and refine the result through human review. Fifteen annotators check each call's necessity, parameters, and placement, removing redundant or mismatched calls, repairing tool and syntax errors, and moving tags after the relevant paragraphs. Samples with systemic factual or structural errors are discarded. Each reference image may be used only once, and calls cannot interrupt sentences, so annotation constrains both informational value and reading rhythm.

2. Unified Tool Planning: define five capability boundaries and encode placement in the output sequence

A document is represented as \(d_i=(T_i,I_i)\), containing text and an associated image set; the document collection is agnostic to whether its source is retrieval or a user upload. The output interleaves textual content with tool instructions. Equation (1) is damaged in the cached text extraction, so the following notation restates only the task definition supported by the adjacent prose, rather than claiming to recover missing original LaTeX:

\[ R=(s_1,\ldots,s_m),\qquad s_j\in\mathcal S_{\mathrm{text}}\cup\mathcal S_{\mathrm{tool}}. \]

Every invocation follows <tool>{"tool_name": ..., "description": ..., "params": ...}</tool>: the name identifies the capability, the description states the purpose, and the parameters specify the operation. Reference cites an in-context image through img_index. Diffusion takes a detailed semantic prompt to synthesize conceptual or creative content absent from the source material. Search retrieves real-world imagery with a targeted query. Code receives a diagram type and data specification for data-driven charts and mathematical plots. Edit takes an img_index and an editing prompt for annotation, local modification, or style previews. The main text does not supply the complete parameter-key schema for every tool, so this note does not invent executable API examples.

These tools are not interchangeable routes to an image. A real landmark needs factual grounding, an experimental curve needs data or an authentic source figure, and a modified room should remain tied to the user's original image. Tags must follow their relevant paragraphs, making placement part of the plan rather than a separate list of illustrations appended at the end. The common protocol allows different MLLMs to be compared under the same capability definitions while temporarily excluding backend execution quality from the primary metrics.

3. Complementary Call Inspection: penalize both invalid visuals and excessive reluctance to use visuals

The Precision Inspector first checks whether a call adds visual value and respects the selected tool's capability boundary. Calls passing these prerequisites are then evaluated for semantic placement, surrounding structural coherence, parameter accuracy, and format correctness, using scores of 0, 1, and 2 for failure, partial satisfaction, and full satisfaction. A successful call must be necessary, semantically appropriate, correctly placed, and use the right tool, parameters, and format. The main text does not fully specify how partial scores enter SR, so a score of 1 must not be silently treated as half a successful call. Nor does a compliant tag demonstrate that a real backend has executed successfully.

The Recall Inspector examines places with no call and identifies visual gaps: references to an absent image, long descriptions replacing needed procedural illustrations, real entities lacking reference or search grounding, and implicit requests for creation, editing, or plotting without the corresponding tool. This complements precision because assessing only existing calls could reward a cautious answer that invokes nothing. MI counts missed image opportunities, with lower values preferred; it is not conventional recall obtained by matching the number of images in one reference answer.

4. Tier-Anchored Holistic Judgment: constrain overall composition scores with two detailed reports

The Chief Judge integrates both reports to score the response's overall interleaved pacing on a 0–100 FS scale. Five anchors guide judgment: 80–100 means near-perfect calls and no missed visual opportunities; 60–80 means a useful answer with minor precision issues or at most one minor omission; 40–60 covers multiple parameter or structural issues, or 1–2 omissions; 20–40 covers marked redundancy, tool-boundary violations, or 2–3 explicit visual contexts left unsupported; 0–20 covers severe formatting failure or loss of the user's core visual intent. These are semantic scoring criteria, not a fixed weighted formula combining SR and MI.

Tool Adoption Rate separately measures the percentage of queries invoking a particular tool at least once, not that tool's fraction of all calls. The authors also compare selected tool-type sets with reference sets using Precision, Recall, and F1. Set matching ignores order, placement, parameters, and repeated use of the same tool, making it complementary to MAM rather than a replacement. The main text does not completely specify score averaging, so recomputing F1 from the reported average Precision and Recall would not justify replacing the published values. By default, Gemini 2.5 Pro fills all three judge roles; role specialization is not equivalent to a vote among three independent model families.

A Worked Example¶

Consider the paper's example of asking which hair colors suit the user's skin tone. The query photograph provides the person's appearance, while source documents may contain red or dark-brown hairstyle references. The model first explains its recommendations from the visible evidence, then places the appropriate visual plans after the corresponding recommendation paragraphs.

Reference is appropriate when showing an existing red-hair example. Edit is the relevant capability when the request is to preview that color on the user's own photograph. Diffusion can provide a conceptual demonstration when no suitable reference exists, but a newly generated portrait should not be presented as a faithful edit of the user. The Precision Inspector checks capability compliance, image indices, and placement; the Recall Inspector looks for promised previews with no corresponding plan. This illustrates the protocol rather than adding a measured experimental trace, and it assumes no successful backend rendering.

Loss & Training¶

The paper trains no new model. It evaluates ten existing MLLMs in a default zero-shot setting and runs ablations on a balanced subset of 800 queries, with 100 from each category. Three-shot prompting tests whether demonstrations improve planning and format adherence; other studies vary tool capability descriptions, substitute judge models, and obtain human evaluations. The cached main paper does not include the referenced supplementary prompts, preventing exact reconstruction of every prompt, decoding setting, or complete tool schema.

Key Experimental Results¶

Main Results¶

The following table combines the published Avg columns from Tables 3–5 without reweighting categories by sample count. FS is an overall score out of 100, SR is a percentage, and MI is an average omission count. SR remains a judge-based assessment of the plan.

Model	FS ↑	SR (%) ↑	MI ↓
Gemini 3 Pro	79.88	81.77	0.49
Claude Sonnet 4.5	69.34	75.77	1.19
Claude Sonnet 4	69.15	73.86	0.99
Grok-4.1	68.18	71.21	1.06
GPT-5	67.18	69.23	1.11
Qwen3-VL-Plus	62.49	63.93	1.44
GPT-4o	60.67	61.63	1.35
Qwen2.5-VL-72B	53.22	39.13	1.68
InternVL3.5-14B	49.09	10.12	1.62
LLaMA-3.2-11B	28.97	18.34	2.16

Gemini 3 Pro exceeds Claude Sonnet 4.5 by 10.54 FS points and 6.00 SR percentage points while reducing MI by 0.70. GPT-5 reaches 93.59 FS in Academic but falls to 48.63 in Renovation, showing that strong textual explanation does not ensure strong visual-tool coordination. Travel generally exposes more omissions, whereas Renovation often exposes invalid calls; neither should be reduced to a generic claim of poor image quality.

Ablation Study¶

These values come from Table 7 and apply only to the balanced 800-query subset. Its zero-shot baselines differ from the full-benchmark results above, so gains must be calculated within this table.

Configuration	FS ↑	SR (%) ↑	MI ↓
GPT-4o, zero-shot	60.35	63.05	1.39
GPT-4o, 3-shot	73.19	82.01	0.83
Qwen2.5-VL-72B, zero-shot	53.88	39.06	1.64
Qwen2.5-VL-72B, 3-shot	72.86	72.88	0.65
LLaMA-3.2-11B, zero-shot	29.60	25.35	2.10
LLaMA-3.2-11B, 3-shot	30.20	26.62	2.01

SR improves by 33.82 percentage points for Qwen2.5-VL-72B and 18.96 for GPT-4o, but only 1.27 for LLaMA-3.2-11B. Demonstrations help some models follow the tool-use protocol, yet the study does not separately isolate learning the format from improving the plan. Attributing all gains to stronger reasoning would therefore overstate the evidence.

Key Findings¶

Judge reliability has several supporting checks. MAM FS rankings correlate with reference tool-set F1 rankings at Spearman 0.879, \(p<0.001\), supporting broad ranking agreement without making the metrics interchangeable.
Human inspection of reports on 400 sampled queries yields agreement rates of 84.00%, 85.88%, and 88.00% for the Precision Inspector, Recall Inspector, and Chief Judge. This is report agreement, not a 400-example end-to-end execution study.
The end-to-end study executes 100 plans per model with in-context references, nano-banana, Google Image Search through Serp API, GPT-5-generated Python scripts, and Doubao Seedream 4.0. Gemini 3 Pro achieves human FS of 3.89/5 and MI of 0.06. MAM and human rankings correlate at 0.8909 for FS and 0.8303 for MI; the five-point human FS cannot be directly compared with the primary 100-point scale.
Judge replacement still changes absolute scores substantially. On the 800-query subset, InternVL3.5-14B receives SR of 11.46 under Gemini 2.5 Pro but 68.30 under GPT-5. High rank correlation does not establish cross-judge calibration of success thresholds.

Highlights & Insights¶

Treating visual gaps as a separate evaluation target is useful. Invalid calls and absent necessary calls are different failures, and inspecting both SR and MI distinguishes overly aggressive behavior from conservative omission.
Explicit boundaries between reference, search, generation, plotting, and editing make factuality and creativity auditable decisions. The same approach can support offline plan review for multimodal assistants without running expensive backends for every evaluation.
The three agents separate local evidence checks from overall reading rhythm rather than merely adding conversation turns. A final score can be traced to particular invalid calls or missing visual opportunities, providing more diagnostic information than one opaque overall judgment.

Limitations & Future Work¶

The authors acknowledge that the setting includes only text and images and five tools, excluding audio, video, and broader agent capabilities. They also do not compare the alternative of captioning images before using a text-only LLM judge.
Plan compliance does not guarantee execution success. The smaller end-to-end experiment provides support but cannot cover search freshness, generation randomness, edit fidelity, or deployment cost and latency; these require separate execution metrics.
Gemini 2.5 Pro is used both for query generation and as the default judge, with all three roles sharing the model. This creates a potential source of correlated preferences. Judge substitution and human evaluation mitigate concern but do not demonstrate that bias has been eliminated.
Tool choices respond to capability-boundary wording, making scores protocol-dependent; abundant supplied images can also encourage Reference. Future analysis should stratify by available visual evidence, mixed-tool requirements, and source authenticity rather than relying only on aggregate scores.
The cached task equation is damaged, and full prompts and some metric implementation details are absent from the main text. This note separates a recoverable task definition from unverifiable implementation details and does not invent training hyperparameters, ablation values, or exact bar heights from figures.

Compared with OpenLEAF, InterleavedBench, and OpenING: these works primarily examine generated interleaved content and its quality, while ATP-Bench emphasizes visual-source selection and call placement. Planning evaluation does not replace checking actual image quality.
Compared with RAG-IGBench and MRAMG-Bench: these benchmarks emphasize grounding through retrieved images, whereas ATP-Bench adds generation, editing, and plotting. The useful extension is to audit evidence citation and new visual creation within one answer rather than preselecting a single route.
Compared with ViperGPT, MM-ReAct, AssistGPT, and CLOVA: these tool-augmented systems focus on execution and feedback workflows, while this paper supplies a task and diagnostic measures for interleaved plans. MAM could serve as an offline audit layer; using it as a training reward is a future possibility, not a demonstrated result of this paper.

Rating¶

Novelty: 4/5. Unifies mixed visual sources and placement-aware planning with explicit omission checks; the contribution is principally the task and evaluation, not a model architecture.
Experimental Thoroughness: 4/5. Covers ten models, few-shot prompting, tool boundaries, judge substitution, and human execution studies, but end-to-end scale and implementation detail remain limited.
Writing Quality: 4/5. The task boundary and three judge roles are clear; aggregation details, success classification, and supplementary prompts require further verification.
Value: 4/5. Useful for diagnosing excessive calls, unsuitable calls, and missing visuals in multimodal assistants, provided it is paired with real execution testing.