PPTArena: A Benchmark for PowerPoint Editing¶
Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/michaelofengend/PPTArena
Area: Multimodal VLM / LLM Agent
Keywords: PowerPoint editing, structure awareness, tool routing, dual-judge evaluation, cross-slide consistency
TL;DR¶
PPTArena turns in-place editing of real presentations into a benchmark that checks both structural correctness and visual quality, while its code/OOXML hybrid agent PPTPilot achieves full-benchmark IF/VQ of 2.57/2.69 with reflection and verification, leaving a substantial gap to reliable complex editing.
Background & Motivation¶
Presentations in everyday work are often revised from previous reports, existing templates, and accumulated materials rather than generated on a blank canvas. When a user asks to standardize subtitle sizes and align two logos, the expected behavior is to modify the correct objects while preserving unrelated content, layout, and theme relationships. Treating a slide only as an image discards editable semantics such as paragraphs, object hierarchies, placeholders, stacking order, and master inheritance. Treating the task as text-to-slide generation can instead permit the model to redesign the entire page, concealing whether it understood the original file. Producing an attractive image and reliably editing a presentation that remains usable are therefore different capability requirements.
Earlier work such as PPTC-R and T2US provides presentation tasks, but this paper argues that short operation sequences or template-based scenarios do not sufficiently expose document-level dependencies. For example, distributing narration stored on one slide to the appropriate speaker notes before deleting the staging slide requires content correspondence, not merely checking whether the target text appears. Translation can also change the space occupied by text boxes, producing occlusion or overflow despite accurate meaning. Conversely, a visually tidy deck may put the right content on the wrong slide or omit requested modifications. Evaluation consequently needs both native structural evidence and rendered visual evidence rather than reducing them to a single similarity measure.
PPTArena links original files, human-created target files, editing instructions, and sample-specific scoring constraints for real presentations. With these checkable targets, a model must go beyond explaining a page to changing its underlying structure through tools, while accounting for collateral modifications. The accompanying PPTPilot tests a more specific hypothesis: interface selection and post-execution verification matter more than always using one editing modality. Core Idea: evaluate PowerPoint editing against both native document structure and visual output, then route between complementary editing interfaces to reduce the conflict between fine-grained control and cross-slide batch processing.
Method¶
Overall Architecture¶
The paper has two layers of contribution: PPTArena defines editing tasks and evaluation, while PPTPilot supplies a structure-aware agent that executes those tasks. On the benchmark side, inputs include an original PPT, natural-language changes, and a human target PPT plus a style target for evaluation. On the agent side, the instruction, original structure, content, and screenshots are inspected before a router selects code-based or direct OOXML editing to produce a modified PPT. A verifier can then inspect the rendered result against the original instruction and report errors for further revision. Final evaluation compares the prediction against original/target decks structurally and visually, producing separate instruction-following and visual-quality scores.
Execution state must be distinguished from evaluation references: the human target deck and rubric are important inputs to benchmark construction and judging. The router described in Section 4 instead receives the instruction, screenshots, and deck contents; this does not establish that the editor receives the target deck directly. Intermediate representations include JSON summaries preserving layout, style, and content information, as well as OOXML for precise document-property edits. Screenshots complement these representations with visual relationships that are difficult to infer directly, including occlusion, alignment, and overall layout balance. This is a benchmark and tool-execution system, not a new visual encoder or a trained end-to-end slide-generation network.
Key Designs¶
1. Native editing tasks: define correctness through document-state changes
The authors scrape more than 18000 online presentations, convert them into JSON traces containing layout, style, and content metadata, and apply automated filtering. From more than 500 manually reviewed candidates, they combine these sources with internal materials contributed by researchers and art/design students to select 100 decks. The candidate pool also contains 25 decks created from scratch by the authors, so the final collection should not be described as exclusively naturally occurring online decks. The selected decks contain 2125 slides, and Table 5 reports 1340 distinct edits. Tasks provide the initial state, a human target state, structured textual instructions, and scoring requirements covering layout, typography, color, and content. This design distinguishes moving text to the wrong slide from making the same textual change correctly, rather than comparing only sets of strings. References constrain the purpose of an edit and help identify structures in the original deck that should remain unchanged.
The taxonomy has 5 parent categories, Content, Layout, Styling, Interactivity, and Structure, encompassing 16 concrete operations. It covers text, shapes, images, tables, charts, SmartArt, audio/video, masters, animations, hyperlinks, notes, and accessibility properties. A case can carry multiple category tags, so the 144 in Table 2 counts tags, not 144 independent decks. Table 3 reports an average of 13.4 operations and 8.3 slides per edit, emphasizing longer modification programs rather than isolated actions. Cross-slide cases require coordinated modifications across at least two slides; high-difficulty cases involve cross-slide dependencies or strong visual-textual reasoning such as chart remapping and translation. The speaker-notes task in Figure 2 first requires placing text from a staging slide into the correct slides' notes and then deleting the staging slide, making order part of correctness. The difficulty is not just operation count: object identification, cross-slide correspondence, and execution order must all be correct.
2. Structural/visual dual judges: check content correctness separately from presentation quality
Instruction Following, or IF, measures whether the request is fulfilled semantically and logically, including object selection, formatting, and subtasks within a complex command. Visual Quality, or VQ, measures layout, alignment, typography, color, and professional polish without equating an attractive result with completion of the instruction. Each judge assigns an integer from 0 to 5, where 0 denotes failure and 5 denotes perfection; result tables report mean scores, not success rates. The scoring basis is not a generic prompt suitable for every page but a per-sample style target specifying the content and style constraints for the edit. The authors provide GPT-5 with JSON summaries and screenshots of both original and human target decks to generate these constraints, then manually verify each for correctness and faithfulness. Providing both states helps emphasize the requested change rather than merely restating every element already present in the target slide. Automatically generating judge prompts therefore does not mean trustworthy annotations are produced fully automatically; human verification remains a significant cost.
The IF judge receives structural differences among original, predicted, and human target decks, such as JSON/XML summaries, to focus on content changes and logical completion. The VQ judge receives rendered screenshots of predicted and human target slides to focus on visual execution; both use the task's style target. These are separate judge calls or roles and may use the same base model, rather than necessarily belonging to two different model families. To avoid excessively long visual contexts in cross-slide tasks, Structural Similarity Index Measure, or SSIM, selects slides with salient changes for the visual judge. SSIM is a page-screening mechanism here, not the final quality score, and cannot replace IF's assessment of semantic correctness. The main experiments use GPT-5.2 judges and check stability through independent Gemini 3.1 Pro re-scoring and expert ratings. Structural differences constrain what changed, while rendered images constrain how the result looks; this evidence separation is more targeted than merely lengthening a judge prompt.
3. Hybrid editing routing: match the interface to modification granularity
PPTPilot first interprets the instruction and deck structure, including masters, placeholders, object trees, text, and visual content, before planning semantic editing operations.
A lightweight routing model combines the instruction, screenshots, and content to select programmatic tools or direct Office Open XML, abbreviated OOXML.
Figure 4 gives GPT-5 nano or Gemini-3.0 flash as router examples and GPT-5.2 for editing calls.
The programmatic branch generates and executes code using python-pptx, making it suitable for repetitive operations such as cross-slide find-and-replace, translation, and bulk normalization.
Its advantage is delegating document-wide repetition to program loops rather than asking the model to generate lengthy XML for many slides individually.
However, high-level APIs may not expose the required master, theme, or fine-grained structural controls, so they cannot cover every modification.
The direct XML branch reads, parses, and rewrites underlying OOXML, including slide and theme XML parts. This lets the model manipulate object positions, theme color slots, font properties, and master-level changes without being fully constrained by a high-level API's expressiveness. The cost is long context and strict formatting: an invalid rewrite can damage the deck, and changes across many slides are particularly susceptible to accumulated errors. The introduction describes strict schemas, XML validation, and deterministic patches to constrain execution formats; these do not guarantee correct object selection or edit semantics. Routing assigns bulk content modifications and fine structural operations to more suitable interfaces rather than claiming that XML is always better than code. The main text reports that 66% of edits use the programmatic branch and 34% use XML, with finer statistics delegated to supplementary material absent from this cache. These proportions describe usage under the current task distribution and should not become fixed routing quotas for future systems.
4. Reflection and verification: turn an edit into a correctable state update
After proposing a change, the system temporarily generates and renders a PPT so that a verifier can inspect it against the original instruction and identify failures. The editor uses this feedback for its next update, repairing the current document state rather than generating an unrelated replacement deck. This addresses a limitation of static structural checks: valid XML can still produce misalignment, occlusion, omissions, or incorrect cross-slide correspondences. Reflection differs from the final dual judges: the former is an execution-time correction tool, while the latter is a reference-grounded evaluation protocol. The reference inputs available to final judges must not automatically be assumed available to the reflection verifier. Table 7 compares hybrid execution without refinement against a version with 3 loops, showing that additional checking and revision improve mean results.
The authors observe that most corrections occur by the second pass, but the main text lacks a complete per-pass curve, so this remains a qualitative observation here. Table 6 reports 1.5 minutes for single-pass PPTPilot and 3 minutes for 3 loops, illustrating the exchange between inference-time cost and reliability. These are end-to-end latencies in each system's original environment, not pure algorithmic speed comparisons under matched hardware, models, and tool budgets. A feedback loop can also keep repairing around an incorrect object judgment, so repeated verification is not a proof of document-level correctness. A concrete failure reported in the paper is that PPTPilot sometimes over-edits master XML when the requested change should apply only to a local slide. In that setting, ChatGPT Agent's GUI interaction can leverage PowerPoint's native handling of masters, footers, and sections and need not be inferior.
Loss & Training¶
The paper introduces no new training loss, and the available main text does not report dedicated fine-tuning of the base models. The principal changes occur at inference time through context organization, tool routing, format constraints, and verification feedback rather than parameter optimization. Human target decks support benchmark construction and evaluation and should not be described as supervision used to train this agent. Model names follow the main text and Figure 4; they record the authors' configurations, and this note does not independently verify the corresponding product versions. Detailed prompts, budgets, routing audits, and full ablations are repeatedly deferred to supplementary material, while the current full-text cache contains only the main paper and references.
Key Experimental Results¶
Main Results¶
The following values are from Table 5, page 13: the full benchmark contains 100 decks and 1340 edits and uses GPT-5.2 dual judges; IF/VQ range from 0 to 5, with higher values better. Category tags overlap, so adding category-specific edit counts does not recover the number of distinct tasks.
| System | IF | VQ | Comparison Scope |
|---|---|---|---|
| PPTPilot | 2.57 | 2.69 | Full benchmark |
| Gemini CLI | 1.21 | 1.98 | Full benchmark |
| ChatGPT | 2.07 | 2.22 | Full benchmark |
PPTPilot exceeds ChatGPT by 0.50 IF points and 0.47 VQ points; these are score differences, not corresponding percentage-point gains in task success. Because of proprietary product limits and costs, additional systems are compared only on a matched subset of 25 decks and 206 edits, comprising the 20 hardest cases plus 5 cases for breadth. In Table 4 on page 12, PPTPilot scores 1.87/1.91, ChatGPT Agent 1.68/1.60, and MiniMax Agent 1.04/0.84, always in IF/VQ order. This deliberately difficult subset must not be combined with Table 5's full-benchmark scores into a single undifferentiated leaderboard.
Ablation Study¶
The following values are from Table 7, page 14: the upper block changes judge configurations, while the lower block changes PPTPilot executors; these are not a single component-removal experiment. The Table 7 caption and main text do not explicitly specify its sample count in the available cache, and complete settings are deferred to the supplement, so this is not presented as a matched full-benchmark reproduction of Table 5.
| Experiment Block | Config | IF | VQ |
|---|---|---|---|
| Judge | Single VLM judge, all signals | 2.31 | 4.26 |
| Judge | Dual judge, no structural diffs | 3.76 | 4.54 |
| Judge | Dual judge with structural diffs | 2.36 | 2.40 |
| Executor | XML-only | 0.95 | 2.85 |
| Executor | python-pptx-only | 2.06 | 2.73 |
| Executor | Hybrid, no refinement | 2.36 | 2.69 |
| Executor | Hybrid with 3 loops | 2.84 | 3.21 |
Within the executor block, adding the loop increases IF from 2.36 to 2.84 and VQ from 2.69 to 3.21, gains of 0.48 and 0.52 points respectively. XML-only has the lowest IF but not the lowest VQ, showing that attractive or orderly changes can still miss the instruction and that visual scores alone do not establish editing reliability. Removing structural diffs raises judge scores substantially, which the authors interpret as more lenient evaluation; this increase is not an improvement in agent capability. The source table reports VQ 2.40 for dual judges with diffs but 2.69 for hybrid execution without refinement; both values are retained rather than silently reconciled.
Key Findings¶
- Full-benchmark scores remain only 2.57/2.69, substantially below the maximum of 5; overall leadership neither solves editing nor implies leadership in every category.
- Table 5 gives Styling IF of 2.32 for PPTPilot versus 2.41 for ChatGPT, and Interactivity IF of 3.00 versus 3.25, placing useful limits on the overall claim.
- Table 8 on page 14 uses a stratified 10% subset rated by 25 experts; human-judge Pearson correlations are 0.72 for IF and 0.81 for VQ.
- Across 5 repeated judge runs on identical inputs, majority agreement is 78% for IF and 95% for VQ; this supports stability but does not prove every judgment correct.
- Independent Gemini 3.1 Pro re-scoring gives PPTPilot full-benchmark IF/VQ of 2.45/2.74, retaining its lead while demonstrating that absolute scores depend on the judge.
Highlights & Insights¶
The most transferable idea is to evaluate document-state changes rather than merely the attractiveness of the final image. This separates wrong-object edits, damaged master inheritance, and visual occlusion instead of asking one vague aggregate score to explain every failure.
Hybrid interfaces are valuable because their expressiveness and execution reliability complement each other, not simply because there are more interfaces. The ablations suggest that additional low-level control improves instruction following only when it is routed to suitable tasks.
The central feature of dual judging is evidence separation and sample-specific constraints, not just calling a large model twice. This offers a reusable evaluation pattern for spreadsheets, web pages, and other structured visual documents: structural evidence checks the purpose of an operation, while rendered evidence checks its presentation.
Limitations & Future Work¶
The authors explicitly identify brittleness in long-horizon edits coupling visual, spatial, and cross-slide reasoning, with conversational refinement for ambiguous intent and cross-application workflows as future directions. Live charts, document-to-deck synthesis, and highly specialized domains also extend beyond what the current benchmark fully covers.
From an evaluation perspective, 100 source decks constrain layout and workflow diversity, and 1340 edits should not be treated as samples from an equal number of independent decks. Interactivity covers only 4 decks and 30 edits in Table 5, so its category rankings particularly warrant caution. SSIM screening may miss visually small but semantically important changes; this is a reader-identified risk of the mechanism, not a failure rate measured by the paper.
The available cache excludes supplementary material, preventing further verification of Table 7's sample scope, full prompts, and budget controls. Some dataset tables have interleaved text extraction; this note uses only values supported by readable rows or corroborating prose and does not reconstruct missing fields. The code URL comes from the paper and was not checked online during this reading; no confirmed arXiv ID for this paper was found, so none is supplied.
Related Work & Insights¶
- PPTC-R and T2US: the former evaluates robustness of task completion, while the latter uses structured data for language-driven editing; PPTArena emphasizes native structure, multiple operations, and cross-slide dependencies while separating content and visual evaluation.
- PPTAgent and Paper2Poster: these focus more on generative presentation or poster workflows; their reported failures under this editing protocol expose preservation of an existing deck as an additional constraint, not a lack of value on their original tasks.
- The other PPTArena work: reference 13 is the GUI PowerPoint benchmark by Gandhi and colleagues; the paper explicitly describes it as concurrent independent work, so it must not be conflated with this paper by Ofengenden and colleagues or its repository.
- Research direction: executors could explicitly restrict edit scope and regression-check objects that were not requested to change; this is a reader suggestion motivated by master over-editing, not a new component already validated in the paper.
Rating¶
- Novelty: 4/5. Native editing tasks, sample-specific targets, and dual-evidence evaluation form a clear combination, while the agent mainly organizes existing mechanisms for this domain.
- Experimental Thoroughness: 4/5. Full-benchmark and matched difficult-subset comparisons, executor/judge ablations, and human alignment are included, but some ablation settings rely on unavailable supplementary material.
- Writing Quality: 4/5. Task challenges and interface trade-offs are concrete, though the configuration correspondence between main results and ablations could be clearer.
- Value: 4/5. Useful for studying real office-document agents while preserving the crucial conclusion that reliable usability remains out of reach.