SlidesGen-Bench: Evaluating Slides Generation via Computational and Quantitative Metrics¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://slidesgenbench.yqy314.top
Area: Multimodal VLM
Keywords: Presentation Generation Evaluation, Multimodal Benchmark, Computational Aesthetics, Presentation Editability Intelligence, Vision-Language Models
TL;DR¶
To resolve the incomparability and unreliability in evaluating heterogeneous slide generation systems, SlidesGen-Bench projects rendered outputs into the visual domain while parsing underlying structural files, establishing a multi-branch benchmark across decoupled QuizBank content fidelity, 4D computational aesthetics formulas, and Presentation Editability Intelligence (PEI) that achieves superior human alignment (Spearman \(\rho = 0.71\)).
Background & Motivation¶
The rapid convergence of Large Language Models (LLMs) and code-driven agentic workflows has catalyzed automated presentation generation into three disparate architectural paradigms: template-based systems that populate predefined design schemas (e.g., Gamma, Quark PPT), code-driven markup pipelines that render structured HTML/CSS or scripts (e.g., Zhipu PPT, Skywork), and end-to-end multimodal image synthesis engines that generate slide layouts directly as raster pixels (e.g., NotebookLM, Skywork-Banana). Presentation slides constitute a uniquely demanding multimodal artifact that fuses high-density semantic facts, complex visual layouts, and downstream operational editability. However, benchmarking these fundamentally incompatible systems within an equitable, unified, and practically grounded framework has remained an intractable bottleneck across both academia and the generative software industry.
Prior slide evaluation protocols fall largely into two fragile regimes. Reference-based metrics rely heavily on paired source decks or target canonical images to measure visual reconstruction error, rendering them ineffective in open-ended synthesis where canonical ground truths do not exist. Conversely, emerging LLM-as-a-Judge paradigms feed rendered images or entire decks directly into monolithic black-box prompts. These evaluators suffer from severe verbosity biases, stochastic reasoning failures across multi-page contexts, and an acute vulnerability to hallucinated aesthetic ratings. Crucially, existing automated judges rarely undergo systematic calibration against human perceptual preferences, leaving their reported numbers untethered from real-world design ergonomics and user utility.
The core tension lies in the structural-perceptual dichotomy of slide generation: while cross-system fairness demands a format-agnostic visual common ground, real-world utility requires deep structural editability within enterprise Office Open XML schemas; and while aesthetic and content fidelity are inherently subjective, an evaluation benchmark must yield deterministic, reproducible, and verifiable metrics. To resolve these challenges, this paper proposes projecting high-level perception into visual renderings while probing low-level capabilities via deterministic AST parsing. Core idea: SlidesGen-Bench establishes a standardized benchmark comprising decoupled VLM-based "open-book" QuizBank exams for content fidelity, cognitive and ergonomic mathematical formulas for computational aesthetics, and a hierarchical knockout-based Presentation Editability Intelligence (PEI) taxonomy, comprehensively validated against human judgments on the novel Slides-Align1.5k preference dataset.
Method¶
Overall Architecture¶
SlidesGen-Bench bridges disparate architectures by separating visual perception from structural engineering. Given an input corpus of 189 complex document-level synthesis tasks (spanning 6 functional business scenarios and 94 diverse knowledge domains, averaging 4,525 characters per document), generated presentations from arbitrary pipelines are routed into three parallel, complementary evaluation branches: Branch A quantifies factual content preservation via a two-stage decoupled VLM exam; Branch B computes visual ergonomics and temporal slide pacing via closed-form mathematical equations; and Branch C inspects the underlying XML or DOM hierarchy through a strict, rule-based knockout parser.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Long Document Inputs & Generation Tasks<br/>(189 Topic & Purpose Scenarios)"] --> B["Heterogeneous Slide Generators<br/>(PPTX Template / HTML Code / Pixel Image)"]
B --> C["Rendering & File Dispatching<br/>(Unified High-Res Images + Raw Project Files)"]
C --> D["Two-Stage Open-Book Exam<br/>(VLM Markdown Extraction + Context-Constrained QA)"]
C --> E["4D Computational Aesthetics Suite<br/>(Color Harmony + Engagement + Usability + VisualHRV)"]
C --> F["Presentation Editability Intelligence (PEI)<br/>(Knockout Schema Parsing Across Levels L0-L5)"]
D --> G["Quantitative Benchmark Report & V-E Matrix"]
E --> G
F --> G
Key Designs¶
1. Two-Stage Open-Book Exam: Explicit Decoupling of Extraction and Semantic Reasoning
Directly feeding slide images to a multi-modal judge to grade "information quality" conflates visual perception errors with reasoning failures, introducing severe evaluator hallucinations. To establish an objective, verifiable content retention metric, the framework introduces a multi-agent QuizBank generation pipeline: a Forensic Analyst agent extracts quantitative data anchors and qualitative concepts from raw 100k+ token source PDFs with verbatim citations; a Critic agent filters hallucinated or trivial points; and an Exam Setter agent compiles a standardized battery of 10 Multiple-Choice Questions (5 high-level Concept MCQs and 5 granular Data MCQs) per task, verified to be unanswerable from general world knowledge alone. During evaluation, a two-stage protocol is enforced: first, a VLM (gemini-3-flash-preview) extracts visible content into structured Markdown (capturing verbatim headlines, core bullet statements, tabular numerical metrics, and semantic icon interpretations); second, the VLM examinee answers the QuizBank MCQs constrained strictly to the extracted slide context. Human audit proves that this explicit separation reduces VLM extraction bottlenecks to only 1.2% of total errors, while an oracle-context sanity check on source texts achieves a 100% answer rate, confirming that score variations purely reflect slide-generation omissions.
2. 4D Computational Aesthetics Suite: Mathematical Modeling of Visual Ergonomics and Temporal Pacing
Traditional image quality assessment (IQA) treats slides as static, disconnected images, failing to model the temporal coherence, cognitive load transitions, and typographic legibility required across a multi-slide presentation. Grounded in cognitive load theory and visual ergonomics, the benchmark introduces four closed-form computational aesthetic metrics: - Color Harmony: Maps slide pixels in the HSV color space against 7 canonical harmonic hue templates \(\mathcal{T} = \{i, I, V, L, T, Y, X\}\). Harmonic deviation \(D(H, S)\) is calculated via saturation-weighted shortest angular distance. The deck-level harmony score rewards mean template fit \(\mu_{\text{deck}}\) while penalizing cross-slide stylistic inconsistency \(\sigma_{\text{deck}}\): $\(S_{\text{harmony}} = w_h \cdot (w_m \mu_{\text{deck}} - w_d \sigma_{\text{deck}})\)$ - Engagement & Pacing: Adapts the Hasler-Süsstrunk colorfulness metric on opponent channels \(rg = R - G\) and \(yb = 0.5(R + G) - B\), computing per-slide colorfulness \(M_{\text{slide}} = \sqrt{\sigma_{rg}^2 + \sigma_{yb}^2} + 0.3 \sqrt{\mu_{rg}^2 + \mu_{yb}^2}\). A dual-Gaussian kernel formulation scores both deck-level vibrancy and pacing standard deviation, penalizing erratic jumpscare shifts as well as ghostly flatlines. - Usability (Figure-Ground Contrast): Because global image contrast histograms overlook unreadable text placed over complex illustrations, the framework employs PP-DocLayout_plus-L to detect precise text bounding boxes. Local relative luminance \(L\) is extracted via sRGB linearization to obtain contrast ratio \(c = \frac{L_{\max} + 0.05}{L_{\min} + 0.05}\), logarithmically mapped to \(S_{\text{contrast}} = \frac{\ln(c)}{\ln(21)}\) (saturating at the WCAG 21:1 ideal standard). - Visual Rhythm (VisualHRV): Inspired by physiological heart rate variability, this metric captures presentation cognitive pacing. Steerable pyramid decomposition across CIE-LAB channels computes per-slide Subband Entropy (\(E_{SE}\)) to measure clutter. Pacing across the temporal sequence is measured via the Root Mean Square of Successive Differences (RMSSD): $\(\text{RMSSD} = \sqrt{\frac{1}{N-1} \sum_{i=1}^{N-1} |S_{i+1} - S_i|^2}\)$ Combining a complexity Gaussian with a centered RMSSD pacing band (\(\tau = 0.03\)) penalizes both monotonous, cognitive-fatigue-inducing flatlines and jarring, erratic slide transitions.
3. Presentation Editability Intelligence (PEI): Knockout Structural Parsing of File Schemas
To expose systems that generate visually compelling but functionally uneditable artifacts, the benchmark establishes the 6-level PEI hierarchy. Rather than inspecting visual approximations, PEI executes automated structural parsing on raw project files (Office Open XML schemas via python-pptx and DOM trees for HTML), governed by a strict dependency-based "Knockout Rule" where failing a lower gate immediately terminates evaluation:
- Level 0 (Static): Flattened raster image or unselectable PDF where text exists purely as bitmap pixels;
- Level 1 (Patchwork): OCR-style fragmented single-line text boxes relying on absolute canvas positioning, failing natural text flow and reflow;
- Level 2 (Vector): Scalable SVG primitives and vector paths that stay crisp at 400% zoom, but lack semantic grouping and master layout logic;
- Level 3 (Structural): The boundary between "toys and enterprise tools", requiring native shape grouping (<p:grpSp>) and global master inheritance (<p:sldMaster>) where layout updates propagate deck-wide;
- Level 4 (Parametric): Visual charts must be instantiated as native <c:chart> XML objects bound to live embedded Excel spreadsheets (.xlsx), enabling users to edit raw data tables directly;
- Level 5 (Cinematic): Enterprise presentation files incorporating timed slide transitions, sequence-aligned animation tags (<p:timing>, <p:bldLst>), and native media playback controls.
Benchmark & Evaluation Protocol¶
The benchmark addresses complexity through a dual-lens methodology: - Instruction Diversity: Derived from an initial corpus of 30k human-authored slides, filtering out decks under 5 or over 40 pages. An LLM annotated topic and purpose schemas, yielding 94 Topic Introduction tasks compiled from Wikipedia with hierarchical outlines and figure descriptions. - Functional Scenarios: 95 purpose-driven tasks were collected across 6 commercial domains: Brand Promotion (15), Business Plans (8), Personal Statements (20), Product Launches (24), Course Preparation (15), and Work Reports (13), creating a final corpus of \(N=189\) complex long-context synthesis tasks (mean: 4,525 characters, 13.3 pages). Tasks are stratified by a composite Content Richness index \(S = 0.7 T_{\text{norm}} + 0.3 I_{\text{norm}}\) into balanced Low, Medium, and High complexity tiers. - Human Preference Testbed (Slides-Align1.5k): A gold-standard dataset of 1,500 slide comparisons sampled across 9 mainstream systems and 7 domains, annotated by a diverse pool of CS Ph.D. students, Art school undergraduates, and professional corporate clerks, attaining an inter-annotator Spearman agreement of \(\rho = 0.85 \pm 0.12\).
Key Experimental Results¶
Main Results¶
The benchmark evaluates 9 leading commercial platforms (Gamma, Kimi-PPT Standard/Smart/Banana, Quark PPT, Skywork, Skywork-Banana, Zhipu PPT, NotebookLM) and 2 adapted academic frameworks (AutoPresent* and PPTAgent*).
Table 1: QuizBank Accuracy (%) across Task Purposes, Topics, and Complexity Tiers
| System & Model | Brand Promote | Business Plan | Personal Statement | Product Launch | Course Prep | Work Report | Topic Intro | High Diff | Low Diff | Med Diff | Overall Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Zhipu PPT | 80.00 | 68.75 | 96.00 | 86.08 | 90.00 | 84.17 | 90.44 | 84.07 | 92.00 | 88.97 | 88.29 |
| Skywork-Banana | 92.14 | 67.50 | 91.50 | 89.57 | 87.69 | 90.00 | 79.33 | 79.67 | 88.00 | 84.31 | 84.77 |
| Kimi-Banana | 84.29 | 77.50 | 96.00 | 90.43 | 84.00 | 85.38 | 86.22 | 83.33 | 91.58 | 86.67 | 86.23 |
| NotebookLM | 83.00 | 45.00 | N/A | 84.21 | 93.00 | 86.15 | 68.33 | 69.81 | 86.00 | 71.16 | 77.28 |
| Skywork | 80.00 | 68.75 | 78.50 | 76.52 | 84.67 | 86.15 | 83.12 | 80.00 | 82.58 | 81.31 | 80.89 |
| Quark PPT | 84.29 | 51.25 | 94.00 | 79.05 | N/A | 68.00 | 82.13 | 69.78 | 90.52 | 82.12 | 77.34 |
| Kimi-Standard | N/A | N/A | 96.47 | N/A | N/A | 80.00 | 76.88 | 75.71 | 88.05 | 75.43 | 80.70 |
| Kimi-Smart | N/A | N/A | 95.00 | N/A | N/A | 84.29 | 79.68 | 78.78 | 88.50 | 78.57 | 81.65 |
| Gamma | 63.57 | 52.50 | 74.00 | 54.44 | 85.00 | 48.46 | 76.34 | 67.26 | 75.56 | 68.45 | 67.92 |
| PPTAgent* | 62.25 | 47.37 | 69.15 | 61.80 | 66.86 | 61.27 | 61.94 | 58.94 | 66.76 | 61.68 | 62.43 |
| AutoPresent* | 56.13 | 42.72 | 62.35 | 55.73 | 60.29 | 55.25 | 55.85 | 53.15 | 60.20 | 55.62 | 55.47 |
Table 2: Human Alignment Performance Comparison on Slides-Align1.5k
| Evaluation Method | Spearman Avg \(\rho\) (↑) | Spearman Std (↓) | Identical Ratio Avg (%) (↑) |
|---|---|---|---|
| SlidesGen-Bench (Ours) | 0.71 | 0.16 | 32.6 |
| LLM-as-Judge Rating (gemini-3-flash) | 0.57 | 0.23 | 20.7 |
| PPTAgent Evaluator (Zheng et al., 2025) | 0.53 | 0.26 | 17.8 |
| LLM-as-Judge Arena (Pairwise Prompting) | 0.52 | 0.27 | 17.3 |
| Human Inter-Annotator Agreement (Upper Bound) | 0.85 | 0.12 | 45.3 |
Table 3: Computational Aesthetics Scores and Structural PEI Level Results
| System | Usability (↑) | Engagement (↑) | Harmony (↑) | Rhythm (↑) | Aesthetics Total (↑) | PEI Level |
|---|---|---|---|---|---|---|
| Skywork-Banana | 5.62 | 8.30 | -0.47 | 13.84 | 27.28 | L1 |
| Kimi-Banana | 5.72 | 6.41 | -0.55 | 15.01 | 26.58 | L1 |
| NotebookLM | 4.13 | 7.32 | -0.35 | 11.72 | 22.82 | L0 |
| Zhipu PPT | 4.87 | 7.52 | -1.60 | 11.27 | 22.06 | L2 |
| Skywork | 4.83 | 7.60 | -1.18 | 9.44 | 20.69 | L2 |
| Kimi-Standard | 4.61 | 6.28 | -1.75 | 10.12 | 19.25 | L2 |
| Kimi-Smart | 4.13 | 7.99 | -1.88 | 8.06 | 18.30 | L2 |
| Gamma | 5.31 | 6.31 | -1.51 | 6.99 | 17.09 | L2 |
| Quark PPT | 5.03 | 7.41 | -1.91 | 6.33 | 16.86 | L3 |
| PPTAgent* | 5.01 | 6.12 | -1.43 | 5.62 | 15.32 | L5 |
| AutoPresent* | 4.28 | 5.93 | -2.15 | 4.57 | 12.63 | L4 |
Ablation Study¶
Table 4: Ablation Study of Aesthetic Metric Combinations against Human Ground Truth
| Metric Configuration | Spearman Avg \(\rho\) (↑) | Spearman Std (↓) | Identical Ratio Avg (%) (↑) |
|---|---|---|---|
| Full Method (All 4 Dimensions) | 0.710 | 0.160 | 32.6 |
| Usability + Harmony + Engagement | 0.667 | 0.206 | 24.4 |
| Usability + Engagement | 0.612 | 0.212 | 23.7 |
| Only Visual HRV (Visual Rhythm) | 0.618 | 0.198 | 24.4 |
| Only Usability (Figure-Ground Contrast) | 0.574 | 0.207 | 21.5 |
| Harmony + Engagement | 0.371 | 0.440 | 20.7 |
| Only Harmony (Color Harmony Templates) | 0.312 | 0.414 | 15.6 |
| Only Engagement (Colorfulness & Pacing) | 0.224 | 0.349 | 14.8 |
Key Findings¶
- The Empty "North Star" Quadrant in the Visual-Editability (V-E) Matrix: As mapped in Figure 4, current models suffer from an acute aesthetic-structural dichotomy. Image-centric generators (Skywork-Banana, Kimi-Banana) occupy the "Gallery" (Q2, visual scores 26.58–27.28) but remain uneditable (L1); standard commercial tools populate the "Depths" (Q3, visual scores 17–20, L2 Vector) with severe structural amnesia; and academic frameworks enter the "Skeleton" (Q4, L4/L5) via rigid structural priors but suffer from poor aesthetics (12.63–15.32). No existing model occupies Q1 (The North Star), proving that unifying studio-grade graphic design with deep native data binding remains unsolved.
- Error Evolution from Content Omission to Numerical Hallucination: Analysis of 2,499 incorrect quiz answers across 13,023 samples reveals that Missing Content accounts for 66.3% of failures, while Value Mismatch accounts for 31.7%. However, top systems (Zhipu, Kimi-Banana) flip this distribution: their omission rate drops below 49%, while value mismatches surge to 49%–52%, demonstrating that as generative models master semantic recall, factual fidelity on detailed figures becomes the primary failure mode.
- Critical Synergy of Visual Rhythm in Human Alignment: Usability serves as the strongest individual perceptual anchor (\(\rho = 0.574\)). Adding VisualHRV boosts the alignment metric to 0.710 and drops variance to 0.160, demonstrating that temporal cognitive pacing across slides is indispensable for human aesthetic satisfaction.
Highlights & Insights¶
- Grounding Slide Aesthetics in Physiological and Ergonomic Theory: By formalizing Visual Heart Rate Variability (VisualHRV) via Subband Entropy and RMSSD, this work establishes a principled, differentiable mathematical proxy for cognitive pacing, superseding ad-hoc prompt-based grading.
- VLM Decoupling as a Robust Benchmark Architecture: By isolating visual extraction from constrained question answering, the protocol suppresses evaluator hallucination and extraction errors to a negligible 1.2%, yielding an objective content-retention metric that withstands backend model swapping (DeepSeek vs MiniMax yielding 60.8% vs 61.1%).
- Formalizing the Structural Knockout Standard for Enterprise Slides: The PEI hierarchy pierces the illusion of superficial vector graphics, defining concrete XML-level criteria (
<p:sldMaster>,<c:chart>,<p:timing>) that establish an actionable engineering roadmap from visual toys to enterprise tools.
Limitations & Future Work¶
- Lack of Temporal Dynamics Evaluation in Rendered Viewers: The benchmark operates exclusively on static rendered slides, currently omitting perceptual evaluation of interactive slide transitions, build-in object animations, and embedded dynamic media in real presentation modes.
- Dependency on VLM Extraction Frontends: Despite rigorous decoupling and low failure rates (1.2%), content grading remains bounded by the OCR and visual layout comprehension limits of the underlying vision model on extremely crowded, artistic infographics.
- Cultural and Domain Aesthetics Calibration: The computational aesthetic weights are optimized around international corporate design standards (Duarte principles), requiring adaptive recalibration for dense scientific posters, complex legal briefs, or culture-specific design palettes.
Related Work & Insights¶
- vs PPTAgent (Zheng et al., 2025): PPTAgent introduced an LLM-based evaluation pipeline (PPT-Eval) that suffers from stochastic scoring drift and lacks calibration; SlidesGen-Bench replaces black-box judges with closed-form ergonomic equations and decoupled QuizBank testing, lifting human preference alignment from 0.53 to 0.71.
- vs AutoPresent (Ge et al., 2025): AutoPresent relies on reconstruction metrics against paired ground-truth presentations; SlidesGen-Bench introduces reference-free evaluation across open-ended long documents and formulates the PEI hierarchy to uncover AutoPresent's lack of dynamic master layout control.
- vs Traditional IQA (NIMA / AIM): Generic aesthetic algorithms developed for photography or web interfaces ignore cross-slide narrative pacing and local text-background contrast; SlidesGen-Bench provides presentation-specific metrics by integrating local layout-aware contrast and temporal VisualHRV.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering unified visual-structural benchmark decoupling content, ergonomic aesthetics, and XML-level editability across heterogeneous architectures.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 11 systems, 189 multi-domain long tasks, 2,499 human-audited failure cases, and a 1.5k preference alignment study.
- Writing Quality: ⭐⭐⭐⭐⭐ Exemplary mathematical rigor, precise cognitive psychology formulations, and thorough architectural diagnostics across text and appendices.
- Value: ⭐⭐⭐⭐⭐ Establishes a reproducible, open-source evaluation standard and diagnostic compass for next-generation agentic presentation systems.