Skip to content

From Content Generation to Learning Support: Pedagogy-Guided Generative Video Tutors for STEM Learning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: educational video generation, multimodal LLMs, pedagogy-guided systems, intelligent tutoring systems, misconception remediation

TL;DR

East China Normal University and the National University of Singapore present PIVOT, an end-to-end framework transforming STEM educational video synthesis from passive visual generation into an interactive, learner-centered tutoring workflow through pedagogy-guided storyboarding, multimodal execution verification, and contrastive misconception-aware remediation.

Background & Motivation

Designing engaging instructional media and recording polished tutorial videos imposes a severe workload on STEM educators, while adapting instruction and remediation to diverse student misunderstandings is constrained by human tutoring scalability. Recent advances in generative AI and code-centric animation synthesis—such as driving the Manim programmatic engine via Large Language Models—have demonstrated remarkable potential in compiling complex textual curricula into dynamic mathematical and scientific explanations. Nevertheless, existing systems such as Code2Video and TheoremExplainAgent remain firmly entrenched in content-generation paradigms rather than holistic learning support, churning out audiovisual assets that lack pedagogical scaffolding, robust fidelity verification, and interactive feedback mechanisms.

This status quo reveals three profound limitations in current educational multimedia synthesis. First, generated video scripts typically lack explicit pedagogical structure: they neglect prerequisite activation, granular objective scoping, and step-by-step cognitive scaffolding, leaving learners struggling to organize new abstractions into working conceptual schemata. Second, unconstrained programmatic rendering frequently triggers visual layout chaos—including overlapping text labels, cluttered coordinate grids, and erratic transitions—alongside subtle yet catastrophic factual misstatements, such as conflating the conservation of mass with atom count conservation. Third, conventional pipelines operate as unidirectional broadcasts without formative assessment, leaving educators and learners blind to comprehension gaps and incapable of delivering targeted corrective feedback when misconceptions arise.

To dismantle these shortcomings, the authors reconstruct educational video generation around the classical teaching lifecycle of preparation, instruction, practice, and adaptive intervention. The core idea is to reframe STEM video generation as a three-stage pedagogy-guided tutoring framework (PIVOT) that translates instructional principles into structured storyboards, enforces multimodal verification across layout, factual, and cognitive dimensions, and couples video delivery with parameter-decoupled formative quizzes that trigger targeted contrastive remediation videos upon detecting learner errors.

Method

Overall Architecture

PIVOT directly mirrors the human pedagogical lifecycle—lesson preparation, classroom instruction, and targeted post-class intervention—through three interconnected functional stages. Given a user-specified STEM topic or curriculum goal, the system executes: (1) Pedagogy-Guided Instructional Planning, constructing an explicit JSON storyboard encompassing Bloom's taxonomy objectives, cognitive scaffolding, and visual cue blueprints; (2) Video Generation with a Pedagogical Verification Harness, iteratively compiling shot-level Manim scripts with layout, factual, and cognitive verifiers alongside aligned TTS narration; and (3) Adaptive Assessment and Remediation, evaluating understanding via parameter-decoupled diagnostic quizzes and synthesizing misconception-targeted explanatory animations upon student failure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Learner Input Topic & STEM Objective"] --> B["Pedagogy-Guided Instructional Planning<br/>Objectives / Prior Activation / Scaffolding / Visual Blueprint"]
    B --> C["Shot-Level Executable Code Synthesis<br/>Translate storyboard into granular Manim scenes"]
    C --> D["Pedagogical Verification Harness & Auto-Repair<br/>Audit layout consistency, factual validity, and cognitive flow"]
    D --> E["Primary Video Rendering & TTS Narration<br/>Verified multimodal STEM instructional video"]
    B --> F["Pedagogically Grounded Quiz Generation<br/>Transfer conceptual probes without raw example reuse"]
    E & F --> G["Learner Assessment & Response Evaluation<br/>Identify specific error choices and reasoning patterns"]
    G -->|Incorrect Answer Triggered| H["Misconception-Aware Remediation Generation<br/>Contrastive video juxtaposing flawed and correct reasoning"]
    H --> I["Learner Knowledge State Reinforcement"]

Key Designs

1. Pedagogy-Guided Instructional Planning: Bridging Cognitive Theory and Generative Blueprints Directly prompting language models to write standalone animation programs produces fragmented visual demonstrations devoid of pedagogical intentionality. PIVOT introduces a formal JSON storyboard as an intermediate semantic representation prior to code synthesis. Grounded in Bloom's taxonomy and Mayer's cognitive theory of multimedia learning, the planning module structures content across six dedicated dimensions: ① Explicit Learning Objectives delineating target conceptual acquisitions; ② Prerequisite Knowledge Activation to reduce extraneous cognitive load; ③ Step-by-Step Concept Explanations employing incremental scaffolding strategies; ④ Visual Representation Plans prescribing spatial coordinates, color-coded focus highlights, and dynamic trajectory markers; ⑤ Worked Examples illustrating intermediate algorithmic or mathematical transformations; and ⑥ Diagnostic Assessment Probes targeting common cognitive traps. This plan is segmented into discrete, shot-level execution blocks, constraining the visual elements and duration budgets of each scene.

2. Pedagogical Verification Harness and Iterative Auto-Repair: Multimodal Quality Control Translating abstract storyboards into executable Manim code inherently risks rendering exceptions and visual degradation. To ensure absolute instructional fidelity and aesthetic cleanliness, PIVOT establishes a multimodal verification harness auditing three distinct layers: - Layout Consistency Verification: Evaluates screen boundaries, element whitespace spacing (\(\text{buff} \ge 0.5\)), font hierarchy constraints, and occlusion across visual elements, ensuring clean visual composition and enforcing structured transitions (e.g., using Transform rather than indiscriminate scene-clearing FadeOut); - Factual Consistency Verification: Systematically inspects symbolic transformations, algorithmic pointer states, and chemical or physical laws to preempt factual drift; - Cognitive Clarity Verification: Verifies instructional continuity between adjacent shots, preventing abrupt narrative jumps.

When verification fails, PIVOT deploys an automated self-healing loop: execution traces and visual layout diagnostics condition low-temperature code patches via GPT-5.2, with persistent execution failures escalating to Claude Sonnet 4.6 for specialized debugging before speech synthesis produces synchronized voiceover.

3. Concept-Transfer Quiz Generation and Misconception-Aware Remediation: Closing the Active Tutoring Loop To evaluate conceptual mastery rather than rote visual memorization, PIVOT's assessment module derives multiple-choice diagnostic quizzes directly from the pedagogical storyboard while enforcing a strict "no-reuse constraint"—prohibiting the verbatim replication of numerical parameters, sample equations, or variable names utilized in the main video. When a learner selects a distractor, the system decodes the latent misconception underpinning that error (e.g., assuming bubble sort positions the smallest value first). It then automatically generates a targeted remediation video that explicitly contrasts the incorrect assumption with the valid procedural rule, establishing cognitive dissonance to dismantle misconceptions and solidify durable understanding.

Benchmark & Evaluation Protocol

The empirical evaluation benchmarks PIVOT against two premier open-source educational code-generation baselines: TheoremExplainAgent and Code2Video. To establish an exhaustive and fair testbed, the benchmark covers 40 canonical instructional topics evenly distributed across 4 core STEM disciplines (10 topics each in Mathematics, Physics, Chemistry, and Computer Science, including the Newton–Raphson Method, Euler's Formula, Bubble Sort, Huffman Coding, Transition Metals, Redox Reactions, Newton's Laws of Motion, and Archimedes' Principle).

Evaluation couples automated multimodal scoring with expert human review: automated assessment leverages Gemini-2.5-Flash under deterministic sampling (\(T=0\)) to score frame-level Layout Quality and Visual Richness (1–10 scale), and GPT-4o to assess narration Accuracy & Depth and Logical Flow (0–1 scale); the human study enlists 32 active secondary and university STEM educators (12 in Computer Science, 12 in Mathematics, 5 in Chemistry, and 3 in Physics) evaluating full tutoring flows across 7 Likert-scale dimensions.

Key Experimental Results

Main Results

Automated video presentation metrics (Table 1) and narration script quality (Table 2) across all 40 STEM topics illustrate PIVOT's across-the-board superiority over baseline paradigms.

Table 1: Automated visual presentation quality evaluated via Gemini-2.5-Flash (Scale 1–10) | Method | Spatial Clarity & Layout (Layout) | Visual Diversity & Coverage (Richness) | Overall Visual Score (Overall) | |---|---|---|---| | Code2Video | 8.18 | 4.22 | 6.20 | | TheoremExplainAgent | 9.00 | 4.23 | 6.62 | | PIVOT (Ours) | 9.10 | 6.06 | 7.58 |

Table 2: Automated narration transcript quality evaluated via GPT-4o (Scale 0–1) | Method | Conceptual Accuracy & Depth (Accuracy) | Instructional Progression (Logical Flow) | Overall Narration Score (Overall) | |---|---|---|---| | Code2Video | 0.78 | 0.72 | 0.75 | | TheoremExplainAgent | 0.85 | 0.82 | 0.84 | | PIVOT (Ours) | 0.87 | 0.91 | 0.89 |

Table 3: Expert educator human evaluation across STEM disciplines (Likert Scale 1–5) | Discipline | Evaluated Paradigm | Clarity | Engagement | Logical Flow | Error Explanation | Teaching Effectiveness | |---|---|---|---|---|---|---| | Physics | TheoremExplainAgent | 3.67 | 3.33 | 2.00 | 1.67 | 1.67 | | | PIVOT | 4.33 | 4.33 | 4.33 | 4.33 | 3.67 | | Computer Science | TheoremExplainAgent | 4.18 | 4.00 | 3.73 | 3.27 | 3.27 | | | PIVOT | 4.45 | 4.36 | 4.18 | 4.18 | 4.00 | | Chemistry | TheoremExplainAgent | 3.67 | 3.33 | 3.67 | 2.00 | 3.33 | | | PIVOT | 4.33 | 4.00 | 4.33 | 4.67 | 4.33 | | Mathematics | TheoremExplainAgent | 4.20 | 4.00 | 4.60 | 4.40 | 4.30 | | | PIVOT | 4.50 | 4.50 | 4.60 | 4.70 | 4.70 |

Ablation Study

The contribution of each core verification pillar was systematically quantified by ablating individual modules, with results detailed in Table 4.

Table 4: Ablation study of core verification harness modules within PIVOT | Ablation Configuration | Layout Score (1–10) | Accuracy & Depth (0–1) | Logical Flow (0–1) | |---|---|---|---| | w/o Layout Verification | 7.87 | – | – | | w/o Factual Verification | – | 0.85 | – | | w/o Cognitive Verification | – | – | 0.89 | | Full Framework (PIVOT) | 9.10 | 0.87 | 0.91 |

Key Findings

  • Layout verification forms the bedrock of visual legibility: Discarding the layout consistency module causes the Layout metric to collapse from 9.10 down to 7.87 (-1.23 points), inducing severe element collisions, overlapping labels, and unreadable equation densities.
  • Remediation delivers the greatest gains in procedurally complex domains: Improvements were especially striking in Physics (+1.67 average gain across metrics) and Computer Science (+0.76 average gain). Conceptual pitfalls in these subjects hinge upon intermediate execution steps and dynamic states that static videos fail to clarify, making targeted error deconstruction exceptionally potent.
  • Educators overwhelmingly validate closed-loop tutoring: A decisive 96.9% of participating STEM instructors affirmed that the closed loop of "instructional video \(\to\) formative quiz \(\to\) contrastive remediation" delivers a far superior learning experience compared to watching standalone instructional clips.

Highlights & Insights

  • From Content to Pedagogy: By integrating Bloom's taxonomic scaffolding and cognitive load management directly into storyboard synthesis, PIVOT demonstrates how generative AI can transcend shallow media production to provide structured educational support.
  • Resilient Dual-Tier Verification: Pairing programmatic layout guardrails with an automated repair escalation loop (GPT-5.2 first-line remediation backed by Claude Sonnet 4.6 fallback debugging) establishes a practical template for reliable code-to-video synthesis.
  • Generalizable Cognitive Remediation: The technique of parameter-decoupled diagnostic probing combined with contrastive remediation offers immediate utility beyond video generation, readily extending to conversational tutoring agents and complex software on-boarding tutorials.

Limitations & Future Work

  • Absence of Persistent Learner Profiles: Adaptation currently operates on single-turn quiz interactions rather than maintaining a persistent longitudinal cognitive model tracking Bayesian knowledge state, forgetting decay, or progressive curriculum difficulty.
  • Higher Computational Latency: Orchestrating multi-turn storyboard generation, programmatic code synthesis, sandbox verification, and contingent remediation introduces significantly higher compute costs and latency than direct end-to-end diffusion models.
  • vs Code2Video & TheoremExplainAgent: While prior code-centric frameworks proved the feasibility of compiling instructions into Manim scripts, they function as stateless content generators lacking pedagogical scaffolding, code quality audits, and interactive post-instruction assessment.
  • vs Video Diffusion Models (Sora, Kling, Veo): Continuous diffusion models remain notorious for spatial hallucination, unstable formula rendering, and non-deterministic physics; PIVOT's programmatic Manim substrate guarantees byte-level symbolic precision and precise modular re-editing.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering framework transforming code-centric educational video generation into an interactive, pedagogy-guided tutoring loop.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 40 representative topics spanning 4 STEM fields with automated VLM verification, comprehensive ablations, and a 32-instructor blind user study.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid narrative prose supported by exhaustive appendices detailing formal storyboard prompts, layout rules, and bilingual case studies.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational design blueprint for deployable AI tutoring agents capable of both synthesis and pedagogical intervention.