Skip to content

ScEdit: Script-based Assessment of Knowledge Editing

Conference: ACL 2025
arXiv: 2505.23291
Code: Yes (https://github.com/asdfo123/ScEdit)
Area: NLP / Knowledge Editing Evaluation
Keywords: Knowledge Editing, Script Generation, Evaluation Benchmark, Procedural Reasoning, Text-level Evaluation

TL;DR

The authors propose ScEdit, a script-based evaluation benchmark for knowledge editing, which extends traditional "What"-style factual recall evaluation to "How"-style procedural reasoning. It introduces a two-tier evaluation system at both token and text levels, revealing significant limitations of existing knowledge editing methods in practical application scenarios.

Background & Motivation

While the field of Knowledge Editing (KE) has developed rapidly in recent years, existing evaluation frameworks suffer from fundamental deficiencies:

Oversimplified evaluation: Current metrics (Efficacy, Generalization, Locality) mainly focus on predicting the next few tokens of "What"-style questions, on which many methods have achieved near-perfect scores.

Disconnection from practical applications: LLMs are increasingly deployed as agents where users ask "How"-style procedural questions (e.g., "How to travel from Beijing to Singapore?") rather than simple factual queries.

Lack of text-level evaluation: Existing evaluations only focus on token probabilities, ignoring the quality of long-text generation.

To illustrate with an example from the paper: Singapore has implemented a visa-free policy for Chinese tourists, but an un-updated LLM would still advise users to apply for a visa. This error is not just a simple factual mistake; it leads the entire action plan (script) astray.

The motivation of ScEdit is to elevate knowledge editing evaluation from "correctly answering factual questions" to "correctly guiding multi-step procedural tasks," which is closer to the real-world challenges faced by LLMs during deployment.

Method

Overall Architecture

ScEdit consists of three core components: 1. Facts: Standard (s, r, o^c) โ†’ (s, r, o) knowledge triple editing. 2. Script Questions: "How"-style multi-step questions based on the edited facts. 3. Scripts: Multi-step responses generated by the model to the script questions.

The evaluation framework is divided into two tiers: - Token-level: Evaluates metrics such as S-ES, S-NS, and S-BO using cloze prompts. - Text-level: Evaluates Executability, Coherence, Consistency, and Completeness via automatic (GPT-4) and human evaluation.

Key Designs

  1. Script Question Generation

    • Function: Generate "How"-style questions that would be affected by the edited facts, starting from each knowledge editing triple.
    • Mechanism:
      • Generate candidate script questions using GPT-4 with few-shot prompting.
      • Manually filter to ensure the questions strictly depend on the edited facts.
      • Example: (Panamera, manufacturer, Porsche โ†’ Ford) โ†’ "How to book a maintenance service for a Panamera?"
    • Design Motivation: Simulate real-world scenarios where LLMs acting as agents are required to answer procedural questions.

  2. Token-level Evaluation Metrics

    • S-ES (Script-based Efficacy Success): Measures whether the edited model prefers the new answer under the script cloze prompt.
      • Truncated the original script up to the first occurrence of the old answer, and concatenate the script question to formulate the cloze prompt.
      • Check if P(o|Qฬƒ) > P(o^c|Qฬƒ).
    • S-NS (Script-based Neighborhood Success): Measures whether editing affects unrelated neighboring facts.
      • Construct neighbor-fact-based script cloze prompts.
      • Check if the model still retains the neighboring facts.
    • S-BO (Script-based Bleedover): Measures the bleedover effect of editing on semantically close facts.
      • Measures the drop in accuracy of neighboring facts after editing.

  3. Text-level Evaluation Metrics (7-point Likert Scale)

    • Executability: Evaluates whether the script steps are logically executable (disregarding knowledge updates).
    • Coherence: Evaluates whether the script is coherent with the updated facts.
    • Consistency: Evaluates whether the script is internally consistent (no confusion between old and new facts).
    • Completeness: Evaluates whether the script adequately addresses all aspects of the question.
    • Design Motivation: Token-level metrics cannot capture the overall generation quality and editing effects of the generated text.

  4. Two Subtasks

    • ScEdit-CF (Counterfactual Editing): Evaluates the performance of counterfactual knowledge editing in script scenarios based on the CounterFact dataset.
    • ScEdit-T (Temporal Editing): Evaluates temporal update editing based on WikiFactDiff, which closer reflects real-world scenarios.

Dataset Statistics

Subtask No. of Edit Cases S-Eff. Samples S-Spec. Samples
ScEdit-CF 1,830 7,342 13,672
ScEdit-T 1,762 7,038 6,597

Key Experimental Results

Main Results (Token-level, ScEdit-CF and ScEdit-T)

Method Model ESโ†‘ S-ESโ†‘ S-NSโ†‘ ESโ†‘(T) S-ESโ†‘(T) S-BOโ†“(T)
Base GPT2-XL 20.55 21.18 81.52 44.27 41.72 0.00
FT GPT2-XL 100.00 71.27 65.08 87.17 52.80 1.15
ROME GPT2-XL 99.95 74.76 80.24 99.15 68.00 0.13
MEMIT GPT2-XL 93.72 58.11 81.16 81.44 52.13 0.03
PROMPT GPT2-XL 96.28 69.63 42.88 99.49 84.39 0.54
FT GPT-J 100.00 83.94 25.81 99.60 97.90 5.47
ROME GPT-J 99.95 86.50 83.35 99.60 74.29 0.28
MEMIT GPT-J 99.95 74.59 85.07 99.09 64.66 0.08

Key takeaway: S-ES drops by 27% on average compared to traditional PS metrics, demonstrating that script scenarios are much more challenging.

Text-level Evaluation (ScEdit-CF, LLAMA3-8B)

Method Exec.โ†‘ Coh.โ†‘ Cons.โ†‘ Comp.โ†‘
Base Model 6.74 2.48 6.86 5.40
FT 2.94 2.97 6.17 2.17
ROME 6.41 4.32 6.57 4.67
MEMIT 6.54 3.67 6.70 4.98
PROMPT 6.36 4.35 6.05 5.49

Key Findings

  1. S-ES is much harder than ES: S-ES is on average 27% lower than traditional PS, indicating that script scenarios significantly increase editing difficulty.
  2. FT practically fails at the text level: Executability drops to only 2.94/7, nearly destroying the fundamental capabilities of the model.
  3. ROME achieves the best overall performance: It performs robustly at both token and text levels, proving that the locate-then-edit strategy remains effective in script scenarios.
  4. MEMIT preserves locality best: It achieves the best S-NS and S-BO, but at the cost of editing efficacy.
  5. PROMPT shows good efficacy but poor locality: S-NS is only 42.88%, indicating that in-context editing severely interferes with neighboring facts.
  6. Token-level and text-level evaluations capture different dimensions: Coherence is only weakly correlated with S-ES, and Consistency is almost uncorrelated with token-level metrics. This shows that the two evaluation tiers offer unique and irreplaceable values.
  7. The edit-efficacy vs. locality trade-off is more pronounced in script scenarios.

Highlights & Insights

  • Valuable problem formulation: For the first time, knowledge editing evaluation is extended from factual recall to procedural reasoning, aligning closer with real-world deployment scenarios.
  • Complementary two-tier evaluation system: Token-level and text-level evaluations capture different dimensions, as verified by correlation analysis.
  • More practical ScEdit-T subtask: It is constructed based on real Wikipedia temporal updates, rather than artificially designed counterfactuals.
  • Identification of cases with high token-level scores but text-level failures: This exposes the blind spots of existing evaluation frameworks.

Limitations & Future Work

  1. Limited model scales: Only models of size <10B were evaluated; larger models might exhibit different behaviors.
  2. Single-edit focus: Large-scale or sequential editing scenarios are not considered.
  3. Data generated by GPT-4: There might be a bias toward causal language models, and some samples might be less natural.
  4. Embodied AI scenarios not covered: Script execution is limited to the human level; robotic execution is an important direction for future work.
  5. Editing methods remain based on the triple paradigm: How to develop editing methods suitable for script scenarios remains an open question.
  6. Potential overestimation/underestimation in automated text-level evaluation: Although manual validation was performed, the coverage remains limited.
  • Complementary to ROME/MEMIT evaluation frameworks: ScEdit provides more challenging and practical evaluation scenarios.
  • Theoretical foundation from script generation work: Studies like (Schank & Abelson, 1975; CoScript) provide the theoretical basis for scripts.
  • Inspirations for "script-aware" knowledge editing methods: Editing should not only modify facts but also ensure that reasoning chains relying on those facts are correctly updated.
  • Connection to multi-hop reasoning evaluation (Ripple Effect, MeLLo): While sharing similar directions, ScEdit takes a step further by focusing on procedural reasoning.

Rating

  • Novelty: โญโญโญโญ โ€” Elevating knowledge editing evaluation to the level of script/procedural reasoning is a valuable contribution, offering a unique "How"-style evaluation perspective.
  • Experimental Thoroughness: โญโญโญโญ โ€” Comprehensive coverage with three models, six editing methods, two-tier token+text evaluations, manual+automatic evaluations, and metric correlation analyses.
  • Writing Quality: โญโญโญโญ โ€” The motivation is clearly articulated, the evaluation system is logically designed, and the examples/diagrams are intuitive.
  • Value: โญโญโญโญ โ€” Proposes an important complement to the evaluation paradigm in the knowledge editing field, revealing the performance gap of existing methods in real-world scenarios.