✏️ Knowledge Editing¶

🧪 ICML2025 · 2 paper notes

Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing: Through synthetic experiments training Transformers on cyclic knowledge graphs, this work reveals that knowledge editing (KE) "shatters" the learned geometric representation manifolds inside the model. The degree of shattering is positively correlated with edit distance (\(r^2=0.905\)). Based on this, "representation shattering" is proposed as a mechanistic hypothesis for how KE impairs model capabilities, and the universality of this phenomenon is validated on Llama 3 and Mamba.
WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs: This paper proposes WikiBigEdit, a large-scale lifelong knowledge editing benchmark containing over 500k real Wikidata knowledge updates, revealing the severe limitations of existing knowledge editing methods under realistic scales—general methods such as retrieval augmentation and continual fine-tuning paired with model merging surprisingly perform better.