Editing Rules in LLMs Requires More Than a Localized Update¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Pepper66/DMLE
Area: Knowledge Editing
Keywords: Knowledge Editing, Rule-Level Editing, Causal Tracing, Multi-Form Knowledge, Distributed Multi-Layer Editing
TL;DR¶
Addressing the failure of localized factual model editors on mathematical and physical rules, this paper reveals via fine-grained causal tracing that rule knowledge exhibits form-specific layer-wise separation (formulas and descriptions reside in early layers while numerical instances peak in middle layers), and proposes Distributed Multi-Layer Editing (DMLE) to perform coordinated cross-layer updates that dramatically boost rule understanding and instance portability.
Background & Motivation¶
Large language models (LLMs) function not only as massive associative factual databases but also store abstract rule knowledge that underpins mathematical derivation and scientific reasoning. Over recent years, model editing techniquesโmost prominently locate-then-edit frameworks such as ROME, MEMIT, and AlphaEditโhave emerged as promising solutions to patch obsolete or erroneous knowledge without full retraining. These conventional editors operate under a localized update hypothesis: they assume that knowledge can be altered by identifying a single layer or a narrow contiguous block of feed-forward network (FFN) layers and applying a rank-one or closed-form parameter update.
However, when extended to structured, rule-level knowledge, this localized update paradigm fails fundamentally. Unlike isolated fact tuples, a scientific or mathematical rule inherently manifests across multiple interdependent forms: symbolic formulas representing abstract mathematical relations, natural-language descriptions conveying conceptual meaning, and concrete numerical instances demonstrating applied calculation. Existing editors often succeed in updating a rule in one specific surface expression (e.g., memorizing a modified symbolic equation) but consistently fail to propagate that update coherently across related descriptions or numerical computations. While prior benchmark studies have underscored this limitation, the mechanistic root causeโwhy localized updates fail to propagate across formsโhas remained largely unexplained.
To resolve this question, the authors investigate how multi-form rules are internally represented inside Transformer architectures using fine-grained causal tracing. Across diverse autoregressive language models, their causal interventions uncover a distinct "form-specific layer-wise separation": symbolic formulas and natural language descriptions share high causal attribution in early Transformer layers, whereas concrete numerical instances rely heavily on middle-layer representations. Core idea: effective rule-level editing requires moving beyond single-site localized parameter adjustments by adopting Distributed Multi-Layer Editing (DMLE), which applies a shared weight update to formulas and descriptions in their causal early layers alongside a decoupled update to numerical instances in their designated middle layers.
Method¶
Overall Architecture¶
The paper introduces Distributed Multi-Layer Editing (DMLE), a framework designed to mirror the empirical layer-wise distribution of rule knowledge discovered through causal tracing. The overall pipeline proceeds through three consecutive stages: first, layer-wise causal attribution identifies the optimal high-impact layer intervals for different forms using an automated top-window selection based on Average Indirect Effect (AIE); second, symbolic formulas and natural language descriptions are aggregated into a joint key-value set to solve a shared closed-form weight update applied to early Transformer layers; third, numerical instances are mapped into a separate instance-level key-value set to solve an independent weight update applied to middle layers, ultimately producing an edited model \(F^*\) that preserves cross-form consistency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Form Rule Input<br/>R = (s, f, d, i)"] --> B["Form-Specific Causal Localization<br/>AIE tracing isolates early/middle windows"]
B --> C["Early-Layer Joint Update<br/>Shared weight offset ฮWfd for formula/description"]
B --> D["Middle-Layer Decoupled Update<br/>Independent weight offset ฮWi for instances"]
C --> E["Cross-Form Coherent Rule Reasoning<br/>Yields globally consistent model F*"]
D --> E
Key Designs¶
1. Form-Specific Causal Localization: Localized editors ignore representational heterogeneity -> AIE-guided automated window partitioning -> Exact identification of layer targets Conventional locate-then-edit methods treat knowledge as localized within a single contiguous block of layers, which fails to capture the multi-faceted nature of rules. DMLE addresses this by performing fine-grained layer-wise causal tracing across separate formula, description, and instance prompts. Given a prompt-target pair \((x, y)\), the clean target probability \(p(y|x)\) is computed, and Gaussian noise \(\epsilon \sim \mathcal{N}(0, \sigma^2)\) is added to the subject token embeddings (with \(\sigma\) set to 2.5โ3 times the empirical standard deviation of input embeddings) to generate corrupted inputs \(\tilde{x}\). By restoring the intermediate hidden state at layer \(l\) back to its clean state \(h_l^{\text{clean}}\), the recovery score is computed as: $\(r_l = p(y \mid \tilde{x}; h_l \leftarrow h_l^{\text{clean}}) - p(y \mid \tilde{x})\)$ Averaging \(r_l\) yields the Average Indirect Effect (AIE). Experiments on GPT-J-6B, GPT2-XL, Qwen2-7B, and Qwen2.5-7B demonstrate that the causal contribution peaks at the final subject token. Crucially, formulas and descriptions exhibit overlapping AIE peaks in early layers (layers 2โ4 in GPT-J; layers 3โ5 in Qwen2/2.5), whereas instances peak in middle layers (layers 13โ15 in GPT-J; layers 12โ14 in Qwen2/2.5). An automated sliding-window criterion selects the contiguous window with highest mean AIE under a fixed layer budget \(L\), avoiding manual visual tuning.
2. Early-Layer Joint Update: Independent updates cause conceptual drift -> Shared early-layer least-squares optimization -> Grounded abstract rule semantics Representational alignment analysis reveals that edit-induced hidden-state shifts between formulas and descriptions exhibit high cosine similarity in early layers. Furthermore, cross-form transfer interaction matrices show that an update derived from formula improves description log-probabilities, and vice versa. Updating them independently risks catastrophic parameter interference and representational divergence. Treating Transformer FFNs as linear associative key-value memories, DMLE merges key-value pairs from formula and description sets, \(D_{fd} = D_f \cup D_d\), and calculates a shared weight update \(\Delta W_{fd}\) applied to the early-layer window: $\(\Delta W_{fd} = \arg\min_{\hat{W}} \left( \sum_{(k_f, v_f) \in D_f} \|\hat{W} k_f - v_f\|_2^2 + \sum_{(k_d, v_d) \in D_d} \|\hat{W} k_d - v_d\|_2^2 \right)\)$ This joint formulation anchors the symbolic representation and the conceptual prose definition into a unified semantic manifold, preventing cross-form divergence.
3. Middle-Layer Decoupled Update: Concrete computation requires distinct neuro-functional pathways -> Independent middle-layer optimization -> Elimination of interference with conceptual abstraction In contrast to abstract equations and prose definitions, numerical instances involve specific arithmetic calculations, variable substitution, and computational workflows. Causal tracing and hidden-state analyses show that instance-induced representation shifts align weakly with formula and description shifts, and instance-derived updates exhibit negligible cross-form transfer to conceptual queries. Forcing instances into the early conceptual layers corrupts high-level semantics and yields sub-optimal instance execution. DMLE isolates numerical instances into an independent key-value set \(D_i = \{(k_i, v_i)\}\) and optimizes a dedicated closed-form weight update \(\Delta W_i\) across the designated middle-layer window: $\(\Delta W_i = \arg\min_{\hat{W}} \sum_{(k_i, v_i) \in D_i} \|\hat{W} k_i - v_i\|_2^2\)$ This structural decoupling creates a natural feed-forward synergy: early layers encode updated abstract rule semantics, while middle layers execute the updated numerical transformation without cross-layer collision.
Evaluation Benchmark and Formal Definition¶
1. Formal Definition of Rule Editing: A rule is formalized as a 4-tuple \(R = (s, f, d, i)\), where \(s\) is the subject entity, \(f\) is the symbolic formula, \(d\) is the natural language description, and \(i\) is the concrete numerical instance. The objective of rule editing is to update model \(F\) into \(F^*\) such that for any query form \(x^{(g)}\) with \(g \in \{f, d, i\}\), the model reliably outputs the counterfactual target \(y^{(g)}\): $\(F^*(x^{(g)}) \approx y^{(g)}, \quad \forall g \in \{f, d, i\}\)$ while preserving the model's outputs on unrelated neighboring knowledge.
2. RuleEdit-200 Benchmark: To overcome the narrow coverage of prior work (such as the 80 geometry-only rules in RuleEdit), the authors construct RuleEdit-200, containing 200 fundamental mathematical and physical rules across 600 aligned samples. Sourced from Wikipedia Mathematics and OpenStax Physics, it covers 8 mathematical sub-disciplines (84.5% total: Geometry 89, Calculus & Analysis 24, Algebra 18, Decision Theory 11, Probability & Statistics 9, Logic 7, Discrete Mathematics 6, Number Theory 5) and 5 physical branches (15.5% total: Classical Mechanics 13, Electricity & Magnetism 6, Optics 5, Oscillations & Waves 4, Thermodynamics 3). Counterfactual rule statements are generated via Gemini and verified through triple-annotator manual review (Fleiss' \(\kappa = 0.81\)).
3. Evaluation Metrics: - Standard Knowledge Editing: Reliability (Rel., accuracy on target edit prompts), Generalization (Gen., accuracy on rephrased prompts), Locality (Loc., accuracy preservation on out-of-scope neighborhood queries). - Rule-Level Specific Metrics: - Rule Understanding (RU): Measures whether the edited rule is consistently captured across formula, description, and instance forms. - Instance Portability (IP): Measures whether the edited rule generalizes to unseen numerical instances. Open-ended generation outputs are judged using DeepSeek-V3.2 and rigorously audited via manual verification.
A Worked Example¶
Consider an edit on the Geometric Mean rule:
- Original Rule: Formula \(G = \sqrt{a \times b}\); Description "taking the nth root of the product"; Numerical instance for numbers 16 and 9: \(\sqrt{16 \times 9} = 12\).
- Target Counterfactual Edit: Formula \(G = \sqrt{a + b}\); Description "taking the square root of the sum of the numbers"; Numerical instance: \(G = \sqrt{16 + 9} = 5\).
- DMLE Execution Steps:
1. Causal Localization: Tracing on GPT-J-6B identifies early layers (Layers 2โ4) for formula and description, and middle layers (Layers 13โ15) for instances.
2. Early Shared Update: Keys and values from prompts "The formula for the geometric mean of two numbers a and b is" and "The geometric mean of two numbers can be described as" are merged to compute \(\Delta W_{fd}\) for Layers 2โ4.
3. Middle Decoupled Update: Key and value for "The geometric mean of 16 and 9 is" are solved to compute \(\Delta W_i\) for Layers 13โ15.
- Model Evaluation After Edit:
- Rephrase Prompt: "The formula for the geometric mean of two values a and b is" \(\rightarrow\) outputs G = sqrt(a + b).
- Rule Portability: "The geometric mean of two numbers can be described as" \(\rightarrow\) outputs the square root of the sum of the numbers.
- Instance Portability (unseen input): "Given numbers 12 and 13, substitute them into the geometric mean formula and compute the result" \(\rightarrow\) outputs G = sqrt(12 + 13) = sqrt(25) = 5.
Key Experimental Results¶
Main Results¶
The authors benchmark DMLE against parametric editors (ROME, MEMIT, AlphaEdit) and non-parametric editors (GRACE, PROMPT) across four LLM backbones on RuleEdit-200.
| Base Model | Method | Rel.โ | Gen.โ | Loc.โ | IPโ | RUโ |
|---|---|---|---|---|---|---|
| GPT-J-6B | ROME | 80.17 | 44.33 | 17.25 | 20.50 | 9.13 |
| MEMIT | 88.88 | 41.83 | 44.75 | 23.08 | 11.25 | |
| GRACE | 94.50 | 1.17 | - | 4.33 | 2.00 | |
| PROMPT | 67.00 | 51.16 | - | 30.92 | 18.88 | |
| AlphaEdit | 92.83 | 52.50 | 66.42 | 25.92 | 13.63 | |
| DMLE (ours) | 95.00 | 58.17 | 46.79 | 59.19 | 84.92 | |
| GPT2-XL | ROME | 74.33 | 50.83 | 24.04 | 17.16 | 10.13 |
| MEMIT | 89.17 | 54.50 | 41.63 | 18.96 | 11.75 | |
| GRACE | 88.83 | 1.83 | - | 4.21 | 1.13 | |
| PROMPT | 86.50 | 55.17 | - | 29.17 | 18.00 | |
| AlphaEdit | 93.50 | 54.67 | 62.00 | 25.33 | 12.63 | |
| DMLE (ours) | 91.33 | 56.83 | 43.50 | 49.04 | 79.13 | |
| Qwen2.5-7B | ROME | 93.17 | 39.83 | 48.08 | 33.91 | 15.38 |
| MEMIT | 97.16 | 51.67 | 48.25 | 36.50 | 20.25 | |
| GRACE | 92.33 | 1.33 | - | 5.25 | 2.13 | |
| PROMPT | 92.59 | 66.67 | - | 52.78 | 27.16 | |
| AlphaEdit | 96.17 | 56.17 | 74.58 | 34.25 | 25.13 | |
| DMLE (ours) | 96.50 | 58.17 | 51.00 | 63.51 | 85.50 | |
| Qwen2-7B | ROME | 95.33 | 35.17 | 50.08 | 38.92 | 19.63 |
| MEMIT | 97.83 | 51.67 | 46.25 | 39.92 | 19.63 | |
| GRACE | 95.16 | 1.67 | - | 3.17 | 2.25 | |
| PROMPT | 93.17 | 63.17 | - | 50.58 | 29.13 | |
| AlphaEdit | 97.00 | 50.30 | 72.58 | 40.75 | 26.38 | |
| DMLE (ours) | 98.17 | 63.33 | 52.08 | 64.58 | 87.63 |
Note: Locality is omitted for non-parametric methods (GRACE and PROMPT) because they inject representations at inference without updating parameters.
Ablation Study¶
Four controlled variants are evaluated on Qwen2-7B and GPT-J-6B under an identical layer budget: - FLSU (Fixed-Layer Separate Updates): Separate updates for each form applied to default contiguous layers; - FLJU (Fixed-Layer Joint Update): Single unified update for all three forms applied to default contiguous layers; - SFSU (Sequential Form-Specific Updates): Formula and description updated separately in early layers, instances in middle layers; - RLA (Reversed-Layer Allocation): Formula/description updated in middle layers, instances in early layers.
| Variant | Qwen2-7B Rel.โ | Qwen2-7B Gen.โ | Qwen2-7B IPโ | Qwen2-7B RUโ | GPT-J-6B Rel.โ | GPT-J-6B Gen.โ | GPT-J-6B IPโ | GPT-J-6B RUโ |
|---|---|---|---|---|---|---|---|---|
| FLSU (Fixed Separate) | 97.17 | 60.33 | 50.58 | 82.50 | 94.50 | 56.83 | 47.79 | 76.63 |
| FLJU (Fixed Joint) | 95.83 | 60.17 | 51.42 | 84.25 | 91.33 | 56.50 | 47.41 | 80.00 |
| SFSU (Decoupled Early) | 96.33 | 61.83 | 63.92 | 84.88 | 92.00 | 57.33 | 55.08 | 80.88 |
| RLA (Reversed Window) | 95.33 | 58.83 | 47.25 | 54.63 | 91.50 | 54.83 | 42.58 | 51.38 |
| DMLE (Full) | 98.17 | 63.33 | 64.58 | 87.63 | 95.00 | 58.17 | 59.19 | 84.92 |
Key Findings¶
- Unprecedented gains on rule-level coherence: Across all models, DMLE achieves massive improvements on IP and RU over existing baselines. Compared to the best baseline on each model, DMLE yields average absolute improvements of 18.22% on IP and 61.00% on RU (e.g., in GPT-J-6B, RU leaps from 13.63% to 84.92%).
- Causal layer assignment is non-negotiable: Swapping the layer windows in RLA precipitates catastrophic performance degradation (RU collapses from 87.63% to 54.63% on Qwen2-7B, and from 84.92% to 51.38% on GPT-J-6B), proving that performance improvements stem directly from aligning updates with neuro-functional storage locations rather than increased parameter budget.
- Shared updates benefit conceptual forms: Removing formula-description weight sharing in SFSU causes consistent drops in IP and RU, confirming that formulas and descriptions share representational geometry and benefit from joint optimization.
- Null-space constraints hinder generalization: While AlphaEdit achieves high locality (62โ74%) by projecting updates onto null spaces, its restrictive projection severely penalizes cross-form generalization, leaving RU and IP scores comparable to basic single-layer editors.
Highlights & Insights¶
- Mechanism-driven architecture design: Rather than treating rule editing as an empirical tuning problem, the authors systematically trace the internal organization of multi-form rules and translate causal interpretability findings directly into a distributed editing algorithm.
- Debunking the localized update myth for rules: The paper provides strong empirical and conceptual evidence that localized, single-layer updates are fundamentally unsuited for multi-form reasoning rules.
- Form-specific functional specialization: The findings reinforce broader transformer interpretability observations: conceptual semantics settle in early layers, whereas procedural arithmetic computation unfolds in middle layers.
Limitations & Future Work¶
- Scope limited to single-line rules: The benchmark is constrained to self-contained mathematical and physical rules expressible as single-line formulas; multi-step conditional rules or non-mathematical logical heuristics remain unexplored.
- Absence of sequential and batch editing evaluation: The method is evaluated strictly in single-edit scenarios; lifelong continual editing and cross-rule interference have not been analyzed.
- Evaluation within templated queries: Assessment relies on standardized prompt formats and does not measure downstream complex multi-step reasoning benchmarks (e.g., full theorem proving or GSM8K / MATH problems).
Related Work & Insights¶
- vs ROME / MEMIT: While ROME and MEMIT established locate-then-edit with closed-form least-squares updates on single layer ranges for facts, DMLE generalizes this mathematical formulation into a distributed, multi-layer architecture aligned with multi-form knowledge.
- vs AlphaEdit: AlphaEdit prevents collateral damage via null-space projection at the expense of cross-form expressivity; DMLE demonstrates that cross-form rule coherence requires distributed parameter freedom.
- vs RuleEdit (Zhang et al., 2026): RuleEdit introduced the 3-form evaluation paradigm on 80 geometry rules without explaining failure modes; DMLE expands the benchmark to 200 comprehensive scientific rules and uncovers the underlying layer-wise separation mechanism.
Rating¶
- Novelty: โญโญโญโญโญ First study to uncover the layer-wise neuro-anatomical separation of multi-form rules and propose distributed multi-layer editing.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous causal tracing, 4 LLM backbones, 5 baselines, detailed ablations, sensitivity checks, and 95% confidence intervals.
- Writing Quality: โญโญโญโญโญ Cohesive narrative with crystal-clear transitions between causal discoveries, algorithmic design, and empirical validation.
- Value: โญโญโญโญโญ Provides crucial foundations and benchmarking tools for reliable rule editing and reasoning alignment in foundation models.