Skip to content

InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Syon-Li/InComeS
Area: LLM Reasoning
Keywords: Model Editing, Context Compression, Cross-Attention Selection, Gist Token, Knowledge Distillation

TL;DR

InComeS compresses each individual edit context into the KV cache of a single gist token and integrates these representations through a specialized cross-attention module equipped with a learnable Zero-Gist token, breaking context window constraints and dramatically accelerating multi-hop reasoning and natural language model editing over large batches.

Background & Motivation

Model editing (or knowledge editing) aims to precisely correct or update specific factual knowledge within large language models (LLMs) without incurring the prohibitive computational overhead of retraining from scratch. Traditional parametric editing paradigms—such as locate-and-edit methods (e.g., ROME, MEMIT, and AlphaEdit)—modify feed-forward network weights under the rigid assumption that edits strictly conform to subject-relation-object triplets. While these methods demonstrate high recall on isolated factual updates, recent studies expose a severe overfitting pathology: they assign excessively high probabilities to the edited target entity, which severely distorts the model's latent representations and causes catastrophic failure when handling multi-hop composition, natural language paragraph edits, and broader reasoning tasks requiring systematic generalization.

Conversely, in-context learning (ICL) editing provides a flexible, non-parametric alternative by simply prepending updated factual statements into the prompt context. Leveraging the model's native contextual comprehension and self-attention reasoning, ICL naturally accommodates free-form natural language edits and multi-step relational deductions without weight perturbation. However, practical deployment demands concurrent batch editing across hundreds or thousands of instances, where vanilla ICL quickly hits fundamental computational and algorithmic bottlenecks. The maximum number of edits is strictly bounded by the model's finite context window; the quadratic complexity of self-attention over extended sequences leads to severe latency and memory spikes; and more critically, as irrelevant prefix contexts accumulate, the model's attention dilutes, resulting in a marked degradation in factual retrieval accuracy and reasoning performance.

Resolving this tension requires an architecture that retains the reasoning fidelity and structural flexibility of ICL while eliminating its prefix length bottleneck and retrieval interference. Core idea: Decouple batch editing into parallel per-edit gist compression and fine-grained token-level cross-attention selection—compressing each edit into the deep-layer KV cache of a single gist token, and adaptively retrieving edit information via a zero-gist-enabled cross-attention module optimized through sensitivity-weighted knowledge distillation.

Method

Overall Architecture

InComeS (Integrating Compression and Selection Mechanisms) circumvents the context window limits and computational burden of ICL by introducing a decoupled two-stage editing pipeline paired with a one-time meta-training phase. First, during Edit Compression, each individual edit is independently encoded by the target language model and condensed into the key-value (KV) representations of an appended special <GIST> token, retaining only the deep-layer KV cache while discarding all intermediate text tokens. Second, during Edit Selection, the query input is processed through the language model, where input tokens dynamically attend to the compressed gist KV cache pool via specialized cross-attention layers. A dedicated "Zero-Gist" token is introduced to grant tokens the option to bypass edit information entirely. Finally, an offline Meta-Training phase aligns the student model's compressed representations with the uncompressed full-context teacher distribution using token-sensitivity reweighting and knowledge distillation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Batch Edit Contexts & Query Input"] --> B["Gist Token Independent Compression<br/>Parallel encoding into single-token KV caches"]
    B --> C["Parallel Deep Cross-Attention Selection<br/>Token-level dynamic routing over gist pool"]
    C --> D["Zero-Gist Skip Mechanism<br/>Learnable key and zero value for selective bypass"]
    D --> E["Sensitivity-Weighted Distillation Training<br/>Loss-difference weighting + KL distribution alignment"]
    E --> F["Autoregressive Generation with Edited Knowledge"]

Key Designs

1. Gist Token Independent Compression: Eliminating Context Explosion via Parallel Encoding

Standard ICL concatenates all edits \(T = \{t_1, t_2, \dots, t_n\}\) into an unwieldy prefix sequence, causing attention complexity to surge quadratically. InComeS extends the concept of gisting to knowledge editing: for each edit instance \(t_i = (t_0^i, t_1^i, \dots, t_{m}^i)\), a special gist token \(t_g = \langle\text{GIST}\rangle\) is appended to the sequence. The target LLM encodes each edit independently in a single forward pass, after which all original token activations are discarded, retaining only the gist token's key-value activations \((g_{K}^i, g_{V}^i)\). This design provides three decisive advantages: it achieves an aggressive compression ratio of approximately \(12:1\) (compressing an average 12-token factual statement into 1 gist representation); it decouples the batch, enabling massive GPU parallelization that slashes prefilling latency by orders of magnitude compared to sequential ICL prefixing; and it utilizes the identical base LM for compression, ensuring that the resulting latent vectors are perfectly aligned with the model's internal representation space.

2. Parallel Deep Cross-Attention Selection: Fine-Grained Token Routing with Layer Specialization

To dynamically retrieve relevant edits from the pooled gist cache during query decoding without external retrievers, InComeS integrates specialized cross-attention modules into the model. Unlike RAG which operates at the coarse query-level prior to decoding, InComeS computes attention over the gist pool at every individual generation token. For any token's query representation \(q\), the cross-attention output \(o_{\text{cross}}\) is computed and residually added directly to the self-attention output: $\(o_{\text{cross}} = \text{Attention}\left(q, \{g_{K}^0, g_{K}^1, \dots, g_{K}^n\}, \{g_{V}^0, g_{V}^1, \dots, g_{V}^n\}\right)\)$ Empirical probing reveals that shallow transformer layers primarily capture local syntactic structures, whereas abstract factual knowledge and multi-hop associations coalesce in the deeper half of the network. Consequently, InComeS introduces cross-attention modules exclusively across the second half of the layers (e.g., layers 8–15 in Llama-3.2-1B, and layers 14–28 in Qwen2.5-7B). This architectural restriction cuts the runtime cross-attention KV cache footprint by 50% without any degradation in reasoning performance. Furthermore, scaling cross-attention logits with an inference temperature \(T = 0.45\) regularizes attention entropy, driving decisive and sharp retrieval of the correct edit vector.

3. Zero-Gist Skip Mechanism: Preserving Locality and Guarding Against Semantic Drift

A fatal risk of applying continuous cross-attention across external knowledge pools is over-retrieval: syntactic glue tokens, punctuation, and query components unrelated to the edit can become corrupted by spurious edit activations, degrading locality and inducing hallucinations. To solve this, InComeS incorporates a synthetic virtual token denoted as Zero-Gist (\(g_0\)). For the zero-gist, the model maintains a learnable key vector \(g_{K}^0\), but enforces a fixed zero vector for its value \(g_{V}^0 = \mathbf{0}\). Whenever an input token does not require knowledge injection, the cross-attention mechanism can naturally divert its probability mass onto \(g_0\). The resulting zero value vector contributes zero perturbation to the self-attention path, gracefully allowing the model to "attend to nothing" and execute native pre-trained generation undisturbed.

4. Sensitivity-Weighted Distillation Training: Pinpointing Pivotal Tokens via Knowledge Transfer

Pre-trained LLMs lack innate inductive biases for decoding compressed gist states. InComeS activates this capability via a one-time meta-training protocol utilizing a teacher-student framework. An unedited instance of the base model serves as the teacher, executing two parallel forward passes: one contextualized with the uncompressed golden edit text, and one uncontextualized. The difference between their token-level cross-entropy losses defines the edit-sensitivity weight \(w_{x_i}\): $\(w_{x_i} = \max\left(0, \text{CE}(x_i \mid x_0, \dots, x_{i-1}) - \text{CE}(x_i \mid \{t_i \mid i \in \mathcal{R}(x)\}, x_0, \dots, x_{i-1})\right)\)$ This reweighting heavily penalizes prediction errors on tokens whose generation strictly depends on the edited knowledge, while ignoring trivial tokens that can be predicted from local context alone. To ensure full distribution matching, the student model's output distribution \(p_S\) (conditioned on compressed gists) is aligned with the teacher's distribution \(p_T\) (conditioned on raw text) via Kullback-Leibler (KL) divergence, imparting full-context reasoning capabilities into the compressed architecture.

Loss & Training

The meta-training optimization objective combines token-weighted cross-entropy with output distribution distillation: $\(\mathcal{L}_{x_i} = w_{x_i} \cdot \text{CE}(x_i) + \mathcal{D}_{\text{KL}}\left(p_T(x_i \mid \{t_i\}_{i\in\mathcal{R}(x)}, x_{<i}) \parallel p_S(x_i \mid \{g_1, \dots, g_n\}, x_{<i})\right)\)$ Meta-training is exceptionally lightweight, requiring only ~1.5 billion tokens drawn from summarization (S2ORC, AG News) and QA datasets (SQuAD, Natural Questions, OpenBookQA, QASC, MedMCQA, NetEval, ZsRE, and COUNTERFACT). Training utilizes DeepSpeed and Liger Kernel over 8 Nvidia GPUs with cosine linear-warmup scheduling (\(lr_{\max} = 1\times 10^{-5}\) for Llama-3.2-1B, taking ~11 hours; \(lr_{\max} = 5\times 10^{-6}\) for Qwen2.5-7B, taking ~35 hours). Batch sizes are dynamically sampled from \(\{8, 16, 32, 64, 128\}\) with probabilities \(\{0.05, 0.05, 0.05, 0.15, 0.7\}\) to maximize batch robustness.

Key Experimental Results

Main Results

InComeS is systematically evaluated across complex model editing scenarios, notably multi-hop reasoning on MQuAKE (evaluating 2-hop, 3-hop, and 4-hop queries on MQuAKE-CF-3k-v2) and free-form natural language editing on DUNE (evaluating New Information, Scientific Reasoning, and Debiasing). Batch evaluations employ a default batch size of 100 edits.

Dataset / Setting Backbone Model Metric InComeS (Ours) ICL (Baseline) Best Parametric Baseline Performance Contrast & Key Observations
MQuAKE (Single Edit) Llama-3.2-1B Accuracy Avg 71.99 56.62 65.56 (LoRA) Outperforms ICL by +15.37%; parametric methods (ROME 3.43) suffer complete collapse
MQuAKE (100 Batch) Llama-3.2-1B Accuracy Avg 53.15 47.43 50.68 (ATBIAS) Surpasses batch ICL by +5.72% and fine-tuning FT-M (48.71)
MQuAKE (Single Edit) Qwen2.5-7B Accuracy Avg 71.41 73.74 73.24 (FT-M) Comparable to raw-text ICL, substantially outperforming AlphaEdit (53.71) and ROME (8.64)
MQuAKE (100 Batch) Qwen2.5-7B Accuracy Avg 52.17 49.61 50.36 (AlphaEdit) Demonstrates +2.56% gain over batch ICL on 7B scale
DUNE (100 Batch) Llama-3.2-1B Accuracy Avg 53.45 50.69 49.08 (LoRA) Consistent lead on unstructured natural language updates
DUNE (100 Batch) Qwen2.5-7B Accuracy Avg 63.37 57.75 57.21 (LoRA) +5.62% absolute advantage over ICL, excelling in debiasing (56.69 vs 40.73)
ZsRE-ext (100 Batch) Llama-3.2-1B Portability 52.23 51.58 54.84 (FT-M) Maintains 87.09% edit success alongside high locality (59.66%)
ZsRE-ext (100 Batch) Qwen2.5-7B Portability 61.22 60.57 46.41 (FT-M) Outperforms FT-M by +14.81% and MEMIT (49.97%), showing strong reasoning transfer

In execution efficiency, compressing 100 edits with InComeS takes only 0.0326 seconds on Llama-3.2-1B, compared to 0.8934 seconds for ICL prefilling (27.4× speedup), while MEMIT requires 112.2s and EMMET requires 158.7s. On scaled sequence decoding (2k tokens), InComeS-Selection completes in 0.0297 seconds, outpacing prefilled ICL generation (0.3545s) by roughly 12×.

Ablation Study

Ablation experiments conducted on MQuAKE using Llama-3.2-1B across both single editing and 100-edit batch settings:

Configuration Single Edit Avg Acc 100 Batch Edit Avg Acc Empirical Analysis & Mechanistic Insight
InComeS (Full Framework) 71.99 53.15 Standard architecture applying deep-half cross-attention with Zero-Gist
- w/o zero-gist 58.98 40.59 Catastrophic drop (-12.56% in batch), proving tokens must be able to skip edit retrieval
- w/ full model layers 72.40 52.34 Marginal difference (+0.41% single, -0.81% batch), validating that shallow KV caches are redundant
- w/o loss on query 59.86 44.78 Severe decline (-8.37% in batch), confirming query supervision is vital for query-edit mapping
- w/ golden loss 71.09 48.56 Significant drop (-4.59% in batch); forcing hard selection disrupts flexible multi-hop attention
- w/o KL distillation loss 55.10 51.71 Massive plunge in single editing (-16.89%), proving distillation transfers contextual reasoning
- w/o token weighting 55.02 52.63 Massive plunge in single editing (-16.97%), showing sensitive token reweighting is indispensable

Key Findings

  • Zero-Gist is the Core Safeguard against Contextual Noise: Eliminating Zero-Gist causes batch editing accuracy on MQuAKE to plummet by 12.56%. Attention visualizations show that after the initial query token, generation tokens allocate over 80% of cross-attention probability to Zero-Gist when processing standard non-entity words, preventing irrelevant factual drift.
  • Deep Layers Carry Concentrated Knowledge: In layers 1–6, Zero-Gist probability remains above 0.8 across tokens, indicating that lower layers barely attend to edit gists. A dramatic shift occurs at layer 7 where gist utilization surges. Restricting cross-attention to the top half of layers saves 50% KV cache memory with negligible impact on performance (53.15 vs 52.34).
  • Soft Attention Beats Explicit Golden Supervision: Adding an explicit cross-entropy loss to force the cross-attention to align with the ground-truth edit ("w/ golden loss") actually degrades batch performance by 4.59%. In complex multi-hop queries, the model benefits from flexibly aggregating multiple gists across different reasoning tokens rather than rigidly peaking on a single supervised target.

Highlights & Insights

  • Decoupled O(1) Per-Edit Compression: Condensing each fact into an isolated Gist token KV cache eliminates the \(O(N^2)\) self-attention cost across concatenated edits, converting batch prefilling into highly efficient, parallel matrix operations.
  • Attend-to-Nothing via Zero-Gist: Using a learnable Key coupled with a hardwired zero Value vector provides an elegant sink for extraneous attention mass, bypassing edit injection without requiring discrete gating heuristics or classifier thresholds.
  • Differential Loss Token Mining: Leveraging the teacher model's loss gap between contextualized and uncontextualized inputs automates the discovery of edit-sensitive pivot tokens, maximizing meta-training efficiency without handcrafted entity labeling.

Limitations & Future Work

  • Fixed Compression Ratio Constraint: The current setup maintains a ~12:1 token compression ratio (one gist token per short edit sentence). For long-form document updates or multi-paragraph guidelines, single-token representations may suffer information bottlenecking, necessitating adaptive multi-gist sequence compression.
  • Architectural Scope: The framework is currently optimized and evaluated on decoder-only autoregressive models up to 8B parameters. Expanding to mixture-of-experts (MoE) architectures and multimodal foundation models represents an essential next step.
  • vs ROME / MEMIT / AlphaEdit: Weight-editing techniques directly alter MLP weights via closed-form optimization, which excels at single-triple recall but severely overfits and fails on multi-hop or natural language inputs. InComeS preserves base model weights and dynamically conditions generation on compact external KV caches, guaranteeing reasoning stability and reversibility.
  • vs Standard ICL / IKE: Vanilla ICL and retrieval-augmented IKE concatenate raw text sequences, rapidly exhausting the context window and suffering from attention distraction. InComeS condenses each edit into a single-token representation and retrieves via specialized cross-attention, cutting prefilling time by 27× and decoding latency by 12×.
  • vs SERAC / External Memory Systems: SERAC relies on a separate scope classifier model and an auxiliary counterfactual generator, introducing multi-model pipeline latency and cascading errors. InComeS performs compression, selection, and generation end-to-end within the same underlying LLM.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Brilliant integration of gist token compression, zero-gist selective cross-attention, and sensitivity-weighted distillation for scalable model editing.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks spanning multi-hop (MQuAKE), free-form natural language (DUNE), portability, scaling up to 10k edits (~120k tokens), and deep layer-wise attention analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, clean mathematical formulation, and exceptionally thorough empirical analysis.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, robust, and computationally efficient paradigm for continuous knowledge integration in production LLMs.