Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zhejianglab/Gengram
Area: LLM Reasoning
Keywords: Genomic Foundation Models, Retrieval-Augmented Memory, Multi-scale K-mers, Local Window Aggregation, MoE Load Balancing
TL;DR¶
Gengram establishes a conditional memory module for genomic foundation models that explicitly stores and retrieves multi-scale (\(k=1\) to \(6\)) motif embeddings via collision-free hashing across a local causal window (\(W=21\) bp), injecting gated contextual representations into Transformer layers to eliminate redundant neural motif inference and stabilize MoE routing.
Background & Motivation¶
Genomic foundation models (GFMs) have emerged as powerful tools for deciphering the sequence grammar governing gene regulation, organismal development, and disease etiologies. Predominant architectures—spanning bidirectional encoders like DNABERT, autoregressive decoders such as Nucleotide Transformer and Genos, and linear state space models like Caduceus—rely predominantly on single-nucleotide tokenization. While single-base resolution is indispensable for evaluating fine-grained single-nucleotide polymorphisms (SNPs) and point mutation effects, functional regulatory elements in biological genomes are fundamentally characterized by multi-nucleotide motifs, including promoter TATA-boxes, RNA splice junctions, and codon triplets. Under standard Transformer paradigms, models possess no explicit mechanism to store or query these sequence combinations; instead, they depend on deep stacks of self-attention and dense feed-forward computations to implicitly reconstruct motif-level semantics from raw base sequences. This indirect formulation squanders extensive parameter capacity on redundant sequence pattern discovery and imposes an artificial performance ceiling on downstream motif-dominated prediction tasks.
In natural language processing, retrieval-augmented architectures like DeepSeek's Engram demonstrate that static \(n\)-gram knowledge can be offloaded into hash-indexed memory tables, decoupling lexical memorization from dynamic contextual reasoning to introduce an orthogonal sparsity axis. However, naive transplantation of natural language memory into biological sequence modeling encounters severe domain mismatch: natural language models grapple with massive vocabularies and long-range lexical correlations, typically retrieving a single preceding \(n\)-gram per position. Conversely, genomic sequences are assembled from a discrete alphabet of merely four canonical bases and an ambiguous character (\(\Sigma = \{\text{A, C, G, T, N}\}\)), which offers a remarkably compact state space naturally amenable to collision-free hash addressing. Crucially, functional genomic signals are rarely encoded by isolated, single-scale \(n\)-grams; biological regulation is orchestrated through spatial clusters of heterogeneous, multi-scale motifs acting collaboratively within compact genomic windows. Querying a single preceding \(n\)-gram therefore risks transient noise and fails to reflect the synergistic nature of local regulatory architectures.
To overcome these structural limitations, the authors introduce Gengram, a biology-aligned conditional memory module tailored for genomic sequence modeling. Gengram maintains dedicated, collision-free hash embedding tables for multi-scale \(k\)-mers ranging from \(k=1\) through \(6\). At each sequence position, it scans a local causal genomic window, deduplicates active substrings to emphasize motif diversity rather than raw frequency, pools their memory embeddings, and injects the aggregated signal via scaled dot-product gating and depthwise convolutions into the backbone. Core idea: exploit the compact genomic alphabet to maintain collision-free multi-scale motif tables, using local window deduplicated aggregation and scaled dot-product gated residual injection to offload motif inference from dense attention networks, accelerating training and stabilizing MoE routing.
Method¶
Overall Architecture¶
Gengram models multi-nucleotide motif lookup as an explicit, lightweight deterministic pathway running alongside standard Transformer blocks. Given an input sequence representation \(X \in \mathbb{R}^{n \times d_{\text{model}}}\), Gengram inspects preceding tokens across a causal local window of width \(W\), retrieves pre-computed embeddings for multi-scale \(k\)-mers (\(k \in \{1, 2, 3, 4, 5, 6\}\)), applies deduplicated mean pooling, and fuses the multi-scale representations through dual linear projections. The resulting gated signal is combined with a short depthwise 1D convolution and residually added back to the hidden states before subsequent self-attention or MoE feed-forward layers.
The entire module preserves linear time complexity \(O(n|N|W)\) with respect to sequence length, decoupling static motif storage from dynamic representation learning:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Single-Base Sequence Tokens<br/>(A, T, C, G, N)"] --> B["Local Window Enumeration & Deduplication<br/>Causal W=21 bp, Extract Unique k∈{1..6} Substrings"]
B --> C["Multi-Scale Hash Retrieval & Mean Pooling<br/>Collision-free base-|Σ| Indexing & Dimension Standardization"]
C --> D["Cross-Scale Concatenation & Dual Projections<br/>Gating Vector z_t & Content Vector u_t"]
D --> E["Scaled Dot-Product Gating & Residual Injection<br/>RMSNorm Interaction, DWConv1D Sequential Smoothing"]
E --> F["Modulated Hidden Representation Y_t<br/>Forwarded to Subsequent Attention / MoE Layers"]
Key Designs¶
1. Local Window Enumeration & Deduplication: High-Frequency Noise -> Causal Sliding Window Deduplication -> Focus on Functional Motif Repertoire while Regulating Scale
Directly accumulating all \(n\)-grams within a genomic region causes repetitive, low-complexity elements (such as long poly-T tracts or tandem repeats) to dominate the retrieved embeddings, drowning out informative functional signals and destabilizing memory magnitudes. Gengram defines a causal window at each position \(t\): \(W_t = \{t-W, \dots, t-1\}\) (with optimal size \(W=21\text{ bp}\)). Within this window, all contiguous substrings of length \(N \in \{1, 2, 3, 4, 5, 6\}\) are enumerated:
$\(G_t^{(N)} = \{(s_j, \dots, s_{j+N-1}) \mid j \in [t-W, t-N]\}\)$
To shift the model's focus toward distinct motif varieties rather than raw physical counts, Gengram applies explicit deduplication:
$\(\tilde{G}_t^{(N)} = \text{Unique}(G_t^{(N)})\)$
Each unique substring is indexed into a dedicated lookup table \(M^{(N)} \in \mathbb{R}^{|\Sigma|^N \times d_m}\) via deterministic base-\(|\Sigma|\) encoding \(h_N(g)\). Uniform mean pooling is then applied over unique substrings:
$\(m_t^{(N)} = \frac{1}{|\tilde{G}_t^{(N)}|} \sum_{g \in \tilde{G}_t^{(N)}} M^{(N)}[h_N(g)]\)$
This deduplicated mean pooling guarantees scale invariance across variable sequence positions, suppressing spiky magnitude variations and preventing low-complexity repeats from distorting downstream activations.
2. Multi-Scale Hash Retrieval & Mean Pooling: Multi-Tiered Genomic Structures -> Channel Concatenation & Dual Projections -> Decoupling Gate Conditioning from Injected Content
Biological regulatory mechanisms operate across vastly different physical scales: splice donor/acceptor sites are determined by dinucleotide boundaries (GT/AG), amino acids are encoded by triplet codons, and transcription factor binding sites (such as TATA boxes or enhancers) span 6 bp or more. To preserve multi-tiered contextual signals without arbitrary feature selection, Gengram concatenates all pooled summaries across \(N \in \{1, \dots, 6\}\) along the channel dimension:
$\(m_t = \text{Concat}(m_t^{(1)}, m_t^{(2)}, m_t^{(3)}, m_t^{(4)}, m_t^{(5)}, m_t^{(6)})\)$
The combined representation \(m_t\) is projected through two independent parameter matrices:
$\(z_t = m_t W_z + b_z, \quad u_t = m_t W_u + b_u\)$
Here \(z_t \in \mathbb{R}^{d_{\text{model}}}\) parameterizes a token-conditioned gating query, while \(u_t \in \mathbb{R}^{d_{\text{model}}}\) carries the actual retrieved motif content. This dual-projection design decouples the decision of whether to inject motif information from what content to inject.
3. Scaled Dot-Product Gating & Residual Injection: Uninformative Background Sequences -> Normalized Gated Filtering & Depthwise Convolution -> High SNR Contextual Fusion with Smooth Sequential Dynamics
Pervasive non-coding intergenic regions in eukaryotic genomes contain millions of uninformative k-mers; adding raw memory embeddings unconditionally would degrade backbone representation quality. Gengram introduces a scaled dot-product gating mechanism utilizing root mean square normalization (RMSNorm):
$\(\tilde{u}_t = \sigma\left( \frac{\text{RMSNorm}(X_t)^T \text{RMSNorm}(z_t)}{\sqrt{d_{\text{model}}}} \right) u_t\)$
The Sigmoid gate \(\sigma(\cdot)\) activates strongly only when the current backbone hidden state \(X_t\) semantically aligns with the retrieved motif vector \(z_t\), acting as an adaptive noise filter across uninformative sequence segments. To ensure sequential continuity across adjacent bases, the gated tensor \(\tilde{U}\) passes through a lightweight depthwise 1D convolution with kernel size 4 and SiLU activation:
$\(\hat{u}_t = \tilde{u}_t + \text{SiLU}(\text{DWConv1D}(\tilde{U}))_t\)$
The refined update is residually added to the previous layer: \(Y_t = X_t^{(\ell-1)} + \hat{u}_t\), ensuring stable gradient flow and minimal runtime overhead.
4. Hierarchical Layer Placement Strategy: Uniform Full-Layer Insertion Inefficiency -> Selective Depth Deployment at Layers {3, 6, 10} -> Mirroring Sensory to Abstract Cortical Processing
Sweeping individual Transformer layers in a 12-layer backbone demonstrates that deeper layers yield greater loss reductions because their representations contain richer semantic abstractions capable of governing the gate effectively. However, multi-layer insertion experiments indicate that saturating every layer is counterproductive. Gengram adopts a sparse hierarchical configuration inserting memory modules at layers \(\{3, 6, 10\}\). This pattern mirrors biological hierarchical processing: Layer 3 provides shallow motif-level scanning, Layer 6 integrates mid-range structural boundaries, and Layer 10 facilitates high-level regulatory abstraction.
Loss & Training¶
The 10B Gengram model was trained within the Megatron-LM framework across distributed 8-GPU nodes using a pipeline parallelism degree of 2. Training data strictly maintained a 1:1 taxonomic ratio between human genomes (HPRC release 2) and non-human primate sequences (NCBI RefSeq). - Two-Stage Pretraining Curriculum: 1. Stage I (Base Pretraining): 200B tokens formatted as 8,192 bp fragments restricted to the forward strand; global batch size 1,024; learning rate decayed from \(1 \times 10^{-4}\) to \(1 \times 10^{-5}\) via cosine schedule; weight decay 0.1; BF16 mixed precision with FlashAttention. 2. Stage II (Continued Pretraining - CPT): 100B tokens formatted into 32,768 bp fragments; introduced randomized dual-strand sampling (56% negative strand, 44% positive strand) and incorporated distal intergenic and repetitive elements to expose the model to complete genomic diversity. - MoE Architecture & Parameter Overhead: The backbone features 8 experts with Top-2 routing and auxiliary load-balancing loss, executed via Grouped GEMM operators. Out of 10B total parameters, only 2.87B are activated per token. Gengram introduces merely ~60M parameters and under 5% wall-clock per-iteration latency overhead.
Key Experimental Results¶
Main Results¶
On standard genomic benchmarks spanning Genomic Structure Understanding, Gene Regulation Prediction, Epigenetic Profiling, Variant Effect & Clinical Impact, and Evolutionary Analysis, representations were extracted under a frozen-backbone protocol and evaluated via downstream task-specific MLPs/XGBoost heads.
| Dataset | Category | Baseline-10B-8k | Gengram-10B-8k | Absolute Gain (AUC) | External SOTA Reference (Evo2-40B / Genos-10B) |
|---|---|---|---|---|---|
| splice_sites_all | Structure Understanding | 0.8453 | 0.9009 | +0.0556 (+6.58%) | 0.9138 (Evo2-40B, 9.3T tokens) |
| multi_species_exon_classification | Structure Understanding | 0.8903 | 0.9832 | +0.0929 (+10.43%) | 0.9332 (Evo2-40B) / 0.9755 (Genos-10B, 2.2T) |
| multi_species_three_prime_utr | Structure Understanding | 0.8759 | 0.9601 | +0.0842 (+9.61%) | 0.9634 (Evo2-40B) |
| multi_species_five_prime_utr | Structure Understanding | 0.8397 | 0.9443 | +0.1046 (+12.46%) | 0.9895 (Evo2-40B) |
| human_enhancers_cohn | Regulation Prediction | 0.7605 | 0.8358 | +0.0753 (+9.90%) | 0.7756 (Evo2-40B) / 0.8552 (Genos-10B) |
| human_ocr_ensembl | Regulation Prediction | 0.7216 | 0.7714 | +0.0498 (+6.90%) | 0.7635 (Evo2-40B) |
| variant_effect_causal_eqtl_8K | Variant Effect & Clinical | 0.6078 | 0.6518 | +0.0440 (+7.24%) | 0.7054 (Evo2-40B) / 0.6773 (Genos-10B) |
| H3 | Epigenetic Profiling | 0.8934 | 0.9434 | +0.0500 (+5.60%) | 0.9311 (Evo2-40B) |
| H3K36me3 | Epigenetic Profiling | 0.7227 | 0.8039 | +0.0812 (+11.24%) | 0.8823 (Evo2-40B) |
| primate_mammal_species_8192 | Evolutionary Analysis | 0.7275 | 0.9611 | +0.2336 (+32.11%) | 0.8867 (Evo2-40B) / 0.9739 (Genos-10B) |
Note: Gengram-10B-8k was trained on only 200B tokens (vs. 2.2T for Genos-10B and 9.3T for Evo2-40B), yet achieved the absolute highest score across all evaluated models on Multi-species Exon Classification (0.9832 AUC).
In cross-context adaptation, Gengram-10B-8k (trained exclusively on 8k sequences) decisively outperformed Baseline-10B-32k (which underwent expensive 32k continued pretraining): scoring 0.9009 vs. 0.7809 on splice site identification (+15.3% relative gain) and 0.8039 vs. 0.6556 on H3K36me3 histone profiling (+22.6% relative gain).
Ablation Study¶
1. Window Size and Local Context Controls (1.2B MoE Model, 50B Tokens)
| Architecture / Branch Variant | Validation Loss | Relative to Baseline | Remarks |
|---|---|---|---|
| Baseline (w/o Gengram Memory) | 0.9600 | Reference baseline | Standard self-attention + MoE |
| Gengram (Window W=21, Layers 3, 6, 10) | 0.9350 | -0.0250 | Optimal full configuration; coincides with ~2 B-DNA helical turns |
| Gengram (Window W=6) | 0.9520 | -0.0080 | Window too narrow to capture multi-codon motifs |
| Gengram (Window W=30) | 0.9410 | -0.0190 | Window too broad; admits redundant intergenic noise |
| Parameter-Matched Conv-Local Control | 1.0200 | Severe loss spikes | Fails to match retrieval-based dynamic memory |
| Parameter-Matched K-mer Pooling Control | 1.0700 | Stagnant loss | Lacks adaptive gating and windowed interactions |
2. Component Ablations: Pooling and Gating (1.2B Validation Set)
| Component Configuration | Validation Loss | Description |
|---|---|---|
| Gengram Baseline (Mean Pooling + Gate) | 1.0177 | Default formulation; robust scale stability and high SNR |
| Max Pooling + Gate | 1.0217 | Susceptible to outlier k-mer embeddings |
| Attention Pooling + Gate | 1.0221 | Prone to overfitting within small local sequence windows |
| Mean Pooling w/o Gate | 1.0487 | Drastic degradation; proves gate is critical to filter background DNA |
3. K-mer Granularity Ablation across Downstream Tasks
| Dataset | Full Range k=1–6 | Short Motifs k=1–3 | Long Motifs k=4–6 | Takeaway |
|---|---|---|---|---|
| splice_sites_all | 0.7681 | 0.7641 | 0.7480 | Splice sites strongly hinge on short dinucleotide boundaries |
| human_ocr_ensembl | 0.7783 | 0.7744 | 0.7730 | Accessible chromatin requires multi-scale cooperative motifs |
| CPC_8192 | 0.9467 | 0.9365 | 0.9376 | Fusing multi-scale representations beats restricted lengths |
| variant_effect_causal_eqtl_8K | 0.6999 | 0.6916 | 0.6959 | Regulatory causal eQTLs depend on multi-tiered context |
Key Findings¶
- Windowed Retrieval Bypasses Long-Context Pretraining Tax: The 8k-context Gengram consistently matched or exceeded the 32k-context baseline model. Stacked causal window aggregations progressively broaden the receptive field across layers, allowing the network to assemble local motifs into global contextual representations without costly extended sequence pretraining.
- Gating is the Foundational Filter for Biological Sequences: Removing the scaled dot-product gate collapsed validation loss from 1.0177 to 1.0487. Because eukaryotic genomes consist largely of non-functional intergenic DNA, the gate acts as an indispensable threshold suppressing unaligned memory reads.
- LogitLens Demonstrates Accelerated Prediction Readiness: LogitLens-KL divergence analysis revealed that Gengram drives intermediate representations toward final predictive distributions much earlier (sharply decreasing KL divergence by Layer 3–6), relieving deeper layers from basic motif pattern reassembly.
- Inherent Stabilization of Sparse MoE Load Balancing: Across MoE configurations with 32, 64, and 128 experts (Top-2 routing), Gengram completely subdued router load-balancing loss oscillations. The gated memory module absorbs repetitive, non-informative k-mers that would otherwise bias routing gates, preventing pathological expert collapse.
Highlights & Insights¶
- Domain-Native Memory for Compact Alphabet Sequences: Unlike NLP models burdened by immense vocabularies and hash collisions, Gengram capitalizes on the compact discrete alphabet (\(\Sigma=5\)) to maintain exact, collision-free hash tables, achieving high-capacity motif retrieval with only ~60M parameter overhead across MHA, MLA, GQA, Dense, and MoE backbones.
- The 21-bp Window and B-DNA Structural Periodicity: The empirically selected optimal window (\(W=21\) bp) naturally coincides with two complete turns of the B-DNA double helix (10.4–10.5 bp/turn). Because spatial orientation along major and minor grooves governs transcription factor accessibility, this data-driven optimum mirrors fundamental physical constraints of the double helix.
- Emergent Biological Alignment in Latent Space: Probing the 3-gram memory space showed that reverse-complement (RC) codon pairs exhibit strong spatial clustering in Layer 3 and Layer 10 (reflecting double-stranded invariance), while Layer 6 maintains strand-specific asymmetry required for transcription and splicing. Furthermore, synonymous codons do not trivially collapse, preserving codon usage bias. Genome-wide in-silico core perturbation validated that Gengram activations are strictly sensitive to local nucleotide ordering rather than simple base composition.
Limitations & Future Work¶
- Static Multi-scale Hyperparameters: Both the \(k\)-mer range ($k=1\text
}6\() and window size (\)W=21$ bp) are globally fixed. Dynamic or learnable window expansion based on local chromatin states remains unexplored. - Diminished Returns on Isolated Single-Base Mutations: While Gengram delivers gains of up to 22.6% on motif-dominated tasks (e.g., exon classification, splice site detection), improvements on tasks governed strictly by isolated single-nucleotide variants without motif context are comparatively modest.
- System Memory Bandwidth Overhead: Enumerating and deduplicating multi-scale \(k\)-mers in software creates memory traffic. Scaling Gengram to multi-megabase contexts will necessitate fused CUDA kernel implementations to eliminate sliding-window tensor instantiation bottlenecks.
Related Work & Insights¶
- vs DeepSeek Engram: Engram targets natural language with split-head hashing to combat huge vocabularies, retrieving only a single preceding \(n\)-gram to serve as a sparsity axis complementary to MoE; Gengram targets genomic sequences with collision-free hashing, multi-scale ($k=1\text{6$) coverage, local window aggregation, and depthwise convolution smoothing, providing uniform acceleration across both Dense and MoE models.
- vs GENERator / Nucleotide Transformer (K-mer Tokenization): Conventional \(k\)-mer tokenizers chunk sequences at the input layer, degrading single-nucleotide spatial resolution and impairing fine-grained mutation analysis; Gengram preserves single-base tokenization at the backbone while incorporating \(k\)-mer priors through a parallel retrieval pathway.
- vs Caduceus / HyenaDNA (State Space Models): SSMs offer linear sequence scaling but still rely on parameter weights to implicitly deduce recurring biological motifs; Gengram proves that explicit memory retrieval pre-empts deep attention layers, allowing shorter nominal contexts (8k) to rival or exceed long-context (32k) models.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers conditional memory retrieval with collision-free multi-scale hash tables and local causal window aggregation for genomic foundation models.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across 18 diverse benchmarks, multiple attention variants, Dense/MoE backbones, parameter-matched controls, and biological interpretability analyses.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Impeccable narrative flow, mathematically precise formulation, and compelling biological grounding.]
- Value: ⭐⭐⭐⭐⭐ [Resolves the fundamental tension between single-base tokenization and multi-nucleotide motif learning, delivering an efficient, architecture-agnostic primitive for next-generation biological foundation models.]