Skip to content

SoftGene: Protein Language Model-Enhanced Soft Prompt Tuning for Genomic Sequence Modeling

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/KU-AI4H/softgene
Area: Medical NLP
Keywords: Gene Set Annotation, Protein Language Models, Soft Prompt Tuning, Hierarchical Attention, Computational Biology

TL;DR

SoftGene introduces a hybrid prompting framework that hierarchically encodes protein amino acid sequences and alternative splice isoforms via ESM-2 into continuous soft prompt tokens, conditioning a local LLM alongside auxiliary biological context to generate accurate and interpretable gene set annotations.

Background & Motivation

Gene set analysis is an indispensable pillar of modern functional genomics and systems biology, serving to group co-regulated genes and translate high-throughput sequencing measurements into coherent biological processes and cellular pathways. Historically, annotating these gene cohorts with concise, descriptive summaries has relied either on labor-intensive manual curation by domain experts or on statistical enrichment methods such as Over-Representation Analysis (ORA) and Gene Set Enrichment Analysis (GSEA). However, manual annotation scales poorly with the staggering volume of emerging genomic data, while statistical enrichment techniques remain intrinsically bounded by pre-existing database catalogs that exhibit substantial biases toward well-studied pathways and leave uncharacterized gene associations poorly explained.

The rapid rise of Large Language Models (LLMs) has opened up promising opportunities for automated biological reasoning and text-based gene function summarization. Nevertheless, prevailing LLM-based solutions typically treat genes as bare alphanumeric tokens within hard prompts or coordinate external multi-agent database searches. Crucially, these frameworks neglect the foundational biological reality that genes encode proteins, whose specific linear amino acid sequences govern molecular interactions, subcellular localization, and downstream metabolic roles. Disregarding protein primary structures exposes text-only LLMs to hallucinated functional claims, while generic narrative outputs lack verifiable, feature-level decision paths linking specific genes or splice isoforms to predicted annotations.

Bridging the divide between discrete natural language descriptions and continuous biochemical sequence features requires conditioning language generation on multi-level molecular representations. Core idea: encode amino acid sequences using a frozen protein language model, aggregate splice isoforms and member genes through a two-level hierarchical attention pooling encoder into a unified gene set embedding, and project this embedding into continuous soft prompt tokens that steer a local language model to synthesize grounded, interpretable gene set annotations.

Method

Overall Architecture

SoftGene decouples the gene set annotation pipeline into two coordinated modules: a Hierarchical Attention-Based Gene Set Encoder and a Hybrid Gene Set Soft Prompting Module. The hierarchical encoder extracts amino acid-level embeddings from a frozen protein language model (ESM-2) and aggregates them through successive attention pooling stages across splice isoforms and individual genes to yield a fixed-size gene set representation. The hybrid prompting module subsequently projects this vector into continuous soft prompt tokens, which are concatenated with task instructions and external context before steering an open-source autoregressive LLM (e.g., Llama-3.1-8B) to generate the target descriptive annotation.

Given an input gene set \(\mathcal{G} = \{g_1, g_2, \dots, g_M\}\), each member gene \(g_i\) corresponds to \(K_i\) annotated protein splice isoforms \(\{p_{i,1}, \dots, p_{i,K_i}\}\). The amino acid sequence of each isoform is mapped to contextual representations via ESM-2 and mean-pooled into a protein vector \(\hat{p}_{i,j}\). An isoform-level attention pooling mechanism computes importance weights across isoforms to produce a composite gene embedding \(\hat{g}_i\). A set-level attention pooling layer then aggregates the gene embeddings into the global gene set vector \(\hat{\mathcal{G}}\). Finally, a multi-layer perceptron maps \(\hat{\mathcal{G}}\) into continuous virtual tokens \(S\), which condition the generation alongside instruction prompt \(H_{\text{inst}}\) and commercial LLM context \(H_{\text{cont}}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Gene Set<br/>G = {g1, g2, ..., gM} and protein isoform sequences"] --> B["ESM-2 Amino Acid Sequence Encoding & Mean Pooling<br/>Extract fixed-dimensional isoform embeddings"]
    B --> C["Hierarchical Attention-Based Gene Set Encoder<br/>Two-level adaptive pooling over isoforms and genes"]
    C --> D["Hybrid Gene Set Soft Prompting Module<br/>MLP soft prompt projection + text context concatenation"]
    D --> E["Frozen Local LLM Autoregressive Generation<br/>Supervised teacher-forcing cross-entropy training"]

Key Designs

1. Hierarchical Attention-Based Gene Set Encoder: Addressing isoform noise and uniform gene weighting to construct a biochemically grounded, interpretable set representation
Standard NLP approaches flatten gene sets into string lists, ignoring variable splice variants and treating all member genes equally regardless of pathway centrality. SoftGene addresses this by processing each isoform sequence \(p_{i,j} = (a_1, \dots, a_{L_{i,j}})\) through frozen ESM-2 and averaging residue embeddings: $$ \hat{p}{i,j} = \frac{1}{L}} \sum_{k=1}^{L_{i,j}} \hat{a{i,j}^k $$ Because certain splice variants mediate primary biological functions while others represent non-functional or minor transcripts, an isoform-level attention pooling layer adaptively scores each isoform: $$ \hat{g}i = \text{AttentionPooling}}}\left({\hat{p}{j=1}^{K_i}\right) = \sum}^{K_i} \alpha_{i,j} \hat{p{i,j}, \quad \alpha} = \frac{\exp(\text{MLP{\text{iso}}(\hat{p}}))}{\sum_{\ell=1}^{K_i} \exp(\text{MLP{\text{iso}}(\hat{p} $$ A secondary, independently parameterized set-level attention pooling layer then aggregates }))\(\{\hat{g}_i\}_{i=1}^M\) into a global gene set embedding \(\hat{\mathcal{G}} = \text{AttentionPooling}_{\text{set}}\left(\{\hat{g}_i\}_{i=1}^M\right)\). In addition to filtering out noise, the learned attention weights directly provide a transparent, feature-level decision path quantifying the relative biological contribution of each individual gene and protein isoform.

2. Hybrid Gene Set Soft Prompting Module: Bridging primary sequence embeddings and natural language reasoning through continuous soft tokens and textual context
Continuous latent embeddings from protein encoders cannot be fed directly into discrete autoregressive language models without modality adaptation. SoftGene bridges this semantic gap by introducing a trainable multi-layer perceptron projector \(\text{MLP}_{\text{soft}}\) that maps \(\hat{\mathcal{G}}\) into a sequence of \(n\) continuous virtual prompt tokens \(S \in \mathbb{R}^{n \times d_{\text{LLM}}}\): $$ S = \text{MLP}_{\text{soft}}(\hat{\mathcal{G}}) $$ Recognizing that smaller local language models may lack broad encyclopedic context for rare pathways, the framework combines \(S\) with hard text prompts. Specifically, an instruction prompt \(H_{\text{inst}}\) specifies stylistic guidelines, while an auxiliary context prompt \(H_{\text{cont}}\) generated by an external LLM provides high-level narrative background. The local LLM processes the concatenated sequence \([S; H_{\text{inst}}; H_{\text{cont}}]\) to generate candidate annotations, successfully combining continuous biophysical sequence signals with macro-level biomedical concepts.

Loss & Training

During the optimization phase, both the protein language model (ESM-2) and the local generative language model (Llama-3.1-8B) remain entirely frozen. Gradients are propagated strictly through the soft prompt projector \(\text{MLP}_{\text{soft}}\) and the multilayer perceptrons inside the hierarchical attention pooling layers. The model is trained using teacher forcing on reference annotations \(y = (y_1, y_2, \dots, y_T)\) by minimizing token-level cross-entropy loss: $$ \mathcal{L}{\text{LLM}} = -\sum) $$ Key hyperparameters include a learning rate }^T \log P(y_t \mid y_{<t}, S, H_{\text{inst}}, H_{\text{cont}\(\alpha = 1 \times 10^{-3}\) for the hierarchical encoder, \(\beta = 5 \times 10^{-4}\) for the soft prompt module, batch size of 4 with 4 gradient accumulation steps, and a soft prompt length of \(n = 10\) virtual tokens. The model is trained for 1 epoch on Gene Ontology (GO) and 5 epochs on MSigDB on NVIDIA A6000 GPUs using Hugging Face Accelerate.

Key Experimental Results

Main Results

Evaluation was conducted on two benchmark datasets: Gene Ontology (GO, containing 18,889 gene sets across Biological Process, Cellular Component, and Molecular Function) and the Molecular Signatures Database (MSigDB C2:CP canonical pathways, containing 2,461 gene sets). Baselines span standard prompt engineering, prompt-tuning methods, and agentic retrieval-augmented architectures.

Dataset Configuration / Method METEOR ROUGE-L BERTScore-F1
Gene Ontology (GO) Zero-shot (Llama-3.1-8B) \(0.024 \pm 0.001\) \(0.133 \pm 0.001\) \(0.773 \pm 0.000\)
Few-shot (Llama-3.1-8B) \(0.018 \pm 0.001\) \(0.164 \pm 0.003\) \(0.740 \pm 0.001\)
One-shot CoT (Llama-3.1-8B) \(0.065 \pm 0.001\) \(0.096 \pm 0.002\) \(0.761 \pm 0.001\)
Gene Symbol Soft Prompting \(0.113 \pm 0.002\) \(0.158 \pm 0.004\) \(0.778 \pm 0.001\)
P-Tuning \(0.148 \pm 0.130\) \(0.196 \pm 0.130\) \(0.783 \pm 0.042\)
Prefix Tuning \(0.142 \pm 0.082\) \(0.173 \pm 0.094\) \(0.766 \pm 0.048\)
Discovera (Agentic) \(0.107 \pm 0.003\) \(0.144 \pm 0.005\) \(0.769 \pm 0.001\)
TALISMAN (Agentic) \(0.108 \pm 0.006\) \(0.146 \pm 0.002\) \(0.766 \pm 0.001\)
GeneAgent (Agentic) \(0.071 \pm 0.003\) \(0.116 \pm 0.002\) \(0.768 \pm 0.001\)
SoftGene (Llama 3.1 8B, Ours) \(\mathbf{0.220 \pm 0.033}\) \(\mathbf{0.267 \pm 0.029}\) \(\mathbf{0.806 \pm 0.006}\)
MSigDB (C2:CP) Zero-shot (Llama-3.1-8B) \(0.087 \pm 0.014\) \(0.127 \pm 0.009\) \(0.762 \pm 0.002\)
Few-shot (Llama-3.1-8B) \(0.100 \pm 0.025\) \(0.136 \pm 0.017\) \(0.771 \pm 0.008\)
One-shot CoT (Llama-3.1-8B) \(0.086 \pm 0.010\) \(0.112 \pm 0.003\) \(0.762 \pm 0.001\)
Gene Symbol Soft Prompting \(0.105 \pm 0.022\) \(0.138 \pm 0.021\) \(0.771 \pm 0.008\)
P-Tuning \(0.113 \pm 0.058\) \(0.161 \pm 0.061\) \(0.716 \pm 0.026\)
Prefix Tuning \(0.080 \pm 0.006\) \(0.113 \pm 0.021\) \(0.756 \pm 0.007\)
Discovera (Agentic) \(0.092 \pm 0.003\) \(0.108 \pm 0.003\) \(0.768 \pm 0.007\)
TALISMAN (Agentic) \(0.086 \pm 0.005\) \(0.107 \pm 0.001\) \(0.760 \pm 0.001\)
GeneAgent (Agentic) \(0.030 \pm 0.006\) \(0.117 \pm 0.003\) \(0.764 \pm 0.002\)
SoftGene (Llama 3.1 8B, Ours) \(\mathbf{0.157 \pm 0.005}\) \(\mathbf{0.215 \pm 0.001}\) \(\mathbf{0.795 \pm 0.003}\)

Ablation Study

To dissect the individual contributions of the continuous soft prompt, external contextual guidance, and pretrained protein sequence embeddings, systematic ablation experiments were conducted with the Llama-3.1-8B backbone.

Configuration GO METEOR GO ROUGE-L GO BERTScore MSigDB METEOR MSigDB ROUGE-L MSigDB BERTScore Description
SoftGene (Full Model) \(\mathbf{0.220 \pm 0.033}\) \(\mathbf{0.267 \pm 0.029}\) \(\mathbf{0.806 \pm 0.006}\) \(\mathbf{0.157 \pm 0.005}\) \(\mathbf{0.215 \pm 0.001}\) \(\mathbf{0.795 \pm 0.003}\) Full pipeline with all components
w/o Soft Prompt (\(S\)) \(0.020 \pm 0.000\) \(0.065 \pm 0.001\) \(0.707 \pm 0.001\) \(0.006 \pm 0.004\) \(0.045 \pm 0.002\) \(0.704 \pm 0.001\) Relying solely on hard prompt text inputs
w/o Context (\(H_{\text{cont}}\)) \(0.182 \pm 0.024\) \(0.232 \pm 0.028\) \(0.795 \pm 0.005\) \(0.145 \pm 0.007\) \(0.202 \pm 0.020\) \(0.790 \pm 0.003\) Excluding auxiliary commercial LLM summary
w/o Protein Embeddings \(0.199 \pm 0.015\) \(0.252 \pm 0.017\) \(0.803 \pm 0.002\) \(0.097 \pm 0.017\) \(0.135 \pm 0.025\) \(0.774 \pm 0.012\) Replacing PLM vectors with random Gaussian noise

Key Findings

  • Soft prompt conditioning is the foundational performance driver: Removing the soft prompt \(S\) causes catastrophic degradation across all metrics (GO METEOR collapses from 0.220 to 0.020, and MSigDB METEOR plunges from 0.157 to 0.006). This underscores that structured continuous representations of gene sets provide vital inductive biases that cannot be compensated by textual prompts alone.
  • Protein embeddings provide amplified gains in heterogeneous pathways: In MSigDB, replacing real protein embeddings with random vectors drops METEOR substantially from 0.157 to 0.097 (a drop of 38.2%). While GO terms are clustered by clean predefined annotations where gene co-occurrence alone carries strong predictive signal, MSigDB gene sets derive from heterogeneous biological signatures where symbolic names are less reliable cues and amino acid sequence embeddings provide indispensable discriminative power.
  • Differential benefit across biological domains: Domain-stratified ablation on GO reveals that protein sequence embeddings deliver the largest performance leap in the Cellular Component (CC) namespace (\(\Delta\text{METEOR} = +0.062\), \(\Delta\text{ROUGE-L} = +0.061\)), moderate gains in Molecular Function (MF) (\(\Delta\text{METEOR} = +0.023\)), and negligible difference in Biological Process (BP) (\(\Delta\text{METEOR} = +0.001\)). This aligns precisely with molecular biology: amino acid sequences directly dictate transmembrane helices and structural localization motifs (CC), whereas complex biological processes (BP) are governed by higher-order regulatory networks.
  • Robustness under gene-disjoint evaluation: In a rigorous gene-disjoint split where test genes do not appear in the training set, SoftGene achieves a METEOR of \(0.205\) and BERTScore of \(0.801\) on GO (compared to \(0.220\) and \(0.806\) under random split), confirming that the network learns transferable sequence-level motifs rather than memorizing gene identity co-occurrences.

Highlights & Insights

  • Center-Dogma aligned hierarchical pooling: The architecture mirrors biological reality by mapping amino acids to protein isoforms via mean pooling, isoforms to genes via gene-level attention pooling, and genes to gene sets via set-level attention pooling, preserving multiscale biological granularity.
  • Unsupervised discovery of functional splice variants: Without any isoform-level supervision, the model learns to assign the highest attention weight to PRKG1\(\alpha\) (ENSP00000363092.5) in vascular smooth muscle relaxation (GO:0045986), capturing the exact functional variant that binds to smooth muscle stress fibers while downweighting non-specific isoforms.
  • Linear-time computational efficiency: By caching whole-proteome ESM-2 embeddings once (taking 1.83 hours on a single A6000 GPU) and employing query-only attention with sparse scatter operations, the hierarchical encoder scales linearly \(\mathcal{O}(n)\) with gene set size, evaluating massive sets of over 10,000 genes in less than 500 ms.

Limitations & Future Work

  • Penalization from semantic granularity mismatches: Qualitative failure analysis indicates that reference-based lexical metrics unfairly penalize semantically accurate outputs that diverge in granularity (e.g., predicting valid parent term "regulation of circadian rhythm" against ground truth "regulation of circadian sleep/wake cycle").
  • Substrate specificity smoothing in catalytic pockets: The model occasionally recognizes the correct transport mechanism but misidentifies the exact biochemical substrate (e.g., predicting "sodium:sugar symporter" instead of "fucose transmembrane transport"), suggesting that global sequence pooling may smooth out subtle residue-level catalytic pocket mutations.
  • Absence of wet-lab validation and uncertainty estimation: Current evaluations are tied to existing curated databases, which carry historical annotation biases; future iterations should incorporate calibrated uncertainty quantification and wet-lab experimental validation for newly discovered gene cohorts.
  • vs Prompt Engineering & Agentic LLMs (Zero-Shot / CoT / GeneAgent / Discovera): Hard-prompted LLMs and search agents rely on symbolic gene tokens and external API lookups, incurring heavy inference latencies and frequent hallucinations; SoftGene directly grounds generation on biological sequences, outperforming GeneAgent on GO METEOR (\(0.220\) vs. \(0.071\)).
  • vs Standard Parameter-Efficient Prompt Tuning (P-Tuning / Prefix-Tuning): Standard prompt tuning optimizes static virtual vectors decoupled from input sequences; SoftGene dynamically computes soft prompts from protein primary sequences via hierarchical attention, outperforming P-Tuning (\(0.220\) vs. \(0.148\) METEOR on GO).
  • vs Protein Language Models (ESM-2 / ProtBERT / ProtGPT2): Traditional PLMs focus exclusively on representation learning or single-sequence design; SoftGene builds a conditioning bridge that translates hierarchical PLM sequence embeddings into coherent natural language summaries via soft prompting.

Rating

  • Novelty: โญโญโญโญโ˜† Elegantly adapts protein language model embeddings into continuous soft prompts via a two-level attention pooling hierarchy tailored to biological structures.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously tested across GO and MSigDB, incorporating fine-grained domain ablations, gene-disjoint splits, and biological case studies.
  • Writing Quality: โญโญโญโญโญ Well-structured, technically crisp, with clear illustrations of attention mechanisms and molecular mechanisms.
  • Value: โญโญโญโญโญ Bridges molecular bioinformatics with natural language processing, establishing an open-source, reproducible baseline for interpretable gene set interpretation.