Skip to content

Unlocking Multimodal Protein Language Models at Inference Time

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/EchoChou990919/mplm_inference
Area: LLM Reasoning
Keywords: Multimodal protein language models, Inference-time computation, Classifier-free guidance, Reward-guided beam search, Exploration-exploitation trade-off

TL;DR

Addressing the longstanding neglect of inference-time strategies in multimodal protein language models (pLMs), this work introduces a three-level control framework spanning sampling distributions, per-step logits, and parallel trajectories, demonstrating that training-free test-time guidance and internal reward beam search boost generalist pLMs to match or surpass specialized state-of-the-art models.

Background & Motivation

Proteins govern virtually all cellular biological functions, strictly obeying the sequence-structure-function paradigm where primary amino acid sequences fold into three-dimensional conformations to dictate biochemical activity. In recent years, generative multimodal protein language models (pLMs), such as ESM3 and the DPLM-2 family, have emerged as unified "all-in-one" foundation models by representing sequence and structure as parallel discrete token tracks. Formulated under discrete diffusion principles, these models aim to address diverse tasks including unconditional sequence-structure co-generation, motif scaffolding, structure prediction, and inverse folding within a unified parameter space. However, prior literature has focused almost exclusively on model architecture scaling and pre-training objectives, leaving inference-time generation governed by task-specific, unverified, and highly heuristic default sampling configurations.

This reliance on default inference protocols introduces severe suboptimality and distorts fair cross-model comparisons. For instance, when decoding the structure of protein 7PWE with ESM3, single-step greedy decoding at temperature = 0.0 successfully recovers the native fold, whereas standard multi-step sampling at temperature = 1.0 triggers catastrophic generation collapse. Furthermore, ESM3 natively lacks implementations for stochastic unmasking, token remasking, and synchronous sequence-structure decoding—features readily accessible in DPLM-2. As a consequence, ESM3 has frequently been labeled an underperforming baseline in multi-task evaluations simply due to crippled inference primitives. More fundamentally, distinct protein modeling tasks inhabit vastly different solution space topologies: unconditional generation demands broad exploratory capacity across the entire biophysical landscape, whereas structure prediction and inverse folding inhabit tightly constrained biophysical valleys requiring greedy exploitation.

To resolve these contradictions, the authors systematically formulate and explore the inference design space of multimodal pLMs through the lens of the exploration-exploitation trade-off. Rather than pursuing expensive parameter fine-tuning or relying on external heavy scoring models, they construct a bottom-up investigation hierarchy spanning distribution-level regularization, logit-level conditioning, and trajectory-level reward guidance. Core idea: formalize the inference-time design space of multimodal protein language models across three orthogonal control axes—standardizing vanilla sampling distribution primitives to match task landscape topologies, applying dual-track classifier-free guidance (CFG) to steer per-step logits toward condition-aligned regions, and employing internal pTM reward-guided beam search over parallel trajectories—unlocking the full latent capacity of foundation pLMs entirely at test time.

Method

Overall Architecture

The proposed inference framework formulates protein generation as reverse discrete diffusion denoising from a fully masked initial pair \((s^{(T)}, z^{(T)})\) toward a clean protein state \((s^{(0)}, z^{(0)})\). The pipeline operates across three bottom-up orthogonal stages: 1. Distribution Level (Vanilla Sampling): Standardizes diffusion primitives across base models and performs grid searches over denoising steps \(T\), unmasking modes (deterministic, stochastic, random), remasking mechanisms, temperature annealing schedules, and cross-track scheduling (synchronous vs. sequential) to center the baseline distribution; 2. Logit Level (Reward-Free Guidance, CFG): Translates each target task into conditional diffusion, extrapolating conditional and unconditional forward logits to steer local step transitions toward condition-consistent regions; 3. Trajectory Level (Reward-Guided Search): Transcends single-trajectory greediness by maintaining \(N\) parallel sampling paths, expanding them by factor \(B\) every \(K\) steps, unmasking them in a single step via QUICKUNMASK, and pruning candidates using internal structural and foldability pTM rewards.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Initial Masked Input<br/>(s^(T), z^(T)) and Task Conditions"] --> B["Standardized Vanilla Sampling<br/>Synchronous decoding + stochastic unmasking/remasking + annealing"]
    B --> C["Dual-Track Classifier-Free Guidance<br/>Evaluate conditional/unconditional logits, extrapolate with track scales"]
    C --> D{"Denoising Step t mod K == 0 ?"}
    D -->|Yes: Periodic Search Step| E["Internal Reward-Guided Beam Search<br/>Branch candidates Bx → QUICKUNMASK → Score via internal pTM"]
    D -->|No: Regular Step| F["Single-step parallel trajectory advance"]
    E --> G["Update Retained Beam Set T^(t-1)"]
    F --> G
    G --> H{"Diffusion Finished t == 0 ?"}
    H -->|No| C
    H -->|Yes| I["Global Optimal Trajectory Output<br/>Co-designed sequence and 3D structure"]

Key Designs

1. Standardized Vanilla Sampling: Reconciling disparate inference primitives with task-specific solution topologies

To eliminate evaluation discrepancies caused by missing implementations in original codebases, this design equips all base models (ESM3, DPLM-2, DPLM-2.1) with a unified discrete diffusion inference interface and systematically sweeps six operational dimensions. For open-ended, exploration-biased tasks such as unconditional co-generation, the optimal setup adopts long trajectories (\(T=L\) or \(500\)), stochastic unmasking via Gumbel perturbation, active remasking of low-confidence tokens, and high-to-low temperature annealing (\(1.0 \to 0.0\)), combined with synchronous track decoding to prevent unilateral sequence-structure drift. In contrast, for exploitation-dominated tasks like structure prediction and inverse folding, where sequence and structure are tightly biophysically constrained, the optimal configuration converges to short trajectories (\(T=8\) or \(L/8\)), deterministic/mildly stochastic unmasking, and near-zero decoding temperatures. This standardization establishes a calibrated baseline for subsequent guidance and search.

2. Dual-Track Classifier-Free Guidance: Steering step-wise logits to suppress modal collapse and enforce constraints

To prevent diffusion trajectories from drifting into physically implausible regions, this design generalizes classifier-free guidance to the dual-track discrete diffusion setting, applying track-wise guidance scales \(w_s\) and \(w_z\):

\[l_{m,(t)}^{\text{guided}} = l_{m,(t)}^{\text{uncond}} + w_m \cdot \left( l_{m,(t)}^{\text{cond}} - l_{m,(t)}^{\text{uncond}} \right), \quad m \in \{s, z\}\]

For unconditional co-generation, the formulation treats synchronous decoding as implicit cross-modal conditioning: conditional logits are obtained from \(f_\theta(s^{(t)}, z^{(t)})\), while unconditional logits drop the cross-track context (\(l_{s,(t)}^{\text{uncond}} = f_\theta(s^{(t)}, \emptyset)\) and \(l_{z,(t)}^{\text{uncond}} = f_\theta(\emptyset, z^{(t)})\)), actively penalizing repetitive residues and structural collapse. For motif scaffolding, a dual-condition CFG is instantiated where the unconditional branch simultaneously removes motif anchors and cross-track context. For structure prediction and inverse folding, the sequence or structure track is completely masked in the unconditional branch, respectively. This mechanism focuses probability mass onto constraint-aligned subspaces without introducing extra parameters.

3. Internal Reward-Guided Beam Search: Exploring parallel trajectories and scoring global quality via detokenizer pTM

Recognizing that token-level step logits only capture local likelihoods rather than overall fold viability, this design introduces trajectory-level beam search maintaining beam width \(N\). Every \(K\) steps, each retained trajectory is branched into \(B\) candidates via independent single-step sampling. Because intermediate candidates remain partially masked and cannot be evaluated directly, the algorithm executes QUICKUNMASK—a single-pass argmax forward step that fills all remaining masked positions. Global reward evaluation \(r(\cdot)\) strictly exploits the internal predicted template modeling (pTM) head embedded in ESM3's structural detokenizer: structural pTM directly scores structural tokens, while foldability pTM evaluates sequence tokens folded into structure via one greedy pass. The selection operator SELECT applies top-\(N\) pruning for exploitation tasks and uniform random sampling over candidates exceeding \(\text{pTM} > 0.8\) for unconditional generation, preserving diversity while enforcing structural validity.

Benchmark & Evaluation Protocol

The empirical investigation adheres strictly to community standard benchmark datasets and metrics: - Unconditional Co-Generation: 100 samples are generated across five chain lengths (100 to 500 residues). Samples with \(\text{scRMSD} < 2.0\text{ \AA}\) upon ESMFold refolding are classified as designable, reporting designable counts, Foldseek cluster numbers, pLDDT, scTM, and secondary structure composition; - Motif Scaffolding: Evaluated on 24 benchmark problems from MotifBench, counting candidates that concurrently satisfy global designability (\(\text{scRMSD} < 2.0\text{ \AA}\)) and motif preservation (\(\text{motif-RMSD} < 1.0\text{ \AA}\)), reporting solved problem counts (#Solved/24), overall success rate, and cluster diversity; - Structure Prediction & Inverse Folding: Assessed on CAMEO2022 and PDB Date Split datasets, tracking C\(\alpha\)-RMSD, TM-score, self-consistency scTM, and native sequence amino acid recovery (AAR); - Computational Cost: Floating-point operations (FLOPs) and wall-clock execution times are logged across all experiments executed on a dedicated NVIDIA H20 (96GB) GPU.

Key Experimental Results

Main Results

Performance improvements across investigation stages and comparisons against specialized state-of-the-art systems are summarized below:

Table 1: Benchmark Evaluation on Unconditional Protein Sequence-Structure Co-Generation

Model & Inference Protocol FLOPs (\(\times 10^{17}\)) Runtime (h) #Design ↑ #Clusters ↑ All pLDDT ↑ All scTM ↑
DPLM-2 Default 2.18 1.17 149.0 76.0 81.920 0.906
DPLM-2 CFG 6.54 3.22 162.4 78.4 82.242 0.913
DPLM-2.1 Default 2.15 1.02 143.4 86.6 85.116 0.904
DPLM-2.1 CFG 6.44 3.00 162.2 96.4 84.995 0.911
ESM3 Default (seq2struct) 1.25 1.68 52.8 34.4 61.387 0.660
ESM3 Vanilla (Optimized) 1.25 1.87 174.8 64.4 68.724 0.683
ESM3 CFG (\(w_s=2, w_z=3\)) 3.75 2.74 325.0 116.6 88.748 0.931
ESM3 Beam (Select=Random) 15.4 11.88 350.4 139.0 90.704 0.964
ESM3 Beam (Select=Best) - - 411.6 126.6 91.577 0.976
ESM3 Official SVDD - - 189.4 88.4 76.478 0.881
Specialist SOTA (La-Proteina) - - 383.0 134.7 83.770 0.953

Table 2: Multi-Stage Performance Progression Across Motif Scaffolding, Structure Prediction, and Inverse Folding

Task & Dataset Metric ESM3 Default ESM3 Vanilla ESM3 CFG ESM3 Beam Specialist Baselines (ESMFold / ProteinMPNN / La-Proteina)
Motif-Scaffolding #Solved / 24 ↑ 19.4 21.6 22.8 23.0 22.0 (La-Proteina scaff)
(MotifBench, 24 tasks) Success Rate ↑ 19.6% 30.9% 37.5% 49.9% 42.7% (La-Proteina scaff)
(MotifBench, 24 tasks) #Clusters ↑ 94.2 110.0 179.4 261.0 256.6 (La-Proteina scaff)
Structure Prediction RMSD (Å) ↓ 5.334 5.563 5.144 5.221 3.961 (ESMFold)
(CAMEO 2022) TM-score ↑ 0.860 0.868 0.870 0.872 0.898 (ESMFold)
Structure Prediction RMSD (Å) ↓ 4.053 4.872 3.969 3.744 2.661 (ESMFold)
(PDB Date Split) TM-score ↑ 0.882 0.885 0.889 0.893 0.931 (ESMFold)
Inverse Folding scTM ↑ 0.899 0.905 0.910 0.914 0.902 (ProteinMPNN)
(CAMEO 2022) AAR (%) 46.4 48.4 45.7 46.0 42.4 (ProteinMPNN)
Inverse Folding scTM ↑ 0.944 0.947 0.954 0.956 0.953 (ProteinMPNN)
(PDB Date Split) AAR (%) 49.6 52.0 49.1 49.3 46.8 (ProteinMPNN)

Ablation Study

Table 3: Ablation on Dual-Condition CFG Formulation for Motif Scaffolding (ESM3)

CFG Condition Formulation #Solved / 24 ↑ Overall Success Rate (%) ↑ Diversity #Clusters ↑
ESM3 Vanilla (No Guidance) 21.6 (21, 22) 30.9 ± 1.1% 110.0 (100, 124)
ESM3 CFG (Motif-only) 22.6 (22, 23) 35.3 ± 0.6% 146.8 (127, 158)
ESM3 CFG (Cross-Modal only) 22.4 (21, 23) 36.6 ± 1.4% 155.0 (144, 166)
ESM3 CFG (Dual Conditioning: Both) 22.8 (22, 23) 37.5 ± 1.0% 179.4 (165, 198)

Table 4: Explore-then-Select Temperature Sensitivity in Beam Search (ESM3)

Task & Evaluation Subset Denoising Temperature Setting Quality Performance Diversity Performance
Unconditional Co-Gen (Designable) Annealed (\(1.0 \to 0.0\)) #Design: 393.8 #Clusters: 117.0
Unconditional Co-Gen (Designable) Constant Exploration (\(1.0\)) #Design: 350.4 #Clusters: 139.0
Motif Scaffolding (MotifBench) Modest Exploration (\(0.7 \to 0.0\)) Success Rate: 48.9 ± 0.9% #Clusters: 214.6
Motif Scaffolding (MotifBench) High Exploration (\(1.0 \to 0.0\)) Success Rate: 49.9 ± 0.7% #Clusters: 261.0

Key Findings

  • Overturning consensus on foundation pLM capability: Under default sampling, ESM3 achieved only 52.8 designable samples and was perceived as an uncompetitive generative backbone. Aligning basic vanilla sampling primitives increased this to 174.8; adding cross-modal CFG doubled it to 325.0; and integrating reward-guided beam search yielded 350.4 designable samples and 139.0 unique clusters, surpassing the specialist model La-Proteina (134.7 clusters). This demonstrates that default inference protocols constitute a major hidden confounder in academic benchmarks.
  • Guidance scaling correlates with base model internalization: CFG delivered substantial gains for ESM3 but only marginal improvements for DPLM-2 and DPLM-2.1. DPLM-2 was adapted from a sequence-only model, resulting in asymmetric track internalization, whereas ESM3 was natively pre-trained on multi-track data, providing balanced conditioning sensitivity across sequence and structure tracks.
  • The Explore-then-Select paradigm: Beam search achieves maximum efficacy when the underlying sampler is deliberately tuned to be slightly more exploratory than its single-trajectory optimum (e.g., maintaining higher sampling temperature or stochastic unmasking). Excessive greediness in candidate generation creates redundant beams, rendering trajectory-level reward discrimination ineffective.

Highlights & Insights

  • Orthogonal three-level control space: Systematically categorizes inference choices into distribution shaping, step-level logit steering, and trajectory-level selection, creating a structured hierarchy for test-time optimization.
  • Closed-loop internal reward guidance: Completely circumvents the need for external, computationally heavy scoring oracles by repurposing ESM3's native structural detokenizer pTM head for both structure and foldability evaluation.
  • Validation of test-time compute in molecular biology: Establishes that scaling inference computation via guidance and search provides a viable, training-free alternative to model parameter scaling for discrete diffusion pLMs.

Limitations & Future Work

  • Increased inference compute overhead: While CFG incurs a manageable 2-3x FLOP overhead that translates to minimal wall-clock delay under spare GPU capacity, beam search expands FLOPs and runtime by an order of magnitude, lacking an explicit resource-constrained Pareto optimization scheme.
  • Evaluation confined to in silico metrics: All findings rely on computational proxies (ESMFold refolding confidence, TM-scores, Foldseek clustering) without wet-lab synthesis, expression, or binding affinity validation.
  • Lack of external environment integration: The current search loop operates strictly within internal model logits, omitting interaction with molecular dynamics simulators, physics-based energy functions, or laboratory active learning pipelines.
  • vs DPLM-2 / DPLM-2.1: DPLM-2 emphasizes architectural innovations like lookup-free quantization (LFQ) but relies on unverified sampling heuristics; this paper demonstrates that standardized inference and CFG can dramatically improve its generation without structural retraining.
  • vs ESM3 Official SVDD: ESM3's native soft value-based decoding operates sequentially atop default unaligned decoding; this work shows that combining synchronous sampling, CFG, and beam search provides strictly superior, complementary performance.
  • vs La-Proteina / ProteinMPNN: While task-specific generative architectures embed explicit geometric inductive biases, this work proves that a generalist multimodal pLM equipped with test-time guidance matches or exceeds specialist architectures across co-generation, scaffolding, and inverse folding.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First systematic exploration and formalization of the inference design space for multimodal pLMs.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 3 base models, 4 fundamental tasks, 6 sampling dimensions, and detailed runtime/FLOP profiles.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear pedagogical progression, rigorous mathematical formulations, and thorough ablation analyses.
  • Value: ⭐⭐⭐⭐⭐ Uncovers inference protocol as a crucial confounding variable and offers concrete test-time recipes for biomolecular design.