MDForge: Agentic Molecular Dynamics Pipeline Design under Sparse Simulator Feedback¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Zehong-Wang/MDForge
Area: Physics & Scientific Computing
Keywords: molecular dynamics, large language model agent, verbal reinforcement learning, process supervision, free energy calculation
TL;DR¶
MDForge frames molecular dynamics (MD) pipeline design as open-ended code generation driven by verbal reinforcement learning (Verbal RL), utilizing a multi-expert PRISM arbitration debate to densify sparse, expensive simulation feedback into stage-wise diagnostics and localized code patches, matching human expert pipelines on SAMPL benchmarks and prospectively discovering a picomolar binder verified by wet-lab competition NMR.
Background & Motivation¶
Molecular dynamics (MD) simulation is the foundational in-silico instrument across computational chemistry, structural biology, and material science, integrating first-principles equations of motion to track molecular phenomena at atomistic resolution. By tracing high-dimensional phase space trajectories over time, MD uncovers transient conformational states, binding free energies, and solvation thermodynamics that are inaccessible or prohibitively expensive to capture in wet-lab experiments. Nevertheless, setting up a dependable, physically sound MD workflow for an unstudied molecular system remains an expert-dependent bottleneck: the joint configuration of initial binding poses, force fields, water models, equilibration schedules, umbrella sampling restraint ladders, and statistical mechanics estimators involves non-trivial physical coupling, meaning pipeline templates optimized for one chemical class rarely transfer to another. While data-driven deep learning predictors predict binding constants rapidly, they lack atomistic mechanistic explanations and degrade silently when evaluated outside their training distributions, leaving rigorous physics-based MD simulation indispensable.
Recent efforts in autonomous scientific agents seek to eliminate manual workflow construction using large language models (LLMs). Prominent tools such as MDCrow, MDAgent, and DynaMate rely on orchestrating predefined toolboxes via structured function calls. However, this tool-calling paradigm restricts the agent to rigid parameter slots within pre-selected modules, failing to reflect how computational chemists actually workโwriting expressive, customized Python code tailored to each target system. More severely, existing agent architectures lack mechanisms to learn iteratively from simulation diagnostics: running an end-to-end free-energy pipeline consumes substantial GPU hours per molecule, restricting realistic design tasks to tiny trial budgets (e.g., fewer than 5 trials). With standard reinforcement learning requiring thousands of rollouts and scalar terminal rewards offering zero subsystem attribution, conventional agentic search methods face severe trial-to-trial instability and catastrophic collapse.
To bridge the gap between open-ended pipeline authoring and the extreme scarcity of simulation feedback, this paper formalizes MD pipeline design as program synthesis governed by in-context verbal reinforcement learning (Verbal RL). The foundational insight is that molecular simulations possess an intrinsic temporal-staged structure and compartmentalized physics: pipeline crashes and statistical divergence leave concrete diagnostic footprints across force field consistency, phase space overlap, and energy drift. Core idea: propose Process-Reward Interpretation via Subsystem Mediation (PRISM), a framework that decomposes sparse terminal simulation feedback across preparation, equilibration, production, and analysis stages, deploying a three-expert panel (Force Field, Sampling, Analysis) in a two-round debate to generate reputation-weighted, subsystem-attributable typed code repairs.
Method¶
Given a target molecular system class \(\mathcal{T}=\{s_1, \dots, s_M\}\) and experimental observables \(\{y_{\text{exp}}(s_m)\}\), MDForge searches for an executable four-stage Python pipeline \(\pi \in \Pi\) that minimizes binding affinity prediction error within an austere budget of \(N=5\) successful trials.
Overall Architecture¶
The operational architecture of MDForge comprises two tightly integrated feedback loops: an inner sandbox execution pipeline and an outer PRISM process-reward densification loop. Prior to code generation, an initial design consultation across physical domains seeds the initial pipeline proposal. The emitted four-stage scripts (01_prep.py, 02_equilibrate.py, 03_production.py, 04_analysis.py) pass through Layer-1 static checks and an automated debugging Engineer agent before execution. During sandbox execution, stage boundaries capture raw structured physics diagnostics \(D_\pi\). Upon run termination or numeric divergence, three specialized LLM agents debate the physical evidence across two deliberation rounds, yielding a reputation-weighted, typed critique that guides the Code Agent to apply surgical in-context code edits.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Molecular System Input T<br/>(complex structure & free-energy target)"] --> B["Code Agent Context Update<br/>(fast-weight in-context rewrite of pipeline ฯ)"]
B --> C["Stage-Wise Physics Diagnostics Dฯ<br/>(Prep/Equil/Prod/Analy boundary metrics)"]
C --> D["Multi-Expert Subsystem Debate<br/>(Force-Field/Sampling/Analysis 2-round review)"]
D --> E["Reputation-Weighted Critique Aggregator Aฯ<br/>(Beta-Bernoulli consistency-driven updating)"]
E -->|typed localized code fix c_post| B
E --> F["Optimized Molecular Dynamics Pipeline ฯ*<br/>(converged free energy & wet-lab validation)"]
Key Designs¶
1. Code Agent Context Update: Gradient-free fast-weight policy adaptation via verbal RL To navigate the prohibitive cost of multi-hour GPU simulations where gradient backpropagation is impossible, MDForge casts trial-to-trial policy improvement into verbal reinforcement learning via context rewriting \(C_t \mapsto C_{t+1}\), behaving as an in-context fast-weight update over a frozen backbone model \(\theta\). At trial \(t\), the Code Agent operates on a tightly structured context bundle \(C_t = \{T, \pi_t, K_t, H_t\}\): \(T\) provides the invariant task specification, \(\pi_t\) contains the full multi-file code from the previous trial (encouraging localized diff repairs over volatile full-script rewrites), \(K_t = \{c_{l1,t}, c_{pre,t}, c_{post,t}\}\) supplies typed critiques from static linters and multi-expert panels, and \(H_t\) aggregates compact execution histories and benchmark metrics. By converting multi-dimensional simulation traces into actionable natural-language programming directives, MDForge maintains monotonic optimization over open-ended code spaces within minimal trials.
2. Stage-Wise Physics Diagnostics: Deconstructing execution boundaries to yield rich physical evidence Standard reinforcement learning operating on a single scalar free energy error suffers from severe temporal and structural credit assignment failure. PRISM resolves this by instrumenting the four canonical MD stages (\(K=4\)) with native diagnostic hooks that harvest ground-truth observables directly from simulation engines (OpenMM, AmberTools, pymbar), preventing LLM hallucinations. Specifically, preparation (01_prep.py) outputs antechamber charge assignments, frcmod parameter gaps, and tleap topology warnings; equilibration (02_equilibrate.py) reports NVT/NPT temperature, pressure, and density drift traces; production (03_production.py) monitors replica exchange acceptance rates and umbrella sampling phase-space overlap; and analysis (04_analysis.py) verifies the full \(K \times K\) MBAR overlap matrix and block-averaged variance thresholds. This structured tuple \(D_\pi\) transforms the expensive terminal outcome into immediate stage-attributed diagnostic evidence without incurring auxiliary GPU overhead.
3. Multi-Expert Subsystem Debate: Non-overlapping jurisdictions with cross-visibility arbitration When physical simulations fail, single generalist critics frequently confound superimposed root causesโsuch as mistaking inadequate alchemical sampling for force field charge over-polarization. PRISM delegates critique generation to a panel of \(J=3\) specialized agents possessing non-overlapping physical jurisdictions: the Force-Field Expert oversees parameterization, atom typing, and water/ion compatibility; the Sampling Expert governs APR/FEP lambda ladders, integration time steps, and equilibration floor lengths; and the Analysis Expert directs thermodynamic cycle algebra, standard-state restraint corrections, and statistical estimators. The panel executes a two-round debate protocol in both pre-execution screening and post-execution review: Round 1 collects independent, parallel structured reviews; Round 2 shares all peer critiques across full visibility, enabling cross-subsystem consensus. The aggregator synthesizes these inputs into a single typed critique pinpointing the exact file, script block, and physical patch required.
4. Reputation-Weighted Critique Aggregator: Beta-Bernoulli posterior calibration across trials To prevent noisy, hallucinated, or uncalibrated expert opinions from destabilizing the code generation loop, MDForge maintains a dynamic reliability prior for each specialist agent. Each expert \(i\) outputs a categorical label \(\ell_i \in \{\text{PASS}, \text{UNCERTAIN}, \text{FAIL}\}\), confidence \(\kappa_i \in [0, 1]\), and strategic insight. The aggregator evaluates a consensus score weighted by confidence and expert reputation \(\rho_i\): $\(S = \frac{\sum_{i=1}^J \rho_i \kappa_i s(\ell_i)}{\sum_{i=1}^J \rho_i \kappa_i}\)$ where \(s(\text{PASS})=1\), \(s(\text{UNCERTAIN})=0.5\), and \(s(\text{FAIL})=0\). Expert reputations follow a conjugate Beta-Bernoulli update rule: starting from a uniform prior \(\text{Beta}(\alpha_i=1, \beta_i=1)\), after each trial the orchestrator compares each expert's pre-execution prediction against the post-execution benchmark outcome. When the expert's judgment agrees with empirical evidence, \(\alpha_i \leftarrow \alpha_i + 1\); otherwise, \(\beta_i \leftarrow \beta_i + 1\). The deterministic reputation weight reflects the posterior mean: $\(\rho_i = \frac{\alpha_i}{\alpha_i + \beta_i}\)$ This Bayesian updating mechanism dynamically suppresses historically miscalibrated experts, stabilizing aggregation within tiny trial budgets.
A Worked Example¶
Consider a representative recovery trace on SAMPL4 CB[7]. In trial 2, the generated pipeline completed production sampling, but at stage 4 the simulator diagnostic emitted mbar_overlap_above_0p03: False inside stage_04_result.json. During the PRISM debate, the Analysis Expert identified that the MBAR estimator was exhibiting false convergence due to adjacent window overlap falling below reliable statistical thresholds, recommending an explicit runtime safeguard. Rather than re-initializing the entire four-stage pipeline, the Code Agent consumed this critique and inserted a localized one-line assertion in 04_analysis.py: if min_adj < 0.05: raise RuntimeError(...). This targeted intervention halted silent false convergences, forcing subsequent trials to densify the lambda schedule between \(\lambda=0.3-0.5\), progressively lifting held-out Kendall \(\tau\) from 0.20 to 0.56 by trial 5.
Key Experimental Results¶
Main Results¶
Evaluations were conducted on three benchmark host-guest challenges from the SAMPL series: SAMPL4 CB[7] (\(n_{\text{held}}=10\)), SAMPL4 OAH (\(n_{\text{held}}=5\)), and SAMPL5 CBClip (\(n_{\text{held}}=6\)). For each system, 4 training guests spanning the experimental affinity spectrum provided verbal feedback, while remaining compounds served as held-out test sets. Baselines and MDForge operated under a hard budget of \(N=5\) successful runnable trials.
| Host Benchmark | Pipeline Strategy / Baseline | Runnable Trials | Train \(R^2\) | Train Kendall \(\tau\) | Test \(R^2\) | Test Spearman \(\rho\) | Test Kendall \(\tau\) |
|---|---|---|---|---|---|---|---|
| CB[7] (\(n=10\)) | No feedback (one-pass) | 0/5 (\(\times\)) | โ | โ | โ | โ | โ |
| LLM critic (zero-shot critique) | 0/5 (\(\times\)) | โ | โ | โ | โ | โ | |
| Step-level feedback (ReAct style) | 1-2/5 (โณ) | 0.16 | 0.33 | 0.14 | 0.21 | 0.16 | |
| Trial-level feedback (Reflexion) | 5/5 (\(\checkmark\)) | 0.29 | 0.33 | 0.32 | 0.32 | 0.24 | |
| MDForge (Ours) | 5/5 (\(\checkmark\)) | 0.68 | 0.67 | 0.58 | 0.68 | 0.56 | |
| OAH (\(n=5\)) | Step-level feedback | 1-2/5 (โณ) | 0.23 | 0.33 | 0.13 | 0.20 | 0.20 |
| Trial-level feedback | 5/5 (\(\checkmark\)) | 0.47 | 0.33 | 0.21 | 0.20 | 0.20 | |
| MDForge (Ours) | 5/5 (\(\checkmark\)) | 0.99 | 1.00 | 0.43 | 0.30 | 0.20 | |
| CBClip (\(n=6\)) | Step-level feedback | 1-2/5 (โณ) | 0.12 | 0.33 | 0.03 | 0.09 | 0.07 |
| Trial-level feedback | 5/5 (\(\checkmark\)) | 0.54 | 0.33 | 0.09 | 0.26 | 0.20 | |
| MDForge (Ours) | 5/5 (\(\checkmark\)) | 0.63 | 0.67 | 0.54 | 0.60 | 0.47 |
When compared directly against the fixed-toolbox paradigm under identical compute budgets, fixed-hyperparameter tool calling attained held-out Kendall \(\tau\) of 0.22 on CB[7], 0.00 on OAH, and 0.33 on CBClip. Manually tuning tool hyperparameters achieved 0.28, 0.00, and 0.47. In contrast, MDForge achieved 0.56, 0.20, and 0.47 purely through open-ended Python synthesis, demonstrating that open program synthesis escapes the representational constraints of predefined libraries.
Ablation Study¶
The ablation analysis investigates the contribution of PRISM's multi-expert debate and stage-wise physics diagnostics on held-out ranking accuracy across all three hosts.
| Configuration | CB[7] Test \(\tau\) | CB[7] Test \(R^2\) | OAH Test \(\tau\) | OAH Test \(R^2\) | CBClip Test \(\tau\) | CBClip Test \(R^2\) | Core Observation |
|---|---|---|---|---|---|---|---|
| MDForge (Full Pipeline) | 0.56 | 0.58 | 0.20 | 0.43 | 0.47 | 0.54 | Stage diagnostics + 3-expert debate + reputation weights |
| w/o Multi-expert debate | 0.33 | 0.53 | 0.00 | 0.26 | 0.33 | 0.23 | Uncurated critiques cause misdiagnosis and overshooting |
| w/o Stage diagnostics | 0.16 | 0.45 | 0.00 | 0.10 | 0.20 | 0.03 | Lacks stage attribution; peaks at trial 3 (\(\tau=0.47\)) then collapses |
| Equal-weighted aggregation | 0.42 | - | - | - | - | - | 5-trial average \(\tau\) drops from \(0.40 \pm 0.14\) to \(0.33 \pm 0.11\) |
Key Findings¶
- Stage diagnostics prevent policy collapse: On CB[7], MDForge monotonically improves held-out Kendall \(\tau\) from 0.20 to 0.56 over 5 trials. In contrast, removing stage diagnostics causes performance to peak at trial 3 (\(\tau=0.47\)) and crash to 0.16 in trial 5, because unguided edits inadvertently overwrite functioning components.
- Error typing suppresses silent false convergence: Rather than merely lowering crash counts, PRISM explicitly classifies failures. Mid-pipeline runtime aborts dropped from 51% to 0%, while analysis-stage convergence rejections emerged deliberately to prevent invalid free energy outputs from being accepted as converged solutions.
- Prospective wet-lab validation discovers picomolar binder: Applying the best autonomous CB[7] pipeline to screen 10 unseen candidate compounds identified Bromantane (Brom) as the top hit. Wet-lab competition \(^1\text{H}\) NMR against the picomolar reference ferrocenylmethyl-trimethylammonium (FMTA, \(K_a \approx 2 \times 10^{12}\ \text{M}^{-1}\)) yielded a relative affinity \(K_{\text{rel}} = 4.26 \pm 0.15\) (\(n=3\)), establishing an absolute binding constant \(K_a \approx 8 \times 10^{12}\ \text{M}^{-1}\) (\(\Delta G_{\text{exp}} \approx -17.6\ \text{kcal/mol}\)). This confirmed prospective discovery of a true picomolar binder, placing Bromantane among the highest-affinity CB[7] binders ever documented.
Highlights & Insights¶
- Verbal process supervision for high-cost scientific workflows: Converts multi-hour, expensive GPU simulation rollouts into granular, interpretable natural language process signals, providing a practical blueprint for reinforcement learning under extreme feedback scarcity.
- Jurisdiction-partitioned multi-agent debate with Bayesian calibration: Assigns physical review authority strictly across force fields, sampling, and analysis, employing Beta-Bernoulli conjugate updates to maintain self-calibrating collective consensus across trials.
- Autonomous convergence to human expert design paradigms: Without manual intervention, MDForge synthesized an Attach-Pull-Release (APR) protocol with GAFF2/AM1-BCC and MBAR overlap guards, mirroring the architecture of canonical gold-standard human pipelines (such as the Gilson Lab's pAPRika framework).
Limitations & Future Work¶
- Target chemical space restricted to rigid macrocycles: Empirical evaluations are currently grounded on SAMPL host-guest macrocycles (CB[7], OAH, CBClip); expanding the framework to flexible biomolecular targets, protein-ligand systems, and membrane complexes remains unvalidated.
- Computational throughput tradeoffs: Experiments operated under a lightweight protocol (\(\approx 2\) GPU hours per guest) to enable systematic benchmarking; scaling to multi-microsecond production sampling or polarizable QM/MM force fields will demand enhanced computational resources.
- Narrow prospective screening breadth: Wet-lab validation confirmed only the top-1 ranked molecule via competition NMR; comprehensive binding curve profiles across the full candidate library were not collected.
Related Work & Insights¶
- vs Fixed-Toolbox MD Agents (MDCrow, MDAgent, DynaMate): Existing systems orchestrate pre-packaged toolboxes through API wrappers, restricting algorithmic novelty and failing to perform runtime bug recovery; MDForge authors raw Python pipelines from scratch and leverages multi-expert verbal feedback for surgical self-debugging.
- vs Generalist Verbal RL (Reflexion, LATS): Reflexion relies on high-frequency, low-cost scalar feedback in toy symbolic benchmarks (e.g., AlfWorld, HotpotQA); MDForge pioneers verbal RL in computationally intensive physical simulations with multi-hour rollout delays.
- vs Human-Designed Reference Pipelines (pAPRika / Gilson Lab): MDForge independently converged on the same methodological family (GAFF2, APR umbrella sampling, MBAR estimation) as human experts, recovering 78โ82% of human ranking fidelity autonomously.
Rating¶
- Novelty: โญโญโญโญโญ Formulates MD pipeline design as open-ended code generation via verbal RL, pioneering PRISM process-reward densification.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous multi-benchmark testing, comprehensive ablation analyses, and successful prospective wet-lab competition NMR validation.
- Writing Quality: โญโญโญโญโญ Clear mathematical POMDP formalization, well-structured agent communication contracts, and transparent failure-mode analysis.
- Value: โญโญโญโญโญ Sets a new benchmark for autonomous AI agents tackling long-horizon, high-cost computational physics and drug discovery pipelines.