Skip to content

AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/lmqfly/AgentFold
Area: LLM Agent
Keywords: LLM Agent, Protein Folding, Closed-Loop Code Search, Monte Carlo Tree Search, AI for Science

TL;DR

AgentFold establishes an autonomous closed-loop multi-agent framework that searches over executable code variants on a compact ESMFold substrate via an MCTS-style tree controller, boosting best lDDT by 7.5% over independent Codex proposals under matched evaluation budgets and uncovering three transferable empirical design patterns for geometric deep learning.

Background & Motivation

Scientific LLM agents have demonstrated considerable promise in literature synthesis, hypothesis generation, and high-level experimental planning. However, existing benchmarks and frameworks predominantly evaluate agents on textual reasoning or isolated, shallow script authoring. In real-world scientific machine learning (Scientific ML) workflows, systems are built upon tightly coupled codebases comprising thousands of lines of engineering code, where modules interact via delicate geometric invariances, multi-objective trade-offs, and iterative recycling loops. Under such demanding conditions, plausible textual proposals or naive code generation fail to deliver actual model improvements.

This challenge is acutely evident in protein structure prediction. Contemporary folding systems such as AlphaFold2 and ESMFold tightly couple sequence and residue-pair representations, iterative geometric refinement (Structure Module / Invariant Point Attention), recycling mechanisms, and composite stereochemical loss formulations (such as Frame Aligned Point Error and structural violation penalties). Evaluating any proposed intervention is computationally expensive, requiring complete GPU training cycles and balanced assessment across local distance precision (lDDT), global topology (TM-score, RMSD), and stereochemical plausibility (MolProbity). Without rigorous execution grounding and structured attribution, code generation agents easily fall victim to syntactic crashes, orphaned parameters, or catastrophic distortion of the underlying geometric manifold.

To bridge this gap, this paper reframes scientific model development as an autonomous closed-loop search over an executable code-variant tree. Rather than operating as passive coding assistants, scientific agents must be capable of implementing interventions in coupled codebases, autonomously recovering from runtime errors, and distilling empirical lessons from both triumphs and failures to guide compute allocation. Core idea: build a dual-loop multi-agent search framework, AgentFold, that leverages an MCTS-style tree controller on a compact ESMFold substrate to orchestrate a "deduplicationโ€“unified planning & codingโ€“interactive debuggingโ€“attribution" pipeline, discovering high-performing folding architectures while distilling reusable empirical design patterns.

Method

Overall Architecture

AgentFold formulates autonomous folding-model development as heuristic search across a tree of concrete, executable code snapshots. It is explicitly designed to yield two coupled artifacts: (i) improved, parameter-efficient model variants, and (ii) a distilled set of empirical design patterns \(\mathcal{P} = \{P_k\}_{k=1}^K\) derived from logged interventionโ€“outcome traces. To navigate the computational expense of training folding networks, AgentFold implements a dual-loop architecture comprising an Inner Exploration Loop and an Outer Periodic Update.

The inner exploration loop takes a sampled parent node and drives it through context summarization, deduplication screening, unified planning and implementation, interactive real-time debugging, full training/evaluation, and multi-objective attribution analysis. The outer periodic update executes in batches (every 10 inner iterations), aggregating objective evaluation scores with Critic Agent assessments to refresh tree node values and dynamically redirect Top-\(k\) search compute toward promising variant neighborhoods.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Experience Pool Tree Nodes<br/>Executable Code Snapshots"] --> B["Dynamic Sampling & Context Summarization<br/>Top-k Promising Branches & Failure Context"]
    B --> C["Self-Evolving Agent Chain<br/>Deduplication โ†’ Unified Planning & Coding"]
    C --> D["Interactive Debugging & Training Execution<br/>Debugger Captures Exceptions & Resolves Errors"]
    D --> E["Multi-Objective Attribution & Pattern Mining<br/>Analyst Attributes Deltas & Distills Patterns"]
    E --> F["Literature Augmentation & Memory Persistence<br/>Searcher Links Evidence to Central Database"]
    F --> G["Periodic Update & Composite Scoring<br/>Objective Metrics & Critic Update Tree Policy"]
    G -->|Update Node Values| A

Key Designs

1. Dynamic Sampling & Context Summarization: Balancing exploitation and exploration under severe compute limits

Given that scientific ML rollouts entail substantial GPU expenditures, AgentFold replaces naive random selection with an MCTS-based dynamic sampling strategy over concrete code nodes. The sampler selects high-scoring parent nodes alongside diverse reference nodes, sampling multiple siblings from the same parent to construct tightly controlled code-level counterfactuals. Before initiating a new code revision, the Summarizer Agent queries the historical database for prior interventions sharing similar edit types or failure modes. It produces a compact brief highlighting successful outcomes, fatal implementation traps, and empirical hypotheses currently supported by accumulated traces, ensuring the subsequent planner never repeats known dead ends.

2. Self-Evolving Agent Chain & Unified Planning: Mitigating architectural and implementation interface mismatch

In complex scientific software, decoupling architectural conception from code implementation frequently leads to subtle tensor mismatches and broken gradient flows. AgentFold establishes a tightly coordinated agent chain: the Deduplicator Agent first filters proposed research directions against historical trials using strict criteria (problem focus, mechanism type, sequence-length regime, and expected contributions) under a conservative policy that minimizes false positives. Approved proposals are forwarded to the Unified Planner, which is solely responsible for both conceptual design and full executable code synthesis. Once launched in the training environment, a dedicated Debugger Agent inspects real-time logs, catches syntax, shape, or runtime anomalies, and collaborates directly with the Unified Planner to execute iterative in-place code fixes until training commences smoothly.

3. Multi-Objective Attribution & Empirical Pattern Mining: Converting trial-and-error traces into design heuristics

Upon completion (or premature termination) of a 150-epoch training run, AgentFold activates a two-stage knowledge distillation procedure. First, the Analyst Agent reviews training dynamics, multi-metric structural logs (lDDT, TM-score, RMSD, MolProbity, and contact precision/F1), and exact code diffs to attribute performance shifts to specific code interventions. Second, the Searcher Agent queries external literature to connect observed failure modes (such as attention collapse or manifold distortion) with theoretical biology and geometry literature, persisting all records into the central database. Systematically analyzing these historical traces yielded three general empirical design patterns: - P1: Bias before geometry: Injecting soft, learnable priors (e.g., residue-index separation biases or smooth sequence distance embeddings) into pair representations or Invariant Point Attention (IPA) logits before coordinates are generated steers attention effectively and boosts local accuracy; conversely, imposing direct displacement offsets onto downstream rigid-frame updates or backbone torsion angles frequently destabilizes training. - P2: Multiplicative refinement: Applying multiplicative scaling (e.g., via Sigmoid gates) to rigid-body updates naturally dampens unconfident or flexible regions; in contrast, additive displacement constraints often force unphysical displacements and break numerical convergence. - P3: Avoid geometry-to-attention feedback: Feeding immature or noisy coordinate-derived distances or differential geometry features back into attention mechanisms creates a destructive error-amplification feedback loop that precipitates structural collapse.

4. Periodic Update & Composite Scoring: Blending objective validation with implementation risk

Unlike canonical MCTS which updates tree values after every individual rollout, AgentFold synchronizes search node valuations in batched periods every 10 iterations to smooth out stochastic training noise. Node value updates are governed by a hybrid composite score:

\[S_{\text{total}}(e) = S_L(e) + S_B(e) + S_A(e)\]

where \(S_L(e)\) represents the objective training loss score and \(S_B(e)\) is the benchmark structural score (quantified by the Normalized Weighted Relative Score, NWRS, over CAMEO2022 development targets across backbone lDDT, all-atom lDDT, oligo GDT-TS, RMSD, and TM-score). To safeguard search compute against fragile hacks, the Critic Agent assigns an auxiliary score \(S_A(e)\) assessing hypothesis coherence, implementation cleanliness, and numerical stability risks. Importantly, \(S_A(e)\) is strictly confined to prioritizing future tree expansions and is never used to claim experimental performance gains.

A Worked Example: Tracing the Evolution of the Top-Performing Variant

The discovery of the top-performing variant esmfold_struct_enhanced_v4 illustrates the end-to-end multi-agent lifecycle: 1. Sampling & Context Brief: Rooted at the baseline ESMFold node, the Summarizer notes that previous attempt #11 (applying additive frame-level regularization directly to BackboneUpdate) suffered complete collapse (lDDT < 0.005). The brief advises exploring pre-coordinate representation biasing. 2. Deduplication & Planning: The Deduplicator confirms no prior exploration of residue-index-conditioned MLP biasing. The Unified Planner synthesizes variant esmfold_struct_enhanced_v4 (#36). It inserts an index-conditioned bias MLP prior to IPA blocks, expanding attention logits with dynamic sequence and structural terms: $\(a_{h,i,j} = \alpha \langle q_{h,i}, k_{h,j} \rangle + b_h(z_{i,j}) + b_{\text{dyn}}^h z_{\text{bias}_{i,j}} + b_{\text{seq}}^h(\Delta \text{residx}_{i,j}) + b_{\text{struct}}^h(s_i, s_j) + \text{point\_term}_{h,i,j} + \text{mask}_{i,j}\)$ Simultaneously, it introduces a learnable vector \(g \in \mathbb{R}^6\) to multiplicatively gate raw rigid-body backbone updates: \(\Delta \leftarrow \Delta \odot \sigma(g)\). 3. Execution & Evaluation: The Debugger verifies tensor dimensions, and the variant undergoes 150 training epochs on the 1,000-chain PDB subset. On CAMEO2022, mean lDDT reaches 0.285 (a +0.053 improvement over baseline), while NWRS attains a search-wide peak of 0.526, requiring a modest parameter overhead of only 1.1% (22.86M vs. 22.61M). 4. Attribution & Pattern Confirmation: The Analyst records that the index-aware bias successfully guides residue routing without perturbing rigid-body updates, while BackboneUpdate gating dampens erratic loop updates, confirming both P1 and P2 heuristics.

Key Experimental Results

Main Results

Evaluations were conducted on a compact ESMFold substrate featuring a single-block Folding Trunk trained on a 1,000-chain representative PDB subset (clustered via MMseqs2 at 40% sequence identity). In a controlled comparison under a matched budget of 36 full model evaluations on the CAMEO2022 development benchmark, AgentFold systematically outperforms both random exploration and independent Codex generation.

Method / Controller Evaluation Budget Best lDDT โ†‘ Top-5 lDDT โ†‘ Best NWRS โ†‘ Top-5 NWRS โ†‘
Random controller 36 evaluations 0.260 0.242 0.510 0.506
Codex proposals 36 evaluations 0.265 0.257 0.512 0.509
AgentFold (Ours) 36 evaluations 0.285 0.267 0.526 0.516

Across representative variants explored during the campaign, AgentFold consistently achieves substantial gains in local structural accuracy (lDDT) and baseline-normalized composite score (NWRS) while preserving global fold topology.

Variant NWRS โ†‘ bb_lddt โ†‘ (Mean / Median) lddt โ†‘ (Mean / Median) oligo_gdtts โ†‘ (Mean / Median) rmsd โ†“ (Mean / Median) tm_score โ†‘ (Mean / Median)
esmfold (Baseline Absolute) 0.500 0.644 / 0.651 0.232 / 0.220 0.564 / 0.570 7.380 / 5.358 0.648 / 0.693
esmfold_struct_enhanced_v4 (#36) +0.026 +0.009 / +0.010 +0.053 / +0.059 +0.005 / -0.003 +0.082 / -0.038 +0.004 / -0.012
esmfold_struct_local_context_v1 (#47) +0.020 +0.002 / +0.006 +0.049 / +0.044 -0.001 / -0.005 +0.176 / -0.129 +0.001 / -0.012
esmfold_struct_dist_aware_v1 (#28) +0.018 +0.011 / +0.014 +0.027 / +0.024 +0.011 / +0.020 -0.088 / -0.134 +0.011 / -0.009
esmfold_struct_enhanced_multiscale_v2 (#50) +0.017 +0.007 / +0.014 +0.045 / +0.049 +0.004 / +0.006 +0.261 / +0.338 +0.001 / -0.019
esmfold_net_conformal_geometric_attention (#31) +0.017 -0.006 / -0.008 +0.043 / +0.046 -0.006 / +0.011 -0.063 / +0.240 -0.005 / -0.001
esmfold_struct_enhanced_v1_dup2 (#58) +0.014 +0.012 / +0.010 +0.023 / +0.025 +0.007 / +0.010 +0.029 / -0.038 +0.007 / -0.009

Ablation Study & Targeted Analysis

Component ablations on esmfold_struct_enhanced_v4 and depth scaling across deeper trunks (8-block Folding Trunk) demonstrate the orthogonality of the discovered mechanisms and their scalability.

Configuration Setting Mean lDDT Mean bb_lddt GDT-TS NWRS Note
ESMFold (Baseline repeated run) 1-block trunk 0.238 0.638 0.552 0.500 Single-layer baseline
Enhanced v4 (Full model) 1-block trunk 0.274 0.648 0.564 0.521 Combined dual-mechanism improvement
Enhanced v4 w/o IPA bias 1-block trunk 0.257 0.643 0.558 0.511 Removing sequence bias drops lDDT by 0.017
Enhanced v4 w/o BB gate 1-block trunk 0.262 0.644 0.560 0.514 Removing update gating drops lDDT by 0.012
ESMFold (Deep trunk baseline) 8-block trunk 0.321 0.665 0.575 0.500 Extended to deep 8-block Evoformer trunk
Enhanced v4 (Deep trunk transfer) 8-block trunk 0.355 0.676 0.588 0.524 Consistent gain maintained (lDDT +0.034)

Targeted physical and structural evaluations highlight the localized nature of the improvements: - Loop regions: Enhanced v4 delivers a loop lDDT gain of +0.060 and a loop backbone lDDT gain of +0.008 over baseline. - Physical stereochemistry: Enhanced v4 reduces MolProbity by -0.157, driven by a 1.435% reduction in Ramachandran outliers, a 3.240% increase in favored residues, and an 18.169 drop in steric clashscore. - Medium-range contacts: In the 12โ€“24 sequence separation bin, contact precision and F1 increase by +0.019 and +0.010, respectively.

Key Findings

  • Gains stem from structural inductive biases, not capacity bloat: The best variant (#36) adds only ~0.25M parameters (+1.1%), and variant #47 adds a mere 0.015M parameters while achieving the second-highest NWRS. Conversely, variants with heavy parameter expansions (e.g., #24 with 28.46M or #57 with 32.49M) perform noticeably worse, and #60 (31.03M) collapses catastrophically (lDDT = 0.000).
  • Complementary roles of biasing and gating: Ablations confirm that IPA attention biasing routes information flow dynamically across flexible residues, while BackboneUpdate gating controls coordinate displacement magnitudes. Neither component alone can recover the full structural performance.
  • Localized and medium-range improvements: Across all top variants, gains are heavily concentrated in local all-atom neighborhoods and flexible loop regions, while global fold metrics (TM-score and RMSD) remain largely invariant.

Highlights & Insights

  • Executable closed-loop ground truth over textual speculation: Resolves the persistent failure mode of scientific LLMs proposing untestable or non-functional architectures by enforcing full compileโ€“debugโ€“trainโ€“evaluate cycles on coupled codebases.
  • MCTS-driven compute allocation under high evaluation costs: Demonstrates that tree-structured exploration coupled with batched re-scoring can effectively focus expensive GPU budgets on productive code neighborhoods.
  • Translating trial-and-error traces into reusable scientific insights: The extraction of concrete empirical patterns (P1, P2, P3) illustrates how autonomous agents can transcend empirical optimization to contribute descriptive design knowledge to computational biology.

Limitations & Future Work

  • Restricted single-block exploration substrate: Primary exploration was constrained to a 1-block Folding Trunk and a 1,000-chain dataset to make 80 full training iterations computationally viable (~5,000 GPU-hours); scaling the search directly on full-scale 48-block multi-gigabyte models remains untested.
  • Exclusion of multimeric and complex modeling: The current setup focuses on monomeric PDB chains, leaving multi-chain assembly, protein-ligand interfaces, and quaternary symmetries outside the search scope.
  • Expansion to broader scientific domains: The dual-loop MCTS search methodology holds strong potential for adaptation to RNA folding, force-field parameterization, and lattice protein design.
  • vs FunSearch / AlphaEvolve: While FunSearch and AlphaEvolve explore algorithmic code fragments or heuristic mathematical constructions, AgentFold tackles a full engineering-scale scientific ML codebase (>2,000 LOC) governed by strict 3D physical and geometric equivariance constraints.
  • vs The AI Scientist / BioReason: Unlike systems focused on open-ended literature review and automated manuscript drafting, AgentFold enforces strict execution grounding where every architectural claim is validated through rigorous multi-objective physical benchmarks.

Rating

  • Novelty: โญโญโญโญโญ Pioneering instantiation of an MCTS-driven multi-agent closed-loop code search system on an engineering-scale protein folding codebase.
  • Experimental Thoroughness: โญโญโญโญโญ Executed ~80 full model training/evaluation runs (~5,000 GPU-hours, 170M tokens) with extensive ablations, multi-objective metrics, and targeted stereochemical analysis.
  • Writing Quality: โญโญโญโญโญ Rigorous problem formulation, transparent analysis of both failures and successes, and clear empirical substantiation.
  • Value: โญโญโญโญโญ Provides an exemplary blueprint for autonomous model design and empirical discovery in AI for Science.