Skip to content

Test-Time Scaling for Scientific Equation Discovery

Conference: EMNLP 2026
arXiv: 2502.14382 ⚠️ refer to the original paper
Paper: ACL Anthology
Code: https://ahong-lin.github.io/ScaleSR-tts/
Area: LLM Reasoning
Keywords: Test-Time Scaling, Scientific Equation Discovery, Compute Allocation, Symbolic Regression, Search Width

TL;DR

This paper formalizes LLM-driven scientific equation discovery as a unified external test-time scaling (TTS) search process, showing across ~4,000 H100 GPU hours that search width is the dominant compute allocation parameter that determines both accuracy and wall-clock efficiency, while population-branching splits and controller algorithm choices are secondary.

Background & Motivation

Test-time scaling (TTS) has emerged as an established paradigm to boost the complex reasoning performance of large language models by allocating additional inference-time compute rather than updating model parameters. However, prevailing TTS literature focuses almost exclusively on closed-ended reasoning tasks such as competition mathematics and code generation, where the objective is to recover a single ground-truth answer and the allocation problem is often simplified to selecting the number of thinking tokens or performing independent sampling (Best-of-N). In contrast, scientific equation discovery (or symbolic regression) represents an open-ended modeling challenge: a system must propose candidate functional forms and parameter-fitting routines, receive numerical fitness feedback from an external verifier evaluating observed data, and iteratively refine candidate hypotheses without any predefined ground-truth string to match.

Recent advances have yielded automated program-evolution discovery systems such as AlphaEvolve, OpenEvolve, and ShinkaEvolve, which demonstrate compelling empirical performance in discovering algorithms and symbolic laws. Nonetheless, these frameworks are typically presented as intricate agent workflows tightly coupled with specialized prompting templates, crossover heuristics, model ensembles, and ad-hoc pruning rules. Consequently, it remains unclear whether their performance gains stem from these bespoke engineering heuristics or simply from how their underlying control flow allocates inference-time compute. In particular, when external verifiers involve costly non-linear parameter fitting and numerical integration, pure sequential refinement (e.g., Self-Refine) incurs severe wall-clock bottlenecks, whereas purely parallel one-shot sampling (Best-of-N) completely discards valuable intermediate feedback across a vast hypothesis space.

The core tension is that once an informative verifier is available, algorithm design should not be obscured by elaborate agent scaffolding, but rather framed as a principled strategic trade-off: how to allocate a fixed test-time compute budget \(N\) between broad exploration and deep exploitation. Core idea: formalize LLM-driven equation discovery as a discrete iterative search process parameterized by sampling, generation, evaluation, and pruning, and demonstrate via minimal parallel controllers (PBeam and PIE) that search width is the first-order allocation parameter dictating both discovery accuracy and the wall-clock Pareto frontier, with optimal width scaling systematically with compute budget.

Method

Overall Architecture

The framework formalizes LLM-driven equation discovery as an iterative search over a dynamic memory state \(M_t\). Initialized with a naive baseline solution \(M_0\) (an affine regression model fitted via ordinary least squares), each iteration \(t\) executes four generic operations under a global budget constraint: 1. Sample: select an active subset of parent candidates \(S_t \subseteq M_t\) of size \(n_t\) according to a selection policy \(q_t(\cdot \mid M_t)\); 2. Generate: prompt the LLM to generate \(k_t\) candidate program proposals for each selected parent, producing an expansion set \(G_t\) of per-step width \(w_t = n_t k_t\); 3. Evaluate and Prune: score each proposal in \(G_t\) via an external numerical verifier on the training data, merge proposals with current memory, and apply a pruning operator \(\Pi_t(M_t \cup G_t)\) to retain surviving solutions in \(M_{t+1}\); 4. Iterate: repeat for \(T\) synchronized rounds until the total generation budget \(\sum_{t=0}^{T-1} \sum_{i=1}^{n_t} k_t^{(i)} \le N\) is exhausted.

To systematically isolate allocation effects from prompt engineering, the search space is parameterized by \((q, n, k, g)\), where \(g\) denotes the number of semi-isolated groups (islands), global width is \(w = nk\), and synchronized search depth is \(T = \max(1, \lfloor N / (nk) \rfloor)\). The authors instantiate this minimal design via Parallel Beam Search (PBeam) and Parallel Iterative Expansion (PIE).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Task specification + Training data + Shared affine baseline M0"] --> B["Unified Search Control Framework<br/>Determine depth T and global width w=nk under budget N"]
    B --> C["Grouped Parallel Sampling<br/>Partition into g sub-groups, each selecting s=n/g parents"]
    C --> D["Single-Step Parallel Generation<br/>Prompt LLM asynchronously for k proposals per parent, yielding w=nk candidates"]
    D --> E["Numerical Verification & Pruning<br/>PBeam retains top-s candidates; PIE samples via Softmax from full history"]
    E -->|Remaining budget T steps| C
    E -->|Budget N exhausted| F["Output: Best-performing equation program evaluated on training Acc0.1"]

Key Designs

1. Unified Search Control Framework: Abstracting Open-Ended Discovery into a Budget Allocation Problem

Existing evolutionary systems interweave algorithmic search logic with complex crossover rules and hand-crafted prompts, making controlled comparisons impossible. The unified formulation establishes that the total number of LLM generation passes \(N\) serves as a rigorous proxy for test-time compute. For a fixed transformer backbone of size \(P\) and a fixed code editing scaffold, the input and output token lengths per generation pass are strictly bounded, so the FLOPs required for each rollout satisfy \(C_{\text{rollout}} \approx 2P(L_{\text{in}} + L_{\text{out}})\). Consequently, total inference FLOPs scale linearly with \(N\): $\(c_{\min} N \le C_{\text{total}} \le c_{\max} N\)$ This unified formulation naturally subsumes conventional external TTS baselines: purely parallel Best-of-N (\(T=1, n=1, k=N\)), purely sequential Self-Refine (\(T=N, n=1, k=1\)), and tree search. It enables researchers to bypass domain heuristics and directly explore optimal compute allocation across the \((q, n, k, g)\) design space.

2. Grouped Parallel Sampling: Balancing Parallel Throughput and Exploration Isolation

In real-world equation discovery workflows, numerical parameter fitting and program evaluation represent significant runtime bottlenecks. To overcome the lack of parallelism inherent in classical MCTS and single-chain refinement, the controller introduces a grouping factor \(g\). The active population \(n\) and compute budget \(N\) are partitioned evenly across \(g\) independent groups (islands). Each group maintains \(s = n/g\) active slots and produces a local width of \(w/g = (n/g)k\) candidates per iteration. Grouping replaces a large global synchronization barrier with finer-grained local synchronizations, effectively mitigating straggler effects caused by heterogeneous execution latencies and maximizing multi-GPU utilization.

3. Single-Step Parallel Generation: Executable Scaffold with Bounded Mutation Scope

To guarantee that the LLM focuses its search capacity on the mathematical functional form and optimization routine rather than syntax errors, the framework provides an intentionally minimal executable scaffold. The prompt concatenates the natural-language task instruction, an immutable prefix EXACT_PREFIX, an immutable suffix EXACT_SUFFIX, and at most one in-context inspiration drawn from the current group's best program. The LLM is restricted to generating code between # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END, implementing scaling_law_func and fit_scaling_law using only standard numerical libraries (numpy, scipy, scikit-learn). This standardized constraint isolates the pure compute allocation effects from multi-prompt engineering confounds.

4. Numerical Verification & Pruning: Contrasting Frontier Beam Pruning with Historical Sampling

During evaluation, candidate programs are executed within fresh sandboxed Python subprocesses to fit free parameters on the training data and calculate accuracy \(\text{Acc}_{0.1}\), defined as the fraction of samples with relative prediction error within 10%: $\(\text{Acc}_{0.1} = \frac{1}{m} \sum_{i=1}^m \mathbb{I}\left(\left|\frac{\hat{y}_i - y_i}{y_i}\right| \le 0.1\right)\)$ Two orthogonal pruning mechanisms are systematically compared: - PBeam (Parallel Beam Search): Implements strict greedy frontier pruning, keeping only the top \(s = n/g\) candidates with the highest \(\text{Acc}_{0.1}\) from \(M_t \cup G_t\) at each iteration, representing an aggressive exploitation-oriented strategy; - PIE (Parallel Iterative Expansion): Relaxes strict beam truncation by accumulating full search history \(H_{t+1} = H_t \cup G_t\), and samples surviving parents via a Softmax probability distribution over clipped training scores: $\(p_{t+1}(x) = \frac{\exp(f(x)/\tau)}{\sum_{x' \in H_{t+1}} \exp(f(x')/\tau)}\)$ This historical sampling prevents promising yet temporarily lower-scoring functional branches from premature extinction, preserving exploration diversity across extended rollouts.

A Worked Example

Consider task BPG1 (logistic growth rate discovery) in the Biology domain: - Initialization: At step \(t=0\), the system evaluates the shared affine baseline on the 4,000 training samples, yielding an initial score of \(\text{Acc}_{0.1} \approx 0.50\); - Configuration: Set total budget \(N=128\), global width \(w=32\) (decomposed into population \(n=4\), branching \(k=8\)), and grouping \(g=1\), resulting in \(T = 128 / 32 = 4\) iterations; - Iteration 1: The system takes the 4 replicated initial slots and queries the LLM asynchronously to generate 8 proposals per slot, dispatching a batch of 32 parallel equation programs to the numerical verifier. The model introduces basic non-linear terms and saturation coefficients, elevating the top training \(\text{Acc}_{0.1}\) to 0.82; - Iterations 2–4: PBeam retains the top 4 candidates to spawn the next synchronized batch of 32 candidates. By iteration 3, an explicit carrying-capacity term \(K_0\) is discovered, and at iteration 4, least-squares parameter optimization converges to the true differential form, achieving a training \(\text{Acc}_{0.1}\) of 0.9844 before final evaluation on the held-out test split.

Key Experimental Results

Main Results

Experiments are conducted on LLM-SRBench covering Biology (24 tasks modeling population ODE \(dP/dt\)) and Material Science (25 tasks modeling stress law \(\sigma(\epsilon, T)\)). The primary backbone is gpt-oss-20b, with Qwen3-30B-A3B used for cross-model validation. Results report mean training \(\text{Acc}_{0.1}\) over three independent random seeds.

Table 1 details the optimal search width \(w^\star\) and winning configuration across compute budgets \(N\) for gpt-oss-20b (with \(g=1\)):

Budget \(N\) Bio Best Algorithm Bio Best \((n, k, T)\) Bio Optimal Width \(w^\star\) Material Best Algorithm Material Best \((n, k, T)\) Material Optimal Width \(w^\star\)
1 PBeam (1, 1, 1) 1 PBeam (1, 1, 1) 1
2 PBeam (1, 1, 2) 1 PBeam (1, 1, 2) 1
4 PBeam (1, 2, 2) 2 PBeam (2, 1, 2) 2
8 PIE (1, 2, 4) 2 PIE (2, 2, 2) 4
16 PIE (1, 2, 8) 2 PBeam (1, 4, 4) 4
32 PBeam (1, 8, 4) 8 PBeam (1, 4, 8) 4
64 PBeam (2, 8, 4) 16 PIE (8, 2, 4) 16
128 PBeam (4, 8, 4) 32 PIE (64, 1, 2) 64

Ablation Study

To verify whether decomposing a fixed width \(w=nk\) into population size \(n\) versus branching factor \(k\) is critical, Table 2 compares the admissible power-of-two decompositions of width \(w=32\) at budget \(N=128\) and \(g=1\) across three structural regimes (population-heavy \(n>k\), balanced \(n \approx k\), and branching-heavy \(k>n\)):

Domain Algorithm Regime Best \((n, k)\) Train \(\text{Acc}_{0.1}\) Relative Gap vs. Best
Biology PBeam Population-heavy (16, 2) 0.9816 0.28%
Biology PBeam Balanced (4, 8) 0.9844 0.00%
Biology PBeam Branching-heavy (1, 32) 0.9801 0.44%
Biology PIE Population-heavy (16, 2) 0.9804 0.00%
Biology PIE Balanced (8, 4) 0.9612 1.95%
Biology PIE Branching-heavy (2, 16) 0.9794 0.10%
Material PBeam Population-heavy (16, 2) 0.9857 1.15%
Material PBeam Balanced (8, 4) 0.9971 0.00%
Material PBeam Branching-heavy (1, 32) 0.9876 0.95%
Material PIE Population-heavy (32, 1) 0.9884 0.92%
Material PIE Balanced (4, 8) 0.9975 < 0.01%
Material PIE Branching-heavy (2, 16) 0.9976 0.00%

Key Findings

  • Search width is the dominant first-order allocation parameter: Across domains and model families, the preferred search width \(w^\star\) systematically increases with the global budget \(N\) (climbing from \(w=1\) up to \(w=32\) in Biology, and up to \(w=64\) in Material Science). Both extreme greedy sequential refinement (\(w=1\)) and extreme one-shot wide sampling (\(w=128, T=1\)) perform significantly worse than balanced intermediate widths.
  • Wider search unlocks substantial wall-clock speedups: Transitioning from narrow sequential search to moderate search width dramatically enhances hardware parallelism across batched LLM decoding and multi-process numerical verification. On the Biology benchmark, shifting from \(w=1\) to \(w=32\) slashes wall-clock runtime by nearly 80% (from 7,622 s to 1,586 s) while concurrently boosting training accuracy \(\text{Acc}_{0.1}\) from 0.965 to 0.984.
  • Internal \((n, k)\) split and controller choice are second-order effects: Once global width \(w\) is properly matched to the budget, variations between population-heavy and branching-heavy splits typically account for less than 1% to 2% performance difference. Furthermore, the performance gap between PBeam and PIE stays within \(\pm 2\%\) percentage points, negligible compared to the 10–20% penalty of operating at an improper width.
  • Grouping is a systems throughput knob rather than a quality driver: Sub-island grouping \(g>1\) yields up to \(1.075\times\) wall-clock speedup by shrinking synchronization units and mitigating stragglers. However, excessive grouping (\(g=8\)) artificially narrows local group width \(w/g\), hurting accuracy by up to 1.54 percentage points in the Biology domain.

Highlights & Insights

  • De-mystifying complex evolutionary agent architectures: The findings demonstrate that the impressive empirical performance of heavily engineered discovery agents (e.g., OpenEvolve) is largely driven by their implicit expansion of parallel search width rather than bespoke crossover heuristics, LLM ensembles, or complex prompt mechanics.
  • A Pareto win-win on accuracy and latency: In contrast to standard TTS assumptions where deeper search inevitably costs more time, open-ended discovery with parallel verification allows moderate search width to simultaneously enhance hypothesis exploration and collapse sequential synchronization delays.
  • Actionable empirical recipe for practitioners: Deploying external search on open-ended discovery follows a clean top-down protocol: maximize total budget \(N\) based on inference constraints; identify the optimal width \(w^\star\) via a rapid power-of-two pilot sweep; then adopt a balanced \((n, k)\) decomposition with mild grouping \(g \le 2\), avoiding wasteful manual controller engineering.

Limitations & Future Work

  • Synthetic benchmark gap vs. real-world scientific discovery: Evaluation is confined to synthetic ODE and stress-strain benchmarks in LLM-SRBench, which do not encompass real-world scientific hurdles such as experimental noise, multi-objective trade-offs, causal intervention, and wet-lab physical validation.
  • Sensitivity to verifier fidelity and overfitting risk: The search is entirely guided by proxy training accuracy. While train-test generalization remained solid in this benchmark, aggressive test-time search under imperfect or misspecified reward proxies in broader scientific settings risks overfitting and Goodhart's law.
  • Uniform rollout cost assumption: The framework measures compute cost purely by proposal count \(N\), omitting variations in compilation overhead, runtime timeouts across complex numerical programs, and non-uniform token generation lengths.
  • vs. OpenEvolve / ShinkaEvolve: These multi-agent evolutionary code systems enforce island models, specialized crossover prompts, and ensemble selection. Under controlled matched-budget settings, OpenEvolve represents an extreme partitioned case of \((n=w, k=1, g=w)\); minimal controllers (PBeam/PIE) with mild grouping (\(g=2\)) consistently match or outperform it without complex heuristic overhead.
  • vs. Internal Reasoning Scaling (o1-style): Internal TTS optimizes sequential token budgets within a single autoregressive response, whereas external TTS coordinates discrete multi-candidate rollouts with an external ground-truth verifier, proving particularly effective for black-box symbolic hypothesis search.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Pioneers the systematic study of test-time scaling and compute allocation in open-ended scientific equation discovery.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducted a comprehensive 4,000 H100 GPU-hour sweep spanning 8 budget tiers, full width-depth grids, and cross-model/cross-domain validations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formalization, compelling narrative structure, and rigorous empirical analysis.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical and reproducible foundation for scaling test-time search in AI for Science.