SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Data¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/LAMDA-NeSy/SymboLLM-FE
Area: LLM Reasoning
Keywords: Automated Feature Engineering, Symbolic Regression, Large Language Models, Neuro-Symbolic AI, Tabular Learning
TL;DR¶
SymboLLM-FE introduces a neuro-symbolic automated feature engineering framework that reduces the symbolic regression search space from exponential \(O(2^n)\) to polynomial \(O(n^2)\) via a Spearman correlation-guided expanding-sliding window, mining mathematically grounded formulas that an LLM subsequently refines into human-interpretable, executable code within single-digit API calls to substantially enhance downstream tabular predictors.
Background & Motivation¶
Tabular data represents the predominant data modality across mission-critical machine learning applications, powering core domains including financial risk assessment, algorithmic trading, clinical diagnosis, and pharmaceutical discovery. Despite its ubiquity, raw tabular data frequently suffers from insufficient feature informativeness and latent high-order non-linear feature interactions. These intrinsic limitations constrain tree-based ensembles and deep tabular architectures from effectively capturing underlying data manifolds, leading to suboptimal predictive generalization. To resolve this representational bottleneck, Automated Feature Engineering (AutoFE) has emerged as a vital research frontier, aiming to systematically discover, transform, and select high-utility features to relieve data scientists of labor-intensive trial-and-error cycles.
However, existing AutoFE paradigms exhibit a stark dichotomy between mathematical efficacy and human interpretability. Traditional AutoFE systems, such as AutoFeat and OpenFE, rely on heuristic search over blind operator stacks. While capable of unlocking predictive gains, they generate deeply nested, opaque formulas that lack domain semantics, severely hindering post-hoc attribution in regulated domains and risking overfitting to spurious correlations. Conversely, contemporary LLM-based AutoFE approaches (e.g., CAAFE, OcTree, FEBP) harness broad domain knowledge to synthesize linguistically coherent feature generation scripts. Nonetheless, operating LLMs as open-ended primary generators introduces severe practical hurdles: they require extensive, computationally prohibitive iterative interaction rounds (often 30 to 50 iterations) with downstream predictors, and remain persistently vulnerable to algorithmic hallucinations, semantic drift, and implicit societal biases that can undermine downstream reliability.
The fundamental tension stems from attempting to force LLMs into unconstrained mathematical combinatorial exploration, where they lack statistical anchoring, or relying exclusively on genetic operators that are oblivious to domain logic. Core idea: decouple feature engineering into a collaborative two-stage neuro-symbolic pipeline, where a Spearman correlation-guided expanding-sliding window restricts symbolic regression to a polynomial search subspace to discover statistically grounded mathematical formulas, while an LLM is repurposed from an unguided generator into a deterministic semantic integrator and code refiner that injects domain priors and eliminates collinear redundancies within single-digit API calls.
Method¶
Overall Architecture¶
The SymboLLM-FE pipeline operates across two strictly coordinated yet functionally decoupled stages: Formula Construction by Symbolic Regression and Feature Generation via LLMs. In the first stage, raw tabular features are sorted monotonically by their Spearman rank correlation with the target variable. A dynamic expanding-sliding window samples contiguous feature subsets, bounding candidate subset exploration to polynomial complexity. A genetic programming regressor/classifier evolves protected mathematical operators with parsimony penalties to discover compact, predictive formulas, populating a candidate formula repository. In the second stage, these formula triples (feature subset, validation metric, symbolic expression) are translated into natural language and fed into an LLM alongside dataset domain context. Guided by chain-of-thought prompting, the LLM performs semantic grounding, deduplication, and code synthesis in Python pandas. The synthesized features are validated on a holdout set, with an automated traceback feedback loop guaranteeing robust, closed-loop execution.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Tabular Features C (n dimensions)"] --> B["Spearman Rank Ordering & Expanding-Sliding Window Sampling"]
B --> C["Genetic Programming Symbolic Regression with Parsimony Regularization"]
C --> D["Candidate Formula Triples Repository R"]
D --> E["LLM Chain-of-Thought Code Synthesis & Semantic Grounding"]
E --> F["Downstream Predictor Validation & Dynamic Feature Pruning"]
F --> G["Final Augmented, Interpretable Feature Set C_final"]
Key Designs¶
1. Spearman Rank Ordering & Expanding-Sliding Window Sampling: Polynomial Complexity Space Reduction
Evaluating symbolic regression across all non-empty subsets of an \(n\)-dimensional feature set \(C\) incurs an intractable exponential search complexity of \(O(2^n)\). To achieve scalable feature combination without sacrificing high-order interactions, the design first ranks all features by their Spearman rank correlation coefficient \(\rho_s\) with the target: $\(\rho_s = \frac{\mathrm{cov}(\mathrm{rg}_X, \mathrm{rg}_Y)}{\sigma_{\mathrm{rg}_X} \sigma_{\mathrm{rg}_Y}} = 1 - \frac{6 \sum_{i=1}^n d_i^2}{n(n^2 - 1)}\)$ Spearman correlation provides a non-parametric metric that robustly captures non-linear monotonic dependencies (achieving Kendall's \(\tau = 0.65\) consistency, surpassing Pearson, SHAP, and mutual information). Following monotonic ascending sorting \(c'_1, \dots, c'_n\), an expanding-sliding window mechanism is deployed starting from window size \(k = \lfloor n/2 \rfloor\). In the expanding phase, \(k\) increments by unit steps up to \(n\). In the sliding phase for each fixed \(k\), the window slides rightward with single-feature increments. The total candidate subsets \(S\) is bounded by an exact arithmetic series: $\(S = \sum_{k=\lfloor n/2 \rfloor}^n (n - k + 1) = \begin{cases} \frac{(n+2)(n+4)}{8}, & n \text{ is even} \\ \frac{(n+3)(n+5)}{8}, & n \text{ is odd} \end{cases}\)$ This formulation algebraically collapses the combinatorial exploration space from exponential \(O(2^n)\) down to strictly polynomial \(O(n^2)\), rendering symbolic search computationally tractable on high-dimensional tables.
2. Genetic Programming Symbolic Regression with Parsimony Regularization: Mathematically Grounded Rule Mining
For each generated subset \(S_i\), symbolic regression evolves syntax trees over a curated set of 14 primitive operators encompassing arithmetic (add, sub, mul, div), transcendental (sin, cos, tan, log), and piecewise functions (max, min, abs). All operators incorporate protection strategies (e.g., zero-division returning 1, absolute-value-protected logarithms) to guarantee numerical stability across \(\mathbb{R}^d\). To counter tree bloat and promote Occam's razor, individuals are evaluated via a composite fitness function incorporating parsimony pressure: $\(\tilde{E}(T_i) = E(T_i) + \Omega \cdot \mathrm{size}(T_i)\)$ where \(E(T_i)\) denotes Mean Squared Error on the training fold, \(\mathrm{size}(T_i)\) counts syntax tree nodes, and \(\Omega\) represents the parsimony coefficient optimized via validation grid search. Using tournament selection (size 100) and subtree crossover (\(p_{cx} = 0.9\)), the evolutionary search discovers optimal analytic expressions \(\tau_i\) within 120 generations. This populates a structured repository \(\mathcal{R} = \{(S_i, P_f(S_i), \tau_i)\}_{i=1}^{|S|}\), ensuring that every candidate passed to the LLM is anchored in verified statistical reality.
3. LLM Chain-of-Thought Code Synthesis & Semantic Grounding: Transforming Opaque Trees into Transparent Features
Raw symbolic expressions often exhibit opaque mathematical nesting—such as \(\tan(-0.607 - \tan(X_5))\) on Titanic or products of label-encoded categorical columns—which obscure operational meaning. SymboLLM-FE serializes repository \(\mathcal{R}\) into structured prompts comprising feature identities, mathematical formulas, and historical validation performance, instructing the LLM via chain-of-thought to act as a semantic interpreter and deduplication filter. The LLM performs two grounded transformations: interpretability enhancement, where abstract operations are mapped to domain-specific phenomena (e.g., recognizing that \(\tan(X_5)\) captures the abrupt age threshold of the "women and children first" maritime evacuation protocol, or translating categorical products into directional travel distance); and redundancy consolidation, where formulas mathematically equivalent to existing features (such as reconstructing weight/height\(^2\) in the presence of an existing BMI column) trigger explicit df.drop() commands to eliminate multicollinearity.
4. Traceback Error-Recovery Sandbox & Closed-Loop Downstream Pruning: Robust Single-Digit API Execution
To eliminate execution crashes typical of unconstrained LLM code synthesis, the framework implements a three-stage parsing and error-handling pipeline: markdown delimiter stripping, line-level validation filtering, and execution inside an isolated try-except sandbox. When runtime exceptions occur (such as missing column references or syntax discrepancies), the structured stack traceback is automatically parsed and returned to the LLM as corrective in-context feedback, prompting autonomous code repair. Concurrently, the executed candidate features \(C_{\mathrm{new}}\) enrich the dataset \(C_{\mathrm{final}} = C \oplus C_{\mathrm{new}}\) and are evaluated using a lightweight downstream estimator (e.g., TabPFN). If generated features degrade validation performance, an adaptive backward pruning mechanism discards them. This targeted feedback loop enables the framework to converge within an average of only 4 LLM API calls, slashing token expenditures while generating compact feature sets.
Benchmark & Evaluation Protocol¶
The framework was evaluated across 6 diverse public benchmarks from OpenML and Kaggle spanning binary classification, multi-class classification, and regression (Credit-g, Spaceship Titanic, Cmc, Academic, Ailerons, Tesla), alongside 4 competitive Kaggle arenas (BNP, Obesity Risk, Manufacturing, House Prices). To strictly eliminate data contamination, all experiments enforced a nested cross-validation protocol: the training set was exclusively accessible for Spearman sorting and symbolic model fitting; the validation set was utilized solely for LLM prompt context scoring and hyperparameter tuning; the test set remained strictly sequestered until final evaluation. Downstream evaluators spanned gradient-boosted decision trees (CatBoost, XGBoost), Multi-Layer Perceptrons (MLP), and tabular foundation models (TabPFN), with all results averaged across three random seeds.
Key Experimental Results¶
Main Results¶
Under the primary TabPFN downstream predictor, SymboLLM-FE established superior or highly competitive performance over both traditional AutoFE baselines (AutoFeat, OpenFE) and leading LLM-based AutoFE methods (CAAFE, OcTree, FEBP, LLM-FE, LLM-RANK).
| Dataset | Task (Metric) | Baseline | OpenFE (Best Trad.) | FEBP (Best LLM) | SymboLLM-FE (Ours) | Relative Gain / Observation |
|---|---|---|---|---|---|---|
| Credit-g | Binary (Acc ↑) | 77.03 ± 0.47 | 76.50 ± 3.34 | 77.50 ± 0.41 | 77.00 ± 1.63 | Outperforms OpenFE with reduced variance |
| Spaceship | Binary (Acc ↑) | 80.79 ± 1.27 | 80.30 ± 1.00 | 80.85 ± 1.29 | 81.27 ± 1.31 | +0.48% (outperforms CAAFE 80.99) |
| Cmc | Multi-class (Acc ↑) | 57.85 ± 0.89 | 57.85 ± 2.35 | 57.93 ± 0.48 | 57.97 ± 0.73 | +0.12% (consistent multi-class lead) |
| Academic | Multi-class (Acc ↑) | 77.33 ± 0.67 | 76.42 ± 0.61 | 77.36 ± 0.35 | 77.89 ± 0.23 | +0.56% (\(p < 0.001\), highly significant) |
| Ailerons | Aero Regression (RMSE ↓) | 5.10 ± 0.48 | 5.26 ± 0.49 | 5.08 ± 0.48 | 5.02 ± 0.46 | -0.08 (lowest error across all baselines) |
| Tesla | Stock Regression (RMSE ↓) | 2.39 ± 0.09 | 2.18 ± 0.06 | 2.41 ± 0.08 | 2.16 ± 0.06 | -0.23 (outperforms baseline and OpenFE) |
On the 4 Kaggle competition test sets (BNP, Obesity Risk, Manufacturing, House Prices), SymboLLM-FE paired with TabPFN consistently outperformed vanilla TabPFN and OpenFE + TabPFN, delivering an average performance gain of 2.5 percentage points (2.5 pp) across private and public leaderboards.
Ablation Study & Computational Efficiency¶
To quantify the necessity of each component, systematic ablations were conducted across multiple random seeds (Table 13, aggregated across benchmarks).
| Variant Configuration | Accuracy (%) | Std | 95% Confidence Interval | \(p\)-value (vs. Ours) | Statistically Significant? |
|---|---|---|---|---|---|
| Vanilla Baseline | 75.81 | ± 0.45 | [74.93, 76.69] | < 0.001 | Yes |
| w/o Spearman Pre-sorting (SP) | 76.07 | ± 0.51 | [75.07, 77.07] | < 0.001 | Yes |
| w/o Expanding-Sliding Window (ES) | 75.92 | ± 0.58 | [74.78, 77.06] | < 0.001 | Yes (steepest degradation) |
| w/o LLM Semantic Refinement (LLM) | 76.84 | ± 0.41 | [76.03, 77.65] | 0.0142 | Yes |
| SymboLLM-FE (Full Framework) | 77.16 | ± 0.34 | [76.49, 77.83] | — | Highest accuracy & lowest variance |
In runtime cost analysis, SymboLLM-FE demonstrated outstanding operational efficiency: on Credit-g, traditional OpenFE generated 1,436 uninterpretable features (score 79.53), while LLM-based OcTree and FEBP required 50 and 29 API iterations respectively; in contrast, SymboLLM-FE generated a compact set of only 70 features, attained the top score of 80.02, and required only 4 LLM API calls. Across all datasets, local symbolic regression scaled predictably with polynomial complexity (0.89 mins on Cmc to 30.24 mins on high-dimensional Ailerons), while LLM calls remained strictly between 2 and 5 across all dimensions, consuming 1,147 to 3,587 tokens per call.
Key Findings¶
- Structured windowing is indispensable for non-linear interactions: Removing the expanding-sliding window mechanism (w/o ES) resulted in the sharpest drop to 75.92% accuracy with substantially higher variance, demonstrating that unguided subset sampling misses high-order feature synergies.
- LLMs provide crucial semantic regularization and deduplication: Relying purely on raw symbolic regression expressions (w/o LLM) achieved 76.84%, but suffered from structural redundancy. The LLM served as a critical semantic filter, driving code execution success to 92.6% while removing multicollinear legacy attributes.
- Enhanced reasoning backbones directly improve feature quality: Evaluating alternative LLM engines (Table 3) revealed that reasoning-focused architectures (DeepSeek-R1 reaching 79.49% on TabPFN, OpenAI o1 reaching 78.58%) significantly outperformed standard GPT-4 (76.83%), proving that automated feature synthesis directly benefits from advanced test-time reasoning.
Highlights & Insights¶
- Repositioning LLMs from unconstrained generators to deterministic integrators: By confining the LLM to refining and explaining mathematically verified symbolic rules, the framework eliminates algorithmic hallucinations and spurious correlations while preserving human interpretability.
- Polynomial bounding of combinatorial feature spaces: The integration of non-parametric Spearman rank sorting with an expanding-sliding window provides a mathematically elegant mechanism to reduce search complexity from \(O(2^n)\) to \(\approx n^2/8\), solving a longstanding scaling bottleneck in genetic programming.
- Self-correcting execution harness: The automated exception parsing and traceback feedback loop provides a highly robust, zero-human-intervention pipeline that achieves a 92.6% execution rate, offering a reusable blueprint for code-generating agentic systems.
Limitations & Future Work¶
- Computational latency on very high-dimensional tables: While the \(O(n^2)\) polynomial bound prevents combinatorial explosion, datasets with hundreds of features still incur non-trivial local evolution times (e.g., ~30 minutes on Ailerons), which may limit immediate applicability in real-time or streaming environments.
- Potential blind spots for non-adjacent variable synergies: Sorting features strictly by marginal Spearman correlation with the target assumes that synergistic variables cluster or appear in contiguous sliding windows. Variables with near-zero individual correlation that only exhibit strong predictive power under exclusive XOR interactions could theoretically be separated in the sequence and overlooked.
- Future Directions: The authors suggest formalizing empirical feature engineering transformations mined across large-scale repositories into structured prompting principles, or using them to instruct-tune specialized tabular reasoning LLMs.
Related Work & Insights¶
- vs. Traditional AutoFE (AutoFeat / OpenFE): AutoFeat and OpenFE rely on brute-force algebraic expansions that produce thousands of opaque features (e.g., nested trigonometric combinations of raw indices) that domain experts cannot audit. SymboLLM-FE constrains total features to a compact set annotated with clear domain rationales, while achieving superior predictive accuracy.
- vs. LLM-based AutoFE (CAAFE / OcTree / FEBP): Existing LLM AutoFE frameworks prompt models to hypothesize features from task descriptions, necessitating 30 to 50 feedback rounds that incur heavy API expenses and risk severe hallucination. SymboLLM-FE grounds LLM synthesis in statistically validated formulas, converging in approximately 4 API calls with zero ungrounded generative deviations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant neuro-symbolic synergy that seamlessly bridges evolutionary symbolic regression with LLM semantic reasoning for tabular learning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 diverse benchmarks, 4 Kaggle competitions, 4 downstream model families, and rigorous Wilcoxon statistical testing.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical derivations, rich appendices providing detailed case studies on semantic grounding, and well-structured empirical validation.
- Value: ⭐⭐⭐⭐⭐ Highly practical, open-sourced solution that tackles the twin bottlenecks of interpretability and iteration cost in automated feature engineering.