Accelerating Neural Architecture Search with Code Semantics¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: Neural Architecture Search, Code Semantic Transitions, Evolutionary Algorithms, Risk-Aware Routing, Pre-Evaluation Screening
TL;DR¶
This paper introduces ECO-SCE, the first code-semantic transition estimator for neural architecture search that models parent-to-proposal evolutionary utility via semantic shifts, structural edits, and local evidence calibration, filtering low-gain code rewrites prior to costly evaluation and cutting downstream NAS GPU hours by 70.88% zero-shot.
Background & Motivation¶
Neural architecture search (NAS) is a cornerstone of automated artificial intelligence model design, striving to discover high-performing deep neural networks while substantially reducing manual engineering overhead. Over the past decade, diverse search paradigms including reinforcement learning, regularized evolution, differentiable relaxation, and recent large language model (LLM) code rewrites have demonstrated formidable optimization capabilities. Despite these algorithmic advances, virtually all existing NAS pipelines are organized around a strict generate–evaluate–refine iterative loop whose throughput is fundamentally throttled by the exorbitant computational cost of candidate evaluation, typically requiring full model training, multi-epoch convergence, or expensive benchmark lookups. With the rapid ascendancy of LLM-based code-evolution agents, generating syntactically valid and executable architecture programs has become effortless; yet the vast majority of generated candidate code revisions yield negligible performance improvements over their parent baselines, squandering substantial GPU budgets on unpromising mutants.
To alleviate this evaluation bottleneck, conventional efficiency strategies typically focus on making individual evaluations cheaper—using weight-sharing supernets, multi-fidelity low-resource training budgets, training-free zero-cost proxies, or learned performance predictors. However, in the context of evolutionary and code-rewriting NAS, these standalone architecture scoring paradigms suffer from two foundational limitations. First, the fundamental decision unit is mismatched: evolutionary search operates on directed parent-to-proposal code modifications where the decisive question is not whether the candidate architecture achieves a respectable standalone proxy score, but whether the specific code modification delivers a meaningful incremental gain relative to its parent. A candidate can exhibit high predicted standalone capacity while remaining completely redundant, marginal, or even regressive in terms of evolutionary trajectory. Second, existing performance predictors and zero-cost proxies routinely rely on target-space statistics, task-specific validation samples, or calibration sets; requiring evaluated architectures to fine-tune a predictor for each new search problem merely shifts the evaluation burden rather than removing it.
This dilemma calls for a parent-relative pre-evaluation screening model that can be trained offline and deployed zero-shot across arbitrary downstream architecture search tasks without task-specific calibration. By shifting the objective from scoring static architectures to predicting the marginal utility of code-level transformations, one can assess whether a candidate's syntactic and functional modification justifies the expense of full evaluation. Core idea: reformulate NAS acceleration as code-semantic transition utility modeling, training a lightweight estimator named ECO-SCE on directed architecture-change pairs to combine global semantic shifts, explicit program structural edits, and non-parametric local neighbor evidence, thereby safely bypassing low-gain code mutations before evaluation while preserving high-potential and uncertain transitions via risk-aware routing.
Method¶
Overall Architecture¶
ECO-SCE is seamlessly deployed as a pre-evaluation filter inserted directly between the architecture generator and the costly evaluation module. When the generator proposes an executable mutated candidate architecture \(a_c\) from an active parent architecture \(a_p\), conventional NAS pipelines immediately route \(a_c\) to the evaluator to acquire its ground-truth task score \(s(a_c)\). In contrast, ECO-SCE extracts code-level transition features over the directed pair \(\tau = (a_p, a_c)\), estimates the hindsight evolutionary gain distribution \(\Delta(\tau) = s(a_c) - s(a_p)\), and executes a risk-aware routing policy: definitively low-gain or harmful transitions are rejected and skipped without evaluation, whereas promising or highly uncertain transitions are forwarded to the full evaluator, whose verified feedback subsequently updates the active population or archive.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Parent and Candidate Architecture Code<br/>(a_p, a_c)"] --> B["Multi-Faceted Transition Representation<br/>dense shift + structural edit + sparse overlap"]
B --> C["Dual-Boundary and Evidence Estimator<br/>boundary classifier + magnitude regressor + KNN bank"]
C --> D["Risk-Aware Transition Routing Policy<br/>partition into reject / uncertain / eval-worthy"]
D -->|Harmful or low-gain mutation| E["Skip Evaluation<br/>discard proposal and retain parent"]
D -->|Promising or uncertain mutation| F["Costly Ground-Truth Evaluator<br/>training / validation / benchmark score"]
F --> G["Update Archive and Population<br/>guide subsequent LLM / evolutionary search"]
Key Designs¶
1. Multi-Faceted Transition Representation: Unifying dense semantic shifts, explicit AST differences, and sparse embedding overlaps
To overcome the blind spots of purely dense embeddings (which can blur localized module rewrites) and purely syntactic AST metrics (which lack semantic understanding of functional intent), ECO-SCE constructs a unified transition feature vector \(x(\tau) = [x_{\text{dense}}; x_{\text{code}}; x_{\text{sparse}}] \in \mathbb{R}^{1316}\) capturing program modifications across complementary abstraction levels: - Global semantic shift \(x_{\text{dense}} = \text{Embed}_{\text{dense}}(C_c) - \text{Embed}_{\text{dense}}(C_p) \in \mathbb{R}^{1024}\) measures the directional difference between the architecture-defining code blocks \(C_c\) and \(C_p\) using a high-capacity code embedding model, characterizing macro-level functional transformations; - Explicit structural edit \(x_{\text{code}} \in \mathbb{R}^{285}\) summarizes concrete program diff statistics, AST operator patterns, replaced/added line counts, module invocation types, and def-use dependency graph shifts; - Sparse semantic overlap \(x_{\text{sparse}} \in \mathbb{R}^7\) quantifies the Jaccard similarity, intersection size, \(\ell_1\) distance, and sparse cosine similarity of active embedding dimensions, distinguishing subtle surgical edits from radical rewrites. This feature vector is standardized using offline training statistics and projected onto an orthogonal subspace \(z(\tau) = \text{PCA}_{64}(\text{Std}(x(\tau))) \in \mathbb{R}^{64}\), ensuring that downstream inference incurs sub-second latency.
2. Dual-Boundary and Evidence Estimator: Decoupling directional boundaries, change magnitude, and local neighbor uncertainty
Direct regression of continuous task performance across unseen downstream domains suffers from severe distribution shift. ECO-SCE decomposes transition utility modeling into directional classification, magnitude regression, and non-parametric local calibration over the shared representation \(z(\tau)\): - Dual-boundary classification introduces a performance change margin \(m = 0.02\) to define negative degradation labels \(y_{\text{bad}}(\tau) = \mathbb{I}[\Delta(\tau) \le -m]\) and positive improvement labels \(y_{\text{good}}(\tau) = \mathbb{I}[\Delta(\tau) \ge m]\). Two linear logistic regressors estimate the boundary posterior probabilities \(p_{\text{bad}}(z)\) and \(p_{\text{good}}(z)\), leaving tiny ambiguous edits within the neutral margin; - Change magnitude regression fits the absolute performance change \(|\Delta(\tau)|\) via linear ridge regression \(\hat{a}(z)\), which is integrated with directional posteriors to form a signed benefit score: $\(\hat{r}(z) = (p_{\text{good}}(z) - p_{\text{bad}}(z))\hat{a}(z)\)$ - KNN evidence bank retrieves the \(K=80\) nearest training transitions \(N_K(z)\) in the PCA space by Euclidean distance, computing local harmful support \(s_{\text{bad}}(z) = \frac{1}{K}\sum_{i \in N_K(z)} y_{\text{bad}}(\tau_i)\) and local beneficial support \(s_{\text{good}}(z)\). It further derives a local uncertainty metric \(u(z) = \text{Quantile}_{0.9}(\{|\Delta(\tau_i) - \hat{r}(z_i)|\})\), penalizing overconfident parametric predictions in poorly explored regions of the transition space.
3. Risk-Aware Transition Routing Policy: Preserving evolutionary potential through conservative rejection and uncertainty fallbacks
In architecture evolution, mistakenly filtering an elite candidate is far more damaging than evaluating a mediocre one. ECO-SCE partitions proposals into reject, uncertain, and evaluation-worthy states, requiring unanimous global and local consensus before any proposal is denied evaluation: $\(R_{\text{rej}}(z) = 1 \iff p_{\text{bad}}(z) \ge \tau_{\text{bad}} \land p_{\text{good}}(z) \le \tau_{\text{protect}} \land s_{\text{bad}}(z) \ge p_{\text{bad}}(z) \land \hat{r}(z) + \lambda u(z) < 0\)$ Hyperparameters are chosen via the Feasible Plateau Representative Score (FPRS) under rigorous safety guarantees (Missed Improvement \(\le 10\%\), Accept Bad \(\le 10\%\), Top-10 Kept \(\ge 90\%\)), selecting the default configuration \((\tau_{\text{bad}}, \tau_{\text{good}}, \tau_{\text{protect}}, \lambda) = (0.40, 0.40, 0.40, 0.25)\). Any candidate not explicitly confirmed as harmful is routed as uncertain or evaluation-worthy and forwarded to the full evaluator, guaranteeing robust safety across evolutionary generations.
A Worked Example¶
During architecture evolution on the CLRS-DFS algorithmic reasoning task, the processor's triplet message-passing branch operates on high-order node and edge outer-products. Two candidate mutations illustrate ECO-SCE's screening behavior:
- Candidate A retains the core triplet tensor interaction and introduces only a localized graph-conditioned Sigmoid gate (triplet_gate_proj(graph_fts)), leaving first-order message passing and residual topologies intact. ECO-SCE extracts low global semantic drift with targeted AST modifications, yielding \(p_{\text{bad}} = 0.12\) and \(p_{\text{good}} = 0.68\), backed by consistent positive neighbor support in the KNN bank. It is routed as eval-worthy and evaluated, improving downstream accuracy to 60.01%;
- Candidate B radically rewrites the triplet block into a multi-head projection mechanism combined with directional pairwise gates, edge-conditioned gates, and DFS-specific activation gating. Its transition representation reveals massive semantic divergence and unstable AST structural shifts. The KNN bank flags elevated prediction residual uncertainty with \(p_{\text{bad}} = 0.82 \ge \tau_{\text{bad}}\), activating the rejection rule and skipping evaluation, thereby preventing wasted compute on an unstable, over-parameterized branch.
Loss & Training¶
ECO-SCE is trained offline on NB101-SemPair, an engineered dataset of 10,000 directed architecture-change transitions derived from NAS-Bench-101 (split 60%/20%/20% by parent architecture). Training the standardization scaler, 64-dimensional PCA projection, dual logistic boundaries, and ridge regressor takes under 1 second on a standard CPU. At test time, online inference consumes only 126.17 ms per proposal (15.23 seconds across an entire 100-round evolution campaign). Once constructed, all representations and estimators are deployed zero-shot across diverse downstream tasks without runtime gradient updates or task-specific tuning.
Key Experimental Results¶
Main Results¶
ECO-SCE was evaluated zero-shot within the OpenEvolve evolutionary framework across seven downstream benchmarks spanning algorithmic reasoning, 1D sequence modeling, natural language classification, tabular bioinformatics, and reinforcement learning control (100 evolution rounds on a single NVIDIA RTX 3090 GPU):
| Dataset | Domain / Setting | OpenEvolve GPU h | +ECO-SCE GPU h | GPU Hour Red. | OpenEvolve Score | +ECO-SCE Score | Score Change |
|---|---|---|---|---|---|---|---|
| CLRS | Algorithmic Reasoning (30 tasks) | 6.01 h | 1.74 h | -71.01% | 62.69 | 86.10 | +23.41 |
| MNIST1D | 1D Sequence Classification | 0.74 h | 0.20 h | -72.97% | 70.25 | 80.10 | +9.85 |
| MNIST1D-shuffle | Permuted 1D Sequence | 1.03 h | 0.31 h | -69.90% | 61.65 | 67.48 | +5.83 |
| 20 Newsgroups | Text Classification | 0.29 h | 0.13 h | -55.17% | 57.10 | 57.91 | +0.81 |
| CartPole-v1 | RL Environment Control | 0.73 h | 0.09 h | -87.67% | 377.60 | 500.00 | +122.40 |
| Yeast | Tabular Biology Classification | 0.21 h | 0.18 h | -14.29% | 61.78 | 62.63 | +0.85 |
| Breast Cancer | Diagnostic Tabular Classification | 0.13 h | 0.01 h | -92.31% | 96.90 | 96.81 | -0.09 |
| Average / Summary | – | 1.31 h | 0.38 h | -70.88% | – | – | 6/7 improved |
On the comprehensive 30-task CLRS algorithmic reasoning benchmark (searched on CLRS-DFS and evaluated zero-shot across all 30 algorithmic tasks):
| Method | Parameters ↓ | DFS Search Acc. ↑ | 30-Task Avg. Acc. ↑ | Avg. Acc. w/o DFS ↑ | Task Wins (out of 30) ↑ |
|---|---|---|---|---|---|
| Triplet-GMPNN | 467.3 K | 47.79% | 75.98% | 76.95% | 1/30 |
| OpenEvolve (Baseline) | 498.6 K | 32.54% | 62.69% | 63.73% | 1/30 |
| EvoPrompting | 478.2 K | 68.14% | 80.88% | 81.32% | 2/30 |
| Spark | 470.2 K | 83.74% | 83.91% | 83.92% | 10/30 |
| OpenEvolve + ECO-SCE (Ours) | 485.9 K | 60.01% | 86.10% | 87.00% | 16/30 |
Controlled filtering comparison on NB101-SemPair with a matched evaluation reduction budget of 75.02%:
| Filtering Strategy | Missed Imp. ↓ | Top-10 Kept ↑ | Evaluation Reduction ↑ |
|---|---|---|---|
| Random Filtering | 74.97% | 25.53% | 75.02% |
| Rule-Based Heuristic Filter | 64.08% | 38.80% | 75.02% |
| Standalone Accuracy Predictor | 54.54% | 99.40% | 75.02% |
| ECO-SCE (Ours) | 8.98% | 94.80% | 75.02% |
Ablation Study¶
Component-level ablation on the MNIST1D-shuffle CNN setting inside OpenEvolve (100 evolution rounds):
| Configuration | Runnable Proposals | Full Evaluator Calls ↓ | Skipped Proposals ↑ | GPU Hours ↓ | Best Test Acc. ↑ |
|---|---|---|---|---|---|
| OpenEvolve Baseline | 100 | 100 | 0 | 1.27 h | 59.10% |
| Full ECO-SCE | 93 | 10 | 83 | 0.46 h | 72.40% |
| w/o Evidence Bank | 85 | 26 | 41 | 1.30 h | 71.40% |
| w/o Magnitude Regressor | 98 | 19 | 71 | 0.88 h | 71.80% |
| w/o Boundary Classifier | 89 | 24 | 59 | 1.11 h | 66.50% |
Feature representation ablation on the NB101-SemPair test split:
| Representation Space | AUROC ↑ | Macro-F1 ↑ |
|---|---|---|
| Semantic Embeddings Only | 76.34% | 51.05% |
| Structural AST Edits Only | 82.34% | 60.25% |
| Combined (Semantic + Structural) | 86.09% | 66.13% |
Key Findings¶
- Directional boundary classification safeguards final architecture quality: Omitting the dual-boundary classifier causes the sharpest drop in downstream performance, lowering best test accuracy from 72.40% to 66.50% (a 5.9-point degradation), confirming that explicit directional supervision is indispensable for separating beneficial from regressive mutations.
- KNN evidence bank acts as the primary budget compressor: Eliminating the local evidence bank causes full evaluator calls to jump from 10 to 26 and nearly triples evolution GPU time (0.46h vs 1.30h), demonstrating that local neighbor calibration provides the necessary confidence for safely bypassing redundant candidates.
- Semantic and structural features offer complementary signal: Structural edits alone form a stronger individual predictor than dense semantics (82.34% vs 76.34% AUROC), but their combination delivers an additional +3.75 AUROC and +5.88 Macro-F1 boost, proving that macro-level functional drift and micro-level syntax changes capture orthogonal failure modes.
- Filtering targets runnable, low-gain mutations rather than syntax errors: In the MNIST1D-shuffle CNN experiment, ECO-SCE evaluated only 10 out of 93 fully executable candidate architectures, skipping 89.25% of valid code proposals while achieving superior downstream convergence.
Highlights & Insights¶
- Transition utility modeling vs. absolute architecture scoring: The paper demonstrates that modeling parent-relative code changes (\(\Delta(\tau)\)) is fundamentally better suited for evolutionary search than predicting standalone model accuracy, bypassing task-specific score calibrations.
- Tri-level safety control via global, local, and uncertainty gating: Combining parametric boundary predictions, non-parametric nearest-neighbor residuals, and conservative safety constraints ensures high filtering ratios without discarding transformative high-gain proposals.
- Zero-shot cross-domain generalizability: Offline training on standard vision cell benchmarks transfers directly to graph algorithms, tabular reasoning, and reinforcement learning agents without any downstream fine-tuning.
Limitations & Future Work¶
- Reliance on localized code-diff segments: The method currently analyzes localized architecture definitions; sprawling, multi-file codebases or heavily modularized frameworks may increase token costs and dilute AST change signatures.
- Exploration space restricted to NAS-Bench-101 origins: NB101-SemPair is constructed from cell-based convolutional architectures; expanding the transition evidence bank to large-scale Transformers and diffusion backbones remains an open frontier.
- Conservative uncertainty fallback overhead: In highly unfamiliar search spaces, out-of-distribution transitions trigger frequent uncertainty fallbacks, potentially diminishing the achievable GPU reduction ratio.
Related Work & Insights¶
- vs. Zero-Cost Proxies (e.g., SynFlow, NASWOT): Zero-cost proxies evaluate single architectures at initialization but cannot assess parent-relative marginal gain, frequently misjudging minor structural modifications; ECO-SCE directly estimates transition utility from code differences.
- vs. Performance Predictors (e.g., BRP-NAS): Standard predictors require task-specific training data and fail under zero-shot domain shifts; ECO-SCE abstracts the problem to domain-agnostic program changes, enabling zero-shot deployment.
- vs. LLM-Based NAS Generators (e.g., EvoPrompting, Spark, LLMatic): While LLM generators excel at proposing diverse candidates, they generate many unproductive mutations; ECO-SCE acts as a complementary plug-in filter to cut their evaluation overhead.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates NAS acceleration as code-semantic transition modeling, shifting the paradigm from absolute scoring to relative edit utility.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across NB101-SemPair, 7 downstream benchmarks, budget-matched filter baselines, and deep ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate motivation, elegant formulation, disciplined mathematical notation, and lucid exposition.
- Value: ⭐⭐⭐⭐⭐ Solves the critical evaluation bottleneck in LLM-driven code evolution and evolutionary NAS with a production-ready, zero-shot tool.