Skip to content

DSEffi-Bench: Demystifying Large Language Models’ Capabilities in Data Science Programming

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Albert-Gong/DSEffi-Bench
Area: Code Intelligence
Keywords: Data Science Code Generation, Code Execution Efficiency, Benchmark Evaluation, Efficiency Deficit Taxonomy, Model Routing

TL;DR

Introduces DSEffi-Bench, the first execution efficiency benchmark for data science code generation (comprising 1,000 human-validated instances from real Stack Overflow pain points paired with stress-testing suites), revealing significant decoupling between functional correctness and runtime efficiency, uncovering that 79.1% of performance deficits stem from domain-specific root causes, and demonstrating up to +14.7% efficiency improvement and 13.0× cost reduction via taxonomy-guided optimization and library-conditioned routing.

Background & Motivation

With the rapid emergence of large language models for automated programming, data science (DS) code generation has established itself as one of the most prominent real-world application domains. In data science workloads, writing high-quality code requires not merely translating algorithmic logic from scratch, but critically selecting and composing highly optimized, low-level library primitives (such as NumPy, Pandas, SciPy, PyTorch, and Polars). However, existing data science benchmarks—such as ExeDS, DS-1000, and DSCodeBench—uniformly treat functional correctness as the sole capability proxy, completely leaving execution latency unexamined.

In practical data processing scenarios, functionally equivalent code pathways can differ in execution speed by several orders of magnitude. For instance, computing a grouped cumulative maximum in Pandas via the native vectorized call df.groupby('k').cummax() versus the row-wise callback df.groupby('k').apply(lambda g: g.cummax()) produces identical outputs, yet the latter incurs an entire order of magnitude higher latency due to Python interpreter callback overhead across every group. Existing code efficiency benchmarks, largely targeting competitive programming such as LeetCode (e.g., EffiBench, Mercury), primarily focus on algorithmic complexity classes (\(O(N^2)\) vs. \(O(N \log N)\)), leaving data science domain-specific bottlenecks—such as framework dispatch overhead, contiguous memory layouts, and vectorized primitive selection—entirely unexplored.

To bridge this critical gap, this work investigates LLMs' capabilities in generating efficient data science code, untangling the relationship between correctness and runtime efficiency, and pinpointing models' systematic blind spots in primitive selection and execution overhead. Core idea: curate DSEffi-Bench, a 1,000-instance DS code efficiency benchmark with stress-testing harnesses and expert-refined reference solutions, formalize log-space relative efficiency spectra alongside a five-category deficit taxonomy to demystify correctness–efficiency decoupling, and demonstrate actionable remedies via taxonomy-guided optimization and library-conditioned model routing.

Method

Overall Architecture

DSEffi-Bench sources real-world performance pain points from Stack Overflow (SO), filtering 17,257 candidates down to 1,000 human-validated instances through a four-stage quality-gated pipeline, and establishes a standardized dual-metric evaluation protocol. The workflow comprises: (1) efficiency-oriented post retrieval; (2) two-stage semantic filtering for question-level relevance and answer-level feasibility via LLM-as-Judge; (3) standardized instance transcription generating unified function signatures, fuzz-style testing harnesses with both correctness and stress sub-domains, and multi-dimensional quality scoring; (4) human-expert reference solution validation and refinement (conducted by authors with >5 years of DS coding experience); and (5) multi-tier model evaluation across 16 LLMs on dedicated server hardware (dual Intel Xeon 8468V + 4× NVIDIA H200), constructing task-level efficiency spectra in log-space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Stack Overflow Post Retrieval<br/>21 DS library tags × 21 efficiency keywords"] --> B["Relevance & Feasibility Filtering<br/>Question-level relevance + Answer-level feasibility"]
    B --> C["Instance Transcription & Multi-Dim Scoring<br/>Generate X & harness T + cross-family judge"]
    C --> D["Human-Validated Reference Solution<br/>50.3% alternative strategy / 32.2% refinement"]
    D --> E["Dual-Metric Evaluation & Log-Space Spectrum<br/>Pass rate + Log-normalized Beyond|Pass"]

Key Designs

1. Quality-Gated Four-Stage Pipeline: From Authentic Community Posts to Stress-Tested Harnesses

To overcome the artificiality of synthetic coding benchmarks and the noise of simple keyword matching, the construction pipeline transforms raw Stack Overflow posts into rigorous, reproducible evaluation instances. First, combining 21 DS library tags across numerical computing, DataFrames, classical ML, deep learning, visualization, and compiled acceleration with 21 efficiency keywords, the authors retrieve 17,257 candidate posts from 2021 to 2025. Then, Claude-Sonnet-4.6 acts as an LLM judge to enforce question-level relevance (confirming Python DS context, concrete efficiency optimization intent, and self-contained I/O) and answer-level feasibility (verifying that the accepted answer offers a concrete, reproducible library optimization rather than high-level prose), pruning the pool to 4,360 posts with high human agreement (Cohen's \(\kappa=0.93\) and \(0.71\)).

During transcription, the model formulates a unified entry signature def solution(input_data=None): across five standardized description fields \(X\) (task description, I/O specifications, and input/output examples). It generates an evaluation harness \(T\) with deterministic fuzz-style testing generators input_generator(seed, domain) spanning both functional correctness sub-domains and heavy stress-testing sub-domains (achieving >95% line and branch coverage on the reference code at \(N=5\) cases per domain) along with float-tolerant output_validator checkers. A cross-family judge (GPT-5.4) scores each candidate on four dimensions (1–3 scale), discarding any instance with a score of 1, resulting in 2,301 high-quality candidates.

2. Human-Refined Reference Solutions and Log-Normalized Efficiency Spectra: Establishing a Grounded Baseline

To prevent sub-optimal accepted answers from distorting the efficiency ground truth, two expert annotators independently studied and re-implemented reference solutions for top-ranked candidates until 1,000 instances were curated. Notably, only 17.5% of reference solutions faithfully adopt the accepted SO answer; 32.2% refine the implementation details to eliminate redundant allocations and branching, while 50.3% adopt an entirely different, superior optimization strategy (e.g., replacing Pandas' \(O(N^2)\) row-wise expanding().apply() callback with a sorted list maintained via Python's C-level bisect routines). This curation guarantees that the human reference provides a strong performance anchor (\(86.6\%\) efficiency score).

Recognizing that execution times span wide ranges (median slowest-to-fastest ratio of \(14.5\times\) and 90th percentile of \(522.0\times\)), standard linear normalization compresses 46.6% of correct solutions into the narrow \(>0.95\) bracket, obscuring meaningful engineering distinctions. The benchmark formalizes a log-space spectrum score. For candidate solution \(S_c\) with execution time \(t_c\) bounded by task extrema \(t_{max}\) and \(t_{min}\):

\[\text{Beyond}(S_c) = \frac{\log t_{max} - \log t_c}{\log t_{max} - \log t_{min}}\]

Solutions failing correctness tests or exceeding the 5-second timeout receive 0. To decouple runtime efficiency from functional correctness, the authors evaluate \(\text{Beyond|Pass}\) (B|P), averaging efficiency strictly across passed instances.

3. Five-Category Efficiency Deficit Taxonomy: Localizing Domain-Specific Root Causes

To diagnose why LLMs produce slow DS code, the authors performed fine-grained qualitative annotation on 868 slow–fast code pairs (where both solutions are functionally correct but differ by \(>1.5\times\) in execution time, Cohen's \(\kappa=0.82\)), establishing a two-level taxonomy of 5 Level-1 categories and 19 Level-2 sub-patterns:

  • Optimized Primitive Utilization (OPU, 23.3%): The slow code fails to call an existing, highly optimized library primitive (e.g., hand-coding triple-nested loops for Euclidean distance instead of calling scipy.spatial.distance.cdist, yielding a \(389.4\times\) slowdown).
  • Complexity-Reducing Computation (CRC, 20.9%): The slow code lacks algorithmic design or incurs redundant work (e.g., computing a batch of \(2\times2\) matrix determinants via per-element np.linalg.det LU decomposition instead of closed-form vectorized \(ad-bc\), running \(183.8\times\) slower).
  • Loop-to-Vectorization (L2V, 19.2%): The slow code relies on Python-level iteration (loops, iterrows, apply) rather than compiled C array/table operations (e.g., summing \(3\times3\) image patches using nested Python loops rather than summing nine vectorized slice views, resulting in a \(167.4\times\) slowdown).
  • Data Layout Optimization (DLO, 20.2%): The slow code causes repeated intermediate allocations and layout conversions rather than in-place reuse or contiguous buffers (e.g., replicating DataFrame rows via multiple concat/repeat calls rather than pre-allocating contiguous NumPy arrays and constructing the DataFrame once, incurring a \(95.5\times\) slowdown).
  • Overhead-Aware Stack Matching (OSM, 16.5%): The slow code uses unnecessarily heavy frameworks, function-level dynamic imports, or defensive wrapper boilerplate (e.g., placing import torch inside a function body with dead validation branches, causing a \(218.4\times\) slowdown; or misapplying Numba JIT on small data sizes, yielding a median slowdown of \(1,131\times\)).

Strikingly, 79.1% of all identified efficiency deficits (OPU, L2V, DLO, OSM) belong to library-specific engineering practices rather than traditional algorithmic complexity (CRC).

Key Experimental Results

Main Results

The evaluation spans 16 models across 3 tiers: Closed Frontier (GPT-5.4, GPT-5.4-mini, Claude-Opus-4.6, Claude-Sonnet-4.6), Open Frontier (DeepSeek-V4-Flash, DeepSeek-V3.2, Kimi-K2.5, MiniMax-M2.5, Qwen3.5-397B/A17B), and Open Coder (Qwen3-Coder-Next, Devstral-2-123B, Qwen2.5-Coder-32B/14B, Codestral-22B, DeepSeek-Coder-33B/V2-Lite). Overall and per-library performance under greedy decoding (\(T=0.0\)) are summarized below:

| Model | Tier | Pass (%) | Beyond (%) | B|P (%) | NumPy B|P (%) | Pandas B|P (%) | PyTorch B|P (%) | SciPy B|P (%) | Polars B|P (%) | |---|---|---|---|---|---|---|---|---|---| | GPT-5.4 | Closed Frontier | 66.9 | 48.0 | 71.7 | 78.9 | 71.2 | 53.6 | 82.3 | 59.7 | | GPT-5.4-mini | Closed Frontier | 62.2 | 44.6 | 71.6 | 77.2 | 71.9 | 60.9 | 67.3 | 57.4 | | Claude-Opus-4.6 | Closed Frontier | 61.3 | 42.4 | 69.2 | 73.4 | 69.6 | 49.4 | 80.3 | 66.9 | | Claude-Sonnet-4.6 | Closed Frontier | 59.7 | 39.4 | 66.1 | 68.3 | 69.6 | 48.2 | 74.6 | 57.4 | | Kimi-K2.5 | Open Frontier | 40.2 | 29.6 | 73.6 | 79.6 | 74.0 | 59.1 | 87.6 | 58.1 | | Qwen3.5-397B/A17B | Open Frontier | 57.6 | 39.1 | 67.9 | 70.2 | 72.2 | 51.0 | 69.8 | 62.0 | | DeepSeek-V4-Flash | Open Frontier | 57.7 | 37.5 | 65.0 | 70.7 | 63.2 | 40.4 | 67.6 | 72.1 | | DeepSeek-V3.2 | Open Frontier | 56.1 | 30.4 | 54.2 | 59.9 | 52.3 | 46.2 | 64.2 | 41.1 | | MiniMax-M2.5 | Open Frontier | 42.8 | 26.0 | 60.6 | 64.4 | 60.3 | 35.5 | 64.9 | 56.0 | | Qwen3-Coder-Next | Open Coder | 50.5 | 28.7 | 56.8 | 58.9 | 57.1 | 53.2 | 54.5 | 56.8 | | Devstral-2-123B | Open Coder | 43.0 | 23.5 | 54.6 | 57.8 | 55.8 | 40.2 | 64.1 | 51.0 | | Qwen2.5-Coder-32B | Open Coder | 34.1 | 18.7 | 54.7 | 56.5 | 52.2 | 73.3 | 47.4 | 69.9 | | Qwen2.5-Coder-14B | Open Coder | 32.8 | 15.2 | 46.4 | 44.0 | 50.6 | 53.2 | 39.9 | 61.0 | | Codestral-22B | Open Coder | 27.4 | 13.8 | 50.5 | 49.5 | 53.5 | 44.0 | 61.1 | 55.2 | | DeepSeek-Coder-33B | Open Coder | 17.5 | 8.0 | 45.5 | 47.2 | 45.2 | 42.4 | 61.7 | 39.5 | | DeepSeek-Coder-V2-Lite | Open Coder | 17.1 | 7.3 | 42.6 | 32.5 | 36.3 | 84.2 | 31.1 | 50.5 | | Human Reference | - | 100.0 | 86.6 | 86.6 | 88.5 | 85.8 | 78.7 | 94.2 | 78.1 |

Ablation & Diagnostic Experiments

1. Taxonomy-Guided Test-Time Optimization: Augmenting the state-of-the-art Effi-Learner (EL) profile-guided reprompting loop with the five-category optimization taxonomy prescription (EL w/ Rx) over 3 iterative rounds across initially correct tasks (\(N_{correct}\)):

| Model | Strategy | Pass (%) | Beyond (%) | B|P (%) | B|P Gain | OPT10 (%) | OPT50 (%) | Regr. (%) | |---|---|---|---|---|---|---|---|---| | DeepSeek-V4-Flash | Basic | 57.7 | 37.5 | 65.0 | - | - | - | - | | DeepSeek-V4-Flash | Vanilla EL | 57.3 | 41.2 | 72.0 | +7.0% | 22.7% | 11.4% | 3.8% | | DeepSeek-V4-Flash | EL w/ Rx (Prescription) | 56.6 | 44.1 | 77.9 | +12.9% | 35.5% | 20.8% | 6.1% | | Kimi-K2.5 | Basic | 40.2 | 29.6 | 73.6 | - | - | - | - | | Kimi-K2.5 | Vanilla EL | 39.4 | 33.7 | 85.5 | +11.9% | 37.6% | 19.4% | 3.2% | | Kimi-K2.5 | EL w/ Rx (Prescription) | 39.3 | 34.7 | 88.3 | +14.7% | 54.5% | 27.6% | 4.5% |

2. Library-Conditioned Routing (Lib-Routing): Capitalizing on library-level model specialization, routing each task on the temporal test set (\(N=304\)) to the top \(K=3\) open-source experts selected via the validation set (\(\alpha=0.35\)), compared against proprietary frontier models:

| Configuration | Model Class | Pass (%) | Beyond (%) | B|P (%) | Cost ($/task) | |---|---|---|---|---|---| | Claude-Opus-4.6 | Closed Best@1 | 63.5 | 43.7 | 68.9 | $0.018 | | Claude-Opus-4.6 Best@3 | Closed Best@3 | 72.0 | 53.8 | 74.6 | $0.052 | | GPT-5.4 | Closed Best@1 | 63.5 | 46.4 | 73.0 | $0.014 | | GPT-5.4 Best@3 | Closed Best@3 | 73.4 | 58.9 | 80.3 | \(0.043 | | **Lib-Routing (K=3)** | Open Routing | **69.4** | **51.4** | **74.1** | **\)0.004** |

Key Findings

  • Decoupling of Correctness and Efficiency: High functional correctness does not imply fast execution. GPT-5.4 leads in Pass (66.9%), but its efficiency score (B|P 71.7%) virtually matches GPT-5.4-mini (71.6%), which solves 47 fewer tasks. Most sharply, Kimi-K2.5 achieves the lowest Pass among frontier models (40.2%) yet ranks #1 in efficiency across all 16 models (B|P 73.6%). Conversely, DeepSeek-V3.2 solves 56.1% of tasks but trails Kimi by 19.4% in efficiency (54.2%).
  • Code-Specialized Fine-Tuning Fails to Bridge Efficiency Gaps: Open Coder models average only 50.2% B|P, substantially trailing generalist Open Frontier models (64.3%) despite extensive code pretraining. The human reference reaches 86.6% B|P (outperforming all LLMs by \(\ge 13.0\%\)), showing ample room for optimization.
  • Migration of Weakness Across Capability Tiers: As model capability scales, efficiency bottlenecks shift from elementary Loop-to-Vectorization (L2V drops from 29.6% in Open Coder to 7.3% in Closed Frontier) to insidious Overhead-Aware Stack Matching (OSM climbs from 9.3% in Open Coder to 29.3% in Closed Frontier). Specifically, 45.5% of GPT-5.4's slow instances are caused by OSM due to defensive wrappers, redundant checks, and local imports.
  • Library-Specific Rotation of Efficiency Leadership: No single model dominates across all libraries. Kimi-K2.5 leads in NumPy (79.6%), Pandas (74.0%), and SciPy (87.6%); GPT-5.4-mini leads PyTorch (60.9%); DeepSeek-V4-Flash leads Polars (72.1%). This library-level specialization enables cheap open routing to challenge closed frontier models.

Highlights & Insights

  • Shifting Benchmarking from LeetCode Algorithms to Library Idioms: The benchmark exposes that 79.1% of efficiency deficits in data science code generation stem from library primitive misuse and system overhead rather than theoretical asymptotic complexity.
  • Human-Validated Ground Truth Outclasses Naive Acceptance: Over 82.5% of reference solutions refine or entirely depart from accepted Stack Overflow answers, ensuring that the benchmark reflects professional DS engineering rather than casual community snippets.
  • Over-Engineering as a Hidden Pitfall of Frontier Models: Advanced frontier models tend to write overly defensive, production-styled scaffolding that severely penalizes execution latency in compute-intensive inner loops.
  • Cost-Effective Routing Approaches SOTA: Library-conditioned routing among open models achieves 74.1% B|P, approaching Claude-Opus-4.6 Best@3 (74.6%) at \(13.0\times\) lower cost ($0.004 vs. $0.052 per task).

Limitations & Future Work

  • Language Scope Restricted to Python: Current evaluation focuses exclusively on Python data science ecosystems, leaving R, Julia, and C++ scientific computing unexplored.
  • Hardware and Environment Sensitivity: Measured runtime depends directly on hardware architecture (CPU/GPU specs, cache hierarchy) and BLAS/LAPACK library versions, requiring unified hardware for apples-to-apples comparisons.
  • Single-Function Task Scope: Tasks evaluate single-function generation rather than multi-step, interactive exploratory analysis across stateful Jupyter Notebooks.
  • vs Mercury / EffiBench: While Mercury and EffiBench benchmark algorithmic efficiency on LeetCode problems, DSEffi-Bench explores data science code where runtime differences are governed by framework dispatch, vectorized primitives, and layout alignment.
  • vs DS-1000 / DSCodeBench: Existing DS code generation benchmarks evaluate functional correctness only. DSEffi-Bench proves that correctness is decoupled from efficiency and provides the first systematic efficiency evaluation in data science.
  • vs Effi-Learner (EL): Standard EL uses raw profiling feedback alone. This paper demonstrates that injecting a structured, domain-specific taxonomy prescription (EL w/ Rx) yields substantially larger test-time speedups (+14.7% vs. +11.9% on Kimi-K2.5).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First benchmark targeting execution efficiency in data science code generation, highlighting critical correctness–efficiency decoupling.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ 16 models across 3 tiers, 1,000 human-validated tasks, 868 fine-grained diagnostic pairs, profile-guided optimization, and routing experiments.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, rigorous methodology, transparent evaluation protocols, and highly insightful error taxonomy.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational standard and actionable taxonomy for developing efficient, practical code generation models in data science.