Skip to content

The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/the-mysh/gsm-symbolic-benchmarking
Area: LLM Reasoning
Keywords: GSM-Symbolic, Generalized Linear Mixed Models, Statistical Significance, Large-Number Effect, Mathematical Reasoning

TL;DR

This paper provides a rigorous statistical re-evaluation of the high-profile GSM-Symbolic benchmark, demonstrating that when accounting for item-level variability via bootstrapped Generalized Linear Mixed Models (GLMMs) and controlling for an unacknowledged systematic shift toward larger numbers, the claimed widespread performance drops vanish for almost all models, revealing diverse behavioral failure profiles (such as variable binding fragility, arithmetic limits, and dual-task interference) rather than a uniform lack of reasoning.

Background & Motivation

As Large Language Models (LLMs) continue to advance toward robust problem-solving, reliable benchmarking of their mathematical and logical reasoning capabilities has become paramount, with grade-school math datasets like GSM8K serving as the community standard. However, the omnipresent risk of benchmark data contamination in modern pretraining corpora has fueled persistent skepticism regarding whether state-of-the-art models genuinely reason or merely memorize training problems. Addressing this concern, the recent GSM-Symbolic study (Mirzadeh et al., 2025) synthesized 50 template-based variants for each of 100 sampled GSM8K questions—creating a 5,000-question dataset—and reported consistent accuracy declines across 25 leading open and proprietary LLMs. From these empirical deltas, the authors concluded that current LLMs lack genuine reasoning capabilities and instead rely on brittle, superficial pattern matching, igniting widespread debate across the AI research community.

Despite the sweeping impact of this claim, the conclusion rests on a precarious statistical foundation that overlooks the classical "language-as-fixed-effect fallacy" first identified in psycholinguistics. By directly contrasting raw aggregate accuracy means between the original 100 questions and the 5,000 generated variants, the original study treated linguistic test items as fixed entities rather than random samples from a broader question distribution. This failure to model the non-independence of items sharing the exact same underlying logic template severely inflates false-positive rates and conflates random template-level variance with genuine model deterioration. Unchecked methodological oversights of this nature directly echo the systemic factors that precipitated historical replication crises in experimental psychology and pre-clinical medicine.

Furthermore, a closer inspection of the generated benchmark data reveals an unacknowledged confounding variable: the distribution of integers within GSM-Symbolic problem texts is systematically and heavily shifted toward significantly larger numerical values compared to the original GSM8K. Because arithmetic execution difficulty scales with digit count and sub-tokenization splits, a performance decline on larger numbers does not inherently indicate a collapse of logical inference structure. Core idea: Rigorous assessment of LLM reasoning requires statistical earnestness via bootstrapped Generalized Linear Mixed Models (GLMMs) with per-question random effects, explicitly disentangling arithmetic large-number confounds and probing behavioral failure profiles through controlled prompt reformulations.

Method

Overall Architecture

The evaluation framework re-assesses 20 open-weight LLMs on the GSM-Base (100 questions) and GSM-Variants (5,000 questions) datasets. The methodology unfolds in three sequential phases: first, replicating the GSM-Symbolic benchmark and fitting a hierarchical logistic GLMM with template-level random intercepts across 2,000 bootstrap iterations; second, formulating an arithmetic load metric to test and statistically control for the large-number distribution shift; and third, deploying four controlled prompt perturbations across natural language and executable code representations to isolate distinct cognitive and behavioral failure mechanisms.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Benchmark Input Data<br/>GSM-Base (100) and GSM-Variants (5000)"] --> B["Hierarchical GLMM Modeling & Bootstrapping<br/>Binary logit (1|Id) over 2000 cluster resamples"]
    B --> C["Large-Number Metric Extraction & Control<br/>Sum of log-integers γ centered in GLMM 2"]
    C --> D["Behavioral Probe Prompt Perturbations<br/>Simple/Structured NL and Executable Code prompts"]
    D --> E["Granular Error Attribution & Profiling<br/>Distinguishing binding fragility, arithmetic & dual-task load"]

Key Designs

1. Hierarchical GLMM Modeling & Bootstrapping: Mitigating the Language-as-Fixed-Effect Fallacy and Non-Convergence
Standard benchmark evaluation calculates raw accuracy deltas \(\Delta_{\text{var}} = A_{\text{GSM-Variants}} - A_{\text{GSM-Base}}\), ignoring the hierarchical clustering wherein 50 variants share an identical root template with each base question. The authors formalize response correctness as a Generalized Linear Mixed Model (GLMM 1) with a binomial family and logit link: $\(\text{Correct} \sim Variant + (1|Id)\)$ Here, \(Variant \in \{0, 1\}\) represents the fixed effect of belonging to the variant dataset, while \((1|Id)\) assigns a random intercept to each template \(Id\), capturing baseline question difficulty heterogeneity. To circumvent asymptotic normality assumptions and resolve degenerate, non-convergent single fits observed in classical Wald tests (e.g., severe gradient tolerance violations), the authors implement a cluster bootstrap with 2,000 resamples over question templates. Odds ratios (OR) are computed from the median clean log-odds estimates, and \(P\)-values are rigorously derived from bilateral tail probabilities, accompanied by Holm-Bonferroni family-wise error rate corrections.

2. Large-Number Metric Extraction & Control: Disentangling Arithmetic Tokenization Load from Logical Reasoning
The paper identifies a pronounced rightward distribution shift in integers between GSM-Base and GSM-Variants (\(D = 0.1201, p < 0.001\) under a two-sample Kolmogorov-Smirnov test), contradicting prior assumptions of arithmetic invariance. To control for this confound, the authors formulate a question-level numeric load metric \(\gamma\) defined as the sum of base-10 logarithms of all explicitly stated integers: $\(\gamma = \sum_{n \in N} \log_{10}(|n|)\)$ where \(N\) is the multiset of numeric integers in the prompt. This metric directly reflects tokenization splits and arithmetic calculation depth. Centering \(\gamma\) to zero mean yields \(\gamma_c\), which is incorporated as a second fixed effect in GLMM 2: $\(\text{Correct} \sim Variant + \gamma_c + (1|Id)\)$ This allows the authors to independently test whether larger numbers predict decreased accuracy and whether the variant effect retains statistical significance after neutralizing the arithmetic load.

3. Behavioral Probe Prompt Perturbations: Isolating Surface Memorization, Variable Binding, and External Execution
To diagnose the true root causes for models demonstrating statistically significant variant deltas, four contrasting prompt formats are developed: - Simple NL Prompt: Preserves mathematical reasoning connectives ("First, calculate...", "Next...") while stripping the explicit "Let's think step by step" prompt directive and original few-shot wording, assessing reliance on exact GSM8K phrasing strings; - Structured NL Prompt: Imposes rigid structural sections ("Given:", "To find:", "Solution:"), providing external working memory scaffolding to test whether models suffer from the fragility of internal variable binding and object attribute tracking; - Simple Code & Structured Code Prompts: Instruct the LLM to implement an argument-free Python function whose execution on a secure interpreter yields the final numerical result, offloading numerical calculation to examine pure symbolic logic formulation.

4. Granular Error Attribution & Profiling: Decomposing Non-Logical Output Collapse
Rather than conflating all incorrect outputs into an undifferentiated error bucket, a comprehensive diagnostic taxonomy is enforced. For natural language prompts, responses are classified into: correct answer, wrong numerical answer, last-number regex fallback match, missing numerical output, and empty response. For code prompts, an isolated Python runtime tracks 10 distinct failure classes including syntax errors, name errors, type/value errors, zero division, and infinite generation loops (e.g., generating endless zeros leading to overflow), allowing the removal of superficial syntax artifacts to inspect pure reasoning accuracy.

Benchmark & Evaluation Protocol

The empirical study evaluates 20 open-weight LLMs up to 27B parameters across Google Gemma, Gemma-2, Microsoft Phi-2, Phi-3/3.5, Mistral, Mathstral, and Meta Llama-3 families. Re-evaluation proceeds from reproducing the original benchmark (Ex1), through evaluating alternative prompts on statistically significant models (Ex2.A–Ex2.D), to fitting GLMM 2 on pooled data to quantify the large-number effect and residual variant impact (Ex3.A–Ex3.D).

Key Experimental Results

Main Results: Reproduction and GLMM Significance Testing under Canonical GSM Prompt (Ex1)

The table below reports baseline accuracy on GSM-Base, accuracy on GSM-Variants, raw accuracy deltas (\(\Delta_{\text{var}}\)), GLMM 1 odds ratios (OR) with 95% confidence intervals, and raw vs. Holm-Bonferroni-corrected \(P\)-values across all 20 evaluated models.

Model GSM-Base Acc (%) GSM-Variants Acc (%) Variant Delta \(\Delta_{\text{var}}\) (pp) Odds Ratio OR [95% CI] Raw \(P\)-value Corrected \(P\)-value Significance (\(\alpha=0.05\))
gemma-2b 19.0 12.0 -6.96 0.42 [0.22, 0.91] 0.030 0.443 Significant (uncorrected)
gemma-2b-it 10.0 7.5 -2.52 0.63 [0.26, 2.10] 0.380 1.000 Not significant
gemma-7b 48.0 47.7 -0.30 0.98 [0.53, 1.71] 0.954 1.000 Not significant
gemma-7b-it 31.0 22.8 -8.22 0.50 [0.25, 0.97] 0.043 0.588 Significant (uncorrected)
gemma-2-2b 24.0 20.9 -3.12 0.73 [0.39, 1.41] 0.321 1.000 Not significant
gemma-2-2b-it 42.0 41.8 -0.18 0.99 [0.52, 1.86] 0.993 1.000 Not significant
gemma-2-9b 69.0 61.1 -7.88 0.43 [0.17, 0.92] 0.024 0.432 Significant (uncorrected)
gemma-2-9b-it 86.0 82.6 -3.40 0.60 [0.18, 1.31] 0.227 1.000 Not significant
gemma-2-27b-it 90.0 87.6 -2.40 0.58 [0.14, 1.81] 0.378 1.000 Not significant
phi-2 60.0 39.7 -20.26 0.23 [0.13, 0.39] <0.001 <0.001 Globally Significant
Phi-3-mini-128k-instruct 80.0 78.7 -1.30 0.85 [0.38, 1.76] 0.710 1.000 Not significant
Phi-3-medium-128k-instruct 91.0 88.2 -2.78 0.56 [0.13, 1.56] 0.304 1.000 Not significant
Phi-3.5-mini-instruct 87.0 79.4 -7.58 0.33 [0.11, 0.79] 0.013 0.250 Significant (uncorrected)
Mistral-7B-v0.1 38.0 33.0 -5.00 0.67 [0.37, 1.18] 0.190 1.000 Not significant
Mistral-7B-Instruct-v0.1 36.0 27.5 -8.54 0.50 [0.28, 0.91] 0.025 0.432 Significant (uncorrected)
Mistral-7B-v0.3 33.0 35.5 +2.50 1.25 [0.64, 2.43] 0.511 1.000 Not significant
Mistral-7B-Instruct-v0.3 52.0 45.3 -6.68 0.60 [0.34, 1.04] 0.072 0.864 Not significant
Mathstral-7B-v0.1 82.0 75.1 -6.88 0.47 [0.19, 0.96] 0.042 0.588 Significant (uncorrected)
Meta-Llama-3-8B 55.0 47.4 -7.56 0.49 [0.25, 0.91] 0.026 0.432 Significant (uncorrected)
Meta-Llama-3-8B-Instruct 77.0 69.9 -7.08 0.48 [0.21, 1.07] 0.078 0.864 Not significant

Ablation & Controlled Study: Prompt Perturbations and Large-Number Control (Ex2 & Ex3)

For the 8 models exhibiting raw significance under Ex1, the table below highlights performance under alternative prompts and details the large-number effect odds ratio (\(\gamma_c\)) and residual variant significance under GLMM 2.

Model Prompt Format GSM-Base Acc (%) GSM-Variants Acc (%) Variant Delta \(\Delta_{\text{var}}\) (pp) GLMM 1 Raw \(P\)-value Large-Number OR (\(\gamma_c\)) Corrected Variant OR Corrected Variant \(P\)-value
phi-2 GSM Prompt 60.0 39.7 -20.26 <0.001 0.76 (\(p=0.003\)) 0.31 <0.001
phi-2 NL-simple 45.0 37.0 -8.04 0.021 0.72 (\(p=0.008\)) 0.68 0.224
phi-2 NL-structured 63.0 46.0 -17.04 <0.001 0.66 (\(p<0.001\)) 0.35 <0.001
phi-2 code-simple 63.0 66.6 +3.64 0.179 0.90 (\(p=0.480\)) 1.70 0.141
phi-2 code-structured 66.0 63.8 -2.24 0.457 1.00 (\(p=0.981\)) 0.76 0.480
gemma-2b GSM Prompt 19.0 12.0 -6.96 0.030 0.85 (\(p=0.065\)) 0.49 0.081
gemma-2b NL-simple 21.0 10.9 -10.12 <0.001 0.80 (\(p=0.033\)) 0.33 0.003
gemma-2b NL-structured 12.0 14.5 +2.50 0.295 0.78 (\(p=0.004\)) 2.10 0.060
gemma-2b code-simple 14.0 13.4 -0.56 0.835 0.98 (\(p=0.778\)) 0.95 0.909
gemma-2-9b GSM Prompt 69.0 61.1 -7.88 0.024 1.04 (\(p=0.722\)) 0.40 0.013
gemma-2-9b NL-structured 72.0 71.2 -0.84 0.829 0.90 (\(p=0.365\)) 1.03 0.967
Mathstral-7B GSM Prompt 82.0 75.1 -6.88 0.042 0.90 (\(p=0.260\)) 0.49 0.046
Mathstral-7B NL-structured 78.0 77.7 -0.34 0.911 0.82 (\(p=0.025\)) 1.28 0.490
Meta-Llama-3-8B GSM Prompt 55.0 47.4 -7.56 0.026 1.09 (\(p=0.463\)) 0.45 0.013
Meta-Llama-3-8B NL-structured 16.0 13.2 -2.78 0.418 1.06 (\(p=0.644\)) 0.63 0.332
Phi-3.5-mini GSM Prompt 87.0 79.4 -7.58 0.013 0.74 (\(p=0.001\)) 0.59 0.239
Phi-3.5-mini code-simple 81.0 76.6 -4.42 0.050 1.03 (\(p=0.828\)) 0.49 0.045

Key Findings

  • Severe Loss of Statistical Significance: Only 8 of the 20 evaluated models exhibit a statistically significant variant effect prior to multiple-testing correction. Following Holm-Bonferroni correction, only a single model (phi-2, \(p < 0.001\)) remains statistically significant. For 19 out of 20 models, the null hypothesis of no variant effect cannot be rejected.
  • Vanishing Decline under Prompt Perturbations: For 5 of the 8 initially significant models (gemma-2-9b, Mathstral-7B, Meta-Llama-3-8B, gemma-7b-it, Mistral-7B-Instruct-v0.1), the variant effect completely dissolves across all four alternative prompts, demonstrating that their degradation was an artifact of rigid formatting sensitivity rather than a general reasoning failure.
  • Large-Number Confound Verified: The K-S test confirms a severe distributional shift towards larger numbers in GSM-Variants (\(D = 0.1201, p < 0.001\)). Controlling for numeric load \(\gamma_c\) directly accounts for the variant effect in half of the remaining cases under the canonical prompt.
  • Arithmetic Invariance via Code Execution: When arithmetic calculation is delegated to Python execution, the large-number effect odds ratio converges to 1.0 and completely loses significance across all models, proving that number sensitivity in LLMs stems from arithmetic execution rather than logical planning.

Highlights & Insights

  • Deconstruction of Blanket Claims: The paper dismantles the narrative that LLMs uniformly lack reasoning, replacing it with nuanced, model-specific behavioral profiles:
  • Fragility of Variable Binding (gemma-2b): Suffers under simple wording perturbations but recovers completely when provided with structured Given/To-find scaffolding (+2.50 pp delta, not significant), indicating that externalizing object-attribute tracking relieves internal state burdens;
  • Arithmetic Limitation (phi-2): Persistently drops on all natural language variations, yet achieves a positive delta (+3.64 pp) without statistical significance when allowed to generate Python code, pinpointing multi-digit calculation as the sole bottleneck;
  • Dual-Task Interference (Meta-Llama-3-8B): Shows steep accuracy drops and widespread empty responses under structured formatting constraints, mimicking cognitive load collapse when forced to juggle rigid syntax compliance alongside problem-solving;
  • Pattern Reliance (gemma-2-9b & Mathstral-7B): Display immunity to large numbers due to scale or specialized math pretraining, exhibiting variant drops strictly tied to the canonical GSM few-shot phrasing.
  • Wald vs. Bootstrap Divergence: The study identifies an 11% disagreement rate between asymptotic Wald tests and bootstrap estimates, driven by non-convergent, degenerate curvature in boundary GLMM fits, offering a crucial methodological warning for NLP evaluation.

Limitations & Future Work

  • Integer-Only Arithmetic Metric: The numeric metric \(\gamma\) abstracts total integer magnitude but does not account for floating-point values, fractions, specific operational complexities (e.g., carrying/borrowing, prime division), or exact BPE token boundaries.
  • Black-Box Cognitive Metaphors: Inferences regarding "dual-task interference" and "variable binding" are drawn from black-box input-output behaviors; verifying these hypotheses requires mechanistic interpretability tools such as circuit tracing and activation patching.
  • Scope Constrained to Open Models (<30B): The study was constrained to open-weight models below 30B parameters due to compute limitations, leaving closed-source frontier reasoning models (e.g., o1, GPT-4o) unexamined under this specific GLMM framework.
  • vs GSM-Symbolic (Mirzadeh et al., 2025): Mirzadeh et al. claimed foundational reasoning limitations based on unmodeled aggregate drops; this paper proves the drops are largely statistically insignificant under hierarchical GLMMs and confounded by large-number distribution shifts.
  • vs GSM-Ranges (Shrestha et al., 2025) & Single-Token Number Embeddings (Zhou et al., 2026): While prior work highlighted arithmetic vulnerabilities across wide numerical intervals, this work empirically proves via code execution that neutralizing calculation burdens abolishes the apparent reasoning gap on symbolic variants.
  • vs Language-as-Fixed-Effect Literature (Clark, 1973; Card et al., 2020): Reinvigorates classic experimental design rigor in modern LLM benchmarking, establishing that item random effects are essential to avoid costly replication crises in NLP.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First principled deconstruction of GSM-Symbolic through rigorous hierarchical mixed-effects modeling and distribution-shift controls.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation of 20 models, 5 prompt modalities, 2,000 bootstrap fits, and fine-grained code runtime error attribution.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exemplary scientific rigor and prose clarity; lucid balance between statistical theory and empirical LLM analysis.
  • Value: ⭐⭐⭐⭐⭐ Establishes an urgent and necessary methodological benchmark for statistical credibility in LLM evaluation.