ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Jinsung-Jeon/ANI_EMNLP
Area: LLM Reasoning
Keywords: Numerical Reasoning, Representation Learning, Adaptive Injection, Fourier Number Embedding, Semantic Probing
TL;DR¶
ANI introduces a plug-and-play adaptive numerical injection framework that identifies the layer of peak contextual maturity via linear probing, discriminates between nominal identifiers and quantitative operands via a context-aware gating network, and injects Fourier numerical topology exclusively at the terminal sub-token via an additive residual connection, boosting MATH by 9.5 points while preserving linguistic integrity.
Background & Motivation¶
Large Language Models have achieved impressive milestones across complex multi-step logical reasoning and open-domain text comprehension, yet they consistently encounter severe performance bottlenecks when tasked with rigorous arithmetic operations and composite numerical reasoning. Standard autoregressive Transformer architectures fundamentally rely on discrete subword tokenization algorithms (such as Byte-Pair Encoding) that fragment continuous numbers into arbitrary digit sub-tokens—for instance, splitting the four-digit integer 1234 into disjoint text pieces such as "12" and "34". This structural fragmentation destroys the inherent numerical magnitude, fractional precision, and modular arithmetic continuity of numbers, forcing the model to infer calculations based on surface-level text prediction heuristics rather than genuine mathematical computation.
To mitigate this representational defect, recent research has explored continuous numerical embeddings (such as xVal and FoNE) that project scalar values into continuous, single-token vector spaces. However, these architectures suffer from context-agnostic substitution: they indiscriminately replace every numerical subword sequence at the input embedding layer with continuous numeric vectors, overlooking the functional duality of numbers in natural language. In everyday contexts, such as "Room 505 has a capacity of 10", the number "505" serves as a non-calculable nominal identifier (an arbitrary room label) that must retain its textual identity, whereas "10" is an operational quantitative value intended for mathematical calculation. Naive input-level replacement strips nominal identifiers of their linguistic semantics, triggering severe semantic erasure and catastrophic forgetting on general-purpose NLP tasks and tool-use agentic routing.
Reconciling the tension between signal entanglement caused by naive additive fusion and semantic erasure induced by static substitution requires determining the functional role of numbers from their linguistic environment. Because input-level representations lack the contextual depth necessary to distinguish identifiers from operands, numerical intervention must be shifted to intermediate Transformer layers where hidden states reach contextual maturity. Core idea: Formulate numerical enhancement as an internal semantic probing mechanism where an optimal intermediate layer—identified via linear diagnostic probing—governs a context-aware gating network that selectively injects Fourier numerical topology at the terminal sub-token bottleneck, unifying arithmetic precision with linguistic preservation.
Method¶
Overall Architecture¶
ANI is implemented as a lightweight, plug-and-play intervention module integrated into the intermediate hidden layers of a pre-trained Decoder-only Transformer. The overarching operational workflow consists of four sequential stages: 1. Optimal Layer Selection: An intermediate layer \(l\) is located by training layer-wise linear diagnostic probes to identify the depth at which the hidden representations attain maximal contextual maturity for disambiguating nominal versus quantitative roles; 2. Topology Encoding (Numerical Path): Raw numerical scalars extracted from the sequence are converted into multi-frequency periodic feature vectors via Fourier Number Embedding (FoNE) and projected into the model's latent hidden dimension using a learnable linear transformation; 3. Semantic Probing (Gating Path): Taking the hidden state of the terminal sub-token (which acts as a causal information aggregation bottleneck) concatenated with the projected numerical vector, a lightweight gating MLP predicts an activation coefficient, regularized by a parabolic binarization loss during training and evaluated via hard thresholding at inference; 4. Adaptive Integration: Quantitative operands activate an additive residual connection that injects the aligned mathematical topology, nominal identifiers close the gate for zero-interference identity preservation, and all non-numerical textual tokens completely bypass the module.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Token Sequence S & Scalar Set V"] --> B["Stage 1: Optimal Layer Selection<br/>Linear Probes Locate Peak Contextual Maturity Layer l"]
B --> C["Stage 2: Topology Encoding (Numerical Path)<br/>FoNE Periodic Encoding + Linear Manifold Alignment"]
B --> D["Stage 3: Semantic Probing (Gating Path)<br/>Terminal Sub-token Bottleneck + Gating MLP Decision"]
C & D --> E["Stage 4: Adaptive Integration<br/>Nominal Preservation α=0 / Quantitative Injection α=1 Residual"]
E --> F["Residual Hidden Layers Forwarding & Output"]
Key Designs¶
1. Optimal Layer Selection: Resolving the Premature Functional Ambiguity of Input-Level Interventions
Input-level embeddings lack the bidirectional linguistic context required to distinguish between nominal identifiers and quantitative operands, forcing static substitution models into premature misclassifications. To overcome this, ANI shifts the intervention to an intermediate layer characterized by high contextual maturity. The authors construct a balanced diagnostic dataset comprising 10,000 quantitative instances from OpenMathInstruct-2 and 10,000 nominal instances from Magpie (strictly filtered to eliminate all mathematical and algorithmic vocabulary). By training frozen layer-wise linear classifiers across candidate layers, the probing accuracy reveals a distinct progression that peaks at Layer 24 for Qwen3-8B with 93% accuracy (compared to 72% at Layer 3 and degraded representations in late layers). This empirically validates that Layer 24 provides the most contextually enriched semantic state for discriminating numerical functionality, providing a reliable foundation for downstream adaptive gating.
2. Topology Encoding and Manifold Alignment: Overcoming Subword Fragmentation and Manifold Mismatch
Standard tokenizers shatter multi-digit integers into arbitrary subwords, distorting arithmetic magnitude and cyclic properties. To maintain mathematical fidelity, ANI adopts Fourier Number Embedding (FoNE) as its foundation encoder, mapping a raw scalar \(x\) across multiple periodic sine and cosine frequencies into a continuous vector \(\phi(x) \in \mathbb{R}^{d_{\text{fourier}}}\). Because raw Fourier coordinates inhabit a mathematical space whose geometric distribution conflicts with the pre-trained semantic space of the LLM, naive injection induces signal interference. ANI bridges this distributional gap by introducing a learnable linear projection matrix \(W_P \in \mathbb{R}^{d_{\text{model}} \times d_{\text{fourier}}}\), mapping the periodic vector into the hidden representation space of the backbone: $\(e_{\text{num}} = W_P \cdot \phi(x)\)$ This projects the holistic mathematical topology of the number into a single aligned latent representation, eliminating the structural flaws of multi-token subword segmentation.
3. Semantic Probing and Terminal Token Gating: Eliminating Multi-token Signal Redundancy via Causal Bottlenecks
Injecting continuous vectors across all subwords of a fragmented number \(S = \{t_1, \dots, t_L\}\) creates \(L\)-fold redundant interference, overwhelming local textual representations with repetitive mathematical noise. ANI circumvents this by exploiting the causal attention mechanism of autoregressive decoders: the terminal sub-token \(t_L\) serves as an information bottleneck that synthesizes the preceding numerical chunk. Consequently, ANI restricts numerical injection strictly to the terminal token's hidden vector \(h_{t_L}^{(l)}\). A two-layer gating MLP evaluates the contextual necessity of numerical intervention by processing the concatenated state: $\(z_{\text{num}} = \text{MLP}_{\text{gate}}([h_{t_L}^{(l)}; e_{\text{num}}]), \quad \alpha_{\text{num}} = \sigma(z_{\text{num}})\)$ To enforce a deterministic binary decision while preserving end-to-end differentiability, training incorporates a parabolic binarization penalty \(\lambda \mathbb{E}[\alpha_{\text{num}}(1 - \alpha_{\text{num}})]\). At inference, the gate executes a strict hard threshold \(\alpha_{\text{num}} = \mathbb{I}(\sigma(z_{\text{num}}) > 0.5)\). When \(\alpha_{\text{num}} = 0\), nominal identifiers remain untouched (\(\tilde{h}_{t_L}^{(l)} = h_{t_L}^{(l)}\)); when \(\alpha_{\text{num}} = 1\), quantitative operands receive additive mathematical topology (\(\tilde{h}_{t_L}^{(l)} = h_{t_L}^{(l)} + e_{\text{num}}\)), ensuring zero semantic erasure for linguistic identifiers alongside maximal computational gain for operands.
Loss & Training¶
The framework is optimized using Quantized Low-Rank Adaptation (QLoRA) with rank \(r=64\) and alpha \(\alpha=128\), freezing the base LLM parameters while updating only the LoRA adapters, the linear projection layer \(W_P\), and the gating MLP (configured with a dropout rate of 0.1). The training loss combines standard Next-Token Prediction (NTP) cross-entropy with the binarization penalty: $\(\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda \mathbb{E}[\alpha_{\text{num}}(1 - \alpha_{\text{num}})]\)$ where \(\lambda\) is set to 0.1. Training is conducted on the 20,000-sample balanced diagnostic dataset using the AdamW optimizer with a constant learning rate of \(5 \times 10^{-5}\), per-device batch size of 1, and 32 gradient accumulation steps. High numerical resolution is preserved by fixing integer and fractional lengths to 10 and 6 digits, supporting sequence lengths up to 4,096 tokens.
Key Experimental Results¶
Main Results¶
The primary evaluation of ANI on Qwen3-8B across mathematical reasoning (MATH, GSM8K) and general language understanding (MMLU) under 4/8-shot Chain-of-Thought prompting is summarized below:
| Method Category | Model / Method | MATH (Acc %) | GSM8K (Acc %) | MMLU (Acc %) |
|---|---|---|---|---|
| Official Reference | Qwen3-8B (Official) | 49.28 | 89.84 | 73.75 |
| Text Fine-tuning | Standard SFT | 55.18 | 87.79 | 72.24 |
| Input Additive | Digit-wise (Additive) | 56.44 | 88.17 | 72.56 |
| Input Substitution | xVal | 58.32 | 90.83 | 72.50 |
| Input Substitution | FoNE | 55.08 | 89.61 | 72.30 |
| In-context Reformat | NumeroLogic | 47.56 | 42.99 | 72.45 |
| Proposed Method | ANI (Ours) | 58.76 | 90.14 | 72.67 |
In the N-Digit Arithmetic Scaling benchmark (exact match accuracy across 100 evaluation problems per length), ANI demonstrates exceptional resilience against token fragmentation:
| Method | 2-digit | 4-digit | 6-digit | 8-digit | 10-digit | 12-digit |
|---|---|---|---|---|---|---|
| Qwen3-8B (Pre-trained) | 75.0 | 48.0 | 6.0 | 0.0 | 0.0 | 0.0 |
| Baseline (SFT) | 100.0 | 95.0 | 88.0 | 73.0 | 44.0 | 50.0 |
| ANI (Ours) | 100.0 | 98.0 | 88.0 | 67.0 | 68.0 | 82.0 |
Scalability evaluations across different model sizes and families confirm the cross-architecture portability of ANI: - Llama-3.1-8B: Pre-trained 20.5% MATH / 55.3% GSM8K \(\to\) Baseline SFT 26.4% / 63.5% \(\to\) xVal 21.2% / 59.5% \(\to\) ANI 27.2% / 63.5%; - Mistral-7B-v0.3: Pre-trained 11.4% / 37.5% \(\to\) Baseline SFT 16.6% / 52.4% \(\to\) xVal 16.2% / 51.0% \(\to\) ANI 15.0% / 54.4%; - Qwen3-4B: Baseline SFT 48.2% MATH / 82.8% GSM8K \(\to\) ANI 49.5% / 85.2%; - Qwen3-32B: Baseline SFT 62.5% MATH / 92.0% GSM8K \(\to\) xVal 61.6% / 89.2% \(\to\) ANI 63.1% / 93.5%.
Ablation Study¶
The ablation analysis validating the gating mechanism and injection locus on Qwen3-8B yields:
| Configuration | MATH (Acc %) | GSM8K (Acc %) | MMLU (Acc %) | Note |
|---|---|---|---|---|
| ANI (Full Model) | 58.76 | 90.14 | 72.67 | Adaptive gating at terminal sub-token bottleneck |
| Static Injection (No Gating) | 8.60 | 62.70 | 71.41 | Indiscriminate continuous replacement collapses reasoning |
| Digit-wise Injection (Per-token Gate) | 58.70 | 90.37 | 72.29 | Broadcasting gate to all subwords degrades language fluency |
Key Findings¶
- Catastrophic Failure of Ungated Substitution: Removing the adaptive gating network (Static Injection) causes MATH accuracy to collapse completely to 8.60% and degrades MMLU to 71.41%. This empirically validates that indiscriminately forcing numerical embeddings into every number token introduces disruptive feature noise that destroys linguistic coherence.
- Sparse Terminal Injection vs. Digit-wise Broadcasting: While Digit-wise Injection matches arithmetic accuracy (58.70% on MATH), its MMLU performance drops to 72.29% due to cumulative multi-token signal redundancy. Injecting solely at the terminal token bottleneck minimizes semantic interference while preserving maximum mathematical gains.
- Fine-grained Domain Discrimination and Tool Routing: On the FiNER-139 financial dataset, ANI accurately identifies and bypasses nominal identifiers with 75.8% precision. Furthermore, in deceptive agentic tool-use scenarios with nominal traps (e.g., "Employee 2048"), ANI reduces argument misrouting from 30.4% in the baseline down to 7.1%, achieving a 92.9% correct tool routing rate.
- Statistically Significant High-Difficulty Gains: A paired bootstrap resampling analysis on MATH reveals that while gains on Levels 1–3 are modest (\(\Delta = -0.0049\), \(p=0.8303\)), ANI achieves a statistically significant improvement on the most challenging Levels 4–5 problems (\(\Delta = +0.0134\), \(95\%\text{ CI } [+0.0004, +0.0268]\), \(p=0.0241\)).
Highlights & Insights¶
- From Input-Level Replacement to Intermediate Contextual Probing: By demonstrating that language models achieve peak disambiguation capacity only at intermediate layers (Layer 24 in Qwen3-8B), ANI provides a powerful conceptual shift away from premature input-level feature modifications.
- Leveraging the Causal Information Bottleneck: The choice to intervene exclusively at the final sub-token of a numerical entity exploits the autoregressive structure of decoder transformers, delivering a mathematically complete representation with minimal intervention overhead.
- Differentiable Binarization Regularization: The introduction of a parabolic penalty \(\lambda \alpha(1-\alpha)\) enables smooth end-to-end gradient backpropagation during training while guaranteeing deterministic binary thresholding at inference for zero-interference nominal retention.
Limitations & Future Work¶
- Finite Decimal Dynamic Range: The underlying FoNE encoder configuration is constrained to 10 integer digits and 6 fractional digits, which may prove insufficient for extreme scientific computing contexts requiring arbitrary-precision arithmetic.
- Probe-Dependent Optimal Layer Identification: Determining the optimal injection layer currently requires offline linear diagnostic probing across individual model architectures; developing an automated, cross-model adaptive routing layer remains an open direction.
- Inference Latency in Ultra-Fast Deployments: While the gating MLP and linear projection overhead are minimal relative to the backbone parameters, intermediate layer branching introduces minor scheduling latency in extreme throughput-sensitive environments.
Related Work & Insights¶
- vs xVal (Golkar et al., 2023): xVal maps numbers to continuous single-token embeddings using magnitude-scaled vectors at the input layer. While effective for pure calculation, its context-agnostic replacement causes semantic erasure; ANI overcomes this by postponing injection to an intermediate contextual layer governed by a binary gate.
- vs FoNE (Zhou et al., 2025): FoNE captures multi-frequency Fourier features for modular arithmetic but relies on static replacement; ANI preserves FoNE's representational strength while eliminating its catastrophic forgetting via sparse additive residual gating.
- vs NumeroLogic (Schwartz et al., 2024): NumeroLogic relies on text-level pre-processing by prefixing digit counts directly in the prompt; ANI performs representation-level latent intervention, avoiding prompt length inflation and brittleness across disparate phrasing.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering the concept of contextual maturity probing for adaptive, role-aware numerical latent routing.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation spanning multiple architectures, N-digit scaling, layer sweeps, bootstrap statistical significance, and agentic tool-use traps.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical formalization, and insightful architectural analyses.
- Value: ⭐⭐⭐⭐⭐ Offers a foundational paradigm for harmonizing symbolic precision with natural language processing in large language models.