Fractional State Space Transition for Long Sequence Modeling¶

Conference: NeurIPS2026
arXiv: 2609.36314
Code: https://github.com/anasiri/frac-ssm
Area: LLM Efficiency / Long-Sequence Modeling
Keywords: fractional dynamics, state-space models, long context, sum-of-exponentials approximation, selective read/write routing

TL;DR¶

Frac approximates fractional long memory with finite exponential modes on a shared geometric timescale bank, then adds token-wise controls and independent read/write routing, preserving bounded recurrent state while raising the 1.3B model's average LongBench score from GDN's 16.0 to 17.9, without leading on every task or throughput measure.

Background & Motivation¶

State-space models (SSMs) compress history into a fixed-size state instead of continuously retaining the full context as self-attention does; this makes the law governing forgetting a central question. Mamba-style methods typically start from ordinary differential equations (ODEs) and their discretizations, with individual modes exhibiting exponential decay. Selective gating can decide whether current content deserves retention, but does not automatically ensure suitable memory weights across distances spanning several orders of magnitude. A single characteristic timescale is often insufficient for the long-range accumulation of sparse events.

Fractional differential equations (FDEs) offer a different starting point: state changes depend on a weighted past trajectory, and their Mittag–Leffler kernels have asymptotically polynomial tails. Directly solving such equations requires history access, however, undermining the finite state and efficient decoding that SSMs aim to preserve. This paper therefore does not attach a larger history cache to an existing SSM; it approximates continuous multi-timescale memory using a finite set of ordinary recurrent states.

This approach also limits the interpretation of its results: a finite exponential sum approximates heavy-tailed behavior over a selected horizon, rather than turning an exponential system into an exact power law over infinite time. Subsequent token selectivity also goes beyond the strict approximation theorem for fixed-coefficient FDEs. Core idea: prescribe forgetting geometry through a theoretically motivated geometric multi-timescale memory bank, then let content determine which modes are written and read, rather than optimizing gating alone over a single exponential forgetting mechanism.

Method¶

Overall Architecture¶

A Frac layer receives hidden representations, applies normalization and an input projection, passes them through FracMixer, and returns updated representations through gated normalization and an output projection. The mixer preserves two branches with different signal origins: the pre-convolution signal generates controls, while the post-convolution signal from a local depthwise causal convolution supplies the content to write. Controls operate on the shared timescale bank, and read/write weights distribute content across its modes and combine their outputs.

The sequence of contributions is as follows: the geometric multi-timescale memory bank supplies fast-to-slow base timescales; interval-frozen controls compute retention factors for the current token; prior-based read/write routing determines where information is stored and retrieved; and the affine scan implementation performs state updates and weighted readout. A learnable direct feedthrough term \(D u_t\) is added to the readout so that local content need not pass entirely through long-term state. Training and prefill use chunked parallel computation, whereas decoding caches mode states and executes a single-step version of the same recurrence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    X["Hidden representations<br/>Normalization and input projection"] --> C["Interval-frozen controls"]
    B["Geometric multi-timescale<br/>memory bank"] --> C
    X --> U["Local causal convolution<br/>Content signal"]
    C -->|Retention and injection factors| R["Prior-based<br/>read/write routing"]
    U --> R
    R --> S["Affine scan implementation"]
    P["Previous mode states"] --> S
    S --> O["Readout and direct feedthrough<br/>Gated normalization and output projection"]
    U -->|Direct feedthrough| O

Solid edges show computational data flow; the training objective is described below rather than being an additional inference module. The memory bank supplies shared base parameters, not a history cache regenerated from the current input.

Key Designs¶

1. Geometric multi-timescale memory bank: represent nonlocal history with finite recurrent modes

The theoretical starting point is a Caputo fractional relaxation equation, where a fixed order and fixed decay parameter determine the continuous system's memory law. Two responses must be distinguished: without external input, the initial state decays according to a Mittag–Leffler function; with external input, historical content accumulates through an impulse-response kernel. Both are heavy-tailed, but their exponents differ and should not be described as one identical power law.

\[ {}^{C}D_t^{\alpha}h(t)=-\lambda h(t)+u(t),\qquad E_{\alpha}(-\lambda t^{\alpha})\sim\frac{t^{-\alpha}}{\lambda\Gamma(1-\alpha)},\qquad g_{\alpha,\lambda}(t)\sim\frac{\alpha\,t^{-\alpha-1}}{\lambda^2\Gamma(1-\alpha)}. \]

These asymptotic relations require fixed \(0<\alpha<1\), \(\lambda>0\), and \(t\to\infty\); the Mittag–Leffler kernels are not exact power laws at every distance. Theorem 1 represents both kernels as continuous nonnegative mixtures of exponential decays. Theorem 2 truncates the timescale range and applies quadrature in logarithmic coordinates, producing shared geometric timescales \(\tau_m=\tau_0q^{m-1}\). Each mode then needs only its own first-order state.

Why use multiple scales rather than one slowly decaying mode? An individual mode has one main memory span; geometric spacing distributes finite modes across several orders of magnitude, with fast modes handling local changes and slow modes covering distant dependencies. The implementation uses \(M=16\) and base timescales from \(1\) to \(2^{17}\), rather than adding states for each new sequence length. This base range exceeds the longest language-model evaluation length, but controls rescale the effective timescales, so endpoint coverage alone is not a memory guarantee for every input.

The theorem gives a uniform error bound for the homogeneous kernel over a finite interval, but an integrated absolute error bound for the impulse response. These different norms are appropriate: the latter has an integrable singularity near zero, whereas finite exponential sums are bounded there and cannot approximate it uniformly pointwise. Proposition 3 uses both errors to bound state approximation for a fixed system, a finite horizon, and bounded inputs. It does not certify arbitrary-length approximation by the trained selective network.

2. Interval-frozen controls: adapt the effective timescales of the same bank to content

For each head and token, the control branch generates a step size \(\Delta_t\), fractional order \(\alpha_t\), and scaling parameter \(\lambda_t\). Softplus keeps the step size and scaling parameter positive, while sigmoid places the order in \((0,1)\). Base timescales are shared, but their effective values are rescaled through \(\lambda_t^{1/\alpha_t}\). The network can therefore change the bank's overall memory span based on input, rather than merely selecting an output among fixed modes.

Freezing controls and input within a token interval makes each mode an ordinary scalar linear system, allowing exact zero-order-hold (ZOH) discretization. The following expression lists both the theoretical injection coefficient and the coefficient actually used in experiments to avoid conflating them.

\[ \tilde\rho_{t,m}=\exp\!\left(-\Delta_t\lambda_t^{1/\alpha_t}/\tau_m\right),\qquad \tilde\beta^{\mathrm{ZOH}}_{t,m}=\frac{1-\tilde\rho_{t,m}}{\lambda_t},\qquad \tilde\beta^{\mathrm{impl}}_{t,m}=1-\tilde\rho_{t,m}. \]

Main-text Eq. (20) uses the exact ZOH coefficient with the divisor. Algorithm 1 and Remark 7 in Appendix D explicitly state that all experiments omit this divisor to decouple timescale control from write amplitude and improve training stability. Only an input-independent, fixed \(\lambda\) makes the two forms equivalent through constant rescaling of state and readout. With token-dependent parameters, the experimental recurrence should not be called an exact discretization of the theoretical FDE.

The implementation also clips \(\Delta\in[10^{-4},1.0]\) and \(\lambda\in[0.25,4.0]\) for numerical stability. For a fixed mode system, “exact” refers to local ODE integration under interval-frozen controls; it does not imply exact solution of a variable-order Caputo FDE when the order changes token by token. Remark 8 characterizes the full network as an architectural generalization and inductive bias derived from the fixed fractional construction, which is more accurate than extending the theorem directly to the learnable network.

3. Prior-based read/write routing: independent storage and retrieval on one timescale coordinate system

Multiple modes still require deciding which content should enter slow states and which modes should supply the current output. Theoretical quadrature coefficients have order-controlled decay at large timescales. Frac therefore uses the negative product of order and log-timescale as a read-routing prior, adding residual logits generated from post-convolution content. Write routing uses the same functional form but an independent map, rather than forcing “worth storing” and “worth reading now” to be the same decision.

\[ c_{t,m}=\operatorname{softmax}_m\!\left(-\alpha_t\log\tau_m+g_{\mathrm{read}}(u_t)_m\right),\qquad b_{t,m}=\operatorname{softmax}_m\!\left(-\alpha_t\log\tau_m+g_{\mathrm{write}}(u_t)_m\right). \]

Both maps are linear projections with learnable scalar sigmoid gates and independent parameters. Softmax normalizes over modes, so priors and content residuals jointly determine relative allocation instead of allowing unconstrained mode amplitudes. The read prior is motivated by theoretical coefficient asymptotics; the write prior is not uniquely prescribed by the theorem. It is the authors' shared addressing scheme, supported empirically by ablations.

Frac is therefore not simply a collection of duplicated independent memory slots. Modes have shared log-timescale semantics that current content can modify; the fractional order participates in both timescale control and routing priors. Separating the multi-timescale bank, learnable controls, and normalized access explains why merely maintaining multiple states does not reproduce the full model.

4. Affine scan implementation: retain parallel prefill and single-step decoding with a long-memory prior

Multiplying the implemented injection factor by the write weight gives each mode's coefficient for current content. Each state update remains “previous state times retention, plus the current write,” and the output is a read-weighted sum of updated states. These weights depend on input rather than recurrent state, allowing scan parameters to be constructed in advance.

\[ s_{t,m}=\tilde\rho_{t,m}s_{t-1,m}+\tilde\beta^{\mathrm{impl}}_{t,m}b_{t,m}u_t,\qquad h_t=\sum_{m=1}^{M}c_{t,m}s_{t,m}. \]

Affine transformations compose associatively, enabling chunked parallel scans for training and prefill; decoding only caches the mode states for each head. The main configuration maintains \(16\times16\times256=65{,}536\) principal recurrent scalars per layer, independent of context length. This count excludes short-convolution buffers and architecture-specific auxiliary caches, and is not the entire model's runtime memory footprint.

The Triton implementation fuses scan-parameter construction, within-chunk state construction, cross-chunk state passing, and read/write mixing, incorporating direct feedthrough into output computation. Within-chunk propagation takes differences of cumulative log-decays and then exponentiates instead of dividing cumulative exponentials, reducing numerical problems such as underflow. Mode count determines the additional cost, so linear sequence-length complexity alone does not establish superiority over every attention implementation.

A Worked Example¶

Consider an early positive event in the synthetic task, followed by a long stretch of background tokens before the model predicts the sign of the weighted event sum. When the event arrives, the local causal convolution produces content, the control branch sets each mode's retention factor, and write routing distributes content across fast and slow states. Subsequent background tokens keep updating the states: fast modes lose the event's contribution quickly, while slow modes may retain a weaker but nonzero contribution.

At the end, independent read routing combines the states and the task head predicts the sign. The mode mixture provides broader timescale coverage than a single exponential over the selected horizon, but the event can still disappear if subsequent controls repeatedly induce strong forgetting. This example explains data flow; it neither assumes that all trained tokens share one fixed power-law kernel nor guarantees exact recovery of arbitrarily many historical events.

Loss & Training¶

Language modeling uses the standard autoregressive next-token prediction objective, without an additional fractional-equation supervision loss. The main experiments train approximately 1.3B-parameter models from scratch on 100B tokens sampled from deduplicated FineWeb-Edu, with the Llama-2 32K vocabulary and fully packed 4K sequences. Frac uses 48 layers, hidden size 2048, and 16 heads. Models are approximately parameter-matched, but layer counts and internal expansion factors are not identical.

Training uses AdamW with initial learning rate \(6\times10^{-4}\), 10% warm-up, weight decay 0.1, global batch size 11M tokens, FSDP, and mixed precision. Ablations instead use a 390M model and 30B tokens, training at 4K and evaluating at 16K; their perplexities are not results from the 1.3B main model.

Key Experimental Results¶

Main Results¶

The following averages come from Tables 2–4, with higher scores being better. LongBench averages combine task scores such as F1, ROUGE, accuracy, and code edit similarity. The short-context average excludes the two perplexity columns. Real-world recall-retrieval truncates inputs to 2K and is not a long-distance extrapolation experiment.

Model	LongBench, 14 tasks	Short context, 8-metric average	Real-world recall-retrieval, 6 tasks
Transformer	14.7	55.3	38.5
Mamba2	12.0	53.8	29.9
GDN	16.0	54.2	30.9
Mamba3-SISO	14.4	54.6	35.1
Mamba3-MIMO	11.4	55.3	36.8
Frac	17.9	55.1	36.4

The LongBench gain over GDN is 1.9 score points, not a relative improvement of 1.9%. The authors count wins on 8 of 14 tasks, not all tasks. For example, Frac scores 11.0 on LCC, well below the Transformer's 26.0, and 12.2 on 2WikiMQA, below Mamba3-SISO's 16.3. Its benefits should not be summarized as stronger performance on every long-text capability.

The synthetic heavy-tail task uses one approximately 200K-parameter layer, trains at length 512, and extrapolates to 128K; Table 6 reports 10 runs. At 128K, Frac achieves accuracy \(63.8\pm1.9\) versus GDN's \(58.4\pm3.8\), but at the training length Frac's \(96.5\pm2.5\) trails Mamba3's \(98.6\pm0.7\). This supports slower degradation at long distances, not superiority at every length. The four-task MADLab average is 75.4 versus Mamba3's 74.8, with saturated ICR and N-ICR omitted from the main table.

Language-model NIAH is a separate experiment: models train at 4K and are tested up to 64K. Figure 3 shows slower degradation overall, but GDN is an important exception on S-NIAH-1 with repetitive filler text. The HG38 DNA experiment uses approximately 7M parameters and multiple training lengths from 1K to 64K, testing at the corresponding training length. It studies benefits from extending context length, not extrapolation from short training sequences to longer test sequences.

Ablation Study¶

Table 9 uses 390M parameters, 30B tokens, and 4K training. All three columns below are perplexities on 16K long documents, where lower is better.

Config	ProofPile	PG19	GovReport
Full Frac	47.2	28.4	11.3
\(\alpha=1\)	61.3	36.2	19.7
Without write prior	68.5	41.6	18.2
Pure multiscale bank	76.3	47.1	17.2
Without read/write softmax	80.4	36.9	15.5
Freely learned timescales	54.1	34.3	13.5
Without \(D\)	70.9	46.4	23.6
\(M=8\)	51.4	33.9	14.5
\(M=32\)	47.9	29.1	10.6

The pure multiscale bank simultaneously fixes the order, removes both priors, and changes timescale initialization. It is a combined ablation, so its full gap cannot be attributed to one factor. Freely learning timescales alone also worsens results, supporting the geometric layout; removing direct feedthrough is particularly damaging on GovReport. Increasing to 32 modes does not improve all three columns, and the authors report 5% more parameters and 7% slower computation.

Appendix F, Table 8 defines the system-cost boundary. All entries are tokens/s; prefill uses batch size 1 and decoding averages generation of 64 tokens. The Transformer decode entry is measured at a 16K context.

Model	Prefill 1K	Prefill 4K	Prefill 16K	Decode	Train 4K
Transformer	55,839	79,173	56,297	62	218,184
Mamba2	15,362	41,060	67,571	34	278,620
GDN	10,455	35,481	58,384	20	215,357
Mamba3-SISO	19,517	44,998	74,314	27	238,924
Mamba3-MIMO	13,692	30,710	46,707	25	200,994
Frac	16,213	49,946	77,490	33	241,598

Key Findings¶

Long-context gains and preservation of short-context ability should be discussed separately: Frac's 55.1 still trails both 55.3 scores, and its 2K recall-retrieval score of 36.4 trails 38.5 and 36.8. Competitive performance is not universal leadership.
Frac leads 16K prefill at 77,490, but the Transformer is faster at 1K and 4K. Frac's decode throughput of 33 trails the Transformer's 62 and Mamba2's 34, and it is not fastest in training either.
Table 10 fits coefficients for a fixed homogeneous kernel under a simplex constraint. Mean maximum absolute errors for 8, 16, and 32 modes are respectively \(1.34\times10^{-3}\), \(6.82\times10^{-4}\), and \(6.45\times10^{-4}\). This is kernel fitting over a particular normalized time interval, not the end-to-end error of learned read/write routing.

Highlights & Insights¶

Treating the memory law as an architectural axis. Forgetting curves, associative update rules, and memory capacity address different problems. Frac mainly changes the first, making it potentially complementary to delta-rule updates rather than a replacement for all associative-memory mechanisms.
Theory guides structure without certifying the entire network. Nonnegative diffusive representations explain geometric timescales and the read prior; the write prior, softmax routing, and implemented injection amplitude remain architectural choices. Keeping these separate clarifies what the theory actually constrains.
Multiple modes need not imply a larger principal recurrent state. The mode bank in this configuration is smaller than the compared models' principal states, suggesting that better timescale allocation can matter more than indiscriminately adding state elements. This observation does not establish lower total runtime memory in every implementation.

Limitations & Future Work¶

The finite bank depends on mode count and timescale coverage; dependencies outside that range or tasks requiring rapid forgetting may benefit less. At sufficiently long times, a fixed finite exponential sum is still governed by its slowest exponential mode, not an exact infinite-horizon power law.
Fixed-system theory does not establish that the token-selective model is an exact variable-order FDE solver, and the implemented injection coefficient deliberately differs from the main-text ZOH form. Future work could explicitly analyze how changing controls and routing affect kernel error and stability.
The 1.3B main comparison approximately matches parameters, not layer counts, state capacity, or implementation optimization. Ablations do not repeat every small change across multiple scales on the main tasks, so one perplexity table cannot establish universal causal effects.
Kernel-approximation experiments cover only a fixed homogeneous system without input, optimizing coefficients over a finite normalized time interval. They do not jointly validate selective input responses, the near-zero singularity, or actual trained routing.
Custom Triton kernels and baseline kernels differ in optimization maturity; throughput conclusions depend on hardware, batch size, and sequence length. Broader hardware and batch coverage, together with separate reporting of caches, convolution buffers, and total runtime memory, would strengthen the evidence.

vs Mamba / Mamba2 / Mamba3: These methods strengthen selectivity, computational structure, or discretization expressiveness; Frac derives timescale modes from fractional kernels. Frac's multiple modes are not Mamba3's MIMO rank projections, and should not be conflated merely because both involve multiple state pathways.
vs GDN / delta-rule models: These models improve storage and retrieval precision through associative updates, whereas Frac emphasizes retention across distances. Combining fractional timescales with corrective associative updates is an author-proposed direction, not a result validated in this paper.
vs RetNet / Mixture-of-Memories: Fixed decay across heads and multiple independent memories can increase timescale coverage or capacity. Frac additionally introduces fractionally motivated geometric modes and token-wise access priors, supported by both combined and individual ablations.
vs FADE: FADE handles a fractional integral equation through historical states and an iterative solver; Frac obtains fixed-size recurrence through a finite exponential sum, more directly serving decoder language models. The trade-offs are finite-horizon approximation and the theoretical gap between a strict FDE and a selective architecture.

Rating¶

Novelty: 4/5 — Implements fractional memory kernels as a scan-compatible selective sequence module with clear structural motivation.
Experimental Thoroughness: 4/5 — Covers synthetic extrapolation, language modeling, DNA, ablations, and throughput, but not every evaluation tests extrapolation.
Writing Quality: 4/5 — The appendix explains the theoretical versus implemented injection coefficients; main-text percentage wording requires interpretation as score points.
Value: 4/5 — Offers a reusable long-memory prior, while short-context retrieval and decoding speed leave room for improvement.