Selective State-Space Adaptation and Retrieval for Language Model Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/atahandokme/malora-mara
Area: LLM Reasoning / Selective State-Space Adaptation
Keywords: State-Space Models, Parameter-Efficient Fine-Tuning, Multi-Hop Reasoning, Retrieval Adapter, Mamba
TL;DR¶
This paper introduces a two-level selective state-space adaptation family that deploys recurrent Mamba modules at both the token level to dynamically modulate LoRA scaling factors (MaLoRA) and the context level to decode supporting evidence directly from frozen intermediate states (MaRA), achieving a +6.4 F1 (+10.0% relative) gain over LoRA on multi-hop reasoning.
Background & Motivation¶
Low-rank adaptation (LoRA) has become the de facto standard approach for parameter-efficient adaptation of frozen language models to downstream tasks. However, canonical LoRA learns a static update matrix that is added identically to every token across all inputs. In complex multi-step reasoning and multi-hop question answering, reasoning trajectories pass through multiple latent state transitions. A static weight update forces the model to rely solely on global patterns aggregated over the training distribution, failing to provide input-dependent and state-adaptive adjustments along the reasoning path.
While recent approaches have attempted to make parameter-efficient adaptation input-dependent (such as TopLoRA), they typically rely on stateless token-wise diagonal scaling where the modulation factor depends only on the current token in isolation, with no recurrent memory propagated across positions. More critically, an empirical inspection of trained token-level modulators exposes a fundamental constraint: token-level scaling readily differentiates structural prompt roles (e.g., format markers, question text, and paragraph content), but within paragraph bodies, supporting evidence and distractor paragraphs receive virtually identical scaling factors (differing by at most 0.002). Consequently, the decision of evidence selection cannot be achieved at the token level, irrespective of what the token modulator learns.
To resolve this granularity misalignment and lack of recurrent state, this work introduces a complementary family of selective state-space adapters operating at two nested levels: token-level adaptation and context-level adaptation. Leveraging Mamba's sub-quadratic sequence modeling and input-dependent selective state transitions, the architecture maintains recurrent memory across tokens to govern low-rank update magnitudes while tracking cross-segment transitions to retrieve supporting paragraphs from frozen representations. Core idea: deploy selective state-space modules at two complementary granularities around a frozen backbone—using token-level recurrence to dynamically modulate LoRA updates (MaLoRA) and context-level recurrence to decode supporting evidence from intermediate representations (MaRA)—yielding mutually orthogonal, additive reasoning improvements.
Method¶
Overall Architecture¶
The system wraps around a completely frozen language model backbone and operates across two complementary stages: pre-generation context-level evidence retrieval (MaRA) and in-generation token-level low-rank modulation (MaLoRA). Before generation begins, the query and candidate paragraphs are processed by MaRA, which reuses the first \(K\) layers of the frozen backbone, applies query-driven additive attention pooling to obtain segment embeddings, and routes them through a segment-level Mamba recurrence to select the top-\(k\) supporting paragraphs via chained selection passes. The pruned prompt is then passed to the language model, where MaLoRA uses a token-level Mamba block to accumulate sequence trajectory memory and emit a scalar \(\lambda(x_t)\) that rescales low-rank updates during auto-regressive generation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query Q and Candidate Segments C"] --> B["Frozen Prefix Representation Extraction & Attention Pooling"]
B --> C["Segment-Level Mamba Recurrence & Evidence Selection"]
C -->|Top-k Supporting Segments| D["Pruned Context & Query Concatenation"]
D --> E["Token-Level Mamba Trajectory Modeling"]
E --> F["Dynamic Scalar Modulated Low-Rank Generation"]
F --> G["Final Multi-Hop Reasoning Output"]
Key Designs¶
1. Frozen Prefix Representation Extraction & Attention Pooling: Zero-Encoder Evidence Grounding Conventional retrieval systems require training and hosting external dense bi-encoders comprising hundreds of millions or billions of parameters, whose representation space often diverges from the downstream generator's internal states. MaRA eliminates the need for an external encoder by directly repurposing the first \(K\) layers of the downstream frozen backbone. Each candidate segment \(C_i\) is concatenated with query \(Q\) and passed through the prefix slice, yielding query-conditioned hidden states \(H_i \in \mathbb{R}^{T_i \times d}\). To compress variable-length token representations into a single segment embedding, a learned additive attention pool parameterised by \(W_k \in \mathbb{R}^{d_p \times d}\) and query vector \(q_a\) computes a weighted reduction:
This mechanism ensures segment embeddings natively capture query-context interactions while introducing only a negligible parameter overhead of around 2.9M to 3.1M parameters.
2. Segment-Level Mamba Recurrence & Evidence Selection: Contextual State Tracking for Multi-Hop Chains In multi-hop reasoning, evidence paragraphs often form an interdependent chain where bridge entities are absent from the surface query, preventing independent text retrievers from scoring intermediate hops accurately. MaRA places a two-layer selective state-space module (Mamba) directly over the sequence of pooled segment embeddings \((p_1, \dots, p_N)\), elevating the recurrence unit from individual tokens to full context segments. The causal recurrence produces contextualised states \(s_i\) conditioned on all preceding segments \(1\) through \(i\). During training, a local head \(r_i = w_\ell^\top s_i\) and a global head \(\tilde{r} = W_g s_N\) are jointly optimized using positive-weighted binary cross-entropy, an intra-example pairwise margin loss, and auxiliary global regularization:
At inference time, an \(N_p=4\) chained selection procedure iteratively re-ranks candidates conditioned on already chosen anchor segments, systematically surfacing multi-hop reasoning paths.
3. Token-Level Mamba Trajectory Modeling: Stateful Sequence Memory for LoRA Existing dynamic LoRA extensions calculate adaptation factors purely based on the current isolated token projection, leaving the adapter oblivious to the path traversed so far. MaLoRA attaches a selective state-space block to each adapted projection (covering attention and MLP projections). By projecting the incoming activation \(x_t\) through low-rank projection matrix \(P \in \mathbb{R}^{r \times d_{in}}\), the Mamba recurrence updates an internal state \(s_t \in \mathbb{R}^{d_s}\) via input-dependent discretization matrices \(\bar{A}_t, \bar{B}_t, \bar{C}_t\):
This design injects genuine recurrent memory across positions within the adapter itself, enabling the adaptation strength at token \(t\) to reflect the cumulative reasoning trajectory rather than merely reacting to local token semantics.
4. Dynamic Scalar Modulated Low-Rank Generation: Stable Suppression and Amplification Rather than outputting a high-dimensional diagonal matrix that increases variance and impairs interpretability, MaLoRA maps the Mamba hidden output \(o_t\) to a single scalar scaling factor via a learned linear head and Softplus activation, \(\lambda(x_t) = \text{softplus}(\phi(o_t))\). This scalar modulates the low-rank branch:
The Softplus activation accommodates both update suppression (\(\lambda < 1\)) and amplification (\(\lambda > 1\)). Empirical analysis reveals that trained modulators are predominantly suppression-dominant (61%–84% of tokens receive \(\lambda < 1\)) and exhibit clear role sensitivity: heavily suppressing formatting markers and query tokens while allocating higher scaling values to substantive content tokens, effectively reshaping parameter updates according to functional token roles.
Loss & Training¶
The framework adopts a decoupled two-stage training strategy (Staged Training) to preserve computational efficiency: - Stage 1 (Token Modulator): LoRA parameters and MaLoRA Mamba modules are jointly trained on target reasoning tasks for 2 epochs using AdamW, effective batch size 4 (MuSiQue) or 8 (2WikiMultihopQA), learning rate \(2 \times 10^{-4}\), over 9,998 stratified training examples with the base LM frozen. - Stage 2 (Retrieval Adapter): The base LM is frozen without the modulator, and the 2.9M–3.1M parameter MaRA module is trained for 2 epochs with learning rate \(5 \times 10^{-4}\) (or \(1 \times 10^{-4}\) for Gemma-2-9B). Loss hyper-parameters use positive BCE weight 8.0, pairwise margin weight \(\beta=0.5\) with margin \(m=1.0\), and auxiliary global weight \(\gamma=0.5\).
Key Experimental Results¶
Main Results¶
Evaluated on MuSiQue (2–4 hops over 20 candidates) and 2WikiMultihopQA (up to 4 hops over 10 candidates) across Qwen-2.5-7B, Llama-3.1-8B, and Gemma-2-9B using exact match (EM) and F1:
| Backbone | Method | MuSiQue (EM / F1) | 2WikiMultihopQA (EM / F1) |
|---|---|---|---|
| Qwen-2.5-7B | Base (Full Context, No Adapter) | 5.1 / 14.4 | 15.3 / 26.0 |
| Qwen-2.5-7B | LoRA (Hu et al., 2022) | 38.4 / 51.1 | 71.7 / 77.3 |
| Qwen-2.5-7B | TopLoRA (Li et al., 2025) | 41.5 / 55.6 | 72.6 / 77.9 |
| Qwen-2.5-7B | MaLoRA (Token Modulator) | 42.6 / 56.4 | 73.7 / 79.3 |
| Qwen-2.5-7B | LoRA + MaRA | 42.2 / 55.6 | 75.4 / 81.0 |
| Qwen-2.5-7B | MaLoRA + MaRA (Full System) | 44.9 / 58.9 | 77.9 / 83.3 |
| Llama-3.1-8B | Base (Full Context, No Adapter) | 6.0 / 17.4 | 20.3 / 31.9 |
| Llama-3.1-8B | LoRA (Hu et al., 2022) | 41.8 / 54.4 | 72.9 / 78.5 |
| Llama-3.1-8B | TopLoRA (Li et al., 2025) | 47.3 / 61.4 | 75.1 / 80.4 |
| Llama-3.1-8B | MaLoRA (Token Modulator) | 48.1 / 62.3 | 78.1 / 83.3 |
| Llama-3.1-8B | LoRA + MaRA | 43.4 / 55.8 | 75.7 / 81.2 |
| Llama-3.1-8B | MaLoRA + MaRA (Full System) | 48.2 / 62.5 | 79.9 / 85.1 |
| Gemma-2-9B | Base (Full Context, No Adapter) | 13.8 / 25.3 | 17.9 / 34.2 |
| Gemma-2-9B | LoRA (Hu et al., 2022) | 45.4 / 59.8 | 71.5 / 77.2 |
| Gemma-2-9B | TopLoRA (Li et al., 2025) | 43.8 / 59.4 | 73.5 / 79.2 |
| Gemma-2-9B | MaLoRA (Token Modulator) | 48.7 / 63.7 | 77.1 / 82.2 |
| Gemma-2-9B | LoRA + MaRA | 43.8 / 57.9 | 74.6 / 80.3 |
| Gemma-2-9B | MaLoRA + MaRA (Full System) | 46.9 / 61.9 | 80.1 / 85.2 |
Ablation Study¶
Architectural ablation of context-level retrieval components on unadapted backbones (single scoring pass, Recall@4):
| Architecture Config | Trainable Params | MuSiQue (Qwen / Llama) R@4 | 2Wiki (Qwen / Llama) R@4 | Note |
|---|---|---|---|---|
| MLP (Per-segment, no cross-segment) | 13.4M | 20.6 / 19.6 | 93.8 / 95.3 | Fails without cross-segment interaction |
| LastTok (Last token representation) | 3.6k | 30.1 / 32.3 | 84.8 / 83.7 | Bottlenecked by single token capacity |
| TokenMamba (Concatenated tokens) | 2.0M | 61.4 / 54.8 | 82.5 / 76.1 | Token-level recurrence trails by 19–29 pp |
| PoolOnly (Attention pool only) | 1.8M | 57.3 / 58.2 | 90.0 / 94.7 | Isolates importance of segment mixer |
| LSTM (2-layer classic recurrence) | 2.9M | 58.6 / 58.6 | 94.9 / 93.5 | Lacks selective state transitions (-20+ pp) |
| GRU (2-layer classic recurrence) | 2.6M | 60.2 / 58.7 | 94.8 / 93.8 | Similar degradation without SSM |
| Transformer (Standard self-attention) | 3.4M | 63.8 / 60.5 | 95.5 / 95.2 | Sub-optimal contextual routing |
| MeanPool + Mamba | 2.9M | 73.8 / 76.8 | 98.6 / 98.1 | Uniform pooling drops 7 pp |
| Mamba (MaRA Full Design) | 2.9M | 80.6 / 84.1 | 99.2 / 99.5 | Decisive lead across all configurations |
Key Findings¶
- Near-additive orthogonality across granularities: Across the 6 benchmark cells, token-level modulation improves LoRA F1 by +2.0 to +7.9, while retrieval selection shifts F1 by -1.9 to +4.5. The two mechanisms exhibit an average overlap of only 6%, proving that token-level structural reweighting and context-level distractor filtering provide decoupled, compounding benefits.
- Latent evidence relevance resides in unadapted models: Unadapted models answering MuSiQue at only 14.4–25.3 F1 yield 80.6–84.1% Recall@4 when decoded by the 2.9M parameter MaRA module, outperforming dedicated 8B dense retrievers such as Qwen3-Embedding-8B (75.5%). The information is already latent in representations; the adapter's role is decoding rather than learning it from scratch.
- Granularity matching governs SSM effectiveness: At the token level, Mamba performs similarly to GRU and LSTM (within random seed variance), indicating that recurrence itself is the key driver. At the segment level, however, where each step encapsulates a complex paragraph, selective state-space transitions outperform classical recurrences by more than 20 recall points.
Highlights & Insights¶
- Dual-scale reuse of the same state-space abstraction: Re-purposing the Mamba selective state-space block at both microscopic token generation and macroscopic segment-level evidence routing achieves a remarkably cohesive and elegant architecture.
- Delineated boundary between token modulation and evidence selection: The paper definitively proves that token-level modulators are blind to contextual paragraph relevance (\(\Delta \lambda \le 0.002\)), resolving ambiguity in prior literature regarding why token-level adapters fail to filter long-context distractions.
- Internal representation reading beats outsourced external retrievers: By tapping directly into the generator's own frozen prefix representations, MaRA bypasses representation misalignment and achieves superior retrieval with 1/2500th of the parameters of external 8B retrieval models.
Limitations & Future Work¶
- Restricted to structured multi-hop paragraph contexts: MaRA assumes inputs have clear paragraph boundaries. Extending segment-level routing to single-context tasks like mathematical reasoning (GSM8K) remains unresolved, as solution-step segmentations did not yield consistent improvements.
- Inference latency from chained pre-passes: The \(N_p=4\) chained passes in MaRA require multiple forward passes through prefix layers, adding 2.1 to 5.6 seconds per sample in inference latency and representing the primary efficiency bottleneck.
- Staged rather than end-to-end co-training: Training the token modulator and retrieval adapter in separate stages leaves potential gains from joint end-to-end co-adaptation unexplored.
Related Work & Insights¶
- vs. LoRA & TopLoRA: Standard LoRA relies on static global weight updates; TopLoRA incorporates stateless, token-isolated diagonal scaling. MaLoRA introduces stateful recurrent dynamics across tokens, demonstrating that scalar modulation is stabler and more interpretable than diagonal vectors.
- vs. READ & SSMLoRA: Prior state-space adapters place recurrence along the depth axis (across Transformer layers); MaLoRA anchors recurrence along the temporal sequence axis (across token positions), aligning with the sequential nature of auto-regressive decoding.
- vs. External Dense Retrievers (BGE / Contriever / Qwen3-Embedding): External retrievers suffer from domain drift and semantic misalignment with downstream generators. MaRA directly decodes relevance from the generator's own representations, achieving higher recall with only ~3M parameters.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant nested reuse of selective state spaces across two distinct operational granularities.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 3 model families, exhaustive component ablations, efficiency profiles, and state-usage probing.
- Writing Quality: ⭐⭐⭐⭐⭐ Compelling narrative progression from identifying token-level limitations to formulating context-level adaptation.
- Value: ⭐⭐⭐⭐⭐ Delivers practical insights for parameter-efficient adaptation, long-context reasoning, and internal representation decoding.