LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Long-Context Reasoning, Reinforcement Learning, Intrinsic Activation Patterns, Saliency-Guided Sparse Updates, GRPO/DAPO
TL;DR¶
Reveals that Query and Key attention representations exhibit persistent high-magnitude outlier dimensions across long sequences, and proposes LongAct, a dynamic saliency-guided sparse reinforcement learning framework that selectively updates only the weight rows tied to these critical activation channels (top 30%), improving Qwen3-8B's LongBench v2 score by ~8% (36.73% vs. 27.04% SFT baseline and 32.80% full-parameter DAPO) with zero inference latency or architectural modifications.
Background & Motivation¶
Reinforcement Learning (RL) has emerged as a critical catalyst for eliciting sophisticated multi-step reasoning capabilities in Large Language Models (LLMs). When applied to short-context tasks such as mathematical problem-solving and code generation, rule-verifiable policy optimization algorithms (e.g., DeepSeek-R1, GRPO) have achieved remarkable breakthroughs. As real-world AI applications rapidly expand into long-dialogue tracking, multi-file code repository comprehension, and extended-trajectory autonomous agent decision-making, researchers have naturally begun exploring RL in long-context domains. However, existing endeavors almost exclusively treat the internal Transformer computations as an opaque black box, concentrating heavily on external signals such as synthesizing long reasoning chains, designing progressive context window curricula, tailoring fine-grained verifiable context rewards (e.g., LongRLVR), or altering model architectures via linear and chunked sparse attention.
Under standard policy optimization, dense, uniform gradient backpropagation across all parameters introduces a severe pathology in long-context scenarios. Complex long-sequence deduction fundamentally depends on continuous, delicate "thought trajectories" sustained across deep latent states. During extended sequence processing, minuscule, uncoordinated gradient updates across billions of parameters accumulate destructive noise within self-attention modules, disrupting the pre-trained structural anchors required for long-range context retrieval. Although attention naturally exhibits sequence-level sparsity—where a small fraction of key tokens determines the global reasoning trajectory—prior long-context RL methodologies have completely ignored this sparsity at the internal representation and parameter gradient levels.
Key inspirations can be drawn from model quantization and activation analysis, which demonstrate that hidden dimensions are fundamentally unequal in importance: a tiny subset of channel dimensions persistently produces high-magnitude activations (outliers) that encode disproportionate contextual knowledge. Perturbing or corrupting these high-magnitude activations immediately collapses long-context reasoning into repetitive failure loops, whereas zeroing out low-magnitude channels leaves logical coherence intact. Core idea: exploit the intrinsic, sequence-stable high-magnitude activation channels in Query and Key representations as structural anchors, proposing LongAct—a saliency-guided sparse RL framework that restricts parameter updates exclusively to the weight rows corresponding to these dominant activation channels, delivering robust, noise-resilient policy optimization with zero inference overhead.
Method¶
Overall Architecture¶
LongAct operates through a two-stage training paradigm: supervised fine-tuning (cold start) followed by saliency-guided reinforcement learning. During the cold-start phase, the base model is aligned on high-quality long-context reasoning trajectories to master explicit <think> ... </think> and <answer> ... </answer> formatting. During the subsequent policy optimization phase, the model samples responses on long-document tasks under a rule-based reward.
At each training step, LongAct monitors the \(\ell_2\)-norm of Query and Key representations across the sequence dimension to construct global activation magnitude matrices. It then performs intra-head Top-\(k\) selection to locate the salient hidden dimensions within each attention head. These indices are mapped directly to corresponding rows in the projection weight matrices \(W_Q\) and \(W_K\) to synthesize binary gradient masks. Gradients for non-salient rows are masked to zero, while \(W_V\), \(W_O\), and all feed-forward network (FFN) parameters receive standard full-parameter updates. At inference time, the gradient masks are discarded entirely, and the model functions as a standard dense Transformer with zero architectural changes or computational overhead.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Long-Context Reasoning Input<br/>(Context c + Question x)"] --> B["Forward Pass & Attention Computation<br/>Q, K Activation Projections"]
B --> C["Activation Magnitude Profiling<br/>Sequence-wide l2-norm Computation"]
C --> D["Intra-Head Salient Dimension Selection<br/>Dynamic Local Top-k Identification"]
D --> E["Weight-Row Dynamic Gradient Masking<br/>Binary Mask Mapping to GQ, GK"]
E --> F["Saliency-Guided Sparse Policy Optimization<br/>Targeted Row Updates on WQ, WK + Rule Reward"]
F --> G["Zero-Overhead Inference Deployment<br/>Masks Discarded, Standard Dense Execution"]
Key Designs¶
1. Activation Magnitude Profiling: Quantifying Latent Channels Carrying Contextual Knowledge
Departing from token-level filtering along the sequence axis—where empirical heatmaps exhibit no persistent high-magnitude locations—LongAct inspects the hidden feature dimensions. Given input hidden states \(H_{in} \in \mathbb{R}^{B \times S \times d_{model}}\), linear projections produce Query activations \(Q \in \mathbb{R}^{B \times S \times H_Q \times D}\) and Key activations \(K \in \mathbb{R}^{B \times S \times H_{KV} \times D}\) via learnable weights \(W_Q \in \mathbb{R}^{H_Q \times D \times d_{model}}\) and \(W_K \in \mathbb{R}^{H_{KV} \times D \times d_{model}}\), where \(H_Q\) and \(H_{KV}\) denote head counts, \(D\) is the per-head dimension, \(B\) is the batch size, and \(S\) is the sequence length. To capture persistent salient features independent of token variations, LongAct computes the \(\ell_2\)-norm across the sequence length and averages across the batch, defining the global magnitude matrix \(M^Q \in \mathbb{R}^{H_Q \times D}\) (and analogously \(M^K \in \mathbb{R}^{H_{KV} \times D}\)):
This formulation collapses sequence fluctuations and reliably isolates the intrinsic, load-bearing feature dimensions across different attention heads.
2. Intra-Head Salient Dimension Selection: Preserving Multi-Head Subspace Diversity
Rather than executing a global Top-\(k\) selection across all heads combined—which risks starving subtle heads or letting dominant heads overwhelm the selection—LongAct enforces an intra-head Top-\(k\) selection protocol. For each individual attention head \(h \in \{0, \dots, H_Q - 1\}\), the algorithm sorts the internal dimension indices by their activation magnitude and selects the top \(k = \lfloor \lambda D \rfloor\) indices to form the critical dimension set \(\mathcal{K}_h\):
where \(\lambda \in (0, 1]\) represents the update sparsity ratio (ablation proves \(\lambda = 0.3\) is optimal). This per-head constraint guarantees that every attention subspace retains its specialized capacity to update its most salient contextual pathways.
3. Weight-Row Dynamic Gradient Masking: Direct Channel-to-Row Gradient Filtering
Because linear projection produces the activation of head \(h\) at dimension \(d\) via the dot product between the input hidden vector and the corresponding row in the weight matrix, an activation output channel maps one-to-one to a specific row in the projection weights. The global row index \(j(h, d)\) within \(W_Q\) (total rows \(H_Q \times D\)) is given by:
LongAct instantiates a binary gradient mask \(G^Q \in \{0, 1\}^{H_Q \times D \times d_{model}}\) where row \(r\) is trainable if and only if it aligns with a selected high-magnitude channel:
During backward propagation, gradients are masked via Hadamard product: \(\nabla W_Q \leftarrow \nabla W_Q \odot G^Q\) and \(\nabla W_K \leftarrow \nabla W_K \odot G^K\). Unselected rows have their gradients set to zero, effectively freezing non-critical attention pathways and shielding the model's contextual foundation from noisy RL updates.
Loss & Training¶
The training process is structured as follows:
- Cold Start (SFT): The base model is fine-tuned using standard cross-entropy loss \(\mathcal{L}_{SFT}(\theta) = -\mathbb{E}_{(x, y) \sim \mathcal{D}_{SFT}} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t})\) on 20k open-source instruction-following samples from AM-DeepSeek-R1-0528-Distilled. Training runs for 900 steps with a cosine learning rate scheduler peaking at \(2 \times 10^{-5}\) (warmup ratio 0.03) and a sequence packing length of 16,384 tokens.
- Reinforcement Learning with DAPO: Built upon verl and Direct Advantage Policy Optimization (DAPO), training uses a mixture of DocQA-RL-1.6K and 1k random samples from MemAgent. The training uses a learning rate of \(1 \times 10^{-6}\), a batch size of 8, a rollout number of 16, a temperature of 1.0, and a maximum sequence length of 32,768 tokens.
- Rule-Based Reward Formulation: The reward function combines structural format adherence with factual answer correctness: \(r(y) = r_{fmt}(y) + r_{ans}(y)\), where \(r_{fmt}(y) = 1\) if \(y\) contains valid
<think>,</think>,<answer>, and</answer>tags (0 otherwise), and \(r_{ans}(y) = 1\) if the extracted answer matches the ground truth (0 otherwise).
Key Experimental Results¶
Main Results¶
Evaluations on LongBench v2 benchmark realistic deep understanding across difficulty levels (Easy, Hard) and input context lengths (Short <32k, Medium 32-128k, Long >128k):
| Model | Overall | Easy | Hard | Short (<32k) | Medium (32-128k) | Long (>128k) |
|---|---|---|---|---|---|---|
| Qwen3-8B* (Official) | 33.60 | 39.58 | 29.90 | 39.44 | 28.37 | 34.26 |
| Qwen3-8B-Base | 20.68 | 19.27 | 21.54 | 28.33 | 17.67 | 13.89 |
| Qwen3-8B-SFT | 27.04 | 28.65 | 26.05 | 32.22 | 24.65 | 23.15 |
| Qwen3-8B-SFT w/ DAPO | 32.80 | 40.10 | 28.30 | 38.33 | 28.37 | 32.41 |
| Qwen3-8B-SFT w/ LongAct | 36.73 | 38.02 | 35.93 | 41.94 | 33.37 | 34.72 |
| Qwen3-4B* (Official) | 31.41 | 34.90 | 29.26 | 35.56 | 27.91 | 31.48 |
| Qwen3-4B-Base | 17.10 | 19.79 | 15.43 | 21.11 | 13.49 | 17.59 |
| Qwen3-4B-SFT | 25.65 | 21.35 | 28.30 | 33.33 | 22.33 | 19.44 |
| Qwen3-4B-SFT w/ DAPO | 30.42 | 28.12 | 31.83 | 33.33 | 31.16 | 24.07 |
| Qwen3-4B-SFT w/ LongAct | 34.24 | 33.07 | 34.97 | 37.92 | 31.63 | 33.33 |
Note: Qwen3-8B* and Qwen3-4B* represent the official post-trained checkpoints released by the Qwen team.
On RULER synthetic retrieval benchmarks (128K and 64K context windows), LongAct consistently improves effective context capacity:
| Model | Ruler-128K (Avg) | Ruler-128K (NIAH) | Ruler-128K (VT) | Ruler-64K (Avg) | Ruler-64K (NIAH) | Ruler-64K (VT) |
|---|---|---|---|---|---|---|
| Qwen3-8B-Base | 36.95 | 51.88 | 47.68 | 41.87 | 61.76 | 28.08 |
| Qwen3-8B-SFT | 44.42 | 61.64 | 51.96 | 43.65 | 62.96 | 31.00 |
| Qwen3-8B-SFT w/ DAPO | 49.63 | 57.64 | 76.56 | 45.26 | 66.13 | 32.52 |
| Qwen3-8B-SFT w/ LongAct | 51.15 | 63.70 | 75.24 | 46.37 | 69.64 | 34.96 |
| Qwen3-4B-Base | 32.15 | 46.74 | 45.64 | 44.74 | 60.55 | 56.40 |
| Qwen3-4B-SFT | 49.57 | 53.02 | 91.44 | 54.56 | 66.20 | 90.92 |
| Qwen3-4B-SFT w/ DAPO | 51.21 | 54.57 | 90.48 | 59.05 | 67.74 | 94.96 |
| Qwen3-4B-SFT w/ LongAct | 52.63 | 56.29 | 90.64 | 61.30 | 69.68 | 94.52 |
On InfiniteBench heterogeneous tasks, Qwen3-8B-SFT w/ LongAct achieves the top average score of 49.39% (vs. SFT 46.68% and DAPO 48.11%), with substantial improvements in English summarization En.Sum (14.18% \(\to\) 15.83%) and multiple-choice reasoning En.MC (54.15% \(\to\) 59.39%).
Ablation Study¶
1. Activation Selection Strategy (LongBench v2)¶
Comparing updates directed at massive values vs. low-magnitude values (min values) vs. random feature selection:
| Strategy | Overall | Easy | Hard | Short | Medium | Long |
|---|---|---|---|---|---|---|
| Qwen3-8B-SFT (Cold-start) | 27.04 | 28.65 | 26.05 | 32.22 | 24.65 | 23.15 |
| + Random selection | 28.63 | 30.21 | 27.65 | 28.33 | 30.70 | 25.00 |
| + Min values (low-magnitude) | 29.82 | 32.81 | 27.97 | 35.00 | 26.98 | 26.85 |
| + Massive values (LongAct) | 36.73 | 38.02 | 35.93 | 41.94 | 33.37 | 34.72 |
| Qwen3-4B-SFT (Cold-start) | 27.04 | 28.65 | 26.05 | 32.22 | 24.65 | 23.15 |
| + Random selection | 29.03 | 30.21 | 28.30 | 30.00 | 28.84 | 27.78 |
| + Min values (low-magnitude) | 30.22 | 23.96 | 34.08 | 27.78 | 31.16 | 32.41 |
| + Massive values (LongAct) | 34.24 | 33.07 | 34.97 | 37.92 | 31.63 | 33.33 |
2. Sparsity Ratio \(\lambda\) Ablation (LongBench v2)¶
Evaluating the fraction of massive values updated during policy optimization:
| Model & Sparsity Setup | Overall | Easy | Hard | Short | Medium | Long |
|---|---|---|---|---|---|---|
| Qwen3-8B-SFT (Cold-start) | 27.04 | 28.65 | 26.05 | 32.22 | 24.65 | 23.15 |
| + 20% massive values | 32.41 | 39.06 | 28.30 | 39.44 | 30.23 | 25.00 |
| + 30% massive values (Default) | 36.73 | 38.02 | 35.93 | 41.94 | 33.37 | 34.72 |
| + 40% massive values | 35.98 | 40.62 | 33.12 | 38.33 | 34.88 | 34.26 |
| Qwen3-4B-SFT (Cold-start) | 27.04 | 28.65 | 26.05 | 32.22 | 24.65 | 23.15 |
| + 20% massive values | 30.42 | 30.21 | 30.55 | 29.44 | 30.23 | 32.41 |
| + 30% massive values (Default) | 34.24 | 33.07 | 34.97 | 37.92 | 31.63 | 33.33 |
| + 40% massive values | 32.80 | 31.77 | 33.44 | 41.11 | 28.37 | 27.78 |
3. RL Algorithm Compatibility (LongBench v2)¶
Testing LongAct across diverse RL policy optimization objectives:
| RL Algorithm | Overall (8B) | Hard (8B) | Long (8B) | Overall (4B) | Hard (4B) | Long (4B) |
|---|---|---|---|---|---|---|
| SFT Baseline | 27.04 | 26.05 | 23.15 | 25.65 | 28.30 | 19.44 |
| + DAPO w/ LongAct | 36.73 | 35.93 | 34.72 | 34.24 | 34.97 | 33.33 |
| + GRPO w/ LongAct | 35.04 | 31.27 | 28.24 | 33.45 | 32.72 | 36.34 |
| + CLIP-conv w/ LongAct | 35.04 | 31.51 | 32.87 | 32.11 | 31.83 | 34.72 |
| + KL-conv w/ LongAct | 34.24 | 32.32 | 28.24 | 33.90 | 35.05 | 37.27 |
4. Perturbation & Mechanistic Clamping Analysis (LongBench v2, 503 Cases)¶
| Perturbation Setting | Correct / Total | Accuracy | Typical Model Behavior |
|---|---|---|---|
| Original Qwen3-8B | 173 / 503 | 34.4% | Normal structured reasoning, correct answer derivation |
| w/o Non High-magnitude (clamp bottom 30%) | 108 / 503 | 21.5% | Retains logical connectors and coherent CoT trajectory |
| w/o High-magnitude (clamp top 30%) | 0 / 503 | 0.0% | Catastrophic collapse into repetitive digit loops (333333...) |
Key Findings¶
- High-Magnitude Activations are the Structural Foundation of Coherence: Disrupting low-magnitude activations causes a mild accuracy dip, whereas clamping the top 30% of high-magnitude channels reduces reasoning accuracy to exactly zero (0/503), immediately triggering degenerate repetitive token loops. This provides empirical proof that these outlier dimensions function as the critical backbone for long-sequence context anchoring.
- Disproportionate Gains on Hard Tasks and Long Sequences: In contrast to standard DAPO, which predominantly gains on easy splits, LongAct achieves its largest margins on error-sensitive Hard questions (+7.63% over DAPO on Qwen3-8B) and Long sequences >128k (+9.26% on Qwen3-4B), proving that sparse gradient updates prevent error compounding across extended reasoning steps.
- Sparsity Sweet Spot at 30%: Performance peaks consistently at a 30% update ratio. Widening updates to 40% degrades accuracy across multiple splits, indicating that overly dense gradient updates reintroduce gradient noise and dilute the optimization of load-bearing parameters.
- Positive Transfer to Short-Context Tasks: On GSM8K (80.13% vs. DAPO 78.08%), HumanEval (73.17% vs. 69.51%), and TruthfulQA (69.52% vs. 68.95%), LongAct consistently outperforms full-parameter RL, confirming that saliency-guided sparse updates act as a general noise-filtering optimizer rather than a context-length-specific heuristic.
Highlights & Insights¶
- Repurposing Quantization Insights for Policy Optimization: Innovatively connects the well-known presence of attention outlier activations in model quantization (e.g., AWQ, Massive-Values) to reinforcement learning gradient dynamics, shifting long-context RL from purely external reward engineering to intrinsic representation-guided optimization.
- Zero-Inference-Cost Sparse Execution: While many long-context methods require custom sparse attention kernels or architectural modifications at test time, LongAct applies gradient masking strictly during backward propagation. The final deployed checkpoint remains a standard dense Transformer.
- Intra-Head Dimensional Alignment: By performing Top-\(k\) selection locally within each attention head, the method prevents inter-head magnitude variance from unbalancing multi-head representation spaces.
Limitations & Future Work¶
- Exploration Limited to 4B and 8B Model Scales: Due to GPU compute limitations, experiments were conducted up to the 8B scale. Investigating whether high-magnitude activation patterns become even more sparse or concentrated in 70B+ models represents a vital avenue for scaling research.
- Uniform Sparsity Across Attention Layers: LongAct applies a fixed 30% sparsity ratio across all layers. Exploring layer-adaptive sparsity—such as allocating denser updates to middle/deep semantic layers and sparser updates to early retrieval layers—could further improve efficiency.
- Extension to MoE and FFN Modules: The current implementation restricts gradient masking to \(W_Q\) and \(W_K\). Adapting saliency-guided sparse updates to Mixture-of-Experts routing gates and MLP intermediate representations remains unexplored.
Related Work & Insights¶
- vs. LongRLVR (Chen et al., 2026): LongRLVR addresses the sparse reward problem via verifiable context-grounded reward shaping; LongAct addresses optimization stability by constraining parameter gradients to intrinsic salient dimensions. The two approaches are orthogonal and complementary.
- vs. AWQ / Massive-Values (Lin et al., 2024; Jin et al., 2025): Prior literature identified high-magnitude attention activations as contextual knowledge carriers for low-bit quantization; LongAct is the first to leverage them as dynamic backward gradient filters in reinforcement learning.
- vs. MIGU (Du et al., 2024): MIGU isolates task-specific parameters across multiple tasks using \(L_1\)-norm to prevent catastrophic forgetting; LongAct identifies task-intrinsic \(L_2\)-norm activation patterns within a single long-context task to filter gradient noise during policy optimization.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Translating intrinsic attention activation outliers into a gradient-level sparse RL regularizer is highly original.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across LongBench v2, RULER, InfiniteBench, standard short benchmarks, plus ablation on sparsity, selection strategies, RL algorithms, and perturbation stress tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formalization, well-structured exposition, and compelling mechanistic visualizations.
- Value: ⭐⭐⭐⭐⭐ Establishes a practical, zero-inference-overhead paradigm for stabilizing long-context reinforcement learning.