LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Long-Context RL, Intrinsic Activation Patterns, Saliency-guided Sparse Updates, DAPO, Multi-head Dimension Sparsity
TL;DR¶
LongAct identifies that high-magnitude outlier activations within query and key projections serve as structural anchors for long-context deduction, dynamically applying intra-head gradient masks during reinforcement learning to update only salient projection parameters and achieving an ~8% overall gain on LongBench v2 with zero inference overhead.
Background & Motivation¶
Reinforcement learning (RL) has emerged as an indispensable driver for eliciting complex reasoning behaviors in large language models, particularly within long-context settings such as extended dialogue tracking, document-level structural analysis, and autonomous agent trajectories. However, existing methodologies treat the model's internal parameter evolution as an uninspected black box, focusing almost exclusively on external optimization heuristics. These approaches prioritize synthesizing complex multi-turn reasoning traces, constructing granular verifiable reward functions to mitigate reward sparsity, or progressively scaling context window lengths across training curricula. Despite notable engineering advancements, these external interventions do not engage with the structural geometry of the model's latent representation space as sequence lengths expand.
Prior mechanistic investigations reveal that advanced multi-step deduction relies on coherent continuous trajectories through internal hidden state spaces rather than straightforward surface-level token autoregression. Work in model quantization and activation pruning confirms that internal hidden dimensions contribute unequally: a sparse fraction of massive outlier activations encodes disproportionately essential contextual grounding signals. Furthermore, long-context reasoning inherently exhibits sparsity, though prior efforts have targeted this property strictly across the temporal token dimension via key-value eviction or token dropping. Empirical analysis reveals that activation norms along the sequence dimension display negligible differentiation, whereas activations across hidden feature dimensions in Query and Key projections manifest sharp, persistent outlier spikes across distinct attention heads.
Subjecting all parameter matrices to uniform dense updates under long-context reinforcement learning introduces destructive gradient noise that perturbs contextual memory representations, leading to distribution collapse and catastrophic repetition loops. These observations demonstrate the necessity of shifting away from uniform policy updates toward saliency-guided optimization. Core idea: LongAct leverages intrinsic high-magnitude activations within Query and Key projections as structural anchors for long-context reasoning, applying dynamic intra-head gradient masks to perform saliency-guided sparse updates that reinforce contextual grounding subnetworks with zero inference latency.
Method¶
Overall Architecture¶
The LongAct pipeline encompasses two successive phases: a supervised fine-tuning (SFT) cold-start stage followed by saliency-guided reinforcement learning. During the RL stage, given a long input context \(c\) and prompt \(x\), the policy model \(\pi_\theta\) generates groups of reasoning trajectories wrapped in structured reasoning tags. In the backward pass, sequence-collapsed L2-norm magnitude matrices are computed across batches for Query and Key projections. Top-k feature dimensions are extracted locally within each attention head and mapped directly to global weight rows in \(W_Q\) and \(W_K\) to construct binary gradient masks. Non-critical rows are frozen while remaining parameters undergo group-relative policy updates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Long Context Input c + Prompt x"] --> B["Cold-Start SFT Model<br/>Standardize thought tags and provide reasoning priors"]
B --> C["Attention Activation Profiling<br/>Compute sequence-collapsed L2-norm matrices"]
C --> D["Intra-Head Saliency Selection<br/>Top-k selection of high-magnitude channels per head"]
D --> E["Dynamic Gradient Masking & Sparse Backprop<br/>Selectively update corresponding weight rows in W_Q and W_K"]
E --> F["Rule-Based Reward Evaluation & Policy Optimization<br/>DAPO / GRPO updates with zero inference overhead"]
Key Designs¶
1. Intra-Head Saliency Selection: Preserving Multi-Head Specialization via Local Top-k
Addressing the risk that global cross-head thresholding would allow dominant heads to exhaust the sparse budget, LongAct enforces intra-head feature selection. For Query projection matrix \(W_Q \in \mathbb{R}^{H_Q \times D \times d_{model}}\) and input hidden state \(H_{in} \in \mathbb{R}^{B \times S \times d_{model}}\), the expected L2-norm magnitude matrix \(M_Q \in \mathbb{R}^{H_Q \times D}\) is computed across sequence length \(S\) and batch \(B\):
Using this matrix, the top \(k = \lfloor\lambda D\rfloor\) feature dimensions are isolated independently within each individual attention head \(h\): \(K_h = \{d \mid d \in \arg k\max_{d'=0}^{D-1} \{M_{Q_{h,d'}}\}\}\), where \(\lambda\) denotes the update sparsity ratio (defaulting to 0.3). This localized ranking guarantees that every head maintains its allocation of updated channels, preserving multi-head representation diversity across long-range contexts.
2. Dynamic Gradient Masking & Sparse Backpropagation: Row-Level Projection Mapping with Zero Inference Cost
To prevent dense policy updates from destabilizing fragile contextual representations, LongAct constructs a direct mapping from output feature dimensions to weight matrix rows. In projection weights, head index \(h\) and internal head dimension \(d\) correspond to global row index \(j(h, d) = h \cdot D + d\). A binary gradient mask \(G_Q \in \{0, 1\}^{H_Q \times D \times d_{model}}\) is dynamically instantiated:
During backpropagation, weight gradients are filtered via element-wise Hadamard product: \(\nabla W_Q \leftarrow \nabla W_Q \odot G_Q\) and \(\nabla W_K \leftarrow \nabla W_K \odot G_K\). Only the top 30% salient projection rows receive gradient updates, whereas \(W_V\), \(W_O\), and MLP layers undergo standard full updates. At inference time, the mask is discarded completely, executing standard dense matrix multiplications without architecture changes or runtime latency.
3. Rule-Based Reward Evaluation & Policy Optimization: Verification-Driven Guidance without Critic Models
Addressing the instability and excessive memory footprint of training an auxiliary neural critic over 32k context tokens, LongAct integrates Group Relative Policy Optimization (DAPO / GRPO) with a dual-component verifiable reward formulation:
The structural format reward \(r_{fmt}(y)\) returns 1 if response \(y\) strictly contains <think>, </think>, <answer>, and </answer> tags (0 otherwise). The objective answer reward \(r_{ans}(y)\) assigns 1 for exact symbolic matches with ground truth answers (0 otherwise). Advantage normalization across sampled trajectory groups \(A_i = \frac{r_i - \text{mean}(\{r_j\})}{\text{std}(\{r_j\})}\) concentrates sparse gradients onto parameter rows responsible for accurate long-sequence information extraction.
Loss & Training¶
The framework executes in two successive stages: 1. Cold Start (SFT): Implemented in LLaMA-Factory using 20k instruction-following samples distilled from AM-DeepSeek-R1. Models are optimized using cross-entropy loss \(\mathcal{L}_{SFT}(\theta) = -\mathbb{E}_{(x,y)} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t})\) with a cosine learning rate scheduler (peak \(2\times 10^{-5}\), warmup ratio 0.03) over sequences packed up to 16,384 tokens for 900 steps. 2. Reinforcement Learning (RL): Implemented in verl utilizing DAPO on a curated blend of DocQA-RL-1.6K and 1k samples from MemAgent, scaling context lengths to 32,768 tokens. The policy trains with learning rate \(1\times 10^{-6}\), batch size 8, rollout size 16, sampling temperature 1.0, and maximum generation length 4,096. A default sparsity ratio \(\lambda = 0.3\) is applied to \(W_Q\) and \(W_K\).
Key Experimental Results¶
Main Results¶
Evaluation results across difficulty splits (Easy / Hard) and context length buckets (<32k, 32-128k, >128k) on LongBench v2:
| Backbone Model | Training Configuration | Overall | Easy | Hard | Short (<32k) | Medium (32-128k) | Long (>128k) |
|---|---|---|---|---|---|---|---|
| Qwen3-8B* (Official) | Official Release | 33.60 | 39.58 | 29.90 | 39.44 | 28.37 | 34.26 |
| Qwen3-8B | Base Model | 20.68 | 19.27 | 21.54 | 28.33 | 17.67 | 13.89 |
| Qwen3-8B | Cold-Start SFT | 27.04 | 28.65 | 26.05 | 32.22 | 24.65 | 23.15 |
| Qwen3-8B | SFT w/ DAPO (Dense Update) | 32.80 | 40.10 | 28.30 | 38.33 | 28.37 | 32.41 |
| Qwen3-8B | SFT w/ LongAct (Ours) | 36.73 | 38.02 | 35.93 | 41.94 | 33.37 | 34.72 |
| Qwen3-4B* (Official) | Official Release | 31.41 | 34.90 | 29.26 | 35.56 | 27.91 | 31.48 |
| Qwen3-4B | Base Model | 17.10 | 19.79 | 15.43 | 21.11 | 13.49 | 17.59 |
| Qwen3-4B | Cold-Start SFT | 25.65 | 21.35 | 28.30 | 33.33 | 22.33 | 19.44 |
| Qwen3-4B | SFT w/ DAPO (Dense Update) | 30.42 | 28.12 | 31.83 | 33.33 | 31.16 | 24.07 |
| Qwen3-4B | SFT w/ LongAct (Ours) | 34.24 | 33.07 | 34.97 | 37.92 | 31.63 | 33.33 |
Evaluation results on synthetic long-context benchmark RULER at 128K and 64K context windows:
| Setting | Ruler-128K Avg | Ruler-128K NIAH | Ruler-128K VT | Ruler-64K Avg | Ruler-64K NIAH | Ruler-64K VT |
|---|---|---|---|---|---|---|
| Qwen3-8B-Base | 36.95 | 51.88 | 47.68 | 41.87 | 61.76 | 28.08 |
| Qwen3-8B-SFT | 44.42 | 61.64 | 51.96 | 43.65 | 62.96 | 31.00 |
| Qwen3-8B-SFT w/ DAPO | 49.63 | 57.64 | 76.56 | 45.26 | 66.13 | 32.52 |
| Qwen3-8B-SFT w/ LongAct | 51.15 | 63.70 | 75.24 | 46.37 | 69.64 | 34.96 |
| Qwen3-4B-Base | 32.15 | 46.74 | 45.64 | 44.74 | 60.55 | 56.40 |
| Qwen3-4B-SFT | 49.57 | 53.02 | 91.44 | 54.56 | 66.20 | 90.92 |
| Qwen3-4B-SFT w/ DAPO | 51.21 | 54.57 | 90.48 | 59.05 | 67.74 | 94.96 |
| Qwen3-4B-SFT w/ LongAct | 52.63 | 56.29 | 90.64 | 61.30 | 69.68 | 94.52 |
Ablation Study¶
1. Activation Selection Strategy & Sparsity Ratio on LongBench v2
| Ablation Dimension | Configuration | 8B Overall | 8B Hard | 8B Long (>128k) | 4B Overall | 4B Hard | 4B Long (>128k) |
|---|---|---|---|---|---|---|---|
| Baseline | SFT Baseline | 27.04 | 26.05 | 23.15 | 25.65 | 28.30 | 19.44 |
| Selection Metric | Random Selection (30%) | 28.63 | 27.65 | 25.00 | 29.03 | 28.30 | 27.78 |
| Selection Metric | Minimum Values (30%) | 29.82 | 27.97 | 26.85 | 30.22 | 34.08 | 32.41 |
| Selection Metric | Massive Values (30%) | 36.73 | 35.93 | 34.72 | 34.24 | 34.97 | 33.33 |
| Sparsity Ratio \(\lambda\) | 20% Massive Values | 32.41 | 28.30 | 25.00 | 30.42 | 30.55 | 32.41 |
| Sparsity Ratio \(\lambda\) | 30% Massive Values (Default) | 36.73 | 35.93 | 34.72 | 34.24 | 34.97 | 33.33 |
| Sparsity Ratio \(\lambda\) | 40% Massive Values | 35.98 | 33.12 | 34.26 | 32.80 | 33.44 | 27.78 |
2. Activation Disruption Perturbation Analysis on LongBench v2 (503 Problems Total)
| Perturbation Setting | Correct Count / Total | Output Behavior & Generation Characteristics |
|---|---|---|
| Qwen3-8B Unperturbed | 173 / 503 | Normal chain-of-thought generation reaching correct options |
| Qwen3-8B w/o Bottom 30% Low Activations | 108 / 503 | Preserves logical coherence, connective reasoning, and correct answers |
| Qwen3-8B w/o Top 30% High Activations | 0 / 503 | Total generation collapse, degrading into repetitive numeric loops (333333...) |
Key Findings¶
- High-magnitude activations anchor contextual reasoning: Clamping the top 30% salient activations to their global mean results in catastrophic failure (0/503 correct), whereas disrupting low-magnitude activations preserves significant capability (108/503). Training on massive activations (36.73) drastically outperforms updating minimal activations (29.82) or random subsets (28.63).
- Substantial gains on complex reasoning: On the LongBench v2 Hard subset, LongAct 8B scores 35.93, surpassing full-parameter DAPO (28.30) by +7.63, demonstrating that sparse anchor tuning stabilizes multi-step inference against cumulative context errors.
- RL algorithmic universality: LongAct provides consistent performance improvements regardless of the underlying RL optimizer, recording 36.73 on DAPO, 35.04 on GRPO, 35.04 on CLIP-conv, and 34.24 on KL-conv.
- Short-context transferability: Saliency-guided updates do not degrade short-sequence reasoning, boosting performance across GSM8K (80.13 vs 78.08), HumanEval (73.17 vs 69.51), and TruthfulQA (69.52 vs 68.95).
Highlights & Insights¶
- Bridging representation quantization and policy optimization: LongAct adapts insights from outlier activation protection in post-training quantization directly into policy gradient filtering, pioneering representation-aware RL.
- Intra-head selection avoids head domination: Sorting dimensions locally within individual attention heads maintains attention diversity and prevents single heads from monopolizing gradient updates.
- Zero inference overhead: Sparse gradients are enforced exclusively through backward computation graphs; deployed models remain identical to dense Transformers.
Limitations & Future Work¶
- Evaluation constrained to mid-scale models: Due to hardware limitations (8x H800 GPUs), experiments were conducted on 4B and 8B backbones without testing extreme-scale models (e.g., 70B+).
- Uniform layer-wise sparsity ratio: The selection ratio \(\lambda = 0.3\) was kept static across all attention layers, leaving dynamic layer-adaptive sparsity exploration for future research.
- Restricted to projection matrices: The current mask applies solely to \(W_Q\) and \(W_K\), while MLP up/down projections also exhibit activation sparsity that could benefit from selective updates.
Related Work & Insights¶
- vs LongRLVR / LongDPO: Prior long-context RL frameworks focus on designing external verifiable rewards or stepwise critique curricula, whereas LongAct targets intrinsic representation geometry; the methods are orthogonal and complementary.
- vs AWQ / KIVI: Quantization methods preserve outlier weights in full precision to minimize reconstruction error, whereas LongAct demonstrates that outlier channels act as neural subnetwork anchors that should receive focused policy gradient updates.
Rating¶
- Novelty: โญโญโญโญโ Translates intrinsic activation outlier patterns into a dynamic gradient masking mechanism for RL.
- Experimental Thoroughness: โญโญโญโญโญ Evaluated across LongBench v2, RULER, InfiniteBench, standard short benchmarks, and detailed perturbation ablations.
- Writing Quality: โญโญโญโญโญ Clear exposition, disciplined mathematical notation, and compelling empirical analysis.
- Value: โญโญโญโญโ Offers an efficient, plug-and-play training paradigm for stabilizing long-context reinforcement learning.