Skip to content

Rethinking Gradient Flow Through Frozen Blocks for Memory-Efficient Block-Coordinate Training

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: block-coordinate descent, frozen block gradient flow, sparse backpropagation, activation memory reduction, adaptive token budget

TL;DR

Addressing the activation memory bottleneck caused by frozen blocks in block-coordinate descent (BCD) training, FROST discards inactive MLP activations during the forward pass and selectively reconstructs token gradients via upstream-gradient norms and effective-rank budget allocation, cutting peak memory by 20%–34% without downstream performance degradation.

Background & Motivation

Fine-tuning large language models is fundamentally constrained by GPU memory limits, where model weights, gradients, optimizer states, and forward activation tensors compete for bounded physical hardware capacity. In response, block-coordinate descent (BCD) frameworks—such as LISA, MISA, and BAdam—have emerged as compelling alternatives to full-parameter optimization. BCD partitions Transformer layers or modules into coordinate blocks and activates only a minor subset (with active-parameter ratio \(\rho \le 0.05\)) per training iteration while freezing the remainder. By cycling the active coordinate set over optimization steps, BCD scales optimizer states down in direct proportion to \((1 - \rho)\), drastically slashing the memory overhead of AdamW moment vectors.

However, existing BCD research focuses almost exclusively on coordinate-selection dynamics—refining importance sampling, layer schedules, or module-level rotations—while treating inactive (frozen) blocks as a fixed, passive overhead. Even though frozen blocks are not parameterized or updated in the current step, the chain rule dictates that backpropagation must traverse the complete computational graph to transmit backward error signals to earlier active layers. Consequently, frozen blocks must still cache high-dimensional forward activations. Memory decomposition reveals that once optimizer states are compressed, activation memory becomes the dominant barrier in BCD training, with frozen blocks accounting for 78%–93% of the total activation footprint.

Crucially, dissecting the internal components of frozen blocks exposes an asymmetric gradient sensitivity: while attention layers couple all positions and suffer catastrophic gradient distortion under token sparsification (dropping gradient cosine similarity below 0.35), inactive MLP modules compute position-wise operations and exhibit high token-level gradient sparsity. In inactive MLPs, a tiny fraction of salient tokens carries nearly all the gradient information needed to update earlier active blocks. Core idea: introduce FROST, a frozen-block sparse gradient approximation method that discards forward activations in inactive MLPs, leverages upstream-gradient norms during the backward pass to selectively recompute Jacobian-vector products for salient tokens, and dynamically balances layer-wise token budgets using Shannon effective rank to achieve lossless gradient propagation under tight memory bounds.

Method

Overall Architecture

FROST acts as an agnostic, drop-in backward operator compatible with any underlying BCD scheduler (e.g., LISA or MISA). Its execution pipeline unfolds across four tightly coupled stages: activation discard during the forward pass, backward-time token selection, partial forward recomputation with residual pass-through, and rank-adaptive layer-budget redistribution. During the forward pass, inactive MLP blocks store zero intermediate activations, directly feeding layer outputs downstream. When the backward pass propagates error gradients into an inactive MLP, FROST inspects the upstream gradient norm from subsequent layers to deterministically identify the top-\(k\) salient token positions. It then recomputes forward activations and evaluates Jacobian-vector products strictly for this sparse token subset, while unselected positions bypass the nonlinear transformation and flow directly through the residual skip connection. Meanwhile, an adaptive allocator monitors the Shannon effective rank of layer-wise gradient energy, dynamically shifting token budgets toward layers with diffuse gradient profiles.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Forward Pass & Inactive Classification<br/>Inactive MLPs discard forward activation caches"] --> B["Gradient-Informed Token Selection<br/>Rank tokens by upstream gradient norm to form subset S"]
    B --> C["Rank-Adaptive Budget Allocation<br/>Redistribute token budgets using Shannon effective rank"]
    C --> D["Sparse Recomputation & Residual Pass-Through<br/>Recompute JVP for subset S while others use skip link"]
    D --> E["Gradient Aggregation for Active Blocks<br/>Propagate high-fidelity gradient stream to active blocks"]

Key Designs

1. Gradient-Informed Token Selection: eliminating forward cache via backward-time proxy routing

To resolve the causal dilemma that token importance cannot be reliably anticipated during the forward pass, FROST defers token selection to the backward pass where upstream feedback is available. For an inactive MLP on a residual edge, \(h_{\text{out},t} = h_{\text{in},t} + \text{MLP}(h_{\text{in},t})\), exact backpropagation requires computing the Jacobian correction \(u_t = g_{\text{out},t} J_{\text{MLP}}(h_{\text{in},t})\) for every token position \(t \in \{1, \dots, T\}\), where \(g_{\text{out},t} = \partial \mathcal{L} / \partial h_{\text{out},t}\). Empirical investigation shows that the \(\ell_2\)-norm of upstream gradient \(\|g_{\text{out},t}\|_2\) exhibits a near-perfect Spearman correlation (\(> 0.97\)) with the true oracle correction norm \(\|u_t\|_2\). FROST therefore evaluates \(\|g_{\text{out},t}\|_2\) on arrival at the inactive MLP to deterministically select the top-\(k\) token indices: $$ S = \text{Top-}k\left( {|g_{\text{out},t}|2}^T \right), \quad k = \lfloor r \cdot T \rfloor $$ The inactive MLP completely foregoes caching hidden activations during the forward pass. During the backward pass, it executes a lightweight partial forward pass to recompute activations only for tokens in \(S\), generating the sparsified gradient: $$ \tilde{g}{\text{in},t} = \begin{cases} g $$ Because unselected tokens retain their exact upstream residual gradient },t} + u_t, & \text{if } t \in S \ g_{\text{out},t}, & \text{otherwise} \end{cases\(g_{\text{out},t}\), global computation-graph connectivity is preserved without incurring expensive Jacobian evaluations for redundant tokens.

2. Rank-Adaptive Gradient Approximation: dynamic inter-layer budget reallocation via Shannon effective rank

To overcome the inefficiency of uniform layer-wise token budgets across heterogeneous network depths, FROST dynamically reallocates token quotas according to gradient dispersion. Deep and shallow Transformer layers exhibit starkly different gradient energy distributions: applying a uniform keep ratio \(r\) severely truncates diffuse gradient signals in sensitive layers while over-allocating tokens to layers with concentrated energy. FROST formalizes the normalized gradient energy distribution of each inactive layer \(\ell\): $$ p_{\ell,t} = \frac{|u_{\ell,t}|2}{\sum|}^T |u_{\ell,t'2} $$ and quantifies its dispersion using the Shannon effective rank: $$ \text{EffRank}\ell = \exp\left( - \sum_{t=1}^T p_{\ell,t} \log p_{\ell,t} \right) $$ A higher \(\text{EffRank}_\ell\) reflects a more uniform, diffuse energy distribution that requires a larger token quota to prevent directional drift. Tracking an exponential moving average \(\hat{e}_\ell\) of the effective rank across iterations (with smoothing factor 0.9 and 100-step warmup), the allocator computes the normalized layer-wise share over the current inactive set \(I_s\): $$ \tilde{r}\ell = |I_s| \frac{\hat{e}\ell}{\sum_{\ell' \in I_s} \hat{e}_{\ell'}} $$ Each inactive layer receives an adapted token budget \(K_\ell = \lfloor \tilde{r}_\ell \cdot r \cdot T \rfloor\), clipped within \([0.1r, 2r]\). This ensures that the global mean token keep ratio remains anchored at \(r\) while adaptively providing higher fidelity to layers with high entropy gradient flows.

Loss & Training

FROST strictly preserves the optimization objective and parameter update protocol of the host BCD framework. Given full parameters \(\theta = (\theta_1, \dots, \theta_M)\) partitioned across blocks, at step \(s\) the host BCD coordinator selects an active subset \(A_s \subseteq \{1, \dots, M\}\) and updates only \(\theta_{A_s}\): $$ \theta_{A_s}^{(s+1)} = \theta_{A_s}^{(s)} - \eta \, \tilde{\nabla}{\theta) $$ where }} \mathcal{L}(\theta^{(s)\(\tilde{\nabla}_{\theta_{A_s}} \mathcal{L}\) denotes the gradient transmitted through FROST's sparsified backward pathways. Inactive parameters remain completely fixed: \(\theta_{A_s^c}^{(s+1)} = \theta_{A_s^c}^{(s)}\). Training operates in bf16 mixed precision with a bf16-state AdamW optimizer. Hyperparameters including learning rate, batch size, and total steps are identical to vanilla BCD baselines without auxiliary regularizers or extra learnable probe weights.

Key Experimental Results

Main Results

FROST is extensively evaluated across four distinct downstream regimes: (1) commonsense reasoning across 8 benchmarks (PIQA, ARC-e, ARC-c, WNGD, HS, OBQA, SQA, CSQA); (2) mathematical chain-of-thought reasoning on Math10K evaluating GSM8K, SVAMP, and AQuA; (3) instruction fine-tuning on MMLU and BBH; and (4) cross-modal transfer on vision and audio classification. The architectures evaluated include Llama-3.2-1B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-Math-1.5B, ViT-Base/16, and AST-Base.

The table below presents commonsense-reasoning accuracy and steady-state peak GPU memory (GB) comparing standard full fine-tuning, LoRA, and BCD variants with and without FROST (reproduced from main paper Table 1, default keep ratio \(r=0.5\)):

Backbone Method PIQA ARC-e ARC-c WNGD HS OBQA SQA CSQA Avg. Acc (%) Peak Mem (GB)
Llama-3.2-1B Full FT 80.4 81.3 61.7 72.9 86.1 75.1 63.2 72.4 74.1 22.4
LoRA (\(r=8\)) 78.4 75.7 58.8 70.7 83.3 72.8 60.9 70.1 71.3 14.8
LISA 78.1 79.3 50.4 67.0 82.8 72.0 62.7 69.1 70.2 17.4
LISA + FROST 77.7 79.4 50.5 69.9 83.1 71.9 62.7 69.4 70.6 12.5 (-28.2%)
MISA (\(\rho=0.05\)) 80.2 80.9 61.0 72.4 85.3 75.2 64.3 73.4 74.1 13.7
MISA + FROST 80.2 81.1 61.2 72.7 85.2 75.4 64.8 73.1 74.2 9.5 (-30.7%)
Qwen2.5-3B LoRA (\(r=8\)) 86.8 93.1 81.9 84.5 93.0 87.4 68.6 83.4 84.8 17.9
LISA 85.0 93.9 82.5 76.4 92.0 85.2 71.3 82.6 83.6 21.9
LISA + FROST 85.5 93.7 82.8 76.6 91.0 85.6 72.0 81.7 83.6 19.9 (-9.1%)
MISA (\(\rho=0.05\)) 87.7 94.7 83.1 83.4 92.9 88.4 74.7 84.4 86.2 18.9
MISA + FROST 87.1 95.0 83.2 83.2 93.3 88.2 74.3 84.2 86.1 13.5 (-28.6%)

In mathematical reasoning with Qwen2.5-Math-1.5B (main paper Table 2), FROST achieves matching benchmark accuracy while reducing peak memory by 18.2% relative to MISA and 30.8% relative to LoRA: - Pre-trained: GSM8K 52.2%, SVAMP 65.2%, AQuA 22.2%, Avg 46.5%; - Full FT: GSM8K 76.7%, SVAMP 74.4%, AQuA 41.3%, Avg 64.1%, Peak Mem 23.9 GB; - LoRA: GSM8K 76.6%, SVAMP 73.2%, AQuA 40.1%, Avg 63.3%, Peak Mem 13.0 GB; - MISA: GSM8K 76.5%, SVAMP 74.6%, AQuA 42.1%, Avg 64.4%, Peak Mem 11.0 GB; - MISA + FROST: GSM8K 76.4%, SVAMP 74.5%, AQuA 41.9%, Avg 64.3%, Peak Mem 9.0 GB (-18.2%).

Ablation Study

To validate the architectural components of FROST, the authors systematically ablate backward selection criteria against forward-time alternatives and evaluate the removal of the rank-adaptive budget allocator (reproduced from main paper Table 5):

Variant Sampling Stage & Scoring Metric PIQA Acc (%) GSM8K Acc (%) Analytical Insight
FROST (Ours) Backward Top-\(k\) \(\|g_{\text{out},t}\|_2\) + Adaptive Budget 80.2 76.4 Accurately captures actual descent direction with zero accuracy degradation
w/ Uniform Sampling Forward random uniform token selection 77.0 (-3.2) 70.5 (-5.9) Drops critical reasoning tokens; induces high gradient variance
w/ Norm Sampling Forward hidden state norm \(\|h_{\text{in},t}\|_2\) 78.5 (-1.7) 73.0 (-3.4) Forward activation magnitude decouples from backward loss sensitivity
w/ Attention Sampling Forward aggregated self-attention weights 78.0 (-2.2) 72.0 (-4.4) Attention maps do not correlate with multi-layer compound loss gradients
w/o Adaptive Budget Uniform layer-wise keep ratio \(r=0.5\) 78.7 (-1.5) 74.5 (-1.9) Fails to accommodate high-entropy layers, causing gradient fidelity loss

Key Findings

  • Dramatic Expansion of Batch and Sequence Limits: As demonstrated on a single 24 GiB GPU (main paper Figure 5), LoRA and vanilla MISA encounter out-of-memory (OOM) failures at batch size 31 and sequence length 1,280 because of their reliance on full activation trees. FROST doubles the maximum batch size to 64 and expands the maximum sequence length to 2,688, unlocking practical long-context training regimes.
  • Negative Wall-Clock Overhead via Autograd Bypass: Counterintuitively, while partial forward recomputation introduces marginal arithmetic FLOPs, it eliminates the expensive dynamic Autograd graph construction for frozen MLP blocks. Physical wall-clock profiling (main paper Figure 6(b)) reveals that FROST runs 15–19 ms faster per step than vanilla MISA across all keep ratios (e.g., at \(r=0.5\), latency drops from 116.5 ms to 99.5 ms per step).
  • Plateau Saturation and Collapse Threshold: Keep-ratio sensitivity analysis (main paper Figure 7) shows a consistent accuracy plateau in \(r \in [0.375, 0.75]\) matching the uncompressed MISA reference. However, scaling below \(r=0.25\) causes performance to collapse (at \(r=0.10\), PIQA drops to 72.7% and GSM8K collapses to 56.7%), as tight lower bounds prevent meaningful adaptive budget reallocation.
  • Modality-Agnostic Transformer Applicability: Evaluated on ViT-Base/16 (Pets-37 92.2%, Flowers-102 93.2%) and AST-Base (ESC-50 95.5%, SC-V2 86.3%) in main paper Table 4, FROST reduces peak memory by 24.6%–25.0% while maintaining identical classification accuracy, proving that frozen MLP gradient sparsity is an architectural feature of Transformers rather than a language-specific artifact.

Highlights & Insights

  • Repositioning the Inactive Computational Path as an Optimization Frontier: Rather than treating frozen blocks as a fixed invariant, FROST is the first method to explicitly treat the backward pathway through inactive layers as an actionable design space for memory reduction.
  • Backward-Time Proxy Routing with Zero Forward Footprint: By evaluating upstream gradients only when backpropagation reaches the inactive layer, the framework avoids storing activation graphs during the forward pass while ensuring selected tokens strictly mirror actual gradient sensitivity.
  • Information-Theoretic Rank Allocation: Using the Shannon effective rank of gradient energy provides a parameter-free, principled mechanism to dynamically steer compute resources toward diffuse layers without manual layer-depth heuristics.

Limitations & Future Work

  • Untouched Inactive Attention Blocks: FROST specifically targets inactive MLPs because self-attention mechanisms exhibit global token coupling and suffer significant gradient distortion when sparsified; formulating faithful attention backward approximations remains an open challenge.
  • Scaling to 10B+ Parameter Models: The empirical validation focuses on 1B–3B parameter backbones; exploring FROST under extreme scale where tensor and pipeline parallelism alter activation-to-optimizer memory balances represents an essential next step.
  • Ultra-Sparse Regime Instability: Training trajectory destabilization below \(r=0.25\) requires specialized gradient clipping or progressive token warming to prevent stochastic divergence under severe memory constraints.
  • vs LISA (Pan et al., 2024) / MISA (Liu et al., 2025): While existing BCD methods concentrate on active coordinate selection and reduce optimizer state footprints, they leave frozen block activations completely unaddressed; FROST seamlessly integrates with both, providing an orthogonal 25%–34% memory reduction.
  • vs LoRA (Hu et al., 2021) / GaLore (Zhao et al., 2024): PEFT methods restrict trainable parameters via low-rank decomposition but still maintain full forward computational graphs; BCD + FROST delivers the optimization benefits of full-parameter coordinate rotation while requiring substantially less peak memory than standard LoRA (e.g., 9.5 GB vs 14.8 GB for a 1B model).
  • vs Selective Activation Checkpointing: Traditional checkpointing recomputes entire layers and introduces up to 30% latency overhead; FROST recomputes only a sparse subset of salient tokens and bypasses autograd graph tracking, yielding a net reduction in step latency.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Innovatively establishes the backward pass of frozen blocks as an explicit compression frontier, combining backward proxy selection with effective-rank budget routing.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across commonsense reasoning, chain-of-thought math, instruction tuning, vision, audio, analytical FLOPs, wall-clock timing, and scaling sweeps.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear theoretical formulation, cohesive narrative flow, and precise mathematical and empirical exposition.
  • Value: ⭐⭐⭐⭐⭐ Delivers immediate practical utility by extending batch and sequence scaling on commodity GPUs with zero downstream degradation.