DIGS: Differentiable, Incremental, Global, Scalable Pruning for Language Models

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: model pruning, structured pruning, large language model, gradient norm control, long-horizon training

TL;DR

Addressing the long-horizon multiplier explosion and late-stage gradient oscillations in dual-ascent \(L_0\) structured pruning, DIGS introduces per-step gate-gradient norm bounds and closed-form dual updates paired with a single-backward engine GNCL-Opt, scaling structured channel pruning smoothly beyond 100B tokens and yielding monotonic accuracy gains under fixed sparsity.

Background & Motivation

The rapid scaling of large language models (LLMs) and vision-language models (VLMs) has sharply surged training and deployment costs, making structured channel pruning an indispensable technique to compress network core compute and storage. In current pruning paradigms, post-hoc non-differentiable methods such as SparseGPT and Wanda rely heavily on local statistics; while requiring only small calibration sets, their accuracy quickly saturates and cannot harvest benefits from larger token budgets. In contrast, training-based surrogate-differentiable methods—such as hard-concrete \(L_0\)-regularized structured pruning—possess global budget awareness across layers and incremental schedules. In principle, as the pruning-phase token budget scales up, task-loss gradients should continuously reshape retained channel weights and gate selections, thereby narrowing the accuracy gap between pruned and dense models.

However, attempting to scale existing \(L_0\) structured pruning to long-horizon data budgets consistently triggers a fundamental numerical instability. Conventional approaches rely on augmented-Lagrangian formulations (dual ascent, e.g., Sheared LLaMA and L0-Dual) to enforce global sparsity constraints. Over extended training horizons, the dual variable (Lagrange multiplier \(\lambda\)) steadily accumulates and explodes (often exceeding 600). Consequently, gate parameter updates become heavily constraint-dominated, amplifying stochastic gradient noise and inducing severe late-stage oscillations in both gate gradients and total loss, leading to performance collapse. This intrinsic bottleneck confines existing pruning token budgets to a few billion tokens, causing models to degrade rather than improve when exposed to more data.

This paper tackles the challenge by moving away from the paradigm of indirect accumulation-based penalty enforcement. Instead, it directly imposes an explicit per-step \(L_2\)-norm upper bound on gate-specific gradients, derives a closed-form dual update from this constraint, and pairs it with an efficient single-backward engine. Core idea: introduce DIGS, a framework that enforces per-step \(L_2\)-norm bounds on gate gradients to derive closed-form multiplier increments, incorporates adaptive saturation scaling and geometric conflict reconciliation, and executes via the single-backward GNCL-Opt engine, eliminating multiplier-driven instability and enabling stable scaling to 100B+ tokens with monotonic performance gains.

Method

Overall Architecture

DIGS retains the advantages of differentiability, incremental scheduling, and global budget awareness while thoroughly resolving the gradient amplification caused by runaway Lagrange multipliers. Starting from a dense pretrained or fine-tuned backbone (e.g., the LLM backbone of Qwen2.5VL while freezing the vision encoder and multimodal projector), DIGS attaches stochastic hard-concrete gates \(z_i \in \{0, 1\}\) to structured units (attention heads, FFN intermediate channels) and propagates gradients via continuous relaxed gates \(\hat{z}_i \in [0, 1]\) during training. The entire optimization pipeline encompasses unweighted dual-branch forward loss computation, closed-form gate-gradient norm constraint solving, adaptive geometric trade-off control, single-backward gradient rescaling, and budget-driven power-law scaling.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Dense Backbone & Multimodal Calibration Data"] --> B["Gradient-Norm Bound & Closed-Form Dual Update<br/>Constrain gate composite gradient L2 norm and solve λ in closed form"]
    B --> C["Adaptive Saturation Target & Geometric Trade-off<br/>Scale target with gate Jacobian and orthogonally project conflicting gradients"]
    C --> D["GNCL-Opt Single-Backward Engine<br/>Decouple unweighted forward branches and rescale pruning gradients backward"]
    D --> E["Data Budget Power-Law Scaling<br/>Establish power-law between token budget and norm target for 100B+ scale"]
    E --> F["Output: Accurate High-Sparsity Structured Compact Model"]

Key Designs

1. Gradient-Norm Bound & Closed-Form Dual Update: suppressing multiplier explosion via explicit gate update constraints
Standard dual ascent updates the Lagrange multiplier \(\lambda\) proportional to cumulative sparsity violation, yielding gate updates \(\hat{z} \leftarrow \hat{z} - \eta (\nabla_{\hat{z}} L_{\text{distill}} + \lambda \nabla_{\hat{z}} L_{\text{prune}})\), where \(\lambda\) inevitably inflates over long horizons. DIGS replaces integral accumulation by directly bounding the \(L_2\)-norm of the composite gate gradient at every single step by a user-specified threshold \(c_g > 0\): $\(\|g_t + \Delta \lambda g_p\|_2 \le c_g\)$ where \(g_t = \nabla_{\hat{z}} L(\lambda)\) is the current composite gate gradient and \(g_p = \nabla_{\hat{z}} L_{\text{prune}}\) is the pruning loss gradient. In the feasible regime, DIGS saturates the upper bound, leading to the quadratic discriminant \(D = (g_t^\top g_p)^2 - \|g_p\|^2 \|g_t\|^2 + \|g_p\|^2 c_g^2\). When \(D \ge 0\), the non-negative root is selected; when infeasible (\(D < 0\)), the least-squares minimizer yielding the smallest attainable norm is applied: $\(\Delta \lambda = \begin{cases} \frac{-g_t^\top g_p + \sqrt{D}}{\|g_p\|^2 + \varepsilon_{\text{den}}}, & D \ge 0 \\[6pt] -\frac{g_t^\top g_p}{\|g_p\|^2 + \varepsilon_{\text{den}}}, & D < 0 \end{cases}\)$ The multiplier is then updated with non-negativity clipping \(\lambda \leftarrow \max(\lambda + \Delta \lambda, 0)\). Because the multiplier update is tied strictly to gradient magnitudes rather than constraint error integration, \(\lambda\) stays tightly bounded (mostly below 100 throughout training), preventing multiplier explosion by construction.

2. Adaptive Saturation Target & Geometric Trade-off: mitigating gate saturation drift and balancing task fitting against sparsity
As hard-concrete gates polarize toward 0 or 1 in late training stages, their sigmoid-like Jacobian derivatives contract rapidly, causing \(\|g_p\|\) to diminish. Retaining a fixed \(c_g\) would artificially inflate \(\Delta \lambda\) due to vanishing denominators. To counter this, DIGS introduces an adaptive target scaling mechanism that defines the effective norm target as: $\(c_g^{\text{eff}} = c_g \cdot \frac{\|g_p\|}{\|g_p\| + \varepsilon_{\text{scale}}}\)$ This preserves nominal control pressure when gates are active, but gracefully attenuates the target when gates saturate and channel selection has crystallized. In addition, DIGS provides geometric trade-off dynamics: letting \(g_d = \nabla_{\hat{z}} L_{\text{distill}}\), when \(\angle(g_d, g_p) < 90^\circ\) (aligned gradients) and \(\|g_d\|\) grows large, the algorithm decreases \(\lambda\) to prevent overly aggressive pruning; when \(\angle(g_d, g_p) > 90^\circ\) (opposing gradients) and distillation pressure suppresses pruning, \(\lambda\) increases to counterbalance task opposition; and in infeasible regions (\(D < 0\) with massive \(\|g_d\|\)), the least-squares solution eliminates the composite update's projection onto \(g_p\), temporarily pausing pruning until task distillation stabilizes.

3. GNCL-Opt Single-Backward Engine: eliminating double-backward compute overhead
A naive implementation of the DIGS control rule would require an initial backward pass to harvest gate gradient statistics for computing \(\lambda\), followed by a second backward pass to update parameters under the refreshed loss coefficient, doubling backward compute. To resolve this, DIGS implements Gradient-Norm Controlled Lagrangian Optimization (GNCL-Opt). During the forward pass, GNCL-Opt duplicates gate variables into two branches contributing independently to \(L_{\text{distill}}\) and \(L_{\text{prune}}\), maintaining an unweighted additive objective \(L = L_{\text{distill}} + L_{\text{prune}}\). In the backward pass, custom backward hooks intercept both gradient streams, compute \(\lambda\) on the fly via closed-form equations, and rescale only the pruning branch gradient before merging. This is mathematically equivalent to optimizing \(L_{\text{distill}} + \lambda L_{\text{prune}}\) in forward, while eliminating the extra backward pass and cutting memory and runtime overhead.

4. Data Budget Power-Law Scaling: budget-driven hyperparameter inversion for large-scale pruning
Because DIGS lets the pruning rate emerge adaptively from the gradient constraint rather than an arbitrary linear schedule, the choice of \(c_g\) dictates the token budget required to reach the target sparsity. By running 4–6 short calibration trials on a logarithmic grid of \(c_g\) at a fixed target sparsity \(R\) (e.g., 30%, 45%, 60%), the required pruning token budget \(\mathbf{B}\) follows a predictable power-law relation: $\(\mathbf{B}(R) \approx a(R) \cdot c_g^{-m(R)} + b(R)\)$ where \(a(R)\) is a scale factor, \(m(R)\) is the power exponent, and \(b(R)\) is an offset. Given a target compute or token budget \(\mathbf{B}^\star\), engineers can directly invert this formula to obtain the optimal constraint coefficient \(c_g^\star = \left(\frac{a(R)}{\mathbf{B}^\star - b(R)}\right)^{1/m(R)}\), replacing trial-and-error hyperparameter tuning with principled budget allocation up to 100B+ tokens.

Loss & Training

The pruning pipeline targets the LLM backbone of Qwen2.5VL-3B, keeping the ViT encoder and multimodal projector frozen. The distillation objective uses forward KL divergence over autoregressive response tokens: $\(L_{\text{distill}} = \mathrm{KL}(p_{\text{teacher}} \,\|\, p_{\text{student}})\)$ The global weighted sparsity metric is defined as \(R(\alpha) = \frac{1}{S} \sum_{i=1}^N s_i \pi_i(\alpha_i)\), where \(s_i\) denotes the parameter count eliminated if unit \(i\) is pruned. The model parameters are trained with a learning rate of \(1\times 10^{-4}\) and gate logits \(\alpha\) with \(1\times 10^{-2}\). A dense warmup of 1,000 steps without pruning is conducted first to decouple learning rate warmup from sparsity scheduling. Subsequent pruning runs do not employ any sparsity warmup schedule, relying entirely on DIGS adaptive gradient-norm control.

Key Experimental Results

Main Results

Evaluation is conducted using VLMEvalKit across 8 diverse multimodal benchmarks: MMBench, MMStar (MMSt), MMMU (MMMU-Dev-Val), MathVista (MathV), HallusionBench (Hallu), AI2D, OCRBench (OCR), and MMVet. Comparisons are performed under rigorously matched token budgets at 30% and 60% sparsity against the primary baseline L0-Dual.

At 30% target sparsity, scaling the token budget from 5.7B to 24.5B shows opposing trends between DIGS and L0-Dual:

Method (Run / Ckpt)	Tok@ckpt (B)	MMBench / Hallu	MMSt / AI2D	MMMU / OCR	MathV / MMVet	Avg	∆Acc vs Dense
Qwen2.5VL-3B (Dense)	–	74.75 / 46.60	56.30 / 81.40	51.20 / 82.80	61.20 / 65.90	65.01	0.00
L0-Dual FR5.7B@30%	3.1	68.17 / 49.55	48.46 / 70.17	40.33 / 78.20	53.70 / 57.52	58.26	-6.75
DIGS FR5.7B@30%	2.7	64.33 / 44.96	51.26 / 67.09	37.33 / 76.70	53.30 / 53.48	56.04	-8.97
L0-Dual FR24.5B@30%	11.9	63.40 / 48.24	49.86 / 67.64	40.77 / 76.60	52.70 / 55.59	56.85	-8.16
DIGS FR24.5B@30%	10.3	68.54 / 44.75	52.33 / 70.88	41.89 / 78.50	57.30 / 55.87	58.75	-6.26

At 60% sparsity, where more than half of the backbone channels are removed, stability differences over long horizons become even more pronounced:

Method (Run / Ckpt)	Tok@ckpt (B)	MMBench / Hallu	MMSt / AI2D	MMMU / OCR	MathV / MMVet	Avg	∆Acc vs Dense
Qwen2.5VL-3B (Dense)	–	74.75 / 46.60	56.30 / 81.40	51.20 / 82.80	61.20 / 65.90	65.01	0.00
L0-Dual FR5.7B@60%	5.70	41.94 / 43.46	43.66 / 47.05	33.00 / 71.30	37.90 / 37.80	44.51	-20.50
DIGS FR5.7B@60%	5.70	50.70 / 36.53	44.60 / 55.21	35.00 / 71.50	47.69 / 42.20	47.92	-17.09
L0-Dual FR24.5B@60%	24.50	26.96 / 40.66	38.41 / 41.25	30.77 / 63.40	36.70 / 33.62	38.97	-26.04
DIGS FR24.5B@60%	24.50	59.74 / 44.43	45.80 / 57.41	36.88 / 74.90	51.40 / 44.91	51.93	-13.08

On standard LLM benchmarks (Gemma and LLaMA2 families), DIGS is evaluated under a unified FT-then-prune protocol against competing pruning methods:

Sparsity	Method	Gemma-2B	Gemma-7B	LLaMA2-7B	LLaMA2-13B
0% (Dense)	Dense Baseline	53.82	71.59	58.76	66.74
25%	LLM-Pruner	40.20	50.29	39.72	39.82
25%	SliceGPT	39.72	52.13	41.97	46.75
25%	PAT	52.98	66.68	60.02	66.58
25%	DIGS (Ours)	53.65	70.73	57.45	67.12
30%	LLM-Pruner	40.05	41.35	39.73	39.70
30%	SliceGPT	39.89	44.30	40.14	46.19
30%	PAT	45.33	64.58	57.81	65.15
30%	DIGS (Ours)	49.81	70.12	58.27	66.15

Ablation Study

Ablations investigate whether simpler heuristic stabilizers can rescue dual ascent, alongside \(c_g\) sensitivity and scaling to 100B+ tokens. Diagnostic evaluations use Avg3 (average of MMStar, MathVista-Mini, and OCRBench).

The stabilizer ablation shows that post-hoc gradient clipping or capping cannot match adaptive norm control:

Variant	#Tokens	Avg3	Mechanism & Failure Analysis
L0-Dual (baseline)	~12B	46.32	No stabilizer; \(\lambda\) exceeds 600 over long horizons, inducing severe oscillations
L0-Dual + Gradient Clipping	~12B	45.69	Post-composite clipping shrinks distillation and pruning gradients indiscriminately
L0-Dual + Multiplier Cap (\(\lambda \le 50\))	~19B	52.74	Capping prevents divergence but severely retards pruning, requiring ~7B extra tokens to hit 60%
DIGS (Ours)	~12B	56.29	Bounding gate gradients before merging provides optimal trade-off and best accuracy

Sensitivity analysis of \(c_g\) confirms smooth budget-scaling behavior at 60% target sparsity:

Control Target \(c_g\)	#Tokens	MMStar	MathVista-Mini	OCRBench	Avg3
\(1.2 \times 10^{-2}\)	1.06B	39.80	34.60	62.80	45.73
\(4.7 \times 10^{-3}\)	2.42B	42.33	42.30	68.60	51.08
\(1.9 \times 10^{-3}\)	5.71B	44.60	47.69	71.50	54.60
\(7.5 \times 10^{-4}\)	12.46B	47.07	49.10	72.70	56.29
\(3.0 \times 10^{-4}\)	24.50B	45.80	51.40	74.90	57.37

Scaling to 100B+ tokens at 33% target sparsity (compressing 3B to ~2B parameters) confirms that capacity loss can be almost entirely recovered:

Stage	Pruning Tokens	Recovery Tokens	8-Benchmark Avg	∆Acc vs Dense
Dense Baseline	–	–	65.01	0.00
Prune-only	114B	–	61.52	-3.49
+ post-pruning distillation	114B	20B	63.94	-1.07
+ FineVision SFT	114B	20B + SFT	64.32	-0.69

Key Findings

• Reversal of Data-Scaling Dynamics: At 60% sparsity, L0-Dual peaks at 5.7B tokens (44.51) and collapses to 38.97 (-5.54) at 24.5B tokens; in stark contrast, DIGS advances monotonically from 47.92 to 51.93 (+4.01), showing that data scaling is viable once optimization is stabilized.
• Superior Convergence Profile: Training curves reveal that L0-Dual plateaus at a final distillation loss of ~1.47 with high variance and \(\lambda > 600\), whereas DIGS maintains \(\lambda < 100\) and achieves a distillation loss of ~0.884 with negligible variance.
• Near-Lossless 100B Scaling: With 114B pruning tokens followed by 20B distillation and SFT, the pruned 2B model achieves 64.32 on multimodal benchmarks, coming within 0.69 points of the dense 3B model (65.01).

Highlights & Insights

• From Penalty Accumulation to Active Norm Bounding: While dual ascent acts as an integrator that accumulates past constraint violations into runaway multipliers, DIGS converts constraint satisfaction into an instantaneous norm-bounding projection on the backward update step, establishing a stable closed-loop regulator.
• Single-Backward Efficiency via GNCL-Opt: By separating forward branches and scaling gradients during backpropagation before fusion, GNCL-Opt matches the exact dynamics of a weighted objective without incurring a second backward pass, making 100B+ token pruning practical on compute-constrained clusters.
• Invertible Budget Power Law: The power-law formulation \(\mathbf{B}(R) \approx a(R) c_g^{-m(R)} + b(R)\) turns \(c_g\) into an intuitive budget knob, allowing practitioners to plan compute budgets systematically rather than guessing scheduling hyper-parameters.

Limitations & Future Work

• Backbone-Only Pruning Scope: The empirical validation focuses exclusively on the LLM backbone while keeping the ViT encoder and projector frozen; extending joint structured pruning across multimodal components remains unexplored.
• Architecture-Specific Calibration: Fitting the budget law coefficients \(a, m, b\) requires running 4–6 short calibration trials whenever switching to a new architecture or data mixture.
• Generalization to Dynamic Sparsity: While formulated for static channel pruning, the gradient-norm bounded Lagrangian framework could naturally be extended to MoE expert routing and KV cache dynamic context pruning.

Related Work & Insights

• vs L0-Dual (Sheared LLaMA, Wang et al.): Both apply hard-concrete gating for structured pruning, but L0-Dual suffers from multiplier explosion and performance collapse on long horizons; DIGS bounds gate-gradient norms explicitly, unlocking stable scaling beyond 100B tokens.
• vs Post-hoc Methods (SparseGPT, Wanda): Post-hoc methods are training-free and fast but saturate quickly on small calibration sets; DIGS operates as a scalable training-time framework that continually leverages vast token streams.
• vs PAT (Pruning-Aware Tuning): PAT relies on heuristic multi-stage tuning using ~0.256B tokens; DIGS achieves superior retention across Gemma and LLaMA2 backbones, especially at high sparsity (30%).

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates a mathematically rigorous, closed-form gradient-norm control paradigm that resolves a long-standing instability in Lagrangian pruning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning 8 multimodal benchmarks, 4 LLM backbones, extensive ablations, and scaling up to 114B tokens.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid narrative, precise mathematical derivations, illuminating geometric analysis, and highly informative figures.
• Value: ⭐⭐⭐⭐⭐ Bridges the gap between structured model pruning and large-scale data scaling, providing substantial practical utility for foundation model deployment.