KATANA: Knowledge-Aligned Topology-Aware Neural Agents for RL-Driven Vision-Language Model Compression

Conference: ECCV 2026
Paper: Official paper page · PDF
Area: Model Compression / LLM Agents / Multimodal VLM
Keywords: program synthesis, GRPO, unstructured pruning, dual-norm activation, sparse reconstruction

TL;DR

KATANA treats the pruning algorithm itself as a learnable program, trains a code agent with rewards measured in a sandbox, and discovers KIRI's combination of cubic sparsity scheduling, Dual-Norm Activation scoring, and in-mask reconstruction; at 70% sparsity, LLaVA-1.5-7B reaches 84.6 MSCOCO CIDEr versus GSOP's 83.3, a 1.3-point gain, with approximately 2.8x decode acceleration on the specified sparse runtime.

Background & Motivation

Weights in a vision-language model support more than language modeling: they connect the vision encoder, cross-modal projector, and language decoder. Removing a numerically small weight need not produce a small output change if that weight belongs to a sensitive channel mapping visual features into language space. Magnitude pruning, Wanda, and SparseGPT offer different importance estimates, but their scoring forms are predetermined by researchers. A fixed rule may not preserve image-text correspondence across modules with different scales and input statistics.

Existing automated compression methods can search layer budgets or subnetworks, but this is different from searching how to write the pruning program. Tuning coefficients inside a fixed Wanda expression cannot independently introduce a product of row and column norms or invent a new iterative sparsity schedule. KATANA therefore expands the action space to Python programs and evaluates candidates using caption quality and latency, rather than optimizing weight reconstruction error alone.

Deployment adds another constraint: setting unstructured weights to zero does not automatically accelerate ordinary dense operators, while low-rank compensation branches can reintroduce inference overhead. Core idea: use task-quality and hardware feedback to let a code agent jointly discover importance scoring and scheduling rules, then restrict recovery updates to the existing sparse mask so algorithm search, accuracy recovery, and sparse execution serve the same compression objective.

Method

Overall Architecture

There are two distinct levels. KATANA is an offline search framework that produces a reusable pruning program. KIRI is the discovered algorithm: it takes a pretrained VLM, calibration samples, and a target sparsity, and returns a model with unchanged tensor shapes but zeroed weight entries. Deployment runs the compressed VLM, without the code agent participating in generation.

The search loop consists of candidate code generation, syntax and sandbox checks, calibration-model pruning, quality and latency evaluation, and GRPO updates. Once KIRI is available, it caches input activations, gradually raises sparsity using a cubic schedule, recomputes weight importance through DNA, constructs masks, and optionally performs in-mask low-rank reconstruction at high sparsity. In this paper, Kernel means the algorithmic program, not a CUDA kernel directly generated by the agent.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Baseline program, objectives<br/>and evaluation history"] --> Search["Reward-Guided Program Search<br/>Code generation, sandbox, GRPO"]
    Search -->|Measured reward feedback| Search
    Search -->|Discover KIRI| Schedule["Cubic Sparsity Scheduling"]
    Model["Pretrained VLM<br/>128 calibration samples"] --> DNA["Dual-Norm Activation Scoring"]
    Schedule --> DNA
    DNA -->|Layer-wise quantile masks| Recon["In-Mask Low-Rank Reconstruction"]
    Recon -->|Recompute weight scores next step| Schedule
    Recon --> Output["Sparse VLM<br/>Specialized sparse runtime"]

Key Designs

1. Reward-Guided Program Search: learn to generate rules, not just search their parameters

The code agent receives a baseline pruning function, historical performance, and task objectives, and generates complete candidate programs. Before execution, candidates undergo parsing and restrictive checks, followed by RestrictedPython tests on dummy tensors; valid candidates are then run on the actual VLM. This ordering filters invalid programs before expensive model evaluation. The paper describes an isolated sandbox, but its full security configuration is absent from the cache, so the system should not be treated as an independently audited secure execution environment.

The outer loop updates the generation policy through GRPO reward feedback, rather than merely inserting the previous score into the next prompt. The total reward is a weighted sum of performance, compression, novelty, diversity, and stability components. The detailed performance definition uses pruned CIDEr divided by dense CIDEr, while latency is the ratio of pruned to dense per-token decode time. Although the overview mentions perplexity, this note follows the explicit definition. Equation (3) is corrupted in the cached extraction, so its complete sigmoid expression cannot be reliably recovered and is not invented here.

The curriculum has 4 phases, moving performance and latency weights from \((0.1,0.9)\) early on to \((0.9,0.1)\) later: aggressive compression exploration precedes stronger quality preservation. Compression below 5% receives a -4.0 penalty. Novelty is evaluated against 10,000 strategy signatures using code structure, parametric behavior, targeted layers, and techniques. Removing superficial AST identifiers before structural comparison helps reject apparent novelty caused solely by renaming variables. Monotonic CIDEr improvements over the latest 5 generations receive a stability bonus, but the cached main text does not fully specify the diversity calculation.

2. Cubic Sparsity Scheduling: postpone aggressive deletion until after gradual updates

KIRI does not jump directly to the final pruning ratio. It starts slowly after warmup and then accelerates sparsification. Algorithm 1 and its surrounding explanation support the following clean rendering over the stated schedule interval; this is a reconstruction of the described algorithm, not an invented replacement for damaged typesetting. The settings are \(t_0=2000\), \(n=10\), \(\Delta t=2000\), initial sparsity \(s_0=0\), and target sparsity \(s_f\).

\[ s(t)= \begin{cases} s_0, & t<t_0,\\ s_0+(s_f-s_0)\left(\dfrac{t-t_0}{n\Delta t}\right)^3, & t_0\le t\le t_0+n\Delta t. \end{cases} \]

This expression must not be extrapolated past the interval, where it would exceed the target; the main text does not fully specify post-endpoint behavior. These are internal KIRI steps, not the outer 1,000 RL search iterations. The pseudocode executes step by step while its initialization also lists intervals, leaving the precise invocation frequency to be confirmed from an implementation. The authors report a 15% reduction in activation-variance spikes relative to linear scheduling. That is an empirical observation, not a property guaranteed by the curve alone.

Gradual scheduling can affect the final mask because each step recomputes norms and magnitudes from current weights, and reconstruction can change subsequent scores. The trajectory is therefore path-dependent. If scores remained fixed and weights never changed, applying a threshold in several stages would not by itself explain a large difference at the same final sparsity. The cubic curve must be understood together with iterative rescoring and reconstruction.

3. Dual-Norm Activation Scoring: evaluate a weight in the context of its input and output channels

KIRI prunes linear layers in the vision encoder, cross-modal projector, and the language model's attention/MLP blocks, while exempting embeddings and the final language-model head. A single calibration pass uses 128 MSCOCO validation samples. Visual and textual token positions are averaged with equal weight so longer text sequences do not dominate activation statistics, after which the hooks are removed. Algorithm 1 and equation (4) complement each other sufficiently to render the DNA score as:

\[ I_{ij}^{(\ell)}= |W_{ij}|\left(\|W_{i:}\|_2\|W_{:j}\|_2\right)^\gamma \left(\frac{\bar a_j}{\operatorname{std}(\bar{\mathbf a})+\epsilon}\right)^\alpha. \]

Here \(i\) is an output channel and \(j\) an input channel, so activation \(\bar a_j\) corresponds to the input column. Magnitude measures the strength of an individual connection, row and column norms characterize its surrounding channel structure, and the activation term reflects how calibration data uses that input channel. Main experiments use \(\gamma=0.5\) and \(\alpha=1.0\). Each layer selects a threshold at the current \(s(t)\) quantile and retains high-scoring weights, rather than sharing one uncalibrated threshold across the entire model.

Two qualifications matter. Activations are calibrated once; the quantities recomputed during pruning are weight magnitudes and structural norms, not freshly collected activations. Also, the standard deviation is a shared denominator within a layer. With fixed exponents and layer-wise ranking, it cannot independently change the ordering of weights. The authors' explanation about protecting high-activation-variance weights therefore does not follow from this denominator alone. The cache also does not clearly specify how signed activations are handled, so \(\bar a_j\) should not silently be rewritten as an absolute value or mean square.

4. In-Mask Low-Rank Reconstruction: recover values without restoring deleted connections

When current sparsity exceeds 0.6 and reconstruction is enabled, KIRI approximates the pruning residual with a low-rank update. The text specifies SVD factor rank \(r=\min(0.1\times\min(m,n),64)\), followed by an element-wise projection onto the existing mask. Equation (5) and Algorithm 1 give the update:

\[ W^{\mathrm{recon}}=W^{\mathrm{pruned}}+M\odot(UV^\top). \]

Unlike attaching a dense LoRA branch at deployment, this folds \(UV^\top\) into the weights offline: entries outside the mask remain zero, and inference uses a single sparse matrix. The authors acknowledge that unconstrained SVD followed by projection is not the optimum of directly constrained in-mask least squares. They report retaining 98% of an exact solver's accuracy, but the cache lacks a corresponding detailed results table, so this should not be treated as independently verified evidence.

The guarantee is that an additional branch does not enlarge the support or theoretical inference computation, not that latency is unchanged on every device. Actual acceleration still depends on cuSPARSE v11.7 and custom SparseTIR-style sparse GEMM. Ordinary dense matrix multiplication does not obtain equivalent gains merely because some entries are zero.

A Worked Example

Consider the paper's LLaVA-1.5-7B configuration with 70% target sparsity. Input statistics are collected from 128 calibration samples, then pruning proceeds along the cubic curve after warmup. At each step, DNA is recomputed and layer-wise thresholds zero weights; in-mask reconstruction can be enabled once sparsity exceeds 0.6. The final model generates captions through the specified sparse runtime without calling the code agent again.

In Table 1, the dense model takes 0.17 s per decoded token and KIRI takes 0.060 s. Their ratio is approximately 2.83, consistent with the reported approximate 2.8x speedup. CIDEr falls from 91.8 to 84.6, a remaining loss of 7.2 points. The correct interpretation is therefore better quality preservation than competing compressed models, not lossless compression. This example connects reported settings and measurements without fabricating the output for an individual image.

Loss & Training

Search uses DeepSeek-Coder-V2-Lite-Instruct 16B with 4-bit quantization and LoRA on attention/MLP modules: rank 64, scaling parameter 128, and dropout 0.05. GRPO uses a learning rate of \(2\times10^{-5}\), cosine scheduling, generation temperature 0.7, and KL penalty coefficient 0.02. This generation temperature belongs to the code policy; candidate VLM captions are evaluated with greedy decoding.

The full search runs for 1,000 iterations with 8 candidates per iteration and a reported cost of 480 A100 GPU-hours. Discovery occurs only on LLaVA-1.5 before the strategy is applied across architectures. Rewards use a held-out set of 256 MSCOCO validation samples rotated every 50 iterations, distinct from the 128 pruning-calibration samples. At 300 iterations and 144 GPU-hours, performance reaches 82.5 CIDEr, approximately 97% of the final 84.6. This is a ratio of absolute scores, not recovery of 97% of the compression-induced loss.

Key Experimental Results

Main Results

The following extracts CIDEr from the paper's Table 1, retaining its metric scaling by 100. Uncertainty for KIRI and GSOP is standard deviation across 3 seeds. The 70% setting refers to KIRI's weight sparsity; token-pruning baselines are matched by theoretical FLOPs according to the authors, not by deleting the same percentage of weights. Model names follow the paper: the exact visual variant of Qwen2.5-7B and BLIP-2 backbone are not adequately identified in the cached main text.

Model / Dataset	Dense CIDEr	GSOP CIDEr	KIRI CIDEr	KIRI vs. GSOP	KIRI decode time, s/token
LLaVA-1.5-7B / MSCOCO	91.8	83.3 ± 0.3	84.6 ± 0.2	+1.3	0.060 ± 0.002
BLIP-2 / MSCOCO	92.5	84.8 ± 0.3	85.4 ± 0.3	+0.6	0.061 ± 0.002
Qwen2.5-7B / MSCOCO	92.4	86.8 ± 0.3	87.9 ± 0.2	+1.1	0.060 ± 0.002
Llama-3.2-Vision / MSCOCO	93.8	89.6 ± 0.3	90.5 ± 0.3	+0.9	0.061 ± 0.002

Cross-dataset results in Table 2 also use LLaVA-1.5-7B. Dense / SparseVLM / KIRI achieve 56.3 / 49.0 / 51.5 CIDEr on Flickr30k and 50.9 / 43.5 / 46.4 on NoCaps. These support transfer to those two captioning datasets, but do not establish preservation of every multimodal reasoning capability.

Ablation Study

The following retains the values and uncertainties from Table 3 rather than substituting Table 1's Full-model standard deviation. The configuration is LLaVA-1.5-7B, MSCOCO, and 70% sparsity; VQA refers to VQAv2. All removal-related drops are relative to Full in this table.

KIRI configuration	CIDEr	SPICE	VQA	Speedup
Full	84.6 ± 0.3	16.9 ± 0.2	78.5 ± 0.3	2.8 ± 0.05
Linear schedule	78.2 ± 0.5	15.4 ± 0.3	65.4 ± 0.5	2.7 ± 0.06
Constant schedule	76.5 ± 0.6	15.1 ± 0.4	59.8 ± 0.6	2.8 ± 0.05
DNA without norms	80.1 ± 0.4	16.0 ± 0.3	71.0 ± 0.4	2.8 ± 0.04
DNA without activations	79.5 ± 0.4	15.8 ± 0.3	69.2 ± 0.4	2.8 ± 0.05
Magnitude-only scoring	77.3 ± 0.5	15.3 ± 0.4	63.5 ± 0.5	2.8 ± 0.05
Without reconstruction	81.3 ± 0.6	16.2 ± 0.3	72.9 ± 0.6	2.9 ± 0.04

Table 4 further distinguishes the effectiveness of the search framework from the effectiveness of KIRI's components, using a budget of 1,000 iterations. Novelty Score is the average novelty reward, not a universal measure of algorithmic originality across papers.

Search framework configuration	CIDEr	Novelty Score	Final reward
Full	84.6 ± 0.3	0.72 ± 0.04	6.8 ± 0.2
Without novelty reward	79.8 ± 0.5	0.41 ± 0.05	5.9 ± 0.3
Without diversity reward	80.4 ± 0.4	0.58 ± 0.04	6.1 ± 0.2
Without curriculum	77.9 ± 0.6	0.52 ± 0.05	5.7 ± 0.3
Without GRPO, random search	76.2 ± 0.7	0.48 ± 0.06	5.4 ± 0.4

Key Findings

• Linear scheduling loses 6.4 CIDEr, removing norms loses 4.5, removing activations loses 5.1, and removing reconstruction loses 3.3. These support the combination's usefulness, but without a complete factorial experiment, individual ablations do not prove strictly non-additive synergy.
• Removing GRPO lowers CIDEr by 8.4 relative to Full; removing the curriculum lowers it by 6.7. This random-search setting should not be conflated with the evolutionary baseline reported at 78.4 CIDEr in the method overview.
• Latency is measured on A100 at batch size 1 after a 512-token prefill, and means per-token decode time, not end-to-end response time or time to first token. The main text also reports 32 ms versus 71 ms on RTX 4090, approximately 2.2x, but the cache lacks the appendix's full breakdown.
• The sparsity-sweep paragraph uses beam search and reports 118.4 CIDEr at 70%, while the main table uses greedy decoding and reports 84.6. Gains must not be calculated by mixing these decoding protocols.

Highlights & Insights

• The search object changes from continuous parameters to executable expressions and control flow. A reusable idea is to subject candidates directly to downstream evaluation rather than equating a reconstruction proxy with final task quality.
• Iterative rescoring connects DNA with the pruning schedule. The research value lies in testing interactions between which weights are deleted and the trajectory of deletion, not merely proposing another magnitude multiplier.
• In-mask reconstruction folds recovery into the final sparse weights. It highlights why compression methods should distinguish offline recovery cost from online execution cost instead of offsetting pruning gains with an extra dense branch.

Limitations & Future Work

• Search cost and transfer boundaries: 480 A100 GPU-hours is a discovery cost, not free compression; applying the strategy across architectures still requires target-model calibration and pruning. The authors propose N:M sparsity and broader tasks, but the cache lacks the preliminary results in Appendix E.
• Runtime dependence: the 2.8x gain depends on specialized sparse execution and cannot be generalized to arbitrary devices or dense fallback. Matching theoretical FLOPs between token removal and weight pruning does not automatically ensure fairness in bandwidth, retained context, or time to first token.
• Incomplete evidence: the cache contains the main paper and references, not Appendices A–E. The full reward expression, generation prompt, exact model checkpoints, activation sign handling, and schedule execution details cannot be completely reproduced from it. The overview's 78.4 for evolution and Table 4's 76.2 for random search lack a unified explanation; Full-model standard deviations also differ between the main and ablation tables.
• Mechanistic claims need qualification: a layer-shared standard deviation cannot independently change ranking, and single-factor ablations cannot establish strict synergy. Table 4's novelty decline from 0.72 to 0.48 is also not the halving claimed in the prose. Follow-up work should add factorial experiments, define rewards explicitly, and report variance across independent search runs.
• Task coverage remains caption-heavy: VQAv2 appears mainly in ablations, while MME and POPE information in the main text is brief. The authors suggest GQA and NLVR2 for compositional reasoning, which would help distinguish preservation of captioning ability from broader image-text understanding.

Related Work & Insights

• vs. Wanda / SparseGPT: Wanda uses activation-aware importance, while SparseGPT emphasizes reconstruction error; KATANA searches program structure, and KIRI combines magnitudes, row/column norms, and activations. Advantages should be assessed together with search and sparse-runtime engineering costs.
• vs. AMC / AutoCompress / OptiShear: layer-budget search, automated compression, and evolutionary metric search already establish precedents for automated design. The more specific distinction is updating the code-generation policy with GRPO, not being the first method to use search for compression.
• vs. Eureka / FunSearch: these share the generate-execute-feedback paradigm for algorithm discovery. KATANA grounds feedback in VLM caption quality, sparsity, and measured latency, then turns the discovered program into a transferable pruning rule.
• vs. FastV / SparseVLM / GSOP: compressing tokens, weights, or operations changes different bottlenecks. Comparisons should report prefill, decode, context loss, and memory overhead together rather than relying on one aggregate speedup.

Rating

• Novelty: 4/5. Combining program-level search with gradual DNA pruning is worth investigating, although individual components have clear precedents.
• Experimental Thoroughness: 3/5. The paper covers 4 models, cross-dataset tests, and two levels of ablation, but lacks unified reproduction details and sufficient tests of synergy.
• Writing Quality: 3/5. The distinction between the search framework and discovered algorithm is clear, while several numbers, protocols, and mechanistic explanations need clarification.
• Value: 4/5. It offers transferable ideas for task-driven algorithm discovery and offline sparse recovery, with practical gains constrained by search budgets and sparse runtimes.