Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models

Conference: ECCV2026
Paper: ECCV Paper
Area: VLM Efficiency
Keywords: visual token compression, discrete cosine transform, low-frequency truncation, adaptation training, inference efficiency

TL;DR

Fourier Compressor transforms visual feature grids into the frequency domain, retains low frequencies, and reconstructs a smaller grid; with adaptation training, compressing LLaVA-v1.5-7B from 576 to 36 visual tokens changes its eight-benchmark average from 64.6 to 62.1 and reduces FLOPs from 8.54 T to 1.38 T.

Background & Motivation

Image inputs to vision-language models (VLMs) are often much longer than their text instructions, making visual tokens a major contributor to large language model (LLM) prefill cost and KV cache usage. LLaVA-v1.5 produces 576 visual tokens for a 336×336 image; high-resolution Qwen-VL inputs and multi-frame videos extend the sequence further. These tokens are not independent information units, since neighboring high-level features often repeat scene structure. Existing methods therefore remove less important positions or merge similar tokens, but pruning can discard evidence and merging changes the feature distribution. Learnable queries can aggregate information again, but add parameters and computation to the compressor, so transmitting fewer tokens does not automatically make the entire system cheaper.

This paper changes the representation used to inspect redundancy: instead of deciding which image positions to retain, it examines the spatial frequencies composing the visual feature map. If semantics concentrate in low frequencies, discarding high-frequency components allows each retained component to continue aggregating information from multiple original positions. However, natural images being amenable to low-frequency approximation does not establish the same property for high-level visual features, motivating the spectrum, perturbation, and semantic morphing analyses. The evidence supports low frequencies emphasizing stable global structure and high frequencies being more sensitive to local changes, rather than proving that high frequencies have no task value. This distinction matters especially for text recognition and fine-grained discrimination, where low-magnitude details may still be essential.

The final design uses fixed low-frequency truncation instead of learning another frequency-selection network, keeping the compression operation simple and transferable across architectures. It also acknowledges that truncation changes visual representations and requires the existing model to adapt through training, so parameter-free describes the compressor, not a training-free workflow. Core Idea: replace compression over spatial token positions with selection over frequency components, preserve global low-frequency structure through 2D DCT, and reconstruct a shorter spatial token sequence through the inverse transform.

Method

Overall Architecture

The input is a feature sequence produced by the vision encoder that can still be mapped to a 2D patch grid, not a JPEG encoding of raw RGB pixels. The compressor restores the grid, applies a channel-wise 2D discrete cosine transform (DCT), keeps the upper-left low-frequency rectangle, and applies an inverse DCT at a smaller size. The output remains real-valued spatial features with the same channel width but fewer grid positions, allowing the original vision-language interface to remain in use. In LLaVA, compression sits after the vision encoder and before the projector; the Qwen-VL implementation instead inserts it after the original MLP merger. The latter is an architectural adaptation explicitly described in Section 5.1, so compression should not be presented as preceding the merger in every model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image or video frames"] --> Encoder["Vision encoder<br/>Original merger retained in Qwen"]
    Encoder --> Transform["Spatial Frequency<br/>Decomposition"]
    Transform --> Crop["Low-Frequency<br/>Rectangular Truncation"]
    Crop --> Reconstruct["Small-Grid Spatial<br/>Reconstruction"]
    Reconstruct --> Output["Original downstream interface + text<br/>LLM answer generation"]

The diagram shows the forward data flow shared by training and inference, without an additional teacher branch, query module, or question-based token scorer. During training, compressed features enter the existing model for adaptation through image-text alignment and instruction learning; inference applies the same deterministic transform with the adapted weights. The video extension also relies on the existing visual processing pipeline, while the compressor remains a 2D spatial transform rather than 3D spectral truncation over time. Consequently, video frames in the diagram identify an additional input source, not a separately designed temporal-frequency module.

Key Designs

1. Spatial Frequency Decomposition: preserve 2D adjacency while changing from positions to frequencies

The vision encoder output has shape \(B\times(N_hN_w)\times h_v\), where \(B\) is batch size, \(N_h,N_w\) are grid dimensions, and \(h_v\) is feature width. Algorithm 1 transposes and reshapes it into \(B\times h_v\times N_h\times N_w\), assigning each hidden channel its own 2D feature map. DCT operates only along the two spatial axes, without mixing hidden channels or arbitrarily truncating the flattened sequence through a 1D operation. The implementation transforms one spatial axis, transposes the spatial axes for a second transform, and then restores their order. This preserves horizontal and vertical grid structure instead of treating the flattening order as the only spatial adjacency relationship.

Low frequencies represent slowly varying spatial patterns and high frequencies represent faster local changes; each coefficient is a weighted combination of the entire feature map. Retaining one low-frequency coefficient therefore does not mean retaining one patch or selecting a single important position through attention. The authors observe low-frequency magnitude concentration in Qwen2-VL features and illustrate it with averaged spectra from 10,000 sampled GQA images in Figure 1, page 5. That figure log-scales the mean absolute value across hidden dimensions; although the paper calls this an energy analysis, it should not be rewritten as a strict squared-energy statistic. The perturbation experiment compares the lowest 25% and highest 25% frequency components, finding low frequencies more stable under replacement, noise, blur, and masking in Figure 2. FreeMorph semantic morphing further supports low-frequency stability when global structure persists, but these are motivational analyses rather than complete downstream causal ablations.

2. Low-Frequency Rectangular Truncation: control the budget through a target grid rather than selecting original tokens

After transformation, the algorithm keeps the first \(C_h,C_w\) components starting at zero along the spatial frequency axes and discards the rest. The following notation summarizes Algorithm 1 and Equation (8) in Section 4.2, with \(\widehat F\) denoting the full spectrum and \(F_c\) the compressed spectrum:

\[ F_c=\widehat F[:, :, 0:C_h, 0:C_w],\qquad N_{\mathrm{out}}=C_hC_w. \]

This is a rectangular low-pass region, not magnitude-ranked top-k selection, a circular frequency threshold, or a text-conditioned adaptive mask. At a fixed budget, different images follow the same truncation rule, avoiding an additional importance network or clustering procedure. The key benefit is that retained coefficients still integrate information across the image rather than eliminating all contributions from particular original positions. The cost is that high-frequency details cannot be fully preserved: retaining global contributions does not mean losslessly preserving every position. The compression ratio in the tables denotes the removed fraction, \(1-C_hC_w/(N_hN_w)\), and should not be confused with the retained token fraction. For example, retaining 36/576 tokens leaves 6.25% and removes 93.75%, rather than compressing away only 6.25%.

3. Small-Grid Spatial Reconstruction: return low-frequency coefficients to real-valued features accepted by the existing model

Frequency coefficients are not passed directly to the LLM as tokens; the algorithm applies inverse DCT along both axes of the retained \(C_h\times C_w\) spectrum. The reconstruction has only \(C_h\times C_w\) spatial positions and is flattened into a short sequence of shape \(B\times(C_hC_w)\times h_v\). This differs from zeroing high frequencies and reconstructing the original grid, which would retain the original token count and fail to shorten the sequence. Each new position can be interpreted as the retained frequency band reconstructed on a smaller spatial grid, not an unchanged copy of an old token. This interpretation also explains the need for training: interface compatibility does not guarantee that feature scale and statistics match the original weights.

Both 2D DCT and inverse DCT can be accelerated with the fast Fourier transform (FFT); Fourier in the name refers to the implementation and frequency-domain approach, while the actual transform is a real-valued DCT. Section 4.3 explains reducing 1D DCT computation through even/odd index rearrangement followed by FFT. For an input grid of side length \(N\) and an output grid of side length \(C\), Equation (15) gives spatial transform complexity \(\mathcal O(N^2\log N+C^2\log C)\). This expression omits batch and channel factors; Table 1 writes the dominant term as \(\mathcal O(Bh_vN^2\log N)\). Here \(N\) is the grid side length and the token count is \(N^2\), so the expression should not be misread as approximately quadratic in the token count. An inexpensive transform and a shorter LLM sequence jointly produce savings, but the original vision encoder still processes the image, so not all visual computation shrinks proportionally.

A Worked Example

In the paper's LLaVA-v1.5-7B image setting, a 336×336 input corresponds to a 24×24 grid containing 576 visual tokens. With a 36-token budget, the compressor restores those 576 positions into a 24×24 feature map and applies 2D DCT independently to each channel. It then retains only a 6×6 low-frequency rectangle and applies inverse DCT to those coefficients, producing 36 new spatial positions. The new features pass through the projector and enter the LLM alongside the text instruction; this is not selection of the 36 most salient original patches. This global low-frequency approximation may suffice for a question about the overall scene, whereas recognizing small text may suffer from discarded detail. The preceding sentence is a mechanism-based reader interpretation, not a specific question-answer example supplied by the paper; the TextVQA degradation provides related empirical evidence. With a 144-token budget, the same pipeline instead reconstructs a 12×12 grid and exposes more frequencies to the downstream model. The two budgets differ in retained bandwidth, not in the choice of a compression network.

Loss & Training

The paper explicitly states that frequency truncation changes feature distributions and requires additional training; it adds neither learnable compressor parameters nor a dedicated spectral loss. Fourier-LLaVA follows two-stage training: align compressed visual features with language embeddings, then learn multimodal instruction following. Table 2 lists 558k and 665k training examples for the two stages, with 1 and 2 epochs respectively and a batch size of 256 for both. The first-stage learning rate is 1e-3; the second-stage rate is 2e-4, with a separate multimodal learning rate of 2e-5 and LoRA \(r/\alpha\) of 128/256. Fourier-Qwen continues fine-tuning on 600k single-image conversation samples from LLaVA-NeXT for 2 epochs with a batch size of 128. Its learning rate is 1e-6, multimodal learning rate is 1e-5, and vision learning rate is 1e-6; both model families use AdamW, a cosine schedule, and a 0.03 warmup ratio. The cached text misaligns the trainable-module checkmarks in Table 2, so a complete stage-specific freezing list is not reconstructed here. Evaluating video directly after image adaptation is the paper's zero-shot video transfer setting, not evidence that the model underwent no adaptation training.

Key Experimental Results

Main Results

Table 3, page 11, compares the LLaVA-v1.5-7B family; the selection below retains task scores and the original eight-benchmark Avg., with higher scores being better. Avg. also includes VQAv2, GQA, POPE, and LLaVAW, which are not expanded here; it is not recomputed from the four displayed tasks. MQT-LLaVA comparison results are mostly taken from its original paper, with TextVQA newly evaluated by these authors; the methods' training recipes do not constitute a strictly controlled ablation.

Model	Image tokens	SciQA	TextVQA	MMBench	MMMU	Eight-task Avg.
LLaVA-v1.5-7B	576	66.8	58.2	64.3	35.3	64.6
Fourier-LLaVA	256	69.9	56.0	66.4	33.1	64.6
Fourier-LLaVA	144	69.0	54.7	65.6	35.3	64.6
MQT-LLaVA	64	67.0	51.7	63.5	34.4	61.9
Fourier-LLaVA	64	69.3	52.6	64.7	34.4	63.3
MQT-LLaVA	36	66.8	50.4	63.4	34.4	61.1
Fourier-LLaVA	36	69.0	51.0	64.3	32.8	62.1

The 36-token version loses 2.5 average points and retains approximately 96.13% based on displayed values; this does not guarantee 96% retention on every task. TextVQA falls from 58.2 to 51.0, a loss of 7.2 points, showing how the average can hide degradation on detail-sensitive tasks. The prose says the 144-token version surpasses the original average, but Table 3 rounds both to 64.6; this note treats them as tied at the reported precision.

Ablation Study

The paper does not report a separate component-ablation table removing DCT, retaining high frequencies, or changing the truncation shape; this section uses the actual budget-efficiency analysis instead of inventing ablations. Table 5, page 13, reports FLOPs at different budgets, measured with calflops on an RTX 4090 24G; it is not an end-to-end latency table.

Model	Image tokens	FLOPs (T)	Reduction from baseline
LLaVA-v1.5-7B	576	8.54	Baseline
Fourier-LLaVA	256	4.30	49.6%
Fourier-LLaVA	144	2.81	67.1%
Fourier-LLaVA	64	1.75	79.5%
Fourier-LLaVA	36	1.38	83.8%

Latency and KV cache are measured separately on an A100 40G in Figure 5, page 13: the 576-token baseline has a TTFT of 84.4 ms, compared with 58.1 ms for the 36-token version. The reported 31.2% TTFT improvement corresponds to reduced time to first token, not a 31.2% increase in generated tokens per second. KV cache usage falls from 312.5 MB to 42.5 MB, a reduction of 86.4%; these measurements must not be presented as RTX 4090 timings.

Key Findings

• At the same 64-token budget, Fourier-LLaVA achieves an Avg. of 63.3 versus 61.9 for MQT-LLaVA. This establishes competitive results without additional compressor parameters, but does not isolate training as a causal factor.
• In Table 4, page 12, Qwen2.5-VL-3B's average image token count falls from 553 to 236 while Avg. changes from 71.1 to 72.6; Qwen2-VL-2B instead falls from 66.8 to 65.7, so cross-architecture outcomes are not uniformly positive.
• On MVBench in Table 6, page 14, Fourier-LLaVA reduces tokens per video from 2304 to 288 while the average changes from 45.6 to 44.0, demonstrating transfer of image-adapted compressed representations to video.

Highlights & Insights

• The useful conceptual shift is from which positions matter to which scales of spatial variation should remain. Budget control no longer requires attention scores or an initial guess about the region relevant to a question.
• The inverse transform is essential for reconnecting frequency selection to the original model interface. Preserving spatial feature form and channel width allows the operation to be reused across vision-language architectures.
• Complexity analysis appears alongside measured FLOPs, TTFT, and KV cache usage, addressing both the inexpensive compressor and the shorter downstream sequence. Hardware and metric distinctions must remain explicit.

Limitations & Future Work

• The authors acknowledge that distribution changes require training, so deployment needs adaptation data and optimization resources. Parameter-free is neither cost-free nor a plug-and-play performance guarantee for arbitrary pretrained VLMs.
• The low-frequency preference rests primarily on statistical and perturbation analyses, without a full equal-budget comparison of low, high, random frequencies, and spatial pooling. The gains cannot be attributed entirely to the specific spectral selection rule.
• TextVQA degrades substantially under aggressive compression, suggesting that small text and other fine-grained evidence may not suit a uniform budget. Retaining limited task-relevant high frequencies is a reader-proposed extension, not a validated module in this paper.
• The current 2D compressor does not explicitly exploit temporal redundancy, and reported video averages still decrease. Temporal-frequency extensions need separate experiments rather than inference from image results.
• Some equation characters and Table 2 checkmark alignment are damaged in the cached text. This note only summarizes transforms verifiable from Algorithm 1 and readable prose, without guessing FFT coefficient formulas or freezing details.

Related Work & Insights

• Versus ATP-LLaVA, reference [33] in the paper: it prunes visual tokens according to importance; this method truncates in frequency coordinates and generates new spatial positions. The two approaches remove different representational units.
• Versus LLaVA-PruMerge [26] and VisionZip [32]: these methods use selection or similarity-based merging, while this paper applies a fixed linear transform to whole-image structure. The rule is simple but cannot actively protect question-relevant details.
• Versus MQT-LLaVA [9] and QueCC [15]: learnable queries or cross-attention can adapt to data but introduce compression-module parameters. Fourier Compressor leaves adaptation to training the existing model.
• Connection to Fourier Transformer [8] and FreqKV [12]: all exploit frequency-domain redundancy, but their targets are sequence hidden states, KV caches, and this paper's 2D visual features respectively. Their empirical conclusions are not interchangeable.
• Connection to DocPedia [5]: it extracts DCT coefficients directly from RGB images for visual encoding, whereas this paper compresses tokens after visual encoding. Transferring the idea requires identifying the representation level where the frequency operation takes place.

Rating

• Novelty: 4/5. Frequency truncation has precedents, but its integration with 2D visual token structure and parameter-free compression is clear.
• Experimental Thoroughness: 3/5. Evaluation spans architectures, images, video, and efficiency, but lacks controlled ablations isolating frequency selection from adaptation training.
• Writing Quality: 4/5. Algorithm 1 clarifies data flow, although some prose claims exceed rounded table precision and parameter-free must be distinguished from training-free.
• Value: 4/5. Useful for deployments with an existing training pipeline that need cheaper visual context; fine-grained tasks require separate validation.