LittleBit: Ultra Low-Bit Quantization via Latent Factorization

Conference: NeurIPS 2025 arXiv: 2506.13771 Code: Available Area: Model Compression Keywords: Ultra low-bit quantization, low-rank factorization, binarization, sub-1-bit, LLM compression

TL;DR

This paper proposes LittleBit, a framework that achieves extreme LLM compression down to 0.1 BPW (bits per weight) via low-rank latent-space matrix factorization, binarization, and a multi-scale compensation mechanism. It compresses Llama2-13B to under 0.9 GB and substantially outperforms STBLLM in the sub-1-bit regime.

Background & Motivation

LLM deployment is constrained by enormous memory and computational requirements. Quantization is the primary compression strategy: - PTQ methods (GPTQ, AWQ) perform well at ~4-bit precision but degrade sharply below 2 bits. - QAT methods (OneBit, BinaryMoS) can sustain 1-bit-level compression. - However, even 1-bit models (e.g., ~15.4 GB for 70B parameters) may remain too large for extremely resource-constrained devices.

Core Problem: How can model performance be preserved under extreme sub-1-bit compression (e.g., 0.1 BPW)?

Two key observations motivate the proposed approach: 1. LLM weight matrices typically exhibit significant low-rank structure; SVD-based decomposition is more stable than pruning at high compression ratios. 2. Binarization causes severe information loss, necessitating multi-dimensional scaling factors (row, column, and latent-space dimensions) for compensation.

Method

Overall Architecture

LittleBit redesigns the linear layers of Transformers into a Primary + Residual dual-path structure: 1. The weight matrix \(\mathbf{W}\) is decomposed as \(\mathbf{W} \approx \mathbf{UV}^\top\). 2. The decomposed factors are binarized: \(\mathbf{U}_{sign} = \text{sign}(\mathbf{U})\). 3. Multi-scale compensation parameters are introduced: row scaling \(\mathbf{h}\), column scaling \(\mathbf{g}\), and latent-dimension scaling \(\boldsymbol{\ell}\). 4. A parallel Residual path compensates for the approximation error of the Primary path.

Key Designs

1. Multi-Scale Compensation Mechanism:

The effective weight of the primary path is: $\(\hat{\mathbf{W}}_{pri} = \text{diag}(\mathbf{h}) \mathbf{U}_{sign} \text{diag}(\boldsymbol{\ell}) \mathbf{V}_{sign}^\top \text{diag}(\mathbf{g})\)$

Beyond conventional row/column scaling, a latent-dimension scaling vector \(\boldsymbol{\ell} \in \mathbb{R}^r\) is introduced to learn the relative importance of each latent dimension. Forward computation is implemented via sequential element-wise multiplication and binary matrix multiplication, avoiding the need to materialize the full effective weight matrix.

1. Dual-SVID Initialization:

To avoid QAT instability from naive initialization, an SVD-based initialization strategy is designed: - Truncated SVD is applied to \(\mathbf{W}\) to obtain \(\mathbf{U}', \mathbf{V}'\). - Binary factors are initialized by taking the sign: \(\mathbf{U}_{sign,0} = \text{sign}(\mathbf{U}')\). - Rank-1 SVD is applied separately to the magnitude matrices \(|\mathbf{U}'|\) and \(|\mathbf{V}'|\) to extract initial values for the row, column, and latent-dimension scaling factors. - This "dual SVD" procedure (Sign and Value Independent Decomposition, SVID) ensures the initial effective weight closely approximates the original weight.

1. Residual Compensation:

Rather than increasing total parameter budget, the fixed bit budget is strategically allocated across two low-rank paths: $\(\hat{\mathbf{W}} = \hat{\mathbf{W}}_{pri} + \hat{\mathbf{W}}_{res}\)$

The residual path is initialized via Dual-SVID applied to the approximation error \(\mathbf{W} - \hat{\mathbf{W}}_{pri,0}\) of the primary path. Both paths are jointly optimized during QAT. Visualizations by the authors show that even at 0.3 BPW, the dual-path initialization quality already surpasses that of a single path at 1.0 BPW.

Loss & Training

QAT with knowledge distillation is adopted: $\(\mathcal{L}_{QAT} = \mathcal{L}_{out} + \lambda \mathcal{L}_{inter}\)$

• \(\mathcal{L}_{out}\): KL divergence at the output layer.
• \(\mathcal{L}_{inter}\): MSE at intermediate layers (\(\lambda = 10\)).
• SmoothSign (forward: sign; backward: \(\tanh(100x)\)) replaces STE for greater stability under ultra-low-bit training.
• The latent rank of K/V projection layers in GQA models is adjusted separately.

Key Experimental Results

Main Results

WikiText-2 perplexity (PPL, lower is better):

Method	BPW	Llama2-7B	Llama2-13B	Llama3-8B	QwQ-32B
FullPrecision	16	5.47	4.88	6.10	6.34
OneBit (QAT)	1.0	8.36	7.41	13.09	9.86
BinaryMoS (QAT)	1.0	7.74	6.95	10.83	8.99
STBLLM (PTQ)	0.55	30.67	27.05	241.95	18.32
LittleBit	0.55	10.47	9.24	18.47	13.57
STBLLM (PTQ)	0.30	1800	893.82	170000	512.01
LittleBit	0.30	12.00	10.48	20.34	16.48
LittleBit	0.10	15.92	15.09	26.11	35.26

LittleBit at 0.55 BPW already surpasses STBLLM at 0.7 BPW. At 0.3 BPW, STBLLM nearly collapses (PPL > 500), while LittleBit remains functional.

Ablation Study

Zero-shot reasoning performance (average accuracy across 7 benchmarks):

Method	BPW	Llama2-7B	Llama2-13B	Llama3-8B
FullPrecision	16	62.97%	64.78%	70.15%
STBLLM	0.55	44.29%	45.04%	39.62%
LittleBit	0.55	47.26%	48.03%	-
STBLLM	0.30	35.78%	38.22%	36.62%
LittleBit	0.30	45.20%	45.94%	-

Key Findings

• LittleBit achieves approximately 31× memory compression at 0.1 BPW, reducing Llama2-13B to under 0.9 GB.
• In the sub-1-bit regime, STBLLM degrades sharply (essentially unusable below 0.3 BPW), while LittleBit remains stable.
• Residual compensation yields substantial gains at low BPW: the dual-path initialization at 0.3 BPW surpasses the single-path initialization at 1.0 BPW.
• The SmoothSign gradient estimator outperforms STE, offering greater stability in ultra-low-bit training.
• Separately adjusting the latent rank of K/V projection layers in GQA models effectively preserves performance.

Highlights & Insights

1. Extreme compression: A compression ratio of 0.1 BPW is unprecedented and theoretically enables 11.6× inference speedup over FP16.
2. Synergy of factorization and binarization: Low-rank decomposition provides a stable compression foundation; binarization further reduces bit cost; multi-scale compensation recovers lost information.
3. Dual-SVID initialization: Magnitude information is elegantly decomposed into three-dimensional scaling factors, providing a strong initialization point for ultra-low-precision QAT.
4. Residual compensation without budget increase: Under the same bit budget, two low-rank paths outperform one high-rank path — a design principle with broad applicability.
5. Wide model coverage: Experiments span 1.3B to 32B parameters across multiple model families including Llama, OPT, Phi-4, and QwQ.

Limitations & Future Work

• The current framework addresses weight quantization only; activation quantization is not considered.
• At 0.1 BPW, while PPL remains acceptable, zero-shot reasoning performance still suffers considerable degradation.
• QAT training incurs high computational cost, requiring a full-precision teacher model and multi-epoch training.
• Practical inference acceleration requires custom binary matrix multiplication kernels, which currently lack broad hardware support.
• Whether different strategies should be applied to attention layers versus FFN layers is not thoroughly investigated.

Related Work & Insights

• Early work on binary networks (BinaryConnect, BNN) established the foundation for weight binarization, but direct application to LLMs results in severe performance degradation.
• STBLLM (sub-1-bit PTQ + N:M sparsity) serves as the primary baseline, and its limitations under extreme low-bit compression are thoroughly exposed.
• OneBit and BinaryMoS (1-bit QAT methods) validate the importance of multi-dimensional scaling; LittleBit extends this by introducing latent-dimension scaling.
• The success of low-rank adaptation methods such as LoRA corroborates the low-rank properties of LLM weight matrices, providing theoretical support for decomposition-based compression.

Rating

• Novelty: ⭐⭐⭐⭐⭐ — First to push quantization to 0.1 BPW; the unified framework of low-rank factorization + binarization + multi-scale compensation is elegantly designed.
• Experimental Thoroughness: ⭐⭐⭐⭐ — Covers diverse model families and scales with dual evaluation via PPL and zero-shot reasoning, but lacks actual inference latency benchmarks.
• Writing Quality: ⭐⭐⭐⭐ — Visualizations (Figure 3 weight reconstruction comparison) are highly intuitive; method description is clear.
• Value: ⭐⭐⭐⭐⭐ — The demand for extreme compression is genuine, with enormous potential for on-device deployment scenarios.

Related Papers

• [NeurIPS 2025] ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization
• [AAAI 2026] SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
• [NeurIPS 2025] Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
• [NeurIPS 2025] FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
• [NeurIPS 2025] Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization