Expert-aware quantisation: near-Q4 quality at near-Q2 size?
The article presents "expert-aware quantisation," a method for compressing Mixture-of-Experts language models that claims to achieve quality close to 4-bit precision while using storage near 2-bit size by allocating different bitwidths to different experts based on their importance.
Background
Mixture-of-Experts (MoE) models are large neural networks composed of many smaller "expert" sub-networks; only a few experts activate per input, making them efficient. Quantisation compresses model weights (e.g., from 4-bit to 2-bit) to reduce memory and speed up inference, but typically degrades quality uniformly across all experts.
This post proposes a method that applies different levels of quantisation to different experts based on their importance, concentrating bit-budget on critical experts while aggressively compressing less important ones. The claim: near-4-bit quality at near-2-bit memory footprint.