Expert-aware quantisation: near-Q4 quality at near-Q2 size?

- Mixture-of-Experts (MoE) is a neural-network design where a model contains many specialised sub-networks ("experts") but only activates a few for any given input. This keeps inference fast and memory use moderate despite a huge total parameter count. - Quantisation compresses a model by reducing the numerical precision of its weights (e.g., from 16-bit to 4-bit integers). Lower precision = smaller file size and less RAM, but usually lower output quality. "Q4" and "Q2" refer to 4-bit and 2-bit quantisation levels. - This post describes a technique, likely developed in the open-source LLM community (around tools like llama.cpp or Hugging Face), that profiles which experts in an MoE model are critical for a particular task (e.g., coding, medicine) and applies heavy quantisation only to the less important ones. - The claimed result — near-4-bit quality at near-2-bit size — matters because running large models locally (on a personal computer rather than a server) is tightly constrained by VRAM/DRAM. If true, the technique could let users run higher-quality models on consumer hardware.