KV cache compression techniques, including Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-head Latent Attention (MLA), and linear-attention hybrids, have evolved to reduce memory overhead in large language models. These developments have quietly enabled the long context windows required for modern agentic LLM applications by making key-value caching more efficient.
Luce KVFlash is a memory-efficient optimization enabling 256K context windows using only 72 MiB of KV cache on the GPU. It reduces memory consumption for long-sequence inference by compressing key-value cache storage.
SubQ 1.1 introduces a linear-scaling sparse attention mechanism that maintains 98% retrieval accuracy at 12 million tokens, significantly extending context length efficiency for large language models while reducing computational overhead compared to full attention methods.
A new Transformer implementation called GateGPT achieves 56,000 tokens per second using KV cache on an FPGA running at 80 MHz.
Subquadratic released SubQ 1.1 Small, a 1.5B open-weight language model using a soft-moe-2x8 architecture. It outperforms larger models like Gemma 2 2.6B and Phi-2 2.8B on several benchmarks. The model uses subquadratic soft-MoE layers (MMA and MMAM) for improved efficiency.