LeanK: Learnable K Cache Channel Pruning for Efficient Decoding
Microsoft · Research, Microsoft · Tsinghua University, Tsinghua University · Microsoft and University of Texas at Austin
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。DOI 10.18653/v1/2025.emnlp-main.1584 ↗
摘要
Large language models (LLMs) enable long-context tasks but face efficiency challenges due to the growing key-value (KV) cache. We propose LeanK, a learning-based method that prunes unimportant key (K) cache channels by leveraging static channel sparsity. LeanK reduces GPU memory and accelerates decoding without sacrificing accuracy. Experiments demonstrate up to 70% K cache and 16%–18% V cache memory reduction, and 1.45× decoding speedup. We also provide insights into model channels and attention heads during long-context inference by analyzing the learned importance distribution. Our code is anonymously available at https://anonymous.4open.science/r/LeanK-7A87/README.md.