xLSTM Distillation: Achieving Teacher-Student Parity Through Efficient Hybrid Architectures
NXAI · NXAI, ELLIS Unit JKU · ELLIS Unit / University Linz · Johannes Kepler University Linz · LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · Johannes Kepler University Linz Austria · ELLIS Unit Linz, LIT AI Lab, Institute for Machine Learning, Johannes Kepler University, Institute for Advanced Research in Artificial Intelligence (IARAI)
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
There have been numerous attempts to distill quadratic attention-based LLMs into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.