L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models
National University of Singapore · The University of Sydney · Harbin Institute of Technology · Huawei Technologies Ltd. · University of the Chinese Academy of Sciences · Central South University · Harbin Institute of Technology (Shenzhen) · National Univ. of Singapore
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Large language models (LLMs) have achieved notable progress. Despite their success, next-token prediction (NTP), the dominant method for LLM training and inference, is constrained in both contextual coverage and inference efficiency due to its inherently sequential process. To overcome these challenges, we propose leap multi-token prediction~(L-MTP), an innovative token prediction method that extends the capabilities of multi-token prediction (MTP) by introducing a leap-based mechanism. Unlike conventional MTP, which generates multiple tokens at adjacent positions, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass. This structured leap not only enhances the model's ability to capture long-range dependencies but also enables a decoding strategy specially optimized for non-sequential leap token generation, effectively accelerating inference. We theoretically demonstrate the benefit of L-MTP in improving inference efficiency. Experiments across diverse benchmarks validate its merit in boosting both LLM performance and inference speed. The source code is available at https://github.com/Xiaohao-Liu/L-MTP.