Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity
Department of Computer Science, Princeton University · Stevens Institute of Technology · University of Pennsylvania · University of Minnesota - Twin Cities · Carnegie Mellon University · Cornell University · CMU, Carnegie Mellon University · Rice University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, applying ZO fine-tuning in memory-constrained settings such as mobile phones and laptops remains challenging since these settings often involve weight quantization, while ZO requires full-precision perturbation and update. In this study, we address this limitation by combining static sparse ZO fine-tuning with quantization. Our approach transfers a small, static subset (0.1%) of "sensitive" parameters from pre-training to downstream tasks, focusing fine-tuning on this sparse set of parameters. The remaining untuned parameters are quantized, reducing memory demands. Our proposed workflow enables efficient ZO fine-tuning of an Llama2-7B model on a GPU device with less than 8GB of memory while outperforming full model ZO fine-tuning performance and in-context learning.