Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?

摘要

In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters. In this work, we uncover several key insights regarding the \textit{singular} components of network parameters based on Singular Value Decomposition (SVD).Firstly, the \textit{principal} singular components with large singular values in pre-trained network parameters can be effectively reused during fine-tuning, whereas the \textit{minor} components with smaller singular values are more task-specific and require substantial adaptation. Secondly, we first establish the theoretical connection that the uncontrolled growth of singular values in LoRA adapters leads to the forgetting of pre-trained knowledge — a well-known issue referred to as \textit{catastrophic forgetting}.Building on these observations, we propose \textbf{SCLoRA}, which injects parameterized singular components with spectral clipping into the pre-trained model in a way that is aware of the spectral distribution of the pre-trained model. \textbf{SCLoRA} effectively adapts to new tasks by focusing updates on components that require adaptation, while simultaneously alleviating catastrophic forgetting. We conduct extensive experiments and demonstrate that \textbf{SCLoRA} not only improves downstream performance but also effectively retains pre-trained knowledge.