PrivSplit: A Lossless Method for Prompt Privacy in Distributed Parameter-Efficient Fine-Tuning

摘要

Distributed Parameter-Efficient Fine-Tuning (PEFT) has emerged as a promising framework for personalizing Large Language Models (LLMs) by leveraging a collaboration between cloud servers and edge devices. However, this paradigm harbors a critical privacy vulnerability: the cloud can perform inversion attacks on the intermediate results (e.g., embeddings) sent from the edge to reconstruct the user's raw input prompt. Existing privacy-preserving techniques present a difficult trade-off: they are either too computationally expensive or they degrade model performance. To address this challenge, we introduce PrivSplit, a novel and lightweight protocol for prompt privacy in the Distributed PEFT framework. The core idea of PrivSplit is a ''Split and Compensate'' strategy: the edge device splits the input embedding into a public part sent to the cloud and a private part retained locally. The influence of this private part is then perfectly restored using a precise, lossless compensation mechanism during the interaction. We theoretically prove that PrivSplit is secure against both reconstruction and distinguishability attacks. Crucially, our method is perfectly lossless, ensuring that the model's performance is mathematically identical to the non-private baseline. This allows the security parameters to be arbitrarily strengthened without any impact on utility, breaking the conventional privacy-performance trade-off.