Diverse and Public Features Cooperation via Gradient Rectification for Federated Prompt Learning

摘要

Federated Prompt Learning (FPL) efficiently alleviates data heterogeneity and reduces communication costs by introducing pre-trained models and prompt tuning. However, local prompts tend to favor diverse features and ignore public features captured under extreme data heterogeneity, which compromises the generalization ability of the global prompt by only aggregating local prompts. To address this challenge, we present Federated Prompt Learning with Gradient Rectification (FedGR), which modifies the gradient directions of local and global prompts to enhance the generalization of the global prompt. Specifically, we first introduce a zero-shot prompt as public knowledge and constrain the gradient of local prompts to consistently deviate from the public feature space to capture diverse features adequately. Then, we compute the angular bisector of local and zero-shot prompt gradients and replace the gradient of the global prompt with the gradient of the angular bisector to capture both diverse features and public features. Finally, the server-side global prompt can enhance generalization by aggregating all client-side global prompts. Extensive experiments with various types of heterogeneities have demonstrated that our FedGR outperforms the state-of-the-art methods.