DualFPT: Handling Data Heterogeneity in Federated Prompt Tuning from both Generalized and Personalized Perspective

摘要

Federated Prompt Tuning (FPT) integrates large pre-trained Vision Transformers (ViT) into Federated Learning (FL) by leveraging Visual Prompt Tuning (VPT), achieving state-of-the-art performance with enhanced efficiency across various visual downstream tasks. However, data heterogeneity, such as feature shift and class imbalance, limits prompts' transferability and robustness in FPT. Existing methods primarily focus on generalized FPT (GFPT) or personalized FPT (PFPT), while only a few methods make initial attempts to integrate both approaches. In this paper, we propose a new FPT framework, dubbed DualFPT, which handles data heterogeneity from both generalized and personalized perspectives. Specifically, DualFPT divides the learnable prompts into global and local prompts to jointly capture general and client-specific information, achieving the harmonization of GFPT and PFPT. The generalization of DualFPT is realized by the Feature Sharing (FS) mechanism, which effectively narrows the distribution gap by allowing clients to securely share a portion of sensitive features. The key feature for improving personalization is the Prompt Composition Scheme (PCS), which weights local prompts with distribution similarity to generate composite prompts, thus achieving automatic distribution adaptation. Extensive experiments under feature shift and class imbalance scenarios demonstrate the superior performance of DualFPT. On DomainNet and CIFAR-100 (D (0.1)), DualFPT surpasses SGPT by 4.35% and 5.89% for generalization along with 7.89% and 9.09% for personalization. Ablation studies further validate the effectiveness, efficiency, and security of DualFPT.