Beyond Class Boundaries: Federated Visual Primitive Sharing with Text-Guided Adaptation

摘要

Personalized Federated Learning (pFL) effectively addresses the challenge of statistical heterogeneity in traditional Federated Learning (FL), with feature alignment methods (e.g., FedProto) standing out due to their communication efficiency and model-agnostic design, making them practically viable in real-world non-IID scenarios. These methods directly align class-level features across clients without requiring model parameter transmission. However, they represent each class as a holistic prototype, which limits the diversity and expressiveness of shared features. This restriction hampers the model's ability to generalize across clients and impedes personalized adaptation, as clients lack sufficient semantic components to reconstruct discriminative features tailored to their local data distributions. To overcome these limitations, we propose Federated Visual Primitive Learning (FedVPL), a novel framework comprising two key components: (1) Visual Primitive Space Sharing, which decomposes class-level features into semantically meaningful and reusable visual primitives, enabling cross-client and cross-class sharing to enrich feature diversity and decouple communication cost from the number of classes, significantly improving efficiency; and (2) Text-Guided Semantic Alignment, a parameter-free personalization mechanism that leverages external language priors to align shared primitives with client-specific semantics, without requiring additional communication overhead. Extensive experiments across diverse non-IID benchmarks demonstrate that FedVPL substantially outperforms state-of-the-art baselines, achieving up to a 5.62% improvement in accuracy, reducing communication overhead by at least 12.5x, and effectively addressing generalization and personalization challenges in heterogeneous federated environments.