MeDKCoOp: Dual Knowledge-guided Graph Prompt Learning for Biomedical Vision-Language Models

摘要

The rapid evolution of vision-language models (VLMs), such as CLIP, has demonstrated remarkable zero-shot capabilities in downstream tasks. Prompt learning paradigms like Context Optimization (CoOp) refine learnable prompts for efficient adaptation. However, their application in the biomedical domain remains limited due to the insufficient utilization of specialized biomedical knowledge and cross-modality structural relationships. To address these, we introduce MeDKCoOp, a Medical Dual Knowledge-guided graph adaptation method that leverages systematic integration of knowledge through three aspects: exploit domain-specific knowledge from both textual and visual branches, formalize it into graph-structured representations, and leverage knowledge-guided relation transfer for learning cross-modality fusion. By dynamically optimizing learnable prompts through relation learning process, our method achieves disentangled visual representation and enhances transferability to downstream tasks. Evaluations across 8 biomedical datasets spanning 7 imaging modalities demonstrate state-of-the-art cross-domain generalization, with an average 15.12% accuracy improvement over baselines. Our work establishes a new paradigm through graph prompt learning in medical vision-language models, advancing robust diagnostic AI in data-scarce clinical scenarios. Our code is available at: https://github.com/WangYijun-OUC/MeDKCoOp.