MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training

摘要

Vision-and-language pretraining (VLP) in medicine leverages contrastive learning on image–text pairs, often enhanced with masked modeling. However, existing methods face two challenges: difficulty reconstructing key pathological features due to limited data, and reliance on either paired or image-only datasets without combining both. To address this, we propose **MMCLIP** (**M**asked **M**edical **C**ontrastive **L**anguage–**I**mage **P**re-training), which introduces two modules: **AttMIM**: Masks image features highly correlated with text to improve reconstruction of fine medical details. **EntMLM**: Masks key medical entities in text and reconstructs them using visual cues. Furthermore, **MMCLIP** incorporates unpaired data through disease-kind prompts, achieving state-of-the-art performance in zero-shot and fine-tuning across five benchmarks.