Enhancing Endoscopic Image Retrieval via Self-Supervised Learning and Large VLM-Based Re-ranking

摘要

Medical image retrieval is essential for clinical diagnosis and medical education, yet remains highly challenging in endoscopic imaging due to limited annotated data, the lack of domain-specific pretrained models, and subtle visual similarities across anatomical regions. In this work, we utilize self-supervised contrastive learning to pretrain a strong image encoder tailored for endoscopic data, which serves as the backbone for downstream retrieval tasks. For text-to-image retrieval, we adopt a multi-modal contrastive learning approach that aligns textual and visual representations based on this pretrained backbone. To further enhance retrieval performance, we propose a novel re-ranking module that leverages the reasoning capabilities of large vision-language models (LVLMs), such as GPT-4o and Gemini. We also provide a comparative analysis of various retrieval strategies, offering insights into their effectiveness in clinical scenarios. Our method achieves top-2 in text-to-image and top-5 in image-to-image retrieval at the ENTRep Challenge 2025, demonstrating its potential value for endoscopic image retrieval. Source code is available at https://github.com/ELO-Lab/ENTRep-LDSF.