Why is a Bird's Caption a Good Demonstration? Towards Effective Multimodal In-Context Learning without Dedicated Data

摘要

Multimodal Large Language Models (MLLMs) have achieved impressive performance across a range of tasks by leveraging Multimodal In-Context Learning (MICL), which uses a few task-specific examples as demonstrations. However, existing approaches assume the availability of pre-prepared curated datasets that serve as support sets, limiting the adaptability of MICL to novel and unseen tasks where dedicated data is unavailable. To fill this research gap, we first explore the effectiveness of MICL using non-customized data. Through systematic evaluations across 17 datasets and five state-of-the-art MLLMs, we demonstrate significant performance gains with MICL compared to zero-shot evaluation. To more thoroughly understand underlying reasons behind this phenomenon, we posit and validate two hypotheses: 1) multimodal demonstrations facilitate cross-modal interactions and 2) demonstrations provide transferable knowledge. Building on these insights, we explore factors that affect MICL and arrive at several key takeaways. First, to address the limitations of existing retrieval methods in MICL without dedicated data, we propose a Fast Maximum Mean Discrepancy based (FMMD) retrieval metric and a Semantics-Modality Relation-Aware (SMRA) retrieval metric to perform inter- and intra-dataset retrieval, respectively. Additionally, we find that increasing demonstrations, combining demonstrations from diverse datasets, and providing instructions for query samples can further boost MICL. We hope this study can inspire future works on improving MICL in real-world scenarios.