PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

摘要

As multimedia content expands, the demand for unified multimodal retrieval (UMR) in real-world applications increases. Recent work leverages multimodal large language models to tackle this task. However, the large number of parameters leads to high training resource demands and low inference efficiency. To address this issue, we propose the PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning, an efficient approach to enhancing the unified retrieval capabilities from both structure and learning perspectives: 1) From the perspective of model structure, we analyze and propose Layer-Pruned Self-Distillation approach. It structurally prunes the model by preserving only the shallow layers, substantially reducing the parameters of MLLM. To mitigate representational degradation, we apply self-distillation, reusing the feature from the original model as the teacher signal to guide the retrieval embedding token from the pruned model, enabling effective capacity retention in a compact model. 2) From the perspective of model learning, to mitigate representation degradation caused by rapid convergence during multimodal contrastive learning, we propose Modality-Adaptive Contrastive Learning Loss (MAC-Loss). It adaptively separates in-batch negative candidates into harder intra-modality and simpler inter-modality groups based on each query's target modality. Assigning each group a temperature coefficient with different strategies allows queries to adaptively focus on harder in-batch negatives, reducing the resource demands of multimodal contrastive learning when UMR training. Experiments demonstrate that our approach achieves double efficiency, significantly reduces resource consumption while maintaining most of the performance.