PREMISE: Individual Preference-aware Multi-modal Cooperation for Survival Prediction

摘要

Multi-modal learning that combines whole-slide images (WSIs) and genomic data has recently emerged as a promising paradigm for improving cancer survival prediction. However, existing methods either utilize genomic data as guidance to integrate WSI features or treat both modalities as equally important across all patients, overlooking individual variations in modality importance. As critical survival-related features can reside in different modalities for different patients, prioritizing the modality with more discriminative information for each patient, referred to as individual modality preference, is crucial for enhancing prediction accuracy. In this paper, we propose a novel Individual PREference-aware Multi-modal CooperatIon framework for Survival PrEdiction (PREMISE), which collaborates with a uni-modal and a cross-modal preference learner to fully exploit individual modality preference. Specifically, the uni-modal preference learner adopts a task-aware preference estimator to dynamically assess the importance of each modality for each patient, thereby identifying the preferred modality for input individual. To promote cross-modal learning, the cross-modal preference learner embeds the obtained preferences as biases to construct a preference-aware mutual-attention module, enabling the individually adaptive focus and interactions between modalities. Meanwhile, inspired by clinical practice where doctors reference prior cases for survival evaluation, we introduce dual-level cross-modal alignment, incorporating both patient-level and group-level preferences. This alignment emphasizes the more discriminative modality and improves risk group separation during cross-modal knowledge transfer. Experiments have validated our superiority.