Dual-Learning based Penalized Multi-Align Clustering for Multi-View Incomplete and Disorderly Data

摘要

Multimodal feature fusion, by integrating the complementary information from each modality, can effectively capture complex features in real-world data. However, in many use cases, such as boiler combustion monitoring, factors including equipment failure, inconsistent sensor sampling frequencies, and network delays often cause data collected from different modalities to suffer from missing modality and temporal asynchrony. This leads to the incompleteness and disorderliness of multimodal data. To address these issues, previous studies have proposed several data fusion methods that align the cluster centers before fusion. However, these approaches have two key limitations: 1) they do not guarantee a high alignment accuracy of data pairs at the sample level, and 2) they do not address the issue of significant discrepancies in data sizes across different classes, which impacts the subsequent data fusion performance. To these ends, we propose a dual-learning based penalized multi-align clustering model (DLPMAC). The Dual-Learning mechanism in the model ensures that it can learn the prior knowledge inherent in each modality's data, specifically the semantic and structural information. This facilitates the maintenance of both semantic consistency and structural similarity-on both local and global levels-across different modalities. Additionally, the Penalized Multi-Align module achieves multi-to-multi data alignment based on a penalty mechanism, which enables a single sample to form data pairs with different samples from other modalities, thereby enhancing the alignment accuracy of data pairs. The introduction of the penalty mechanism prevents data aggregation phenomena, thereby avoiding situations where excessive samples are linked to a single sample. Experimental results validate the efficacy of this model in addressing data alignment(from the sampling and the clustering perspectives) and fusion challenges. The code and related supporting materials are available at: https://github.com/Autism-mm/DLPMAC.