WFF: Wavelet-based Information Fusion for Multimodal Knowledge Graph Link Prediction

摘要

Multimodal link prediction on multimodal knowledge graphs is an inference task aimed at finding missing triples, which seeks to improve prediction accuracy by leveraging a wide range of information. However, current multimodal knowledge graph link prediction methods are primarily designed in the spatial domain, necessitating ever-growing complexity in fusion strategies. In addition, most of them focus on only three modalities (text, image, and structure). Forward-looking approaches, however, should accommodate a broader array of modalities. Motivated by the operational simplification enabled by transforming features into the frequency (or time-frequency) domain, we propose a wavelet-transform-based multimodal link prediction method, WFF, which offers high modal extensibility and low fusion complexity. Specifically, for unimodal information, we designed a Unimodal Time-Frequency Knowledge Enhancement module, UTFKE, which extracts time-frequency features via discrete wavelet transform and enhances information quality through adaptive filtering. To address the challenge of multimodal fusion, we devised a Multimodal Time-Frequency Knowledge Fusion module MTFKF that supports high modal extensibility and enables effective, efficient integration. Extensive experiments on multiple well-known datasets demonstrate that WFF outperforms strong baselines and achieves state-of-the-art performance. In addition, WFF extends modality to audio and video, further validating the model's effectiveness. Our code is available at https://github.com/xxd12315/WFF.