Self-Supervised Continual Graph Learning via Adaptive Spaced Replay on Node Proxies

摘要

Most self-supervised graph learning studies typically follow an offline training paradigm, assuming that all data are readily available. This assumption, however, is not always tenable in real-world scenarios as many graph data are generated continuously. Although several continual graph learning models have emerged and achieved empirical success, they almost all rely on external supervision, making it difficult to adapt to applications with a large amount of unlabeled data from the wild. To be honest, research on self-supervised continual graph learning is still surprisingly in its infancy. Therefore, we select several well-known self-supervised graph embedding models as representatives and explore whether they are resistant to catastrophic forgetting in a continual learning setting. Empirical studies find that self-supervised representation models may be potentially better continual learners than supervised counterparts. Driven by this advantage, we propose a self-supervised continual graph representation learning framework based on adaptive spaced replay on node proxies, named Trace. Inspired by the Complementary Learning System theory, Trace employs a dual-system architecture to simulate the functionality and cooperation of the hippocampus and neocortex in the brain. Among them, the fast-learning system efficiently encodes the current input graph to acquire new knowledge and adaptively extracts node proxies from it as important knowledge cached into the memory through progressive clustering. Drawing inspiration from the Ebbinghaus forgetting curve, the slow-learning system implements adaptive spaced replay based on the memory retention rate of each preceding task instead of the widely used consecutive replay scheme for promising flexibility and efficiency. Experiments under task-incremental and class-incremental learning settings on multiple datasets corroborate the superiority of our proposed Trace, even sometimes surpassing supervised graph continual learning solutions.