CWCP: Generalizing Virtual Reality to Real World with Contextual-Weather Correlation Pairing for Deraining and Desnowing

摘要

Rain and snow in real scenes have diverse appearances that are tightly related to shooting angle, light sources, and background elements. Most existing learning-based methods utilized synthetic weather/clean pairs to handle a single weather type. However, synthetic scenarios more focus on the property of image or weather layers independently without considering their mutually exclusive relationship, which lead to domain gap between real and synthetic scenes. This makes it difficult to generalize to complex real-world weather conditions for existing methods, even though they perform well in synthetic scenes. In this paper, we highlight the importance of contextual influence information and utilize virtual reality to effectively simulate real scenes. Specifically, we propose a novel weather removal framework, called Contextual-Weather Correlation Pairing (CWCP), which contains modules Weather Style Adaptation (WSA) and Continual Contextual Learning (CCL). The WSA learns the style correlation of noises by pairing local blocks with weather similarity information, which is highly effective for handling a large amount of similar noises. CCL learns knowledge from the entire image while contrasting it with the features of local blocks based on contextual similarity pairing, which reduces the color difference among adjacent local blocks. Moreover, we propose a novel complex weather dataset, namely SkyLine-Weather, which is a virtual rain and snow removal dataset contains ~10K noisy images and their corresponding ground truth images, by using 3D computer graphic platform. Experiments on 8 datasets demonstrate that our framework achieves SOTA results on deraining and desnowing tasks, across real, synthetic, and virtual scenes.