Unsupervised Image-to-Video Adaptation via Category-aware Flow Memory Bank and Realistic Video Generation

摘要

Image-to-Video adaptation is proposed to train a model using labeled images and unlabeled videos to facilitate the classification of unlabeled videos. The latest work synthesizes videos using still images to mitigate the modality gap between images and videos. However, the synthesized videos are not realistic due to the camera movements are only simulated in 2D space. Therefore, we generate realistic videos by simulating arbitrary camera movements in 3D scenes, and then the model can be trained using the generated source videos. Unfortunately, the optical flows from the generated videos have unexpected negative impacts, resulting in suboptimal performance. To address this issue, we propose the Category-aware Flow Memory Bank, which replaces optical flows in source videos with real target flows, and the new composed videos are beneficial for training. In addition, we leverage the video pace prediction task to enhance the model's perception of speed. Our method achieves state-of-the-art performance and comparable performance on three widely used benchmarks. Our code is available at https://github.com/KenanHuang/mm2024\_cfb4i2v.