Skynet-V1: Towards Early Warning of Video Abnormal Events via A Spatial-temporal Causal-enhanced MoE Framework

摘要

In the literature, prior studies on Video Anomaly Detection (VAD) primarily focus on anomalies that have already occurred (i.e., consequential anomaly), but cannot identify the causative anomalies (i.e., the cause of final anomaly), while this type of causative anomaly could be powerfully beneficial to early warning against the anomalies. Meanwhile existing work mainly focuses on classifying whether each video clip is abnormal, and couldn't extract structured video information, such as what is the abnormal type, which people or things are involved, whereas such structured information can potentially contribute to building an efficient system to monitor the above causative and consequential anomalies. To this end, this paper proposes a new chat-paradigm Video Abnormal Events' Early Warning (VAE-EW) task, aiming to localize and extract not only the consequential abnormal event quadruples but also the causative abnormal event quadruples (i.e., subject, predicate, object, and event type). Further, this paper believes that this new task faces two key challenges, i.e., Spatial-temporal modeling challenge and temporal highlighting challenge. On this basis, this paper proposes a new Skynet-V1 Model with a spatial-temporal causal-enhanced Mixture-of-Expert (MoE) Framework, i.e., acting like Skynet in movie 'The Terminator' to track and early warn against abnormal events, for VAE-EW task. Specifically, this model designs a Spatial-temporal Aware MoE Block (SAMB) and a Causal-guided Temporal Enhancing Block (CTEB) to address the two challenges respectively. Extensive experiments on our VAE-EW dataset show the superiority of our model in localizing and extracting abnormal events, especially the causative events, compared to other advanced baseline models, highlighting the importance of the new VAE-EW task and the effectiveness of Skynet-V1 in addressing such task.