Advanced SpikingYOLOX: Extending Spiking Neural Network on Object Detection with Spike-based Partial Self-Attention and 2D-Spiking Transformer

摘要

Brain-inspired Spiking Neural Networks (SNNs) have garnered significant attention due to their bio-plausibility and low power consumption advantages compared to Artificial Neural Networks (ANNs). However, the application of SNN in computer vision remains limited, primarily due to their inferior performance. In this work, we aim to bridge the performance gap between ANNs and SNNs in object detection by our Advanced SpikingYOLOX. The proposed approach extends the SpikingYOLOX with two key innovations: PSA-SNN and 2D-Spiking Transformer, both designed to enhance object detection performance. PSA-SNN extends spike-based self-attention by incorporating high-speed partial self-attention with an SNN-based 2D-Spiking Transformer in the deepest layer of the backbone, significantly improving feature extraction. The 2D-Spiking Transformer redefines the role of spiking neurons in Transformer sequences (Key, Query, Value), demonstrating that applying an additional spiking layer solely to the Value sequence yields the best performance while maintaining computational efficiency in spike-driven Transformers. We conduct extensive experiments on static images and the Advanced SpikingYOLOX achieves state-of-the-art performance among other SNN-based object detection methods. This work paves the way for more advanced SNN applications in object detection and broader computer vision tasks.