Ear with Eye: Lightweight Multimodal Audio-Visual Network Inspired by Bionic Structures

摘要

In cutting-edge domains such as unmanned aerial vehicles and autonomous driving, edge-based audio-visual systems struggle to strike an optimal balance between complexity and performance. Unlike prevailing approaches that typically rely on pruning and knowledge distillation to streamline unimodal or hybrid models, we propose the Bio-Inspired Multimodal Network (BIMNet), which achieves an efficient audio-visual shared architecture. BIMNet integrates bio-inspired audio-visual modules that emulate the hierarchical sensory integration observed in nocturnal birds, to replicate equivalent biological information flow for both multiscale night vision and noise-adaptive hearing. Experimental findings show that BIMNet achieves superior performance and efficiency in diverse image datasets (varying in spatial scales and lighting conditions), audio datasets (encompassing various types of human and environmental sound), and audio-visual joint event detection tasks. Project support is available at: https://github.com/Mental-Scholar/BIMNet.