OV-DAVEL: Towards Open-Vocabulary Dense Audio-Visual Event Localization in Untrimmed Videos

摘要

The Dense Audio-Visual Event Localization (DAVEL) task aims to temporally localize all audio-visual events within untrimmed videos. Existing methods typically operate under a closed-set assumption, which limits their ability to generalize to test videos containing previously unseen event categories-an essential capability for open-world scenarios. To this end, we propose a novel task setting, Open-Vocabulary Dense Audio-Visual Event Localization (OV-DAVEL), along with a one-stage method Open-DAVTR designed to detect events that were not observed during training. Open-DAVTR consists of two core components: a class-agnostic foreground-aware generator and a multi-modal semantic-aware classifier. Specifically, the generator is a Detection Transformer-based module that produces event proposals while adaptively attending to discriminative foreground snippets for downstream classification. The classifier leverages rich temporal representations and context-aware textual semantics to effectively recognize events, regardless of whether they were seen during training. In addition, we establish comprehensive OV-DAVEL benchmarks across various settings. Experimental results show that our model significantly outperforms existing baselines in detecting both seen and unseen events, highlighting its effectiveness in open-vocabulary event localization. Code and data are available at: https://github.com/yujialele/OV-DEAVEL.