Semantics-Enriched Cross-Modal Alignment for Complex-Query Video Moment Retrieval

摘要

Video moment retrieval (VMR) aims to search for a video segment that matches the search intent in a query sentence, which has received increasing attention in recent years, due to its practical values in various fields. Existing efforts devoted to this interesting yet challenging task typically encode the query sentence and video segments into unstructured global representations for cross-modal interaction and fusion, which may fail to accurately capture the search intent in complex queries with multi-granularity semantics. To fill the research gap, this paper presents a novel solution termed semantics-enriched video moment retrieval method (SVMR), which can effectively and explicitly model the hierarchical multi-granularity semantics of complex textual query. Specifically, we first explore cross-token relations to offer multiple granularity query representations with hierarchical semantic contexts of semantically associated tokens for fine-grained cross-modal interaction and fusion, which contributes to mining rich visual motion cues semantically related to different activities and entities in complex queries. Furthermore, to fully leverage fine-grained cross-modal cues for moment retrieval, we design a specific temporal boundary reasoning module by explicitly generating start and end time-aware filter kernels with visual cues to perceive the moment boundaries. Extensive experiments and analyses on three public benchmarks clearly demonstrate the advantage of our proposed SVMR over existing state-of-the-art approaches, especially in retrieving complex query-based video moments.