MISP-QEKS: A Large-Scale Dataset with Multimodal Cues for Query-by-Example Keyword Spotting

摘要

We present MISP-QEKS, the first large-scale tri-modal benchmark for query-by-example keyword spotting (QEKS). Specifically, MISP-QEKS comprises 610,000 enrollment-query pairs with real-world noise, and covers 9,830 keywords. We also introduce a cross-modal enrolment-query matcher (XEQ-Matcher) as the baseline, which computes wake-word likelihoods by pairwise similarity between pre-trained enrolment and query embeddings. We also propose two plug-and-play modules: a visual gating module (VGM) that filters noise using lip movements, and a multimodal alignment module (MAM) that enforces phone-level consistency across modalities. Experiments show that XEQ-Matcher delivers peak performance with visual enrollment and query, achieving 82.82% AUC/24.23% EER and 79.79% AUC/26.20% EER on in-vocabulary and out-of-vocabulary splits. Incorporating the VGM further improves performance by +1.24% AUC/-0.89% EER and +2.65% AUC/-1.43% EER, respectively. And adding the MAM on top of VGM yields an additional +1.88% AUC/-1.74% EER and +3.00% AUC/-3.28% EER, respectively. These results confirm that tri-modal fusion significantly enhances robustness and generalisation of QEKS. MISP-QEKS with code is available at https://github.com/coalboss/MISP-QEKS.