Alethia: a Foundational Encoder for Voice Deepfakes

摘要

Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines *bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction*. The outcome, *Alethia*, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note *Alethia* significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of *continuous embedding* prediction and *generative pretraining* for capturing deepfake artifacts.