An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Common Crawl Foundation · University of Helsinki · University of Oslo · Prompsit Language Engineering · University of Edinburgh · Aveni · Charles University · University of Turku · Warsaw University of Technology · University of Turku and University of Turku · Edinburgh University, University of Edinburgh · University of Edinburgh, University of Edinburgh · Universidad de Alicante · Charles University Prague
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。DOI 10.18653/v1/2025.acl-long.854 ↗
摘要
Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.