OLMoE: Open Mixture-of-Experts Language Models
Stanford University · Allen Institute for AI · Allen Institute for Artificial Intelligence · Facebook · University of Washington, Seattle · University of Texas at Austin · Google · Princeton University · Alibaba Group · University of Washington · Facebook AI Research · University of Washington/AI2
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present novel findings on MoE training, define and analyze new routing properties showing high specialization in our model, and open-source all our work: model weights, training data, code, and logs.