$\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild
University of North Carolina at Chapel Hill · University of Maryland, College Park · The University of Texas at Austin · The Hong Kong University of Science and Technology · UNC-Chapel Hill · University of Texas at Austin · Carnegie Mellon University · University of Illinois at Urbana-Champaign · Research, Google · CMU · Hong Kong University of Science and Technology · CMU, Carnegie Mellon University · Massachusetts Institute of Technology
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
As Large Language Models (LLMs) excel across tasks and specialized domains, scaling LLMs based on existing models has gained significant attention, which is challenged by potential performance drop when combining disparate models. Various techniques have been proposed to aggregate pre-trained LLMs, including model merging, Mixture-of-Experts, and stacking. Despite their merits, a comprehensive comparison and synergistic application of them to a diverse model zoo is yet to be adequately addressed.In light of this research gap, this paper introduces $\texttt{Model-GLUE}$, a holistic LLM scaling guideline. First, our work starts with a benchmarking of existing LLM scaling techniques, especially selective merging, and variants of mixture. Utilizing the insights from the benchmark results, we formulate a strategy for the selection and aggregation of a heterogeneous model zoo characterizing different architectures and initialization.Our methodology involves clustering mergeable models, selecting a merging strategy, and integrating model clusters through model-level mixture. Finally, evidenced by our experiments on a diverse Llama-2-based model zoo, $\texttt{Model-GLUE}$ shows an average performance enhancement of 5.61\%, achieved without additional training.Codes are available at https://github.com/Model-GLUE/Model-GLUE.