SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
Shanghai AI Laboratory; CUHK · MMLab of CUHK & Shanghai AI Laboratory · Shanghaitech University · Shanghai Jiaotong University · South China University of Technology · East China Normal University · Rutgers University · The Chinese University of Hong Kong · Peking University · Shanghai AI Laboratory · Shanghai Artificial Intelligence Innovation Center · Shanghai AI Lab · The Chinese University of Hong Kong, The Chinese University of Hong Kong · Stanford University · Shanghai Aritifcal Intelligence Laboratory · shanghai ai lab
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We propose SPHINX-X, an extensive Multi-modality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multi-modal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama-1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral-8$\times$7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.