SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs
ICT · Institute of Computing Technology, Chinese Academy of Sciences · Beijing University of Technology · KTH Royal Institute of Technology · University of the Chinese Academy of Sciences · , Chinese Academy of Sciences · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Chinese Academy of Science · University of Aberdeen
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Recent multimodal large language models (MLLMs) marry modality-specific vision or audio encoders with a shared text decoder. While the encoder is compute- intensive but memory-light, the decoder is the opposite, yet state-of-the-art serving stacks still time-multiplex these complementary kernels, idling SMs or HBM in turn. We introduce SpaceServe, a serving system that space-multiplexes MLLMs: it decouples all modality encoders from the decoder, and co-locates them on the same GPU using fine-grained SM partitioning available in modern runtimes. A cost-model-guided Space-Inference Scheduler (SIS) dynamically assigns SM slices, while a Time-Windowed Shortest-Remaining-First (TWSRFT) policy batches en- coder requests to minimise completion latency and smooth decoder arrivals. Evaluation shows that SpaceServe reduces time-per-output-token by 4.81× on average and up to 28.9× on Nvidia A100 GPUs. SpaceServe is available at https://github.com/gofreelee/SpaceServe