H-MAS: Hierarchical Multi-Agent Scheduling for Multi-Tenant LLM Serving

摘要

Multi-tenant Model-as-a-Service (MaaS) LLM serving must maintain stringent quality of service (QoS) despite heterogeneous requests competing for constrained GPU resources. In practice, MaaS workloads exhibit non-stationarity across multiple time scales, including request bursts, request-composition drift, and persistent workload shifts. Because workloads change across multiple time scales, existing request schedulers often rely on a single fixed policy (e.g., First-Come-First-Served, FCFS) that remains unchanged at runtime, which can lead to unstable QoS. We propose H-MAS, a hierarchical multi-agent scheduler that operates in a layered closed loop: a perception/prediction layer infers lightweight request attributes and cost signals; a feedback layer summarizes runtime metrics into short- and long-horizon QoS states; a hierarchical control layer updates the active scheduling policy over longer horizons and tunes execution parameters over shorter horizons; and an execution layer applies these decisions during inference. Experiments with load scaling and Azure-trace replays show that H-MAS achieves 1.2\times–3.0\times higher Goodput than SGLang and vLLM, and maintains more stable QoS under workload drift, diverse request lengths and heterogeneous SLO targets.