PolyAudio: Advancing Multi-Audio Reasoning in Large Audio Language Models with Interleaved Multi-Audio Contexts

摘要

Large Audio Language Models have shown impressive performance on single-clip audio language tasks such as automatic speech recognition, captioning, and sound event recognition. Yet, their ability to reason over interleaved multi-audio contexts-where answering a query requires relating information across multiple audio clips-remains limited. We present PolyAudio, a LALM built on Audio Flamingo 3 that targets multi-audio understanding via instruction tuning rather than massive-scale pre-training, and PolyAudio-Instruct, a high-quality instruction-tuning dataset consisting of 1.3M+ QA pairs, spanning over 14 task subsets to empower multi-audio understanding and reasoning. PolyAudio uses an explicit interleaved representation with clip indexing to encourage faithful grounding and reduce ambiguity in multi-clip references. We evaluate PolyAudio on a diverse suite of multi-audio benchmarks alongside standard single-audio tasks. PolyAudio achieves strong performance on multi-audio reasoning, outperforming competitive baselines that are also often limited to reasoning over up-to 2 audio clips, while preserving robust single-clip performance. Overall, our results suggest that precise, academic-scale multi-audio instruction tuning can unlock advanced cross-clip reasoning capabilities, enabling more capable audio-centric assistants.