AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
Institute of automation, Chinese academy of science, Chinese Academy of Sciences · Institute of Automation, University of the Chinese Academy of Sciences · Kuaishou · Kuaishou- 快手科技 · Peking University · Peking University · Institute of Automation, Chinese Academy of Sciences · Kling Team, Kuaishou Technology · Kuaishou Technology · CASIA
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present **AVoCaDO**, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities. We propose a two-stage post-training pipeline: (1) **AVoCaDO SFT**, which fine-tunes the model on a newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) **AVoCaDO GRPO**, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC benchmark under visual-only settings. The model will be made publicly available to facilitate future research in audiovisual video understanding and generation.