A Filtering Framework for Semi-online Referring Video Object Segmentation

摘要

Referring video object segmentation (RVOS) extracts objects from videos based on provided text narrations. Previous approaches typically work on all video frames simultaneously through offline processing, but this is not always possible. Offline processing becomes also ineffective for long videos, as directly cutting the video into short clips to fit memory limits results in the loss of temporal consistency. To make RVOS more applicable to real-world video streams or long video scenarios, we introduce a Filtering Framework for RVOS (FF-RVOS), the first model capable of operating in online, semi-online, and offline modes with just one training session. We redesign RVOS as a stochastic optimization problem and leverage filtering to optimize object states across temporal sequences. Our method enhances temporal consistency for video streams and when a long video is processed in shorter clip sequences due to memory limitations. FF-RVOS demonstrates superior performance compared to previous state-of-the-art methods on public benchmarks with clear improvements, especially when the video is cut into short clips for processing. Our framework can also be embedded into different offline methods to boost temporal consistency. The project page is https://github.com/haliphinx/FF-RVOS.