Farther Than Mirror: Explore Pattern-Compensated Depth of Mirror with Temporal Changes for Video Mirror Detection

摘要

Current video mirror detection models demonstrate satisfactory performance by analyzing different attributes of mirrors and incorporating temporal information. However, these models still struggle to detect mirrors in complex and dynamic scenarios. A simple yet critical visual cue is that objects reflected in a mirror appear to be farther away than the mirror itself. Motivated by this observation, some studies propose to explicitly analyze the Depth of Mirror (DOM) to effectively localize mirrors - DOM refers to distinct perceived distances that make mirror regions appear farther away from their surroundings. However, merely analyzing the DOM is insufficient in some scenes where the object behind the mirror also appears distant. Meanwhile, the changes in the DOM across different video frames are also important for video mirror detection, yet this aspect has not been fully explored. To address these issues, we devise a novel framework called FTM-Net, which includes two main contributions: a Pattern-Compensated DOM estimation strategy and a Dual-Granularity Affinity module. The Pattern-Compensated DOM estimation strategy uses multiple visual mirror patterns to refine the DOM, enhancing the accuracy of mirror localization in a single image. Furthermore, the Dual-Granularity Affinity module can effectively detect mirrors in video sequences by tracking and integrating DOM changes across video frames. Experimental results on two benchmark datasets show that our model significantly outperforms other state-of-the-art methods in the video mirror detection task. We shall release our trained models, code, and results.