REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion

摘要

As an important and challenging problem in computer vision, Panoramic Semantic Segmentation (PASS) aims to provide complete scene perception based on an ultra-wide angle of view. Most PASS methods often focus on spherical geometry with RGB input or use the depth information in original or HHA format, which does not make full use of panoramic image geometry. To address these shortcomings, we propose REL-SF4PASS with our REL depth representation based on cylindrical coordinate and Spherical-dynamic Multi-Modal Fusion (SMMF). REL is made up of Rectified Depth, Elevation-Gained Vertical Inclination Angle, and Lateral Orientation Angle, which fully represent 3D space in cylindrical coordinate style and the surface normal direction. SMMF aims to ensure the diversity of fusion for different panoramic image regions and reduce the breakage of cylinder side surface expansion in ERP projection, using different fusion strategies to match different regions. Experimental results show that REL-SF4PASS considerably improves performance and robustness on popular benchmarks (e.g., Stanford2D3D Panoramic datasets). It gains 2.35% average mIoU improvement on all 3 folds and reduces the performance variance by approximately 70% when facing 3D disturbance.