Segment Any 3D-Part in a Scene from a Sentence

摘要

This paper aims to achieve the segmentation of any 3D part in a scene based on natural language descriptions, extending beyond traditional object-level 3D scene understanding and addressing both data and methodological challenges. Due to the expensive 3D acquisition and annotation burden, existing datasets and methods are typically limited to object-level comprehension. To overcome these limitations, we introduce the 3D-PU dataset, the first large-scale 3D scene dataset with dense 3D part annotations, created through a cost-effective method for constructing synthetic 3D scenes with fine-grained part-level annotations, paving the way for advanced 3D-part scene understanding. On the methodological side, we propose OpenPart3D, a system to effectively tackle the challenges of part-level segmentation with three integrated modules that converts a 3D scene into 3D-part segmentation masks. Extensive experiments demonstrate the effectiveness in open-vocabulary 3D scene understanding tasks at part level, with strong generalization across real-world 3D scene datasets.