PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum

摘要

Existing open-vocabulary 3D semantic segmentation methods typically supervise 3D segmentation model by merging text-aligned features (e.g., CLIP) extracted from multi-view images onto 3D points. However, such approaches treat multi-view images merely as intermediaries for transferring open-vocabulary information, overlooking their rich semantic content and cross-view correspondences, and thus limiting the model's effectiveness. To this end, we propose PGOV3D, a novel framework that introduces Partialto-Global curriculum to improve Open-Vocabulary 3D semantic segmentation. The key innovation of our work is a two-stage training strategy. In the first stage, we pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry. Partial point clouds are derived from multi-view RGB-D inputs via pixel-wise depth projection. To enable openvocabulary learning, we leverage a multi-modality large language model (MLLM) and a 2D segmentation foundation model to generate open-vocabulary labels for each viewpoint, providing rich and aligned supervision. An auxiliary inter-frame consistency module is introduced during this stage to enforce feature consistency under viewpoint variations and enhance spatial understanding. In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex. To support this, we aggregate the partial vocabularies associated with each scene and generate pseudo labels using the pre-trained model, effectively bridging the semantic gap between dense partial observations and large-scale 3D environments. Extensive experiments on ScanNet, ScanNet200 and S3DIS benchmarks demonstrate that PGOV3D, achieves competitive performance in open-vocabulary 3D semantic segmentation. The code will be released.