SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation
Tsinghua University · Shanghai Jiaotong University · University of Illinois Urbana-Champaign · Galbot · Peking University · Georgia Institute of Technology · University of Illinois at Urbana-Champaign · BAAI · ShanghaiTech University · Eastern Institute of Technology, Ningbo · Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University · University of Science and Technology of China · Tsinghua University, Tsinghua University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation—a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the ''plug-in'' direction of a USB or the ''handle'' direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7\% successful rate on Open6DOR and zero-shot 74.9\% successful rate on SIMPLER-Env.