OpenFly: A COMPREHENSIVE PLATFORM FOR AERIAL VISION-LANGUAGE NAVIGATION
Northwest Polytechnical University Xi'an · Shanghai Artificial Intelligence Laboratory · Beijing University of Aeronautics and Astronautics · Shanghai Jiaotong University · The University of Hong Kong · Zhejiang University · University of Science and Technology of China · East China University of Science and Technology · Shanghai AI Lab · Fudan University · Institute of automation, Chinese academy of science, Chinese Academy of Sciences · OneStar Robotics · Shanghai AI Laboratory · Shanghai AI Lab · Shanghai Artificial Intelligence Labortory · China Telecom
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Aerial Vision-Language Navigation (VLN) seeks to guide UAVs by leveraging language instructions and visual cues, establishing a new paradigm for human-UAV interaction. However, the collection of VLN data demands extensive human effort to construct trajectories and corresponding instructions, hindering the development of large-scale datasets and capable models. To address this problem, we propose OpenFly, a comprehensive platform for aerial VLN. Firstly, OpenFly integrates 4 rendering engines and advanced techniques for diverse environment simulation, including Unreal Engine, GTA V, Google Earth, and 3D Gaussian Splatting (3D GS). Particularly, 3D GS supports real-to-sim rendering, further enhancing the realism of our environments. Secondly, we develop a highly automated toolchain for aerial VLN data collection, streamlining point cloud acquisition, scene semantic segmentation, flight trajectory creation, and instruction generation. Thirdly, based on the toolchain, we construct a large-scale aerial VLN dataset with 100k trajectories, covering samples of diverse scenarios and assets across 18 scenes. Moreover, we propose OpenFly-Agent, a keyframe-aware VLN model emphasizing key observations to promote performance and reduce computations. For benchmarking, extensive experiments and analyses are conducted, where our navigation success rate outperforms others by 14.0\% and 7.9\% on the seen and unseen scenarios, respectively. The toolchain, dataset, and codes will be open-sourced.