Outdoor Vision-and-Language Navigation Needs Object-Level Alignment

• Published in: Sensors (Basel, Switzerland) Vol. 23; no. 13; p. 6028
• Main Authors: Sun, Yanjun, Qiu, Yue, Aoki, Yoshimitsu, Kataoka, Hirokatsu
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 29.06.2023; MDPI
• Subjects: Coffeehouses; Datasets; embodied AI; landmark-based navigation; Language; Natural language; Reagents; Semantics; Sensors; Urban environments; vision-and-language navigation; Visual perception; United States; landmark-based navigation; vision-and-language navigation; embodied AI
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23136028

In the field of embodied AI, vision-and-language navigation (VLN) is a crucial and challenging multi-modal task. Specifically, outdoor VLN involves an agent navigating within a graph-based environment, while simultaneously interpreting information from real-world urban environments and natural language instructions. Existing outdoor VLN models predict actions using a combination of panorama and instruction features. However, these methods may cause the agent to struggle to understand complicated outdoor environments and ignore the details in the environments to fail to navigate. Human navigation often involves the use of specific objects as reference landmarks when navigating to unfamiliar places, providing a more rational and efficient approach to navigation. Inspired by this natural human behavior, we propose an object-level alignment module (OAlM), which guides the agent to focus more on object tokens mentioned in the instructions and recognize these landmarks during navigation. By treating these landmarks as sub-goals, our method effectively decomposes a long-range path into a series of shorter paths, ultimately improving the agent's overall performance. In addition to enabling better object recognition and alignment, our proposed OAlM also fosters a more robust and adaptable agent capable of navigating complex environments. This adaptability is particularly crucial for real-world applications where environmental conditions can be unpredictable and varied. Experimental results show our OAlM is a more object-focused model, and our approach outperforms all metrics on a challenging outdoor VLN Touchdown dataset, exceeding the baseline by 3.19% on task completion (TC). These results highlight the potential of leveraging object-level information in the form of sub-goals to improve navigation performance in embodied AI systems, paving the way for more advanced and efficient outdoor navigation.