Automatic ship collision avoidance using deep reinforcement learning with LSTM in continuous action spaces

• Published in: Journal of marine science and technology Vol. 26; no. 2; pp. 509 - 524
• Main Authors: Sawada, Ryohei, Sato, Keiji, Majima, Takahiro
• Format: Journal Article
• Language: English
• Published: Tokyo Springer Japan 01.06.2021; Springer; Springer Nature B.V
• Subjects: Algorithms; Automotive Engineering; Avoidance behaviour; Avoidance learning; Collision avoidance; Collisions; Deep learning; Distance; Engineering; Engineering Design; Engineering Fluid Dynamics; Long short-term memory; Machine learning; Mechanical Engineering; Offshore Engineering; Original Article; Redesign; Sensors; Ship handling; Ships; Training; Virtual sensors; Multiple ships; OZT; Collision avoidance; Reinforcement learning
• ISSN: 0948-4280; 1437-8213
• DOI: 10.1007/s00773-020-00755-0

This paper presents an automatic collision avoidance algorithm for ships using a deep reinforcement learning (DRL) in continuous action spaces. Obstacle zone by target (OZT) is used to compute an area where a collision will happen in the future based on dynamic information of ships. Agents of DRL detects the approach of multiple ships using a virtual sensor called the grid sensor. Agents learned collision avoidance maneuvering through Imazu problem, which is a scenario set of ship encounter situations. In this study, we propose a new approach for collision avoidance with a longer safe passing distance using DRL. We develop a novel method named inside OZT that expands OZT to improve the consistency of learning. We redesign the network using the long short-term memory (LSTM) cell and carried out training in continuous action spaces to train a model with longer safe distance than the previous study. The bow cross range in collision detection proposed in this paper is effective to COLREGs-compliant collision avoidance. The trained model has passed all scenarios of Imazu problem. The model is also validated by a test scenario which includes more ships than each scenario of Imazu problem.