Learning obstacle avoidance and predation in complex reef environments with deep reinforcement learning

• Published in: Bioinspiration & biomimetics Vol. 19; no. 5; pp. 56014 - 56028
• Main Authors: Hou, Ji, He, Changling, Li, Tao, Zhang, Chunze, Zhou, Qin
• Format: Journal Article
• Language: English
• Published: England IOP Publishing 01.09.2024
• Subjects: Algorithms; Animals; Avoidance Learning - physiology; Biomimetics - methods; Computer Simulation; Coral Reefs; Deep Learning; deep reinforcement learning; Ecosystem; Fishes - physiology; fluid-structure interaction; immersed boundary lattice Boltzmann method; intelligent fish; Models, Biological; Predatory Behavior - physiology; Reinforcement, Psychology; sparse reward; Swimming - physiology; intelligent fish; fluid-structure interaction; deep reinforcement learning; immersed boundary lattice Boltzmann method; sparse reward
• ISSN: 1748-3182; 1748-3190; 1748-3190
• DOI: 10.1088/1748-3190/ad6544

The reef ecosystem plays a vital role as a habitat for fish species with limited swimming capabilities, serving not only as a sanctuary and food source but also influencing their behavioral tendencies. Understanding the intricate mechanism through which fish adeptly navigate the moving targets within reef environments within complex water flow, all while evading obstacles and maintaining stable postures, has remained a challenging and prominent subject in the realms of fish behavior, ecology, and biomimetics alike. An integrated simulation framework is used to investigate fish predation problems within intricate environments, combining deep reinforcement learning algorithms (DRL) with high-precision fluid-structure interaction numerical methods-immersed boundary lattice Boltzmann method (lB-LBM). The Soft Actor-Critic (SAC) algorithm is used to improve the intelligent fish’s capacity for random exploration, tackling the multi-objective sparse reward challenge inherent in real-world scenarios. Additionally, a reward shaping method tailored to its action purposes has been developed, capable of capturing outcomes and trend characteristics effectively. The convergence and robustness advantages of the method elucidated in this paper are showcased through two case studies: one addressing fish capturing randomly moving targets in hydrostatic flow field, and the other focusing on fish counter-current foraging in reef environments to capture drifting food. A comprehensive analysis was conducted of the influence and significance of various reward types on the decision-making processes of intelligent fish within intricate environments.