Efficient hindsight reinforcement learning using demonstrations for robotic tasks with sparse rewards

• Published in: International journal of advanced robotic systems Vol. 17; no. 1
• Main Authors: Zuo, Guoyu, Zhao, Qishen, Lu, Jiahao, Li, Jiangeng
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.01.2020; Sage Publications Ltd; SAGE Publishing
• Subjects: Algorithms; Computer simulation; Robot learning; Robotics; CAHER; demonstrations; Robot learning; sparse reward; reinforcement learning
• ISSN: 1729-8814; 1729-8806; 1729-8814
• DOI: 10.1177/1729881419898342

The goal of reinforcement learning is to enable an agent to learn by using rewards. However, some robotic tasks naturally specify with sparse rewards, and manually shaping reward functions is a difficult project. In this article, we propose a general and model-free approach for reinforcement learning to learn robotic tasks with sparse rewards. First, a variant of Hindsight Experience Replay, Curious and Aggressive Hindsight Experience Replay, is proposed to improve the sample efficiency of reinforcement learning methods and avoid the need for complicated reward engineering. Second, based on Twin Delayed Deep Deterministic policy gradient algorithm, demonstrations are leveraged to overcome the exploration problem and speed up the policy training process. Finally, the action loss is added into the loss function in order to minimize the vibration of output action while maximizing the value of the action. The experiments on simulated robotic tasks are performed with different hyperparameters to verify the effectiveness of our method. Results show that our method can effectively solve the sparse reward problem and obtain a high learning speed.