Deep reinforcement learning for shared control of mobile robots

• Published in: IET cyber-systems and robotics Vol. 3; no. 4; pp. 315 - 330
• Main Authors: Tian, Chong, Shaik, Shahil, Wang, Yue
• Format: Journal Article
• Language: English
• Published: Oxford John Wiley & Sons, Inc 01.12.2021; Wiley
• Subjects: Algorithms; control; Control systems; Controllers; Deep learning; deep learning (artificial intelligence); delays; Human performance; human robot interaction; Integer programming; Machine learning; Manual control; mean square error methods; mobile robots; Neural networks; optimisation; Optimization; Reagents; reinforcement learning; Robot control; Robots; telerobotics; Velocity; Workload; Workloads
• ISSN: 2631-6315; 2631-6315
• DOI: 10.1049/csy2.12036

Shared control of mobile robots integrates manual input with auxiliary autonomous controllers to improve the overall system performance. However, prior work that seeks to find the optimal shared control ratio needs an accurate human model, which is usually challenging to obtain. In this study, the authors develop an extended Twin Delayed Deep Deterministic Policy Gradient (DDPG) (TD3X)‐based shared control framework that learns to assist a human operator in teleoperating mobile robots optimally. The robot's states, shared control ratio in the previous time step, and human's control input is used as inputs to the reinforcement learning (RL) agent, which then outputs the optimal shared control ratio between human input and autonomous controllers without knowing the human model. Noisy softmax policies are developed to make the TD3X algorithm feasible under the constraint of a shared control ratio. Furthermore, to accelerate the training process and protect the robot, a navigation demonstration policy and a safety guard are developed. A neural network (NN) structure is developed to maintain the correlation of sensor readings among heterogeneous input data and improve the learning speed. In addition, an extended DAGGER (DAGGERX) human agent is developed for training the RL agent to reduce human workload. Robot simulations and experiments with humans in the loop are conducted. The results show that the DAGGERX human agent can simulate real human inputs in the worst‐case scenarios with a mean square error of 0.0039. Compared to the original TD3 agent, the TD3X‐based shared control system decreased the average collision number from 387.3 to 44.4 in a simplistic environment and 394.2 to 171.2 in a more complex environment. The maximum average return increased from 1043 to 1187 with a faster converge speed in the simplistic environment, while the performance is equally good in the complex environment because of the use of an advanced human agent. In the human subject tests, participants' average perceived workload was significantly lower in shared control than that in exclusively manual control (26.90 vs. 40.07, p = 0.013).