Robot Policy Improvement With Natural Evolution Strategies for Stable Nonlinear Dynamical System

• Published in: IEEE transactions on cybernetics Vol. 53; no. 6; pp. 4002 - 4014
• Main Authors: Hu, Yingbai, Chen, Guang, Li, Zhijun, Knoll, Alois
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.06.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cloning; Dynamic stability; Dynamical system; Dynamical systems; Evolution; exponential natural evolution strategies (NESs); Heuristic algorithms; imitation learning; Impedance; Motion planning; Motion stability; Obstacle avoidance; Parameters; policy improvement of robustness and adaptability; Probabilistic models; Robot dynamics; Robots; Robustness; Stability analysis; Stability criteria; Stiffness; Supervised learning; Task analysis; Task complexity
• ISSN: 2168-2267; 2168-2275; 2168-2275
• DOI: 10.1109/TCYB.2022.3192049

Robot learning through kinesthetic teaching is a promising way of cloning human behaviors, but it has its limits in the performance of complex tasks with small amounts of data, due to compounding errors. In order to improve the robustness and adaptability of imitation learning, a hierarchical learning strategy is proposed: low-level learning comprises only behavioral cloning with supervised learning, and high-level learning constitutes policy improvement. First, the Gaussian mixture model (GMM)-based dynamical system is formulated to encode a motion from the demonstration. We then derive the sufficient conditions of the GMM parameters that guarantee the global stability of the dynamical system from any initial state, using the Lyapunov stability theorem. Generally, imitation learning should reason about the motion well into the future for a wide range of tasks; it is significant to improve the adaptability of the learning method by policy improvement. Finally, a method based on exponential natural evolution strategies is proposed to optimize the parameters of the dynamical system associated with the stiffness of variable impedance control, in which the exploration noise is subject to stability conditions of the dynamical system in the exploration space, thus guaranteeing the global stability. Empirical evaluations are conducted on manipulators for different scenarios, including motion planning with obstacle avoidance and stiffness learning.