A covariance matrix adaptation evolution strategy in reproducing kernel Hilbert space

• Published in: Genetic programming and evolvable machines Vol. 20; no. 4; pp. 479 - 501
• Main Authors: Dang, Viet-Hung, Vien, Ngo Anh, Chung, TaeChoong
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.12.2019; Springer Nature B.V
• Subjects: Adaptation; Algorithms; Artificial Intelligence; Biological evolution; Biomedical Engineering and Bioengineering; Compilers; Computer Science; Covariance matrix; Domains; Electrical Engineering; Function space; Functionals; Gaussian process; Hilbert space; Interpreters; Kernels; Machine learning; Operators (mathematics); Optimization; Programming Languages; Programming Techniques; Software Engineering/Programming and Operating Systems; Covariance matrix adaptation-evolution strategies (CMA-ES); Kernel methods; Reproducing kernel Hilbert space; Functional optimization; Reinforcement learning; Robot learning; Policy search
• ISSN: 1389-2576; 1573-7632
• DOI: 10.1007/s10710-019-09357-1

The covariance matrix adaptation evolution strategy (CMA-ES) is an efficient derivative-free optimization algorithm. It optimizes a black-box objective function over a well-defined parameter space in which feature functions are often defined manually. Therefore, the performance of those techniques strongly depends on the quality of the chosen features or the underlying parametric function space. Hence, enabling CMA-ES to optimize on a more complex and general function class has long been desired. In this paper, we consider modeling the input spaces in black-box optimization non-parametrically in reproducing kernel Hilbert spaces (RKHS). This modeling leads to a functional optimisation problem whose domain is a RKHS function space that enables optimisation in a very rich function class. We propose CMA-ES-RKHS, a generalized CMA-ES framework that is able to carry out black-box functional optimisation in RKHS. A search distribution on non-parametric function spaces, represented as a Gaussian process, is adapted by updating both its mean function and covariance operator. Adaptive and sparse representation of the mean function and the covariance operator can be retained for efficient computation in the updates and evaluations of CMA-ES-RKHS by resorting to sparsification. We will also show how to apply our new black-box framework to search for an optimum policy in reinforcement learning in which policies are represented as functions in a RKHS. CMA-ES-RKHS is evaluated on two functional optimization problems and two bench-marking reinforcement learning domains.