A Population-based Hybrid Approach for Hyperparameter Optimization of Neural Networks

• Published in: IEEE access Vol. 11; p. 1
• Main Authors: Japa, Luis, Serqueira, Marcello, Mendonca, Israel, Aritsugi, Masayoshi, Bezerra, Eduardo, Gonzalez, Pedro Henrique
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Ablation; Bayes methods; Bayesian analysis; Datasets; Exploitation; Gaussian processes; Genetic algorithms; Hyperparameter optimization; Machine Learning; Neural networks; Optimization; Optimization algorithms; Population-based studies; Random walk; Search problems; Searching
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2023.3277310

Hyperparameter optimization is a fundamental part of Auto Machine Learning (AutoML) and it has been widely researched in recent years; however, it still remains as one of the main challenges in this area. Motivated by the need of faster and more accurate hyperparameter optimization algorithms we developed HyperBRKGA, a new population-based approach for hyperparameter optimization. HyperBRKGA combines the Biased Random Key Genetic Algorithm with an Exploitation Method in order to search the hyperparameter space more efficiently than other commonly used hyperparameter optimization algorithms, such as Grid Search, Random Search, CMA-ES or Bayesian Optimization. We develop and test two different alternatives for this Exploitation Method: Random Walk and Bayesian Walk. We also discuss and implement other schemes, such as a Training Data Reduction Strategy and a Diversity Control strategy, in order to further improve the efficacy of our method. We performed several computational experiments on 8 different datasets to assess the effectiveness of the proposed approach. Results showed that HyperBRKGA could find hyperparameter configurations that outperformed in terms of predictive quality the baseline methods in 6 out of 8 datasets while showing a reasonable execution time. Lastly, we conducted an ablation study and showed that the addition of every component was relevant to achieving high quality results.