COCO: a platform for comparing continuous optimizers in a black-box setting

• Published in: Optimization methods & software Vol. 36; no. 1; pp. 114 - 144
• Main Authors: Hansen, Nikolaus, Auger, Anne, Ros, Raymond, Mersmann, Olaf, Tušar, Tea, Brockhoff, Dimo
• Format: Journal Article
• Language: English
• Published: Abingdon Taylor & Francis 02.01.2021; Taylor & Francis Ltd
• Subjects: Algorithms; benchmarking; Benchmarks; black-box optimization; Computer programming; Computer Science; derivative-free optimization; Machine Learning; Mathematical Software; Multiple objective analysis; Neural and Evolutionary Computing; Numerical Analysis; Numerical optimization; Optimization; performance assessment; runtime distributions; software; Statistics; test functions; runtime distributions; test functions; Numerical optimization; software; Black-box optimization; Benchmarking; performance assessment; Optimization; derivative-free optimization
• ISSN: 1055-6788; 1029-4937
• DOI: 10.1080/10556788.2020.1808977

We introduce COCO, an open-source platform for Comparing Continuous Optimizers in a black-box setting. COCO aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent. The platform and the underlying methodology allow to benchmark in the same framework deterministic and stochastic solvers for both single and multiobjective optimization. We present the rationals behind the (decade-long) development of the platform as a general proposition for guidelines towards better benchmarking. We detail underlying fundamental concepts of COCO such as the definition of a problem as a function instance, the underlying idea of instances, the use of target values, and runtime defined by the number of function calls as the central performance measure. Finally, we give a quick overview of the basic code structure and the currently available test suites.