Evaluating state-of-the-art # SAT solvers on industrial configuration spaces

• Published in: Empirical software engineering : an international journal Vol. 28; no. 2; p. 29
• Main Authors: Sundermann, Chico, Heß, Tobias, Nieke, Michael, Bittner, Paul Maximilian, Young, Jeffrey M., Thüm, Thomas, Schaefer, Ina
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2023; Springer Nature B.V
• Subjects: Compilers; Complex systems; Computation; Computer Science; Configurations; Design of experiments; Interpreters; Product lines; Programming Languages; Software Engineering/Programming and Operating Systems; Software Product Lines and Variability-rich Systems (SPLC); Solvers; State-of-the-art reviews; Configurable systems; Configuration counting; SAT; Benchmark; Model counting; Product lines; Feature models
• ISSN: 1382-3256; 1573-7616
• DOI: 10.1007/s10664-022-10265-9

Product lines are widely used to manage families of products that share a common base of features. Typically, not every combination (configuration) of features is valid. Feature models are a de facto standard to specify valid configurations and allow standardized analyses on the variability of the underlying system. A large variety of such analyses depends on computing the number of valid configurations. To analyze feature models, they are typically translated to propositional logic. This allows to employ # SAT solvers that compute the number of satisfying assignments of the propositional formula translated from a feature model. However, the # SAT problem is generally assumed to be even harder than SAT and its scalability when applied to feature models has only been explored sparsely. Our main contribution is an investigation of the performance of off-the-shelf # SAT solvers on computing the number of valid configurations for industrial feature models. We empirically evaluate 21 publicly available # SAT solvers on 130 feature models from 15 subject systems. Our results indicate that current solvers master a majority of the evaluated systems (13/15) with the fastest solvers requiring less than one second for each successfully evaluated feature model. However, there are two complex systems for which none of the evaluated solvers scales. For the given experiment design, the solvers that consumed the least runtime are sharpSAT (2.5 seconds in sum for the 13 systems) and Ganak (3.5 seconds).