A Detailed Historical and Statistical Analysis of the Influence of Hardware Artifacts on SPEC Integer Benchmark Performance

The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system perform...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on computers Vol. 73; no. 5; pp. 1 - 12
Main Authors Wang, Yueyao, Furman, Samuel, Hardy, Nicolas, Ellis, Margaret, Back, Godmar, Hong, Yili, Cameron, Kirk
Format Journal Article
LanguageEnglish
Published New York IEEE 01.05.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Artifact identification
Benchmark testing
Benchmarks
Central Processing Unit
Central processing units
Computers
CPU Benchmark
CPUs
Data Visualization
Hardware
Libquantum
Performance evaluation
Questions
Regression Analysis
Sensitivity Analysis
SPEC Normalization
Statistical analysis
System performance
Online AccessGet full text

Cover

More Information
Summary:The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?-have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?-can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance.
ISSN:0018-9340
1557-9956
DOI:10.1109/TC.2024.3365941