A Detailed Historical and Statistical Analysis of the Influence of Hardware Artifacts on SPEC Integer Benchmark Performance
The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system perform...
Saved in:
Published in | IEEE transactions on computers Vol. 73; no. 5; pp. 1 - 12 |
---|---|
Main Authors | , , , , , , |
Format | Journal Article |
Language | English |
Published |
New York
IEEE
01.05.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?-have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?-can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance. |
---|---|
AbstractList | The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?-have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?-can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance. |
Author | Hardy, Nicolas Ellis, Margaret Hong, Yili Wang, Yueyao Furman, Samuel Back, Godmar Cameron, Kirk |
Author_xml | – sequence: 1 givenname: Yueyao surname: Wang fullname: Wang, Yueyao organization: School of Statistics and Mathematics, Zhejiang Gongshang University, Hangzhou, Zhejiang, China – sequence: 2 givenname: Samuel surname: Furman fullname: Furman, Samuel organization: Department of Computer Science, Virginia Tech, Blacksburg, VA – sequence: 3 givenname: Nicolas surname: Hardy fullname: Hardy, Nicolas organization: Department of Computer Science, Virginia Tech, Blacksburg, VA – sequence: 4 givenname: Margaret surname: Ellis fullname: Ellis, Margaret organization: Department of Computer Science, Virginia Tech, Blacksburg, VA – sequence: 5 givenname: Godmar surname: Back fullname: Back, Godmar organization: Department of Computer Science, Virginia Tech, Blacksburg, VA – sequence: 6 givenname: Yili surname: Hong fullname: Hong, Yili email: yilihong@vt.edu organization: Department of Statistics, Virginia Tech, Blacksburg, VA – sequence: 7 givenname: Kirk surname: Cameron fullname: Cameron, Kirk organization: Department of Computer Science, Virginia Tech, Blacksburg, VA |
BookMark | eNpNkM1PAjEQxRuDiYCevXho4nmh3bb7ccQVhYREEvDcdLtTWVx2sS0xxH_eIhw8Td7L701m3gD12q4FhO4pGVFK8vG6GMUk5iPGEpFzeoX6VIg0ynOR9FCfEJpFOePkBg2c2xJCkpjkffQzwc_gVd1AhWe1852ttWqwaiu88soH509PWtUcXe1wZ7DfAJ63pjlAq-FkzJStvpUFPLG-Nkr7gLV4tZwWgfPwARY_BXazU_YTL8Gazu5UyN6ia6MaB3eXOUTvL9N1MYsWb6_zYrKIdMy5j6gpqS651jmlpdKmSoFkqeKmNFyprKpIWQUtYpIlnBgOFdcJMMO1EqURwIbo8bx3b7uvAzgvt93Bho-cZISJOM0Ep4EanyltO-csGLm3dTj5KCmRp4blupCnhuWl4ZB4OCdqAPhHcxbuiNkvzcB6xw |
CODEN | ITCOB4 |
Cites_doi | 10.1109/HPCA.2018.00032 10.1145/2181796.2181798 10.1145/1250662.1250713 10.1109/ISPASS.2005.1430555 10.1145/2851141.2851159 10.1145/306225.306228 10.1145/1241601.1241615 10.18637/jss.v072.i01 10.1145/1241601.1241616 |
ContentType | Journal Article |
Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
DOI | 10.1109/TC.2024.3365941 |
DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005-present IEEE All-Society Periodicals Package (ASPP) 1998-Present IEEE/IET Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
DatabaseTitleList | Technology Research Database |
Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library Online url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Engineering Computer Science |
EISSN | 1557-9956 |
EndPage | 12 |
ExternalDocumentID | 10_1109_TC_2024_3365941 10436402 |
Genre | orig-research |
GroupedDBID | --Z -DZ -~X .DC 0R~ 29I 4.4 5GY 6IK 85S 97E AAJGR AASAJ ABQJQ ACGFO ACIWK ACNCT AENEX AKJIK ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD HZ~ IEDLZ IFIPE IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIC RIE RNS RXW TAE TN5 TWZ UHB UPT XZL YZZ .55 3EH 3O- 5VS AAYOK AAYXX ABFSI AETIX AI. AIBXA ALLEH CITATION E.L F20 H~9 IAAWW IBMZZ ICLAB IFJZH MVM RIG RNI RZB UKR VH1 X7M XJT XOL YXB YYQ ZCG 7SC 7SP 8FD JQ2 L7M L~C L~D |
ID | FETCH-LOGICAL-c244t-1fb1cb4cc911bacfd7e087a4fbf4aa8dd0bd87a5208640f4ed4c6e3f4ca5bf5e3 |
IEDL.DBID | RIE |
ISSN | 0018-9340 |
IngestDate | Thu Oct 10 17:47:46 EDT 2024 Thu Sep 26 16:18:26 EDT 2024 Mon Nov 04 12:13:46 EST 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 5 |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-c244t-1fb1cb4cc911bacfd7e087a4fbf4aa8dd0bd87a5208640f4ed4c6e3f4ca5bf5e3 |
ORCID | 0000-0003-1720-9540 0000-0002-1679-5448 0000-0003-2017-9506 |
PQID | 3035278541 |
PQPubID | 85452 |
PageCount | 12 |
ParticipantIDs | crossref_primary_10_1109_TC_2024_3365941 ieee_primary_10436402 proquest_journals_3035278541 |
PublicationCentury | 2000 |
PublicationDate | 2024-05-01 |
PublicationDateYYYYMMDD | 2024-05-01 |
PublicationDate_xml | – month: 05 year: 2024 text: 2024-05-01 day: 01 |
PublicationDecade | 2020 |
PublicationPlace | New York |
PublicationPlace_xml | – name: New York |
PublicationTitle | IEEE transactions on computers |
PublicationTitleAbbrev | TC |
PublicationYear | 2024 |
Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
References | (ref14) 2021 ref20 Hardy (ref11) 2021 Danowitz (ref18) 2014 ref10 ref2 ref1 ref19 (ref15) 2021 Furman (ref8) 2021 ref7 ref4 Vandierendonck (ref3) 2004 Soup (ref12) 2021 ref6 (ref16) 2021 ref5 (ref9) 2023 (ref17) 2021 (ref13) 2021 |
References_xml | – year: 2021 ident: ref16 article-title: AMD product specifications – ident: ref2 doi: 10.1109/HPCA.2018.00032 – year: 2021 ident: ref11 article-title: A data schema for aggregating disparate sources of computer system and benchmark information contributor: fullname: Hardy – year: 2021 ident: ref13 article-title: Pandas – ident: ref7 doi: 10.1145/2181796.2181798 – ident: ref5 doi: 10.1145/1250662.1250713 – ident: ref4 doi: 10.1109/ISPASS.2005.1430555 – year: 2021 ident: ref15 article-title: Green500 lists – year: 2021 ident: ref17 article-title: Intel product specifications – ident: ref19 doi: 10.1145/2851141.2851159 – start-page: 57 volume-title: Proc. Workshop Comput. Archit. Eval. Commercial Workloads (CAECW) year: 2004 ident: ref3 article-title: Many benchmarks stress the same bottlenecks contributor: fullname: Vandierendonck – ident: ref10 doi: 10.1145/306225.306228 – year: 2021 ident: ref14 article-title: Top500 lists – ident: ref1 doi: 10.1145/1241601.1241615 – year: 2023 ident: ref9 article-title: SPECs benchmarks and tools – year: 2014 ident: ref18 article-title: Stanford CPU DB contributor: fullname: Danowitz – ident: ref20 doi: 10.18637/jss.v072.i01 – year: 2021 ident: ref12 article-title: Beautiful soup documentation contributor: fullname: Soup – ident: ref6 doi: 10.1145/1241601.1241616 – year: 2021 ident: ref8 article-title: iLORE: Discovering a lineage of microprocessors contributor: fullname: Furman |
SSID | ssj0006209 |
Score | 2.4593146 |
Snippet | The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an... |
SourceID | proquest crossref ieee |
SourceType | Aggregation Database Publisher |
StartPage | 1 |
SubjectTerms | Artifact identification Benchmark testing Benchmarks Central Processing Unit Central processing units Computers CPU Benchmark CPUs Data Visualization Hardware Libquantum Performance evaluation Questions Regression Analysis Sensitivity Analysis SPEC Normalization Statistical analysis System performance |
Title | A Detailed Historical and Statistical Analysis of the Influence of Hardware Artifacts on SPEC Integer Benchmark Performance |
URI | https://ieeexplore.ieee.org/document/10436402 https://www.proquest.com/docview/3035278541 |
Volume | 73 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3dS8MwEA-6J31wOidOp-TBB1862yXpx-OcDhUcAyfsrSTpRWHYydwQ9J_3krbODwTf2nJNQi6Xu0vufkfISRiFuqsN9xTqPo9rk3hKCvAkxCpWqNITaY8Gbofh1T2_mYhJmazucmEAwAWfQcc-urv8bKaX9qgMJZyzkFvoyPUoSYpkrc9tN6ziOQKUYMb9Escn8JOzcR8dwS7vMBaKhAffVJCrqfJrI3baZVAnw2pcRVDJtLNcqI5--wHZ-O-Bb5Ot0s6kvWJh7JA1yBukXtVwoKVIN8jmF0DCXfLeoxcuphQyugIQoTLPqLVKHaizbbVEMqEzQ9GApNdVpRP7wcYCvMo5uL5t3gSS5fRudNmn9vjxAXs_R9rHJzmf0tEqb6FJ7geX4_6VV5Zn8DTaBAsvMCrQimuN-6WS2mQR-HEkuVGGSxlnma8yfBdd9Jq4bzhkXIfADNdSKCOA7ZFaPsthn1AWWtyxCG0PiXQCEq19xbRgBkyEPlaLnFYcS58LFI7UeS9-ko77qWVuWjK3RZp2_r-QFVPfIu2KxWkppi8ps2iwUSx4cPDHb4dkw7ZehDi2SW0xX8IRmiELdeyW3we439tT |
link.rule.ids | 315,783,787,799,27936,27937,55086 |
linkProvider | IEEE |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LT9wwEB4hegAOUF5i223xgQOXLMn6keS43YKW1wqJReIW2c4YJNRsteyqUvvnO3YSoFRI3JJoElsej-ezM_MNwIFKle1bJyJDvi8S1uWR0RIjjZnJDLn0XPujgcuxGt2Is1t52ySrh1wYRAzBZ9jzl-Fffjm1C39URhYuuBKeOvIDAetM1elaTwuvaiM6ErJhLuKGySeJ86PJkLaCfdHjXMlcJP84oVBV5b-lOPiXkw0Ytz2rw0oeeou56dnfr0gb3931j7DeIE02qKfGJixhtQUbbRUH1hj1Fqy9oCTchj8D9j1ElWLJnilEmK5K5nFpoHX2X224TNjUMYKQ7LStdeIf-GiAX3qGoW2fOUFiFbu-Oh4yfwB5R61_I9n7H3r2wK6eMxd24ObkeDIcRU2BhsgSKphHiTOJNcJaWjGNtq5MMc5SLZxxQuusLGNT0r3sk65E7ASWwirkTlgtjZPId2G5mla4B4wrzzyWEvrQJCcxtzY23Eru0KW0y-rAYaux4mfNw1GE_UucF5Nh4ZVbNMrtwI4f_xdi9dB3oNuquGgM9bHgng82zaRIPr3x2j6sjCaXF8XF6fj8M6z6luqAxy4sz2cL_EKgZG6-hqn4F1Y_3p4 |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Detailed+Historical+and+Statistical+Analysis+of+the+Influence+of+Hardware+Artifacts+on+SPEC+Integer+Benchmark+Performance&rft.jtitle=IEEE+transactions+on+computers&rft.au=Wang%2C+Yueyao&rft.au=Furman%2C+Samuel&rft.au=Hardy%2C+Nicolas&rft.au=Ellis%2C+Margaret&rft.date=2024-05-01&rft.issn=0018-9340&rft.eissn=1557-9956&rft.volume=73&rft.issue=5&rft.spage=1262&rft.epage=1274&rft_id=info:doi/10.1109%2FTC.2024.3365941&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TC_2024_3365941 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9340&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9340&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9340&client=summon |