A Detailed Historical and Statistical Analysis of the Influence of Hardware Artifacts on SPEC Integer Benchmark Performance

The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system perform...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on computers Vol. 73; no. 5; pp. 1 - 12
Main Authors Wang, Yueyao, Furman, Samuel, Hardy, Nicolas, Ellis, Margaret, Back, Godmar, Hong, Yili, Cameron, Kirk
Format Journal Article
LanguageEnglish
Published New York IEEE 01.05.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Online AccessGet full text

Cover

Loading…
Abstract The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?-have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?-can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance.
AbstractList The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?-have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?-can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance.
Author Hardy, Nicolas
Ellis, Margaret
Hong, Yili
Wang, Yueyao
Furman, Samuel
Back, Godmar
Cameron, Kirk
Author_xml – sequence: 1
  givenname: Yueyao
  surname: Wang
  fullname: Wang, Yueyao
  organization: School of Statistics and Mathematics, Zhejiang Gongshang University, Hangzhou, Zhejiang, China
– sequence: 2
  givenname: Samuel
  surname: Furman
  fullname: Furman, Samuel
  organization: Department of Computer Science, Virginia Tech, Blacksburg, VA
– sequence: 3
  givenname: Nicolas
  surname: Hardy
  fullname: Hardy, Nicolas
  organization: Department of Computer Science, Virginia Tech, Blacksburg, VA
– sequence: 4
  givenname: Margaret
  surname: Ellis
  fullname: Ellis, Margaret
  organization: Department of Computer Science, Virginia Tech, Blacksburg, VA
– sequence: 5
  givenname: Godmar
  surname: Back
  fullname: Back, Godmar
  organization: Department of Computer Science, Virginia Tech, Blacksburg, VA
– sequence: 6
  givenname: Yili
  surname: Hong
  fullname: Hong, Yili
  email: yilihong@vt.edu
  organization: Department of Statistics, Virginia Tech, Blacksburg, VA
– sequence: 7
  givenname: Kirk
  surname: Cameron
  fullname: Cameron, Kirk
  organization: Department of Computer Science, Virginia Tech, Blacksburg, VA
BookMark eNpNkM1PAjEQxRuDiYCevXho4nmh3bb7ccQVhYREEvDcdLtTWVx2sS0xxH_eIhw8Td7L701m3gD12q4FhO4pGVFK8vG6GMUk5iPGEpFzeoX6VIg0ynOR9FCfEJpFOePkBg2c2xJCkpjkffQzwc_gVd1AhWe1852ttWqwaiu88soH509PWtUcXe1wZ7DfAJ63pjlAq-FkzJStvpUFPLG-Nkr7gLV4tZwWgfPwARY_BXazU_YTL8Gazu5UyN6ia6MaB3eXOUTvL9N1MYsWb6_zYrKIdMy5j6gpqS651jmlpdKmSoFkqeKmNFyprKpIWQUtYpIlnBgOFdcJMMO1EqURwIbo8bx3b7uvAzgvt93Bho-cZISJOM0Ep4EanyltO-csGLm3dTj5KCmRp4blupCnhuWl4ZB4OCdqAPhHcxbuiNkvzcB6xw
CODEN ITCOB4
Cites_doi 10.1109/HPCA.2018.00032
10.1145/2181796.2181798
10.1145/1250662.1250713
10.1109/ISPASS.2005.1430555
10.1145/2851141.2851159
10.1145/306225.306228
10.1145/1241601.1241615
10.18637/jss.v072.i01
10.1145/1241601.1241616
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
DBID 97E
RIA
RIE
AAYXX
CITATION
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
DOI 10.1109/TC.2024.3365941
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005-present
IEEE All-Society Periodicals Package (ASPP) 1998-Present
IEEE/IET Electronic Library (IEL)
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Technology Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DatabaseTitle CrossRef
Technology Research Database
Computer and Information Systems Abstracts – Academic
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts Professional
DatabaseTitleList
Technology Research Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library Online
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
Computer Science
EISSN 1557-9956
EndPage 12
ExternalDocumentID 10_1109_TC_2024_3365941
10436402
Genre orig-research
GroupedDBID --Z
-DZ
-~X
.DC
0R~
29I
4.4
5GY
6IK
85S
97E
AAJGR
AASAJ
ABQJQ
ACGFO
ACIWK
ACNCT
AENEX
AKJIK
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EBS
EJD
HZ~
IEDLZ
IFIPE
IPLJI
JAVBF
LAI
M43
MS~
O9-
OCL
P2P
PQQKQ
RIA
RIC
RIE
RNS
RXW
TAE
TN5
TWZ
UHB
UPT
XZL
YZZ
.55
3EH
3O-
5VS
AAYOK
AAYXX
ABFSI
AETIX
AI.
AIBXA
ALLEH
CITATION
E.L
F20
H~9
IAAWW
IBMZZ
ICLAB
IFJZH
MVM
RIG
RNI
RZB
UKR
VH1
X7M
XJT
XOL
YXB
YYQ
ZCG
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
ID FETCH-LOGICAL-c244t-1fb1cb4cc911bacfd7e087a4fbf4aa8dd0bd87a5208640f4ed4c6e3f4ca5bf5e3
IEDL.DBID RIE
ISSN 0018-9340
IngestDate Thu Oct 10 17:47:46 EDT 2024
Thu Sep 26 16:18:26 EDT 2024
Mon Nov 04 12:13:46 EST 2024
IsPeerReviewed true
IsScholarly true
Issue 5
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c244t-1fb1cb4cc911bacfd7e087a4fbf4aa8dd0bd87a5208640f4ed4c6e3f4ca5bf5e3
ORCID 0000-0003-1720-9540
0000-0002-1679-5448
0000-0003-2017-9506
PQID 3035278541
PQPubID 85452
PageCount 12
ParticipantIDs crossref_primary_10_1109_TC_2024_3365941
ieee_primary_10436402
proquest_journals_3035278541
PublicationCentury 2000
PublicationDate 2024-05-01
PublicationDateYYYYMMDD 2024-05-01
PublicationDate_xml – month: 05
  year: 2024
  text: 2024-05-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on computers
PublicationTitleAbbrev TC
PublicationYear 2024
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References (ref14) 2021
ref20
Hardy (ref11) 2021
Danowitz (ref18) 2014
ref10
ref2
ref1
ref19
(ref15) 2021
Furman (ref8) 2021
ref7
ref4
Vandierendonck (ref3) 2004
Soup (ref12) 2021
ref6
(ref16) 2021
ref5
(ref9) 2023
(ref17) 2021
(ref13) 2021
References_xml – year: 2021
  ident: ref16
  article-title: AMD product specifications
– ident: ref2
  doi: 10.1109/HPCA.2018.00032
– year: 2021
  ident: ref11
  article-title: A data schema for aggregating disparate sources of computer system and benchmark information
  contributor:
    fullname: Hardy
– year: 2021
  ident: ref13
  article-title: Pandas
– ident: ref7
  doi: 10.1145/2181796.2181798
– ident: ref5
  doi: 10.1145/1250662.1250713
– ident: ref4
  doi: 10.1109/ISPASS.2005.1430555
– year: 2021
  ident: ref15
  article-title: Green500 lists
– year: 2021
  ident: ref17
  article-title: Intel product specifications
– ident: ref19
  doi: 10.1145/2851141.2851159
– start-page: 57
  volume-title: Proc. Workshop Comput. Archit. Eval. Commercial Workloads (CAECW)
  year: 2004
  ident: ref3
  article-title: Many benchmarks stress the same bottlenecks
  contributor:
    fullname: Vandierendonck
– ident: ref10
  doi: 10.1145/306225.306228
– year: 2021
  ident: ref14
  article-title: Top500 lists
– ident: ref1
  doi: 10.1145/1241601.1241615
– year: 2023
  ident: ref9
  article-title: SPECs benchmarks and tools
– year: 2014
  ident: ref18
  article-title: Stanford CPU DB
  contributor:
    fullname: Danowitz
– ident: ref20
  doi: 10.18637/jss.v072.i01
– year: 2021
  ident: ref12
  article-title: Beautiful soup documentation
  contributor:
    fullname: Soup
– ident: ref6
  doi: 10.1145/1241601.1241616
– year: 2021
  ident: ref8
  article-title: iLORE: Discovering a lineage of microprocessors
  contributor:
    fullname: Furman
SSID ssj0006209
Score 2.4593146
Snippet The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Publisher
StartPage 1
SubjectTerms Artifact identification
Benchmark testing
Benchmarks
Central Processing Unit
Central processing units
Computers
CPU Benchmark
CPUs
Data Visualization
Hardware
Libquantum
Performance evaluation
Questions
Regression Analysis
Sensitivity Analysis
SPEC Normalization
Statistical analysis
System performance
Title A Detailed Historical and Statistical Analysis of the Influence of Hardware Artifacts on SPEC Integer Benchmark Performance
URI https://ieeexplore.ieee.org/document/10436402
https://www.proquest.com/docview/3035278541
Volume 73
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3dS8MwEA-6J31wOidOp-TBB1862yXpx-OcDhUcAyfsrSTpRWHYydwQ9J_3krbODwTf2nJNQi6Xu0vufkfISRiFuqsN9xTqPo9rk3hKCvAkxCpWqNITaY8Gbofh1T2_mYhJmazucmEAwAWfQcc-urv8bKaX9qgMJZyzkFvoyPUoSYpkrc9tN6ziOQKUYMb9Escn8JOzcR8dwS7vMBaKhAffVJCrqfJrI3baZVAnw2pcRVDJtLNcqI5--wHZ-O-Bb5Ot0s6kvWJh7JA1yBukXtVwoKVIN8jmF0DCXfLeoxcuphQyugIQoTLPqLVKHaizbbVEMqEzQ9GApNdVpRP7wcYCvMo5uL5t3gSS5fRudNmn9vjxAXs_R9rHJzmf0tEqb6FJ7geX4_6VV5Zn8DTaBAsvMCrQimuN-6WS2mQR-HEkuVGGSxlnma8yfBdd9Jq4bzhkXIfADNdSKCOA7ZFaPsthn1AWWtyxCG0PiXQCEq19xbRgBkyEPlaLnFYcS58LFI7UeS9-ko77qWVuWjK3RZp2_r-QFVPfIu2KxWkppi8ps2iwUSx4cPDHb4dkw7ZehDi2SW0xX8IRmiELdeyW3we439tT
link.rule.ids 315,783,787,799,27936,27937,55086
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LT9wwEB4hegAOUF5i223xgQOXLMn6keS43YKW1wqJReIW2c4YJNRsteyqUvvnO3YSoFRI3JJoElsej-ezM_MNwIFKle1bJyJDvi8S1uWR0RIjjZnJDLn0XPujgcuxGt2Is1t52ySrh1wYRAzBZ9jzl-Fffjm1C39URhYuuBKeOvIDAetM1elaTwuvaiM6ErJhLuKGySeJ86PJkLaCfdHjXMlcJP84oVBV5b-lOPiXkw0Ytz2rw0oeeou56dnfr0gb3931j7DeIE02qKfGJixhtQUbbRUH1hj1Fqy9oCTchj8D9j1ElWLJnilEmK5K5nFpoHX2X224TNjUMYKQ7LStdeIf-GiAX3qGoW2fOUFiFbu-Oh4yfwB5R61_I9n7H3r2wK6eMxd24ObkeDIcRU2BhsgSKphHiTOJNcJaWjGNtq5MMc5SLZxxQuusLGNT0r3sk65E7ASWwirkTlgtjZPId2G5mla4B4wrzzyWEvrQJCcxtzY23Eru0KW0y-rAYaux4mfNw1GE_UucF5Nh4ZVbNMrtwI4f_xdi9dB3oNuquGgM9bHgng82zaRIPr3x2j6sjCaXF8XF6fj8M6z6luqAxy4sz2cL_EKgZG6-hqn4F1Y_3p4
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Detailed+Historical+and+Statistical+Analysis+of+the+Influence+of+Hardware+Artifacts+on+SPEC+Integer+Benchmark+Performance&rft.jtitle=IEEE+transactions+on+computers&rft.au=Wang%2C+Yueyao&rft.au=Furman%2C+Samuel&rft.au=Hardy%2C+Nicolas&rft.au=Ellis%2C+Margaret&rft.date=2024-05-01&rft.issn=0018-9340&rft.eissn=1557-9956&rft.volume=73&rft.issue=5&rft.spage=1262&rft.epage=1274&rft_id=info:doi/10.1109%2FTC.2024.3365941&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TC_2024_3365941
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9340&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9340&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9340&client=summon