The Rational Agent Benchmark for Data Visualization
Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpre...
Saved in:
| Published in | IEEE transactions on visualization and computer graphics Vol. 30; no. 1; pp. 338 - 347 |
|---|---|
| Main Authors | , , , , |
| Format | Journal Article |
| Language | English |
| Published |
United States
IEEE
01.01.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects | |
| Online Access | Get full text |
| ISSN | 1077-2626 1941-0506 1941-0506 |
| DOI | 10.1109/TVCG.2023.3326513 |
Cover
Loading…
| Abstract | Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses. |
|---|---|
| AbstractList | Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses. Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses. |
| Author | Hullman, Jessica Guo, Ziyang Hartline, Jason Wu, Yifan Mamakos, Michalis |
| Author_xml | – sequence: 1 givenname: Yifan orcidid: 0000-0002-4299-8169 surname: Wu fullname: Wu, Yifan email: yifan.wu@u.northwestern.edu organization: Northwestern University, USA – sequence: 2 givenname: Ziyang orcidid: 0009-0004-4200-6774 surname: Guo fullname: Guo, Ziyang email: ziyangguo2027@u.northwestern.edu organization: Northwestern University, USA – sequence: 3 givenname: Michalis surname: Mamakos fullname: Mamakos, Michalis email: michailmamakos2022@u.northwestern.edu organization: Northwestern University, USA – sequence: 4 givenname: Jason orcidid: 0000-0001-5505-6819 surname: Hartline fullname: Hartline, Jason email: hartline@northwestern.edu organization: Northwestern University, USA – sequence: 5 givenname: Jessica orcidid: 0000-0001-6826-3550 surname: Hullman fullname: Hullman, Jessica email: jhullman@northwestern.edu organization: Northwestern University, USA |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/37871058$$D View this record in MEDLINE/PubMed |
| BookMark | eNp9kMtOwzAQRS1URB_wAUgIRWLDJsWPJLaXpUBBqoSESreW406oS5qUOFnA1-O0BaEuWHkW5874nj7qFGUBCJ0TPCQEy5vZfDwZUkzZkDGaxIQdoR6REQlxjJOOnzHnIU1o0kV951YYkygS8gR1GRec4Fj0EJstIXjRtS0LnQejNyjq4BYKs1zr6j3Iyiq407UO5tY1OrdfW_AUHWc6d3C2fwfo9eF-Nn4Mp8-Tp_FoGhoW4zoUlNKUp0JjkXKcJcBMqgkhCQdg1PAsozg1whhpGGSRYQudEk0lASpE5FsN0PVu76YqPxpwtVpbZyDPdQFl45THCI24jCOPXh2gq7KpfCVPSZzEgrOkXXi5p5p0DQu1qayv-al-dHiA7ABTlc5VkP0iBKtWuWqVq1a52iv3GX6QMbbeeqorbfN_kxe7pAWAP5f8j6WM2DfNT4ug |
| CODEN | ITVGEA |
| CitedBy_id | crossref_primary_10_1109_MCG_2024_3360881 crossref_primary_10_1109_TVCG_2024_3456182 |
| Cites_doi | 10.1073/pnas.1915841117 10.1038/s41586-021-03659-0 10.1109/tvcg.2021.3114813 10.1145/3490486.3538338 10.1017/s0140525x20001685 10.1038/nrn3475 10.1109/tvcg.2019.2934287 10.1179/1743277414y.0000000099 10.1109/tvcg.2013.126 10.1162/99608f92.3ab8a587 10.1109/tvcg.2011.279 10.1145/1168149.1168158 10.1117/12.643631 10.1145/1377966.1377974 10.1109/TVCG.2023.3326516 10.1080/01621459.1984.10478080 10.1109/tvcg.2020.3030335 10.1145/3173574.3173718 10.1017/CBO9780511984037 10.1037/a0024558 10.1109/tvcg.2021.3114824 10.1371/journal.pone.0142444 10.1109/tvcg.2009.111 10.1109/tvcg.2018.2864889 10.1142/9789814417358_0006 10.1109/tvcg.2020.3028984 10.1145/2858036.2858558 10.1145/1600150.1600175 10.1109/tvcg.2020.3030395 10.1086/718371 10.1002/acp.2932 10.1109/visual.2005.1532781 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| DBID | 97E RIA RIE AAYXX CITATION NPM 7SC 7SP 8FD JQ2 L7M L~C L~D 7X8 |
| DOI | 10.1109/TVCG.2023.3326513 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef PubMed Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional MEDLINE - Academic |
| DatabaseTitle | CrossRef PubMed Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional MEDLINE - Academic |
| DatabaseTitleList | Technology Research Database PubMed MEDLINE - Academic |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: RIE name: IEEE Electronic Library (IEL) url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1941-0506 |
| EndPage | 347 |
| ExternalDocumentID | 37871058 10_1109_TVCG_2023_3326513 10290994 |
| Genre | orig-research Journal Article |
| GroupedDBID | --- -~X .DC 0R~ 29I 4.4 53G 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ H~9 IEDLZ IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P PQQKQ RIA RIE RNI RNS RZB TN5 VH1 AAYOK AAYXX CITATION RIG NPM 7SC 7SP 8FD JQ2 L7M L~C L~D 7X8 |
| ID | FETCH-LOGICAL-c350t-8222b7b8a08b70f6e3cba11167ee32c7ff20bc8cc9c3ef4c3dab1a291e2884023 |
| IEDL.DBID | RIE |
| ISSN | 1077-2626 1941-0506 |
| IngestDate | Fri Jul 11 00:30:13 EDT 2025 Mon Jun 30 06:35:10 EDT 2025 Mon Jul 21 06:06:34 EDT 2025 Tue Jul 01 02:12:19 EDT 2025 Thu Apr 24 23:02:08 EDT 2025 Wed Aug 27 02:12:08 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 1 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c350t-8222b7b8a08b70f6e3cba11167ee32c7ff20bc8cc9c3ef4c3dab1a291e2884023 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 |
| ORCID | 0000-0001-6826-3550 0009-0004-4200-6774 0000-0002-4299-8169 0000-0001-5505-6819 |
| PMID | 37871058 |
| PQID | 2906587362 |
| PQPubID | 75741 |
| PageCount | 10 |
| ParticipantIDs | pubmed_primary_37871058 proquest_journals_2906587362 crossref_primary_10_1109_TVCG_2023_3326513 ieee_primary_10290994 crossref_citationtrail_10_1109_TVCG_2023_3326513 proquest_miscellaneous_2881247954 |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2024-01-01 |
| PublicationDateYYYYMMDD | 2024-01-01 |
| PublicationDate_xml | – month: 01 year: 2024 text: 2024-01-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States – name: New York |
| PublicationTitle | IEEE transactions on visualization and computer graphics |
| PublicationTitleAbbrev | TVCG |
| PublicationTitleAlternate | IEEE Trans Vis Comput Graph |
| PublicationYear | 2024 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref15 ref14 ref31 ref30 ref11 ref33 ref10 ref32 ref2 ref1 ref17 ref16 ref19 ref18 Coe (ref4) 2002; 12 Knill (ref24) 1996 ref23 ref26 ref25 ref20 ref22 ref21 ref28 ref27 ref29 ref8 ref7 ref9 ref3 ref6 ref5 |
| References_xml | – volume: 12 start-page: 14 volume-title: British Educational Research Association Annual Conference year: 2002 ident: ref4 article-title: Its the effect size, stupid – ident: ref1 doi: 10.1073/pnas.1915841117 – ident: ref10 doi: 10.1038/s41586-021-03659-0 – ident: ref5 doi: 10.1109/tvcg.2021.3114813 – ident: ref26 doi: 10.1145/3490486.3538338 – ident: ref32 doi: 10.1017/s0140525x20001685 – ident: ref2 doi: 10.1038/nrn3475 – ident: ref11 doi: 10.1109/tvcg.2019.2934287 – ident: ref23 doi: 10.1179/1743277414y.0000000099 – ident: ref16 doi: 10.1109/tvcg.2013.126 – ident: ref12 doi: 10.1162/99608f92.3ab8a587 – ident: ref25 doi: 10.1109/tvcg.2011.279 – ident: ref30 doi: 10.1145/1168149.1168158 – ident: ref33 doi: 10.1117/12.643631 – ident: ref15 doi: 10.1145/1377966.1377974 – ident: ref18 doi: 10.1109/TVCG.2023.3326516 – ident: ref3 doi: 10.1080/01621459.1984.10478080 – ident: ref19 doi: 10.1109/tvcg.2020.3030335 – ident: ref6 doi: 10.1145/3173574.3173718 – start-page: 825 volume-title: Perception as Bayesian Inference year: 1996 ident: ref24 doi: 10.1017/CBO9780511984037 – ident: ref8 doi: 10.1037/a0024558 – ident: ref20 doi: 10.1109/tvcg.2021.3114824 – ident: ref14 doi: 10.1371/journal.pone.0142444 – ident: ref28 doi: 10.1109/tvcg.2009.111 – ident: ref13 doi: 10.1109/tvcg.2018.2864889 – ident: ref17 doi: 10.1142/9789814417358_0006 – ident: ref22 doi: 10.1109/tvcg.2020.3028984 – ident: ref21 doi: 10.1145/2858036.2858558 – ident: ref27 doi: 10.1145/1600150.1600175 – ident: ref9 doi: 10.1109/tvcg.2020.3030395 – ident: ref7 doi: 10.1086/718371 – ident: ref29 doi: 10.1002/acp.2932 – ident: ref31 doi: 10.1109/visual.2005.1532781 |
| SSID | ssj0014489 |
| Score | 2.4297926 |
| Snippet | Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study... |
| SourceID | proquest pubmed crossref ieee |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 338 |
| SubjectTerms | Bayes methods Behavioral sciences Benchmark testing Data visualization decision-making Design of experiments Errors Evaluation Information retrieval rational agent Scientific visualization scoring rule Task analysis Uncertainty Visualization |
| Title | The Rational Agent Benchmark for Data Visualization |
| URI | https://ieeexplore.ieee.org/document/10290994 https://www.ncbi.nlm.nih.gov/pubmed/37871058 https://www.proquest.com/docview/2906587362 https://www.proquest.com/docview/2881247954 |
| Volume | 30 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LT8MwDLaAExx4DhgMVCROSC1p0kdyHI8xIcEBMcStStJMIGBD0F349dhNNwESiFulpkmaz5G_2LENcIiQk55gYZmVLET-70JjpQ2lynQuszzJSrJDXl1n_UFyeZ_eN8HqdSyMc66-fOYieqx9-eXYTshUhjucK2Q0yTzM48nNB2vNXAY4jvIXDPOQI01vXJgxU8e3d6cXEdUJjwSylTSm4jkCJRW5hfymj-oCK79zzVrn9Fbgejpbf9XkKZpUJrIfPxI5_vt3VmG5YZ9B14vLGsy50TosfclJuAECBSe4aUyEQZcir4ITFOWHF_32FCDFDc50pYO7x3cKx_RBnC0Y9M5vT_thU1khtCJlVUiswORGaiZNzoaZE9bomFwyzglu8-GQM4LMKivcMLGi1CbWXMWOSzwRcrEJC6PxyG1DkJaxsJlT2IFMnIoNJcTjWqbcskxZ3gY2Xd_CNmnHqfrFc1EfP5gqCJ2C0CkadNpwNPvk1efc-Ktxi1b2S0O_qG3oTFEsmm35XlBu-1TmqLTbcDB7jRuKvCR65MYTbCOJ8-QqxS62PPqzzqdCs_PLoLuwiHNLvImmAwvV28TtIWmpzH4trJ_3BuJC |
| linkProvider | IEEE |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LT-MwEB7xOOxy4LHLozyzEqeVEhw7D_vIu7x6WBXELbIdV6BCiyC98OuZidMKkFhxixTHdvyNNZ9nPDMAuwg56QkWllnJQuT_LjRW2lCqTOcyy5OsJDvkVSdrXyfnt-ltE6xex8I45-rLZy6ix9qXXw7tiExluMO5QkaTTMMsKv409uFaE6cBjqT8FcM85EjUGydmzNRe9-bwNKJK4ZFAvpLGVD5HoKwiu5AfNFJdYuVrtllrnZMF6Izn6y-b9KNRZSL7-imV47d_aBHmG_4Z7HuBWYIpN_gFc--yEv4GgaIT_GuMhME-xV4FByjMd4_6uR8gyQ2OdKWDm_sXCsj0YZzLcH1y3D1sh01thdCKlFUh8QKTG6mZNDnrZU5Yo2NyyjgnuM17Pc4INKuscL3EilKbWHMVOy7xTMjFCswMhgO3BkFaxsJmTmEHMnEqNpQSj2uZcssyZXkL2Hh9C9skHqf6Fw9FfQBhqiB0CkKnaNBpwd_JJ08-68b_Gi_Tyr5r6Be1BZtjFItmY74UlN0-lTmq7Rb8mbzGLUV-Ej1wwxG2kcR6cpViF6se_UnnY6FZ_2LQHfjR7l5dFpdnnYsN-InzTLzBZhNmqueR20IKU5ntWnDfAPGj5Ys |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Rational+Agent+Benchmark+for+Data+Visualization&rft.jtitle=IEEE+transactions+on+visualization+and+computer+graphics&rft.au=Wu%2C+Yifan&rft.au=Guo%2C+Ziyang&rft.au=Mamakos%2C+Michalis&rft.au=Hartline%2C+Jason&rft.date=2024-01-01&rft.issn=1941-0506&rft.eissn=1941-0506&rft.volume=30&rft.issue=1&rft.spage=338&rft_id=info:doi/10.1109%2FTVCG.2023.3326513&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1077-2626&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1077-2626&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1077-2626&client=summon |