Information Visualization Evaluation Using Crowdsourcing
Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little un...
Saved in:
| Published in | Computer graphics forum Vol. 37; no. 3; pp. 573 - 595 |
|---|---|
| Main Authors | , , , , |
| Format | Journal Article |
| Language | English |
| Published |
Oxford
Blackwell Publishing Ltd
01.06.2018
|
| Subjects | |
| Online Access | Get full text |
| ISSN | 0167-7055 1467-8659 |
| DOI | 10.1111/cgf.13444 |
Cover
Loading…
| Abstract | Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research. |
|---|---|
| AbstractList | Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research. |
| Author | Lee, B. Bach, B. Borgo, R. McGee, F. Micallef, L. |
| Author_xml | – sequence: 1 givenname: R. surname: Borgo fullname: Borgo, R. organization: Kings's College London – sequence: 2 givenname: L. surname: Micallef fullname: Micallef, L. organization: Aalto University – sequence: 3 givenname: B. surname: Bach fullname: Bach, B. organization: University of Edinburgh – sequence: 4 givenname: F. surname: McGee fullname: McGee, F. organization: Luxembourg Institute of Science and Technology (LIST) – sequence: 5 givenname: B. surname: Lee fullname: Lee, B. |
| BookMark | eNp1UMFOAjEUbAwmAnrwD0g8eVh4ZV-326PZAJKQeBGvTe22pGTZxXaR4NdbWU5G3-W9SWbmTWZAenVTG0LuKYxpnIne2DFNEfGK9ClmPMkzJnqkDzTeHBi7IYMQtgCAPGN9ki9r2_idal1Tj95cOKjKfXVo9qmqQ3eug6s3o8I3xzI0B68juiXXVlXB3F32kKzns9fiOVm9LJbF0yrRac4wEe9ZbrDMBXLIreUCrFCoUm2g5BpyFIIKylmJhlsskZVUTZUG0EwDapMOyUPnu_fNx8GEVm5jgjq-lFPIOAWkKY2sScfSvgnBGyu1a8_ZW69cJSnIn3pkrEee64mKx1-KvXc75U9_ci_uR1eZ0_9EWSzmneIbwrB12Q |
| CitedBy_id | crossref_primary_10_1109_TVCG_2024_3456341 crossref_primary_10_1016_j_infsof_2024_107545 crossref_primary_10_3389_fbinf_2023_1232671 crossref_primary_10_1016_j_ijhcs_2024_103343 crossref_primary_10_1109_TVCG_2020_3030365 crossref_primary_10_1515_ijeeps_2022_0118 crossref_primary_10_2308_JETA_2021_030 crossref_primary_10_1080_13658816_2024_2370076 crossref_primary_10_3390_su132212725 crossref_primary_10_1016_j_jsis_2020_101593 crossref_primary_10_1177_14738716231189218 crossref_primary_10_3390_info10120366 crossref_primary_10_1109_TVCG_2020_3030428 crossref_primary_10_1016_j_displa_2021_102059 crossref_primary_10_1109_TVCG_2019_2934784 crossref_primary_10_2200_S01094ED1V01Y202104VIS012 crossref_primary_10_1080_10447318_2024_2402571 crossref_primary_10_21605_cukurovaumfd_1334205 crossref_primary_10_1109_TVCG_2024_3456405 crossref_primary_10_3390_su14063616 crossref_primary_10_1007_s00779_019_01256_7 crossref_primary_10_1109_TVCG_2019_2903956 crossref_primary_10_1186_s40649_021_00101_3 crossref_primary_10_1177_14738716221126992 crossref_primary_10_3390_jtaer17040084 crossref_primary_10_1016_j_visinf_2024_10_005 crossref_primary_10_1109_TVCG_2020_3030396 crossref_primary_10_1109_TVCG_2021_3098240 crossref_primary_10_1109_TVCG_2022_3209491 crossref_primary_10_1007_s12650_023_00937_z crossref_primary_10_1111_cgf_13610 crossref_primary_10_3390_forecast5040035 crossref_primary_10_1007_s00371_019_01789_1 crossref_primary_10_1109_TVCG_2019_2934397 crossref_primary_10_1109_TVCG_2020_3029413 crossref_primary_10_1111_cgf_14329 crossref_primary_10_1109_TVCG_2021_3107297 crossref_primary_10_1177_14738716231180892 crossref_primary_10_1109_TVCG_2023_3326914 crossref_primary_10_1109_TCBB_2019_2938501 crossref_primary_10_1007_s10479_021_04465_7 crossref_primary_10_1177_2399808319873923 |
| Cites_doi | 10.1201/b10683 10.1145/3025453.3025969 10.1109/TVCG.2010.186 10.1109/TVCG.2012.199 10.1145/1926180.1926195 10.1109/PASSAT/SocialCom.2011.203 10.1145/2441776.2441847 10.1145/2835776.2835835 10.1145/2858036.2858237 10.1177/1071181311551228 10.1109/VAST.2011.6102445 10.1145/2660398.2660403 10.1109/TVCG.2011.279 10.1109/TVCG.2015.2467759 10.1145/2470654.2466424 10.1145/2858036.2858558 10.1145/2928269 10.1145/3131275 10.1145/2598153.2598168 10.1177/0272989X07304449 10.1007/s10791-012-9205-0 10.1109/TVCG.2015.2467758 10.1109/TVCG.2014.2315995 10.1145/2598153.2602249 10.1109/IV.2014.47 10.1007/978-3-319-66435-4_2 10.1111/cgf.12657 10.1145/3025453.3025781 10.1111/cgf.12634 10.1007/978-3-319-20267-9_14 10.1145/2702123.2702608 10.1145/1978942.1979193 10.1145/2702123.2702594 10.1007/978-3-319-66435-4_4 10.1109/QoMEX.2012.6263866 10.1007/978-3-319-66435-4_3 10.1145/2063576.2063860 10.1145/1978942.1979157 10.1518/155723409X448017 10.1145/2702123.2702545 10.1145/1869086.1869094 10.1145/2047196.2047199 10.1109/TVCG.2016.2599058 10.1109/CCE.2014.6916756 10.1109/VAST.2011.6102470 10.1109/TVCG.2010.174 10.1109/TVCG.2015.2467671 10.1145/2702613.2732934 10.1145/2380116.2380125 10.1109/TVCG.2012.236 10.1109/TVCG.2016.2598862 10.1145/3025453.3025477 10.1111/cgf.12233 10.1145/2556288.2557379 10.1109/TVCG.2016.2598543 10.1145/2598153.2598182 10.1109/TVCG.2017.2746018 10.1145/2858036.2858101 10.1109/MCG.2017.23 10.1145/3025453.3025922 10.1109/TVCG.2016.2599106 10.1007/978-3-319-66435-4_5 10.1145/2858036.2858063 10.1145/3119930 10.1109/TVCG.2014.2346979 10.17705/1jais.00441 10.1145/3027063.3053113 10.1145/2499149.2499168 10.1111/cgf.13009 10.1145/2702123.2702452 10.1145/1746259.1746260 10.1016/j.jebo.2013.03.003 10.1145/2598153.2602225 10.1109/PacificVis.2012.6183570 10.1007/978-3-540-70956-5_2 10.1109/TVCG.2013.164 10.1109/TVCG.2012.220 10.1017/CBO9780511844522 10.1145/2810012 10.1007/978-3-319-66435-4 10.1007/978-3-319-15168-7_56 10.1109/VAST.2010.5652890 10.1109/TVCG.2012.196 10.1109/TVCG.2013.234 10.1111/cgf.13195 10.1145/2702123.2702443 10.1109/TVCG.2008.155 10.1145/2470654.2481410 10.1145/2675133.2675246 10.1111/cgf.12127 10.1145/3025453.3025870 10.1109/TVCG.2017.2674978 10.1145/2598153.2602248 10.1111/cgf.12888 10.1109/TVCG.2016.2598591 10.1109/TVCG.2014.2346978 10.1007/978-3-319-24132-6_17 10.1109/TVCG.2014.2346320 10.1371/journal.pbio.1002456 10.1145/3025453.3025820 10.1145/2858036.2858440 10.1145/1240624.1240781 10.1145/989863.989880 10.1145/2470654.2481373 10.1145/2207676.2207709 10.1145/1978942.1978975 10.1145/2468356.2468529 10.1109/VAST.2014.7042496 10.1163/156856897X00357 10.1145/2858036.2858107 10.1145/2858036.2858300 10.1145/2631775.2631819 10.1016/j.jesp.2009.03.009 10.1109/TVCG.2013.183 10.1109/VAST.2012.6400540 10.1145/2207676.2208556 10.1609/hcomp.v1i1.13088 10.1145/1958824.1958865 10.1145/2591677 10.1145/2669557.2669564 10.1145/1753326.1753357 10.1145/2818048.2820005 10.1145/2441776.2441912 10.1145/2556288.2557200 10.1145/2506364.2506366 10.1145/3025453.3025917 10.1109/T-AFFC.2012.19 10.1145/2858036.2858280 10.5194/isprsannals-II-3-W5-325-2015 10.1145/2531602.2531718 10.1145/3025453.3025592 10.1145/1837885.1837889 10.1145/2470654.2481281 10.1145/2810188.2810192 10.1109/TVCG.2013.119 10.1145/2541288 10.1109/TVCG.2015.2467451 10.1109/TVCG.2015.2500240 10.1364/JOSA.35.000268 10.1145/2488388.2488489 10.1145/1453152.1453156 10.1109/TVCG.2012.230 10.1109/TVCG.2015.2467201 10.1145/1314683.1314684 10.1145/1963405.1963473 10.1109/PACIFICVIS.2017.8031598 10.1109/TVCG.2012.234 10.1109/TVCG.2013.247 10.1111/cgf.12387 10.1201/b17511 10.1111/cgf.13018 10.1109/TVCG.2014.2346575 10.1111/j.1467-8659.2009.01442.x 10.1109/PACIFICVIS.2016.7465249 |
| ContentType | Journal Article |
| Copyright | 2018 The Author(s) Computer Graphics Forum © 2018 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd. 2018 The Eurographics Association and John Wiley & Sons Ltd. |
| Copyright_xml | – notice: 2018 The Author(s) Computer Graphics Forum © 2018 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd. – notice: 2018 The Eurographics Association and John Wiley & Sons Ltd. |
| DBID | AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| DOI | 10.1111/cgf.13444 |
| DatabaseName | CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | CrossRef Computer and Information Systems Abstracts |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1467-8659 |
| EndPage | 595 |
| ExternalDocumentID | 10_1111_cgf_13444 CGF13444 |
| Genre | article |
| GroupedDBID | .3N .4S .DC .GA .Y3 05W 0R~ 10A 15B 1OB 1OC 29F 31~ 33P 3SF 4.4 50Y 50Z 51W 51X 52M 52N 52O 52P 52S 52T 52U 52W 52X 5GY 5HH 5LA 5VS 66C 6J9 702 7PT 8-0 8-1 8-3 8-4 8-5 8UM 8VB 930 A03 AAESR AAEVG AAHHS AAHQN AAMNL AANHP AANLZ AAONW AASGY AAXRX AAYCA AAZKR ABCQN ABCUV ABDBF ABDPE ABEML ABPVW ACAHQ ACBWZ ACCFJ ACCZN ACFBH ACGFS ACPOU ACRPL ACSCC ACUHS ACXBN ACXQS ACYXJ ADBBV ADEOM ADIZJ ADKYN ADMGS ADNMO ADOZA ADXAS ADZMN ADZOD AEEZP AEGXH AEIGN AEIMD AEMOZ AENEX AEQDE AEUQT AEUYR AFBPY AFEBI AFFNX AFFPM AFGKR AFPWT AFWVQ AFZJQ AHBTC AHEFC AHQJS AITYG AIURR AIWBW AJBDE AJXKR AKVCP ALAGY ALMA_UNASSIGNED_HOLDINGS ALUQN ALVPJ AMBMR AMYDB ARCSS ASPBG ATUGU AUFTA AVWKF AZBYB AZFZN AZVAB BAFTC BDRZF BFHJK BHBCM BMNLL BMXJE BNHUX BROTX BRXPI BY8 CAG COF CS3 CWDTD D-E D-F DCZOG DPXWK DR2 DRFUL DRSTM DU5 EAD EAP EBA EBO EBR EBS EBU EDO EJD EMK EST ESX F00 F01 F04 F5P FEDTE FZ0 G-S G.N GODZA H.T H.X HF~ HGLYW HVGLF HZI HZ~ I-F IHE IX1 J0M K1G K48 LATKE LC2 LC3 LEEKS LH4 LITHE LOXES LP6 LP7 LUTES LW6 LYRES MEWTI MK4 MRFUL MRSTM MSFUL MSSTM MXFUL MXSTM N04 N05 N9A NF~ O66 O9- OIG P2W P2X P4D PALCI PQQKQ Q.N Q11 QB0 QWB R.K RDJ RIWAO RJQFR ROL RX1 SAMSI SUPJJ TH9 TN5 TUS UB1 V8K W8V W99 WBKPD WIH WIK WOHZO WQJ WRC WXSBR WYISQ WZISG XG1 ZL0 ZZTAW ~IA ~IF ~WT AAMMB AAYXX ADMLS AEFGJ AEYWJ AGHNM AGQPQ AGXDD AGYGG AIDQK AIDYY CITATION 7SC 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c3854-9b68e4d894708ff790f9a4a3ce0d7c0849919175d4e7f4d45d1a2ac00c5c04ce3 |
| IEDL.DBID | DR2 |
| ISSN | 0167-7055 |
| IngestDate | Fri Jul 25 05:53:26 EDT 2025 Thu Jul 10 09:50:55 EDT 2025 Thu Apr 24 22:57:05 EDT 2025 Wed Jan 22 16:43:28 EST 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c3854-9b68e4d894708ff790f9a4a3ce0d7c0849919175d4e7f4d45d1a2ac00c5c04ce3 |
| Notes | First and second author share equal contribution to the work and are presented in no particular order. ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| OpenAccessLink | https://www.research.ed.ac.uk/portal/en/publications/information-visualization-evaluation-using-crowdsourcing(5e6ddd37-26d7-4f11-9ea5-cf48681dd9d0).html |
| PQID | 2067104131 |
| PQPubID | 30877 |
| PageCount | 23 |
| ParticipantIDs | proquest_journals_2067104131 crossref_citationtrail_10_1111_cgf_13444 crossref_primary_10_1111_cgf_13444 wiley_primary_10_1111_cgf_13444_CGF13444 |
| PublicationCentury | 2000 |
| PublicationDate | June 2018 |
| PublicationDateYYYYMMDD | 2018-06-01 |
| PublicationDate_xml | – month: 06 year: 2018 text: June 2018 |
| PublicationDecade | 2010 |
| PublicationPlace | Oxford |
| PublicationPlace_xml | – name: Oxford |
| PublicationTitle | Computer graphics forum |
| PublicationYear | 2018 |
| Publisher | Blackwell Publishing Ltd |
| Publisher_xml | – name: Blackwell Publishing Ltd |
| References | 2009; 45 2015; 34 2010; 16 2010; 17 1945; 35 2012; 18 2016; 2016 2016; 35 2014; 21 2014; 20 2013; 19 2015; 9394 2013; 16 2017; 37 1997; 10 2017; 36 2007; 27 1976; 102 2015; 2 2012 2011 2010 2017; 24 2010; 200 2017; 23 2009 2008; 14 2011; 31 2008 2013; 90 2008; 15 2007 1996 2006 2005 2004 2016; 17 2016; 14 2018; 24 2009; 28 2012; 3 2013; 32 2015; 22 2017 2016 2015 2014 2009; 5 2013 2014; 33 2016; 23 2016; 22 e_1_2_10_44_2 e_1_2_10_21_2 e_1_2_10_109_2 Glassner A. (e_1_2_10_56_2) 2009 e_1_2_10_131_2 e_1_2_10_154_2 e_1_2_10_177_2 Chung D. H. (e_1_2_10_22_2) 2016; 2016 e_1_2_10_158_2 Hirth M. (e_1_2_10_66_2) 2017 e_1_2_10_70_2 e_1_2_10_116_2 e_1_2_10_139_2 e_1_2_10_93_2 e_1_2_10_18_2 e_1_2_10_74_2 e_1_2_10_112_2 e_1_2_10_135_2 e_1_2_10_150_2 e_1_2_10_173_2 e_1_2_10_37_2 e_1_2_10_97_2 e_1_2_10_78_2 e_1_2_10_6_2 e_1_2_10_14_2 e_1_2_10_79_2 e_1_2_10_32_2 Lebreton P. (e_1_2_10_110_2) 2015; 9394 e_1_2_10_51_2 e_1_2_10_120_2 e_1_2_10_143_2 e_1_2_10_166_2 Ekstrom R. B. (e_1_2_10_40_2) 1976 e_1_2_10_147_2 e_1_2_10_105_2 e_1_2_10_128_2 e_1_2_10_82_2 e_1_2_10_29_2 e_1_2_10_63_2 Kosara R. (e_1_2_10_101_2) 2016 e_1_2_10_124_2 e_1_2_10_162_2 e_1_2_10_48_2 e_1_2_10_86_2 e_1_2_10_67_2 e_1_2_10_45_2 e_1_2_10_68_2 e_1_2_10_41_2 e_1_2_10_129_2 Wang J. (e_1_2_10_161_2) 2011; 31 Oppenheimer D. M. (e_1_2_10_126_2) 2009; 45 e_1_2_10_155_2 e_1_2_10_178_2 e_1_2_10_90_2 e_1_2_10_159_2 e_1_2_10_19_2 e_1_2_10_71_2 e_1_2_10_94_2 e_1_2_10_117_2 e_1_2_10_3_2 Martin D. (e_1_2_10_115_2) 2017 e_1_2_10_52_2 e_1_2_10_136_2 e_1_2_10_170_2 e_1_2_10_15_2 e_1_2_10_38_2 e_1_2_10_75_2 e_1_2_10_98_2 e_1_2_10_151_2 e_1_2_10_7_2 e_1_2_10_132_2 e_1_2_10_174_2 e_1_2_10_57_2 e_1_2_10_33_2 e_1_2_10_10_2 e_1_2_10_118_2 e_1_2_10_144_2 e_1_2_10_167_2 e_1_2_10_148_2 e_1_2_10_60_2 e_1_2_10_83_2 e_1_2_10_106_2 e_1_2_10_125_2 Borgo R. (e_1_2_10_16_2) 2017 e_1_2_10_26_2 e_1_2_10_49_2 e_1_2_10_64_2 e_1_2_10_87_2 e_1_2_10_102_2 e_1_2_10_140_2 e_1_2_10_121_2 e_1_2_10_163_2 e_1_2_10_23_2 e_1_2_10_69_2 e_1_2_10_42_2 Gadiraju U. (e_1_2_10_58_2) 2017 e_1_2_10_107_2 MacEachren A. M. (e_1_2_10_113_2) 2004 e_1_2_10_156_2 e_1_2_10_91_2 e_1_2_10_72_2 e_1_2_10_114_2 e_1_2_10_137_2 e_1_2_10_171_2 e_1_2_10_39_2 e_1_2_10_95_2 e_1_2_10_53_2 e_1_2_10_4_2 e_1_2_10_76_2 e_1_2_10_133_2 e_1_2_10_152_2 e_1_2_10_175_2 e_1_2_10_35_2 e_1_2_10_99_2 e_1_2_10_11_2 e_1_2_10_34_2 e_1_2_10_8_2 e_1_2_10_30_2 e_1_2_10_119_2 e_1_2_10_145_2 e_1_2_10_168_2 e_1_2_10_149_2 e_1_2_10_80_2 e_1_2_10_61_2 e_1_2_10_103_2 e_1_2_10_160_2 e_1_2_10_84_2 e_1_2_10_27_2 e_1_2_10_65_2 e_1_2_10_122_2 e_1_2_10_141_2 e_1_2_10_164_2 e_1_2_10_46_2 e_1_2_10_88_2 e_1_2_10_20_2 e_1_2_10_43_2 e_1_2_10_108_2 e_1_2_10_130_2 e_1_2_10_176_2 e_1_2_10_157_2 e_1_2_10_92_2 e_1_2_10_138_2 e_1_2_10_17_2 e_1_2_10_73_2 e_1_2_10_96_2 e_1_2_10_5_2 e_1_2_10_54_2 e_1_2_10_134_2 e_1_2_10_172_2 e_1_2_10_13_2 e_1_2_10_36_2 e_1_2_10_77_2 e_1_2_10_111_2 e_1_2_10_153_2 e_1_2_10_9_2 e_1_2_10_12_2 e_1_2_10_59_2 e_1_2_10_31_2 Andrienko N. (e_1_2_10_2_2) 2005 e_1_2_10_50_2 e_1_2_10_165_2 e_1_2_10_146_2 e_1_2_10_169_2 e_1_2_10_81_2 Carpendale S. (e_1_2_10_25_2) 2008 e_1_2_10_127_2 e_1_2_10_28_2 e_1_2_10_62_2 e_1_2_10_85_2 e_1_2_10_104_2 e_1_2_10_123_2 e_1_2_10_24_2 e_1_2_10_47_2 e_1_2_10_89_2 Gawande A. (e_1_2_10_55_2) 2010 e_1_2_10_100_2 e_1_2_10_142_2 |
| References_xml | – year: 2011 – volume: 23 start-page: 631 issue: 1 year: 2017 end-page: 640 article-title: Optimizing hierarchical visualizations with the minimum description length principle publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 21 start-page: 6 issue: 1 year: 2014 article-title: Understand usersâĂŹ comprehension and preferences for composing information visualizations publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – start-page: 1157 year: 2013 end-page: 1166 – start-page: 398 year: 2014 end-page: 400 – start-page: 1349 year: 2015 end-page: 1354 – year: 2005 – start-page: 1142 year: 2017 end-page: 1153 – volume: 17 start-page: 16 issue: 2 year: 2010 end-page: 21 article-title: Analyzing the amazon mechanical turk marketplace publication-title: XRDS: Crossroads, The ACM Magazine for Students – volume: 22 start-page: 469 issue: 1 year: 2016 end-page: 478 article-title: Beyond weber's law: A second look at ranking visualizations of correlation publication-title: IEEE transactions on visualization and computer graphics – volume: 28 start-page: 911 year: 2009 end-page: 918 article-title: Preconceptions and individual differences in understanding visual metaphors publication-title: Computer Graphics Forum – year: 2016 article-title: Judgment Error in Pie Chart Variations – start-page: 142 year: 2015 end-page: 151 – start-page: 5580 year: 2017 end-page: 5591 – start-page: 405 year: 2014 end-page: 408 – start-page: 19 year: 2008 end-page: 45 article-title: Evaluating information visualizations – volume: 19 start-page: 2198 issue: 12 year: 2013 end-page: 2206 article-title: Identifying redundancy and exposing provenance in crowdsourced data analysis publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 23 start-page: 351 issue: 1 year: 2017 end-page: 360 article-title: Hindsight: encouraging exploration through direct encoding of personal interaction history publication-title: IEEE transactions on visualization and computer graphics – start-page: 1 year: 2014 end-page: 11 – start-page: 5028 year: 2016 end-page: 5033 – start-page: 13 year: 2012 end-page: 18 – year: 2014 – start-page: 1609 year: 2016 end-page: 1614 – start-page: 1445 year: 2014 end-page: 1455 – start-page: 336 year: 1996 end-page: 343 – start-page: 25 year: 2012 end-page: 32 – start-page: 193 year: 2014 end-page: 200 – volume: 27 start-page: 672 issue: 5 year: 2007 end-page: 680 article-title: Measuring numeracy without a math test: development of the subjective numeracy scale publication-title: Medical Decision Making – start-page: 1375 year: 2015 end-page: 1378 – volume: 22 start-page: 489 issue: 1 year: 2016 end-page: 498 article-title: Guidelines for effective usage of text highlighting techniques publication-title: IEEE transactions on visualization and computer graphics – start-page: 639 year: 2013 end-page: 648 – volume: 35 start-page: 209 year: 2016 end-page: 216 article-title: Image recoloring with valence‐arousal emotion model publication-title: Computer Graphics Forum – volume: 16 start-page: 138 issue: 2 year: 2013 end-page: 178 article-title: An analysis of human factors and label accuracy in crowdsourcing relevance judgments publication-title: Information retrieval – start-page: 12 year: 2010 – start-page: 3551 year: 2017 end-page: 3556 – volume: 14 start-page: 1333 issue: 6 year: 2008 end-page: 1339 article-title: Perceptual organization in user‐generated graph layouts publication-title: Visualization and Computer Graphics, IEEE Transactions on – start-page: 1029 year: 2007 end-page: 1038 – start-page: 1717 year: 2011 end-page: 1726 – volume: 33 start-page: 311 issue: 3 year: 2014 end-page: 320 article-title: Glyphs for exploring crowd‐sourced subjective survey classification publication-title: Computer Graphics Forum – start-page: 551 year: 2014 end-page: 560 – volume: 22 start-page: 27 issue: 6 year: 2015 article-title: Visualizing sets with linear diagrams publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – volume: 14 start-page: 16 issue: 4 year: 2008 article-title: Proactive displays: Supporting awareness in fluid social environments publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – volume: 21 start-page: 14 issue: 3 year: 2014 article-title: Using metrics of curation to evaluate information‐based ideation publication-title: ACM Transactions on Computer‐Human Interaction (ToCHI) – volume: 17 start-page: 648 year: 2016 end-page: 673 article-title: Why individuals participate in microtask crowdsourcing work environment: Revealing crowdworkers’ perceptions publication-title: Journal of the Association for Information Systems – start-page: 2049 year: 2013 end-page: 2058 – start-page: 147 year: 2014 end-page: 151 – start-page: 96 year: 2017 end-page: 138 article-title: Crowdsourcing for information visualization: Promises and pitfalls – start-page: 245 year: 2014 end-page: 252 – volume: 90 start-page: 123 year: 2013 end-page: 133 article-title: Breaking monotony with meaning: Motivation in crowdsourcing markets publication-title: Journal of Economic Behavior & Organization – start-page: 121 year: 2016 end-page: 130 – start-page: 1095 year: 2012 end-page: 1104 – volume: 20 start-page: 1933 issue: 12 year: 2014 end-page: 1942 article-title: Learning perceptual kernels for visualization design publication-title: Visualization and Computer Graphics, IEEE Transactions on – start-page: 477 year: 2011 end-page: 486 – volume: 200 year: 2010 – volume: 16 start-page: 1009 issue: 6 year: 2010 end-page: 1016 article-title: Laws of attraction: From perceptual forces to conceptual similarity publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 22 start-page: 639 issue: 1 year: 2016 end-page: 648 article-title: Suggested interactivity: Seeking perceived affordances for information visualization publication-title: IEEE transactions on visualization and computer graphics – start-page: 227 year: 2011 end-page: 236 – year: 2016 – year: 2010 – start-page: 109 year: 2004 end-page: 116 – volume: 35 start-page: 268 issue: 4 year: 1945 end-page: 275 article-title: Tests for the detection and analysis of color‐blindness. i. the ishihara test: an evaluation publication-title: JOSA – start-page: 401 year: 2014 end-page: 404 – volume: 23 start-page: 431 issue: 1 year: 2017 end-page: 440 article-title: Evaluating the impact of binning 2d scalar fields publication-title: IEEE transactions on visualization and computer graphics – start-page: 1461 year: 2011 end-page: 1470 – start-page: 227 year: 2012 end-page: 228 – volume: 23 start-page: 1588 issue: 6 year: 2017 end-page: 1599 article-title: Towards perceptual optimization of the visual design of scatterplots publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 24 start-page: 30 issue: 4 year: 2017 article-title: Using worker self‐assessments for competence‐based preselection in crowdsourcing microtasks publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – start-page: 887 year: 2014 end-page: 896 – volume: 18 start-page: 2441 year: 2012 end-page: 2448 article-title: Graphical tests for power comparison of competing designs publication-title: IEEE Transactions on Visualization And Computer Graphics – start-page: 1375 year: 2017 end-page: 1386 – start-page: 5092 year: 2016 end-page: 5103 – start-page: 273 year: 2011 end-page: 274 – start-page: 3 year: 2015 end-page: 8 – volume: 2016 year: 2016 article-title: How ordered is it? on the perceptual orderability of visual channels publication-title: EuroVis – volume: 19 start-page: 2306 issue: 12 year: 2013 end-page: 2315 article-title: What makes a visualization memorable? publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 23 start-page: 19 issue: 3 year: 2016 article-title: Diminished control in crowdsourcing: an investigation of crowdworker multitasking behavior publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – volume: 14 issue: 5 year: 2016 article-title: Badges to acknowledge open practices: a simple, low‐cost, effective method for increasing transparency publication-title: PLoS biology – volume: 22 start-page: 349 issue: 1 year: 2016 end-page: 358 article-title: Hola: Human‐like orthogonal network layout publication-title: IEEE transactions on visualization and computer graphics – volume: 16 start-page: 990 issue: 6 year: 2010 end-page: 998 article-title: Perceptual guidelines for creating rectangular treemaps publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 35 start-page: 117 year: 2016 end-page: 126 article-title: A study on designing effective introductory materials for information visualization publication-title: Computer Graphics Forum – volume: 20 start-page: 1943 issue: 12 year: 2014 end-page: 1952 article-title: Ranking visualizations of correlation using weber's law publication-title: IEEE transactions on visualization and computer graphics – start-page: 27 year: 2017 end-page: 69 article-title: Understanding the crowd: Ethical and practical matters in the academic use of crowdsourcing – volume: 15 start-page: 13 issue: 3 year: 2008 article-title: Performing perceptionâĂŤstaging aesthetics of interaction publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – year: 2013 – start-page: 766 year: 2011 end-page: 773 – year: 2009 – start-page: 1364 year: 2015 end-page: 1378 – start-page: 1 year: 2010 end-page: 9 – volume: 20 start-page: 1530 issue: 11 year: 2014 end-page: 1541 article-title: How to display group information on node‐link diagrams: An evaluation publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 1090 year: 2011 end-page: 1094 – start-page: 23 year: 2013 end-page: 28 – start-page: 1187 year: 2015 end-page: 1190 – volume: 20 start-page: 935 issue: 6 year: 2014 end-page: 943 article-title: The perception of visual uncertaintyrepresentation by non‐experts publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 967 year: 2013 end-page: 972 – start-page: 1708 year: 2017 end-page: 1717 – volume: 19 start-page: 2406 issue: 12 year: 2013 end-page: 2415 article-title: A deeper understanding of sequence in narrative visualization publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 40 year: 2016 end-page: 47 – volume: 23 start-page: 391 issue: 1 year: 2017 end-page: 400 article-title: Map lineups: effects of spatial structure on graphical inference publication-title: IEEE transactions on visualization and computer graphics – volume: 37 start-page: 12 issue: 2 year: 2017 end-page: 18 article-title: Expanding research methods for a realistic understanding of personal visualization publication-title: IEEE computer graphics and applications – start-page: 1449 year: 2015 end-page: 1458 – start-page: 3101 year: 2013 end-page: 3110 – start-page: 70 year: 2017 end-page: 95 article-title: Crowdsourcing technology to support academic research – start-page: 5421 year: 2016 end-page: 5432 – start-page: 1941 year: 2011 end-page: 1944 – volume: 18 start-page: 1520 issue: 9 year: 2012 end-page: 1536 article-title: Empirical studies in information visualization: Seven scenarios publication-title: IEEE transactions on visualization and computer graphics – start-page: 5475 year: 2017 end-page: 5484 – volume: 18 start-page: 2104 issue: 12 year: 2012 end-page: 2113 article-title: Human computation in visualization: Using purpose driven games for robust evaluation of visualization algorithms publication-title: IEEE transactions on visualization and computer graphics – start-page: 510 year: 2014 end-page: 515 – volume: 20 start-page: 1663 issue: 12 year: 2014 end-page: 1672 article-title: Finding waldo: Learning about users from their interactions publication-title: Visualization and Computer Graphics, IEEE Transactions on – start-page: 203 year: 2014 end-page: 212 – start-page: 993 year: 2016 end-page: 997 – start-page: 2697 year: 2013 end-page: 2706 – year: 2004 – volume: 32 start-page: 401 year: 2013 end-page: 410 article-title: Selecting semantically‐resonant colors for data visualization publication-title: Computer Graphics Forum – volume: 24 start-page: 657 issue: 1 year: 2018 end-page: 666 article-title: Taking word clouds apart: An empirical investigation of the design space for keyword summaries publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 227 year: 2012 end-page: 236 – volume: 23 start-page: 261 issue: 1 year: 2017 end-page: 270 article-title: Annotation graphs: A graph‐based visualization for meta‐analysis of data based on user‐authored annotations publication-title: IEEE transactions on visualization and computer graphics – volume: 24 start-page: 36 issue: 5 year: 2017 article-title: Bubbleview: an interface for crowdsourcing image importance maps and tracking visual attention publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – start-page: 461 year: 2014 end-page: 468 – start-page: 218 year: 2014 end-page: 223 – volume: 9394 year: 2015 article-title: Bridging the gap between eye tracking and crowdsourcing publication-title: Human Vision and Electronic Imaging XX – start-page: 55 year: 2012 end-page: 62 – year: 2015 – volume: 102 year: 1976 – start-page: 6813 year: 2017 end-page: 6824 – volume: 18 start-page: 2613 issue: 12 year: 2012 end-page: 2620 article-title: An empirical model of slope ratio comparisons publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 14 year: 2013 – start-page: 169 year: 2015 end-page: 182 – start-page: 13 year: 2011 end-page: 22 – start-page: 1387 year: 2017 end-page: 1396 – volume: 22 start-page: 2107 issue: 9 year: 2016 end-page: 2121 article-title: Lightness constancy in surface visualization publication-title: IEEE Transactions on Visualization and Computer Graphics – volume: 10 start-page: 433 issue: 4 year: 1997 end-page: 436 article-title: The Psychophysics Toolbox publication-title: Spatial Vision – start-page: 1469 year: 2015 end-page: 1478 – start-page: 3511 year: 2017 end-page: 3522 – start-page: 241 year: 2014 end-page: 250 – volume: 2 start-page: 325 year: 2015 article-title: Pitfalls and potentials of crowd science: A meta‐analysis of contextual influences publication-title: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences – volume: 18 start-page: 2769 year: 2012 end-page: 2778 article-title: Evaluating sketchiness as a visual variable for the depiction of qualitative uncertainty publication-title: IEEE Transactions on Visualization & Computer Graphics – volume: 36 start-page: 377 year: 2017 end-page: 387 article-title: Visual narrative flow: Exploring factors shaping data visualization story reading experiences publication-title: Computer Graphics Forum – volume: 22 start-page: 529 issue: 1 year: 2016 end-page: 538 article-title: Improving bayesian reasoning: the effects of phrasing, visualization, and spatial ability publication-title: IEEE transactions on visualization and computer graphics – volume: 34 start-page: 221 year: 2015 end-page: 230 article-title: An evaluation of the impact of visual embellishments in bar charts publication-title: Computer Graphics Forum – volume: 45 issue: 4 year: 2009 article-title: Instructional manipulation checks: Detecting satisficing to increase statistical power publication-title: Journal of Experimental Social Psychology – start-page: 3143 year: 2016 end-page: 3154 – volume: 19 start-page: 2316 issue: 12 year: 2013 end-page: 2325 article-title: Perception of average value in multiclass scatterplots publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 38 year: 2016 end-page: 48 – volume: 32 start-page: 255 year: 2013 end-page: 264 article-title: Learning and applying color styles from feature films publication-title: Computer Graphics Forum – start-page: 1631 year: 2015 end-page: 1640 – start-page: 1203 year: 2013 end-page: 1212 – volume: 20 start-page: 2152 issue: 12 year: 2014 end-page: 2160 article-title: Four experiments on the perception of bar charts publication-title: IEEE transactions on visualization and computer graphics – year: 2012 – volume: 18 start-page: 2536 issue: 12 year: 2012 end-page: 2545 article-title: Assessing the effect of visualizations on bayesian reasoning through crowdsourcing publication-title: IEEE Transactions on Visualization and Computer Graphics – start-page: 225 year: 2016 end-page: 236 – volume: 34 start-page: 451 year: 2015 end-page: 460 article-title: Graphunit: Evaluating interactive graph visualizations using crowdsourcing publication-title: Computer Graphics Forum – start-page: 5444 year: 2016 end-page: 5455 – start-page: 163 year: 2010 end-page: 170 – start-page: 6 year: 2017 end-page: 26 article-title: Crowdsourcing versus the laboratory: Towards human‐centered experiments using the crowd – year: 2006 – start-page: 81 year: 2011 end-page: 90 – volume: 17 start-page: 6 issue: 2 year: 2010 article-title: Personalization via friendsourcing publication-title: ACM Transactions on Computer‐Human Interaction (TOCHI) – start-page: 2949 year: 2013 end-page: 2958 – volume: 31 year: 2011 article-title: Estimating the completion time of crowdsourced tasks using survival analysis models publication-title: Crowdsourcing for search and data mining (CSDM 2011) – start-page: 267 year: 2016 end-page: 276 – volume: 18 start-page: 2699 issue: 12 year: 2012 end-page: 2708 article-title: Intelligent graph layout using many users’ input publication-title: IEEE Transactions on Visualization and Computer Graphics – year: 2017 – start-page: 220 year: 2017 end-page: 229 – volume: 3 start-page: 456 issue: 4 year: 2012 end-page: 468 article-title: Crowdsourcing facial responses to online videos publication-title: IEEE Transactions on Affective Computing – volume: 5 start-page: 41 issue: 1 year: 2009 end-page: 81 article-title: Selected human factors issues in information visualization publication-title: Reviews of human factors and ergonomics – ident: e_1_2_10_162_2 doi: 10.1201/b10683 – ident: e_1_2_10_31_2 doi: 10.1145/3025453.3025969 – ident: e_1_2_10_89_2 doi: 10.1109/TVCG.2010.186 – ident: e_1_2_10_65_2 – ident: e_1_2_10_116_2 doi: 10.1109/TVCG.2012.199 – ident: e_1_2_10_99_2 doi: 10.1145/1926180.1926195 – volume-title: Exploratory Analysis of Spatial and Temporal Data: A Systematic Approach year: 2005 ident: e_1_2_10_2_2 – ident: e_1_2_10_170_2 doi: 10.1109/PASSAT/SocialCom.2011.203 – ident: e_1_2_10_63_2 doi: 10.1145/2441776.2441847 – ident: e_1_2_10_105_2 doi: 10.1145/2835776.2835835 – ident: e_1_2_10_28_2 doi: 10.1145/2858036.2858237 – ident: e_1_2_10_98_2 doi: 10.1177/1071181311551228 – ident: e_1_2_10_17_2 – ident: e_1_2_10_172_2 doi: 10.1109/VAST.2011.6102445 – ident: e_1_2_10_57_2 doi: 10.1145/2660398.2660403 – ident: e_1_2_10_106_2 doi: 10.1109/TVCG.2011.279 – ident: e_1_2_10_148_2 doi: 10.1109/TVCG.2015.2467759 – ident: e_1_2_10_108_2 doi: 10.1145/2470654.2466424 – ident: e_1_2_10_92_2 doi: 10.1145/2858036.2858558 – ident: e_1_2_10_48_2 doi: 10.1145/2928269 – ident: e_1_2_10_84_2 doi: 10.1145/3131275 – ident: e_1_2_10_13_2 doi: 10.1145/2598153.2598168 – ident: e_1_2_10_47_2 doi: 10.1177/0272989X07304449 – ident: e_1_2_10_95_2 doi: 10.1007/s10791-012-9205-0 – ident: e_1_2_10_127_2 doi: 10.1109/TVCG.2015.2467758 – ident: e_1_2_10_81_2 doi: 10.1109/TVCG.2014.2315995 – ident: e_1_2_10_85_2 doi: 10.1145/2598153.2602249 – ident: e_1_2_10_124_2 doi: 10.1109/IV.2014.47 – start-page: 6 volume-title: Evaluation in the Crowd. Crowdsourcing and Human‐Centered Experiments year: 2017 ident: e_1_2_10_58_2 doi: 10.1007/978-3-319-66435-4_2 – ident: e_1_2_10_125_2 doi: 10.1111/cgf.12657 – ident: e_1_2_10_52_2 – ident: e_1_2_10_51_2 doi: 10.1145/3025453.3025781 – ident: e_1_2_10_143_2 doi: 10.1111/cgf.12634 – ident: e_1_2_10_157_2 – ident: e_1_2_10_123_2 doi: 10.1007/978-3-319-20267-9_14 – ident: e_1_2_10_130_2 doi: 10.1145/2702123.2702608 – ident: e_1_2_10_39_2 doi: 10.1145/1978942.1979193 – volume: 2016 year: 2016 ident: e_1_2_10_22_2 article-title: How ordered is it? on the perceptual orderability of visual channels publication-title: EuroVis – ident: e_1_2_10_111_2 doi: 10.1145/2702123.2702594 – start-page: 70 volume-title: Evaluation in the Crowd. Crowdsourcing and Human‐Centered Experiments year: 2017 ident: e_1_2_10_66_2 doi: 10.1007/978-3-319-66435-4_4 – ident: e_1_2_10_91_2 doi: 10.1109/QoMEX.2012.6263866 – start-page: 27 volume-title: Evaluation in the Crowd. Crowdsourcing and Human‐Centered Experiments year: 2017 ident: e_1_2_10_115_2 doi: 10.1007/978-3-319-66435-4_3 – ident: e_1_2_10_94_2 doi: 10.1145/2063576.2063860 – ident: e_1_2_10_59_2 doi: 10.1145/1978942.1979157 – ident: e_1_2_10_133_2 doi: 10.1518/155723409X448017 – ident: e_1_2_10_168_2 – ident: e_1_2_10_70_2 doi: 10.1145/2702123.2702545 – ident: e_1_2_10_79_2 doi: 10.1145/1869086.1869094 – volume-title: Manual for kit of factor‐referenced cognitive tests year: 1976 ident: e_1_2_10_40_2 – ident: e_1_2_10_137_2 doi: 10.1145/2047196.2047199 – ident: e_1_2_10_42_2 doi: 10.1109/TVCG.2016.2599058 – ident: e_1_2_10_73_2 doi: 10.1109/CCE.2014.6916756 – volume: 9394 year: 2015 ident: e_1_2_10_110_2 article-title: Bridging the gap between eye tracking and crowdsourcing publication-title: Human Vision and Electronic Imaging XX – ident: e_1_2_10_36_2 doi: 10.1109/VAST.2011.6102470 – ident: e_1_2_10_177_2 doi: 10.1109/TVCG.2010.174 – ident: e_1_2_10_88_2 doi: 10.1109/TVCG.2015.2467671 – ident: e_1_2_10_80_2 – ident: e_1_2_10_83_2 doi: 10.1145/2702613.2732934 – ident: e_1_2_10_138_2 doi: 10.1145/2380116.2380125 – ident: e_1_2_10_169_2 doi: 10.1109/TVCG.2012.236 – ident: e_1_2_10_11_2 doi: 10.1109/TVCG.2016.2598862 – ident: e_1_2_10_165_2 doi: 10.1145/3025453.3025477 – ident: e_1_2_10_167_2 doi: 10.1111/cgf.12233 – ident: e_1_2_10_132_2 doi: 10.1145/2556288.2557379 – ident: e_1_2_10_174_2 doi: 10.1109/TVCG.2016.2598543 – ident: e_1_2_10_8_2 doi: 10.1145/2598153.2598182 – ident: e_1_2_10_43_2 doi: 10.1109/TVCG.2017.2746018 – ident: e_1_2_10_96_2 doi: 10.1145/2858036.2858101 – ident: e_1_2_10_153_2 doi: 10.1109/MCG.2017.23 – ident: e_1_2_10_26_2 doi: 10.1145/3025453.3025922 – ident: e_1_2_10_129_2 doi: 10.1109/TVCG.2016.2599106 – ident: e_1_2_10_32_2 – volume-title: The checklist manifesto: How to get things right year: 2010 ident: e_1_2_10_55_2 – start-page: 96 volume-title: Evaluation in the Crowd. Crowdsourcing and Human‐Centered Experiments year: 2017 ident: e_1_2_10_16_2 doi: 10.1007/978-3-319-66435-4_5 – ident: e_1_2_10_117_2 doi: 10.1145/2858036.2858063 – ident: e_1_2_10_50_2 doi: 10.1145/3119930 – ident: e_1_2_10_76_2 doi: 10.1109/TVCG.2014.2346979 – ident: e_1_2_10_38_2 doi: 10.17705/1jais.00441 – ident: e_1_2_10_69_2 doi: 10.1145/3027063.3053113 – ident: e_1_2_10_44_2 doi: 10.1145/2499149.2499168 – ident: e_1_2_10_154_2 doi: 10.1111/cgf.13009 – ident: e_1_2_10_10_2 doi: 10.1145/2702123.2702452 – ident: e_1_2_10_20_2 doi: 10.1145/1746259.1746260 – ident: e_1_2_10_29_2 doi: 10.1016/j.jebo.2013.03.003 – ident: e_1_2_10_7_2 doi: 10.1145/2598153.2602225 – ident: e_1_2_10_178_2 doi: 10.1109/PacificVis.2012.6183570 – start-page: 19 volume-title: Information visualization year: 2008 ident: e_1_2_10_25_2 doi: 10.1007/978-3-540-70956-5_2 – ident: e_1_2_10_163_2 doi: 10.1109/TVCG.2013.164 – ident: e_1_2_10_77_2 doi: 10.1109/TVCG.2012.220 – ident: e_1_2_10_131_2 doi: 10.1017/CBO9780511844522 – ident: e_1_2_10_139_2 doi: 10.1145/2810012 – ident: e_1_2_10_5_2 doi: 10.1007/978-3-319-66435-4 – ident: e_1_2_10_135_2 doi: 10.1007/978-3-319-15168-7_56 – volume-title: EuroVis 2016 ‐ Short Papers year: 2016 ident: e_1_2_10_101_2 – ident: e_1_2_10_82_2 doi: 10.1109/VAST.2010.5652890 – ident: e_1_2_10_152_2 doi: 10.1109/TVCG.2012.196 – ident: e_1_2_10_21_2 doi: 10.1109/TVCG.2013.234 – ident: e_1_2_10_118_2 doi: 10.1111/cgf.13195 – ident: e_1_2_10_54_2 doi: 10.1145/2702123.2702443 – ident: e_1_2_10_160_2 doi: 10.1109/TVCG.2008.155 – ident: e_1_2_10_72_2 doi: 10.1145/2470654.2481410 – ident: e_1_2_10_136_2 doi: 10.1145/2675133.2675246 – ident: e_1_2_10_107_2 doi: 10.1111/cgf.12127 – ident: e_1_2_10_33_2 doi: 10.1145/3025453.3025870 – volume: 31 year: 2011 ident: e_1_2_10_161_2 article-title: Estimating the completion time of crowdsourced tasks using survival analysis models publication-title: Crowdsourcing for search and data mining (CSDM 2011) – ident: e_1_2_10_121_2 doi: 10.1109/TVCG.2017.2674978 – ident: e_1_2_10_134_2 doi: 10.1145/2598153.2602248 – ident: e_1_2_10_145_2 doi: 10.1111/cgf.12888 – ident: e_1_2_10_159_2 doi: 10.1109/TVCG.2016.2598591 – ident: e_1_2_10_34_2 doi: 10.1109/TVCG.2014.2346978 – ident: e_1_2_10_149_2 doi: 10.1007/978-3-319-24132-6_17 – ident: e_1_2_10_155_2 doi: 10.1109/TVCG.2014.2346320 – ident: e_1_2_10_97_2 doi: 10.1371/journal.pbio.1002456 – ident: e_1_2_10_46_2 doi: 10.1145/3025453.3025820 – ident: e_1_2_10_90_2 doi: 10.1145/2858036.2858440 – ident: e_1_2_10_74_2 – volume-title: Interactive storytelling: Techniques for 21st century fiction year: 2009 ident: e_1_2_10_56_2 – ident: e_1_2_10_75_2 doi: 10.1145/1240624.1240781 – ident: e_1_2_10_128_2 doi: 10.1145/989863.989880 – ident: e_1_2_10_24_2 doi: 10.1145/2470654.2481373 – ident: e_1_2_10_164_2 doi: 10.1145/2207676.2207709 – ident: e_1_2_10_114_2 doi: 10.1145/1978942.1978975 – ident: e_1_2_10_158_2 doi: 10.1145/2468356.2468529 – ident: e_1_2_10_175_2 doi: 10.1109/VAST.2014.7042496 – ident: e_1_2_10_19_2 doi: 10.1163/156856897X00357 – ident: e_1_2_10_151_2 – ident: e_1_2_10_166_2 doi: 10.1145/2858036.2858107 – ident: e_1_2_10_4_2 doi: 10.1145/2858036.2858300 – ident: e_1_2_10_53_2 doi: 10.1145/2631775.2631819 – volume: 45 issue: 4 year: 2009 ident: e_1_2_10_126_2 article-title: Instructional manipulation checks: Detecting satisficing to increase statistical power publication-title: Journal of Experimental Social Psychology doi: 10.1016/j.jesp.2009.03.009 – ident: e_1_2_10_49_2 doi: 10.1109/TVCG.2013.183 – ident: e_1_2_10_61_2 doi: 10.1109/VAST.2012.6400540 – ident: e_1_2_10_23_2 doi: 10.1145/2207676.2208556 – ident: e_1_2_10_146_2 doi: 10.1609/hcomp.v1i1.13088 – ident: e_1_2_10_15_2 – ident: e_1_2_10_142_2 doi: 10.1145/1958824.1958865 – ident: e_1_2_10_68_2 – ident: e_1_2_10_104_2 doi: 10.1145/2591677 – ident: e_1_2_10_9_2 doi: 10.1145/2669557.2669564 – ident: e_1_2_10_60_2 doi: 10.1145/1753326.1753357 – ident: e_1_2_10_87_2 doi: 10.1145/2818048.2820005 – ident: e_1_2_10_14_2 – ident: e_1_2_10_112_2 doi: 10.1145/2441776.2441912 – ident: e_1_2_10_3_2 doi: 10.1145/2556288.2557200 – ident: e_1_2_10_45_2 doi: 10.1145/2506364.2506366 – ident: e_1_2_10_78_2 doi: 10.1145/3025453.3025917 – ident: e_1_2_10_119_2 doi: 10.1109/T-AFFC.2012.19 – ident: e_1_2_10_67_2 doi: 10.1145/2858036.2858280 – ident: e_1_2_10_102_2 doi: 10.5194/isprsannals-II-3-W5-325-2015 – ident: e_1_2_10_173_2 doi: 10.1145/2531602.2531718 – ident: e_1_2_10_100_2 doi: 10.1145/3025453.3025592 – ident: e_1_2_10_27_2 doi: 10.1145/1837885.1837889 – ident: e_1_2_10_140_2 doi: 10.1145/2470654.2481281 – volume-title: How Maps Work ‐ Representation, Visualization, and Design year: 2004 ident: e_1_2_10_113_2 – ident: e_1_2_10_109_2 doi: 10.1145/2810188.2810192 – ident: e_1_2_10_62_2 doi: 10.1109/TVCG.2013.119 – ident: e_1_2_10_171_2 doi: 10.1145/2541288 – ident: e_1_2_10_86_2 doi: 10.1109/TVCG.2015.2467451 – ident: e_1_2_10_150_2 doi: 10.1109/TVCG.2015.2500240 – ident: e_1_2_10_71_2 doi: 10.1364/JOSA.35.000268 – ident: e_1_2_10_147_2 doi: 10.1145/2488388.2488489 – ident: e_1_2_10_37_2 doi: 10.1145/1453152.1453156 – ident: e_1_2_10_144_2 – ident: e_1_2_10_141_2 – ident: e_1_2_10_30_2 doi: 10.1109/TVCG.2012.230 – ident: e_1_2_10_12_2 doi: 10.1109/TVCG.2015.2467201 – ident: e_1_2_10_120_2 doi: 10.1145/1314683.1314684 – ident: e_1_2_10_64_2 doi: 10.1145/1963405.1963473 – ident: e_1_2_10_35_2 doi: 10.1109/PACIFICVIS.2017.8031598 – ident: e_1_2_10_6_2 doi: 10.1109/TVCG.2012.234 – ident: e_1_2_10_156_2 doi: 10.1109/TVCG.2013.247 – ident: e_1_2_10_103_2 doi: 10.1111/cgf.12387 – ident: e_1_2_10_122_2 doi: 10.1201/b17511 – ident: e_1_2_10_93_2 doi: 10.1111/cgf.13018 – ident: e_1_2_10_18_2 doi: 10.1109/TVCG.2014.2346575 – ident: e_1_2_10_176_2 doi: 10.1111/j.1467-8659.2009.01442.x – ident: e_1_2_10_41_2 doi: 10.1109/PACIFICVIS.2016.7465249 |
| SSID | ssj0004765 |
| Score | 2.4961057 |
| SecondaryResourceType | review_article |
| Snippet | Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments,... |
| SourceID | proquest crossref wiley |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 573 |
| SubjectTerms | Categories and Subject Descriptors (according to ACM CCS) Crowdsourcing Demographics Experiments I.3.3 [Computer Graphics]: Picture/Image Generation—Line and curve generation Quality assurance Reproducibility Researchers Scientific visualization Visualization |
| Title | Information Visualization Evaluation Using Crowdsourcing |
| URI | https://onlinelibrary.wiley.com/doi/abs/10.1111%2Fcgf.13444 https://www.proquest.com/docview/2067104131 |
| Volume | 37 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnZ1LS8NAEICHohc9-BartQTx4CVl053dJHgqtbWIelArPQhhd7MrolSx7cVf725eraIg3hLYvGZnMjPJzLcAx1zFRgid-lK3tY_WxfiScNexjKgD3hY0r7a45oMhXozYqAanZS9MzoeoPrg5y8je187AhZwsGLl6NK2AIjoWqKvVcgHRzRwdhSFnJdfbEWMKqpCr4qmO_OqL5gHmYpia-Zn-OjyUd5iXlzy3ZlPZUh_f4I3_fIQNWCviT6-TK8wm1PR4C1YXqITbEBUtSm7KvPuniWu7zJs1vV7FBveyWgOva7P4NPsBYPd2YNjv3XUHfrHAgq9oxNCPJY80plGMIYmMCWNiYoGCKk3SUJHIZkMunWMp6tBgiiwNRFsoQhRTBJWmu7A0fh3rPfACEVnbN0zytkYWEok8MEEcSkVjFhNah5NS1Ikq6ONuEYyXpMxCrDCSTBh1OKqGvuXIjZ8GNcr5SgqrmyQORW_Ty4AG9nKZ4H8_QdI972cb-38fegArVpmjvFKsAUvT95k-tDHJVDZhuXN2dXnbzJTwE9ov3Mw |
| linkProvider | Wiley-Blackwell |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LT8MwDLbGOAAH3ojBgApx4NIpaZy2kbigaWPA2AFtiAuq2jRFCDQQ2y78epK03QYCCXFrpfQRx679pfZngBNfiiyOVeomylMuahfjJsQ3FcuIivpezPJsi57fGeDVPb-vwFlZC5PzQ0w33Ixl2O-1MXCzIT1n5fIxa1CGiAuwaDp6W0B1OyOPwsDnJbO34YwpeIVMHs_00q_eaBZizgeq1tO01-ChfMc8weS5MRknDfnxjb7xv5NYh9UiBHXOc53ZgIoabsLKHDHhFoRFlZJZNefuaWQqL_N6Tac1pQd3bLqB09RAPrX_APTZNgzarX6z4xY9FlzJQo6uSPxQYRoKDEiYZYEgmYgxZlKRNJAk1IDIIDqeogoyTJGnNPZiSYjkkqBUbAeqw9eh2gWHxqE2_4wnvqeQByRBn2ZUBIlkggvCanBayjqSBQG56YPxEpVARAsjssKowfF06FvOuvHToHq5YFFheKPIsNFrhEkZ1Y-zkv_9BlHzom0P9v4-9AiWOv2bbtS97F3vw7LW7TBPHKtDdfw-UQc6RBknh1YTPwG4EN9T |
| linkToPdf | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnZ1LS8NAEICHWkH04FusVg3iwUvKJpndbPAkfVgfFBErPQgh2eyKKLXY9uKvdzevVlEQbwlsXrMzmZlk5luAEyYCFUUysWPpShu1i7FjwkzHMqJ0mBt5WbVFj3X7eDWggwqcFb0wGR-i_OBmLCN9XxsDHyVqzsjFk2o4HiIuwCIywo1Kt-5m7Cj0GS3A3gYZk2OFTBlPeehXZzSLMOfj1NTRdNbgsbjFrL7kpTGdxA3x8Y3e-M9nWIfVPAC1zjON2YCKHG7CyhyWcAt43qNk5sx6eB6bvsusW9Nql3BwKy02sJo6jU_SPwB6bxv6nfZ9s2vnKyzYwuMU7SBmXGLCA_QJV8oPiAoijDwhSeILwnU6ZPI5mqD0FSZIEydyI0GIoIKgkN4OVIdvQ7kLlhNxbfyKxsyVSH0SI3OUE_ix8AIaEK8Gp4WoQ5Hjx80qGK9hkYZoYYSpMGpwXA4dZcyNnwbVi_kKc7Mbh4ZFr_NLx3P05VLB_36CsHnRSTf2_j70CJZuW53w5rJ3vQ_LWrF5VjVWh-rkfSoPdHwyiQ9TPfwETVfeCw |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+Visualization+Evaluation+Using+Crowdsourcing&rft.jtitle=Computer+graphics+forum&rft.au=Borgo%2C+R&rft.au=Micallef%2C+L&rft.au=Bach%2C+B&rft.au=McGee%2C+F&rft.date=2018-06-01&rft.pub=Blackwell+Publishing+Ltd&rft.issn=0167-7055&rft.eissn=1467-8659&rft.volume=37&rft.issue=3&rft.spage=573&rft.epage=595&rft_id=info:doi/10.1111%2Fcgf.13444&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0167-7055&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0167-7055&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0167-7055&client=summon |