Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

• Published in: Organizational behavior and human decision processes Vol. 165; pp. 228 - 249
• Main Authors: van Assen, Marcel A.L.M., Liu, Yang, Heer, Jeffrey, Mohamed, Zainab, Amireh, Hashem, Venkatesh Prasad, Vaishali, Bernstein, Abraham, Snellman, Kaisa, Madan, Nikhil, Silberzahn, Raphael, Murase, Toshio, Mandl, Benjamin, Kelchtermans, Stijn, Naseeb, Chan, Richard Chan, C.S., Adie, Prestone, Alspaugh, Sara, Alstott, Jeff, Ariño de la Rubia, Eduardo, Arzi, Adbi, Bahník, Štěpán, Baik, Jason, Banker, Sachin, Barros-Rivera, Brenda, Briers, Robert A., Castrillo, Laura, Catlett, Timothy, Chen, Olivia, Cohn, Brent, Cugueró-Escofet, Natàlia, Cyrus-Lai, Wilson, Danielsson, Henrik, Russo, Rosaria de F.S.M., de Silva, Niko, Dondelinger, Frank, Duarte de Souza, Carolina, Tyson Dube, B., Dubova, Marina, Mark Dunn, Ben, Fox, Nick, Gnambs, Timo, Gong, Yuanyuan, Greenawalt, Brandon, Han, Dan, Hong, Antony B., Huang, Lilian, Hui, Kent N., Hultman, Keith A., Kamdar, Jash, Kappler, Gregor, Kaszubowski, Erikson, Kleinberg, Bennett, Lavbič, Dejan, Liverani, Silvia, Loh, Bianca, MacDonald, Kyle, Madan, Christopher R., Hjorth Madsen, Lasse, Maimone, Christina, Marshall, Adrienne, Ester Matskewich, Helena, Mavon, Kimia, McLain, Katherine L., Mertens, Ulf, Moore, Ben, Moore, Andrew, Nantz, Eric, Nasrullah, Ziauddin, Nejkovic, Valentina, Nell, Colleen S, Arthur Nelson, Andrew, Nilsonne, Gustav, Nolan, Rory, O'Brien, Christopher E., O'Shea, Kieran, Palsetia, Diana, Protzko, John, Riddle, Travis, Rosenberg, Joshua M., Schulte-Mecklenbeck, Michael, Sharma, Nirek, Shotwell, Gordon, Stedden, William, Stodden, Victoria, Stoltzman, Scott, Subbaiah, Subashini, Tatman, Rachael, Thibodeau, Paul H., Tomkins, Sabina, Duncan Wadsworth, W., Wanders, Florian, Watts, Krista, Whelpley, Christopher E., Won, Andy, Wu, Lawrence, Yip, Arthur, Youngflesh, Casey, Zhang, Leilei, Zibman, Chava, Luis Uhlmann, Eric
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 2021
• Subjects: Analysis-contingent results; Crowdsourcing data analysis; Psychology; psykologi; Research reliability; Researcher degrees of freedom; Scientific robustness; Scientific transparency; Scientific robustness; Researcher degrees of freedom; Analysis-contingent results; Crowdsourcing data analysis; Research reliability; Scientific transparency

•A new platform (DataExplained) helps analysts justify preferred and rejected analytical paths in real time.•Independent analysts used DataExplained to test two hypotheses on the same dataset in a crowdsourced initiative.•Analysts conducted radically different analyses and reported dispersed effect...