Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel; Barnabas Szaszi; Gustav Nilsonne; Olmo R. Van Den Akker; Casper J. Albers; Marcel Alm Van Assen; Jojanneke A. Bastiaansen; Daniel Benjamin; Udo Boehm; Rotem Botvinik-Nezer; Laura F. Bringmann; Niko A. Busch; Emmanuel Caruyer; Andrea M. Cataldo; Nelson Cowan; Andrew Delios; Noah Nn Van Dongen; Chris Donkin; Johnny B. Van Doorn; Anna Dreber; Gilles Dutilh; Gary F. Egan; Morton Ann Gernsbacher; Rink Hoekstra; Sabine Hoffmann; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Kai J. Jonas; Alexander T. Kindel; Michael Kirchler; Yoram K. Kunkels; D. Stephen Lindsay; Jean Francois Mangin; Dora Matzke; Marcus R. Munafò; Ben R. Newell; Brian A. Nosek; Russell A. Poldrack; Don Van Ravenzwaaij; Jörg Rieskamp; Matthew J. Salganik; Alexandra Sarafoglou; Tom Schonberg; Martin Schweinsberg; David Shanks; Raphael Silberzahn; Daniel J. Simons; Barbara A. Spellman; Samuel St-Jean; Jeffrey J. Starns; Eric Luis Uhlmann; Jelte Wicherts; Eric Jan Wagenmakers

doi:10.7554/eLife.72185

Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel^*, Barnabas Szaszi^*, Gustav Nilsonne, Olmo R. Van Den Akker, Casper J. Albers, Marcel Alm Van Assen, Jojanneke A. Bastiaansen, Daniel Benjamin, Udo Boehm, Rotem Botvinik-Nezer, Laura F. Bringmann, Niko A. Busch, Emmanuel Caruyer, Andrea M. Cataldo, Nelson Cowan, Andrew Delios, Noah Nn Van Dongen, Chris Donkin, Johnny B. Van Doorn, Anna DreberGilles Dutilh, Gary F. Egan, Morton Ann Gernsbacher, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Kai J. Jonas, Alexander T. Kindel, Michael Kirchler, Yoram K. Kunkels, D. Stephen Lindsay, Jean Francois Mangin, Dora Matzke, Marcus R. Munafò, Ben R. Newell, Brian A. Nosek, Russell A. Poldrack, Don Van Ravenzwaaij, Jörg Rieskamp, Matthew J. Salganik, Alexandra Sarafoglou, Tom Schonberg, Martin Schweinsberg, David Shanks, Raphael Silberzahn, Daniel J. Simons, Barbara A. Spellman, Samuel St-Jean, Jeffrey J. Starns, Eric Luis Uhlmann, Jelte Wicherts, Eric Jan Wagenmakers

^*Corresponding author for this work

Department of Neurobiology

Research output: Contribution to journal › Article › peer-review

32 Scopus citations

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Original language	English
Article number	e72185
Journal	eLife
Volume	10
DOIs	https://doi.org/10.7554/eLife.72185
State	Published - Nov 2021

Funding

Funders	Funder number
Horizon 2020 Framework Programme	640638, 681466, 726361
National Science Foundation	1760052
Not added	406.17.568

Access to Document

10.7554/eLife.72185

Cite this

Aczel, B., Szaszi, B., Nilsonne, G., Van Den Akker, O. R., Albers, C. J., Van Assen, M. A., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-Nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., Van Dongen, N. N., Donkin, C., Van Doorn, J. B., ... Wagenmakers, E. J. (2021). Consensus-based guidance for conducting and reporting multi-analyst studies. eLife, 10, Article e72185. https://doi.org/10.7554/eLife.72185

@article{43dd525c50714bd88f51982b0de2fc4b,

title = "Consensus-based guidance for conducting and reporting multi-analyst studies",

abstract = "Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.",

author = "Balazs Aczel and Barnabas Szaszi and Gustav Nilsonne and \{Van Den Akker\}, \{Olmo R.\} and Albers, \{Casper J.\} and \{Van Assen\}, \{Marcel Alm\} and Bastiaansen, \{Jojanneke A.\} and Daniel Benjamin and Udo Boehm and Rotem Botvinik-Nezer and Bringmann, \{Laura F.\} and Busch, \{Niko A.\} and Emmanuel Caruyer and Cataldo, \{Andrea M.\} and Nelson Cowan and Andrew Delios and \{Van Dongen\}, \{Noah Nn\} and Chris Donkin and \{Van Doorn\}, \{Johnny B.\} and Anna Dreber and Gilles Dutilh and Egan, \{Gary F.\} and Gernsbacher, \{Morton Ann\} and Rink Hoekstra and Sabine Hoffmann and Felix Holzmeister and Juergen Huber and Magnus Johannesson and Jonas, \{Kai J.\} and Kindel, \{Alexander T.\} and Michael Kirchler and Kunkels, \{Yoram K.\} and Lindsay, \{D. Stephen\} and Mangin, \{Jean Francois\} and Dora Matzke and Munaf{\`o}, \{Marcus R.\} and Newell, \{Ben R.\} and Nosek, \{Brian A.\} and Poldrack, \{Russell A.\} and Ravenzwaaij, \{Don Van\} and J{\"o}rg Rieskamp and Salganik, \{Matthew J.\} and Alexandra Sarafoglou and Tom Schonberg and Martin Schweinsberg and David Shanks and Raphael Silberzahn and Simons, \{Daniel J.\} and Spellman, \{Barbara A.\} and Samuel St-Jean and Starns, \{Jeffrey J.\} and Uhlmann, \{Eric Luis\} and Jelte Wicherts and Wagenmakers, \{Eric Jan\}",

note = "Publisher Copyright: {\textcopyright} Aczel et al.",

year = "2021",

month = nov,

doi = "10.7554/eLife.72185",

language = "אנגלית",

volume = "10",

journal = "eLife",

issn = "2050-084X",

publisher = "eLife Sciences Publications Ltd",

}

Aczel, B, Szaszi, B, Nilsonne, G, Van Den Akker, OR, Albers, CJ, Van Assen, MA, Bastiaansen, JA, Benjamin, D, Boehm, U, Botvinik-Nezer, R, Bringmann, LF, Busch, NA, Caruyer, E, Cataldo, AM, Cowan, N, Delios, A, Van Dongen, NN, Donkin, C, Van Doorn, JB, Dreber, A, Dutilh, G, Egan, GF, Gernsbacher, MA, Hoekstra, R, Hoffmann, S, Holzmeister, F, Huber, J, Johannesson, M, Jonas, KJ, Kindel, AT, Kirchler, M, Kunkels, YK, Lindsay, DS, Mangin, JF, Matzke, D, Munafò, MR, Newell, BR, Nosek, BA, Poldrack, RA, Ravenzwaaij, DV, Rieskamp, J, Salganik, MJ, Sarafoglou, A, Schonberg, T, Schweinsberg, M, Shanks, D, Silberzahn, R, Simons, DJ, Spellman, BA, St-Jean, S, Starns, JJ, Uhlmann, EL, Wicherts, J & Wagenmakers, EJ 2021, 'Consensus-based guidance for conducting and reporting multi-analyst studies', eLife, vol. 10, e72185. https://doi.org/10.7554/eLife.72185

TY - JOUR

T1 - Consensus-based guidance for conducting and reporting multi-analyst studies

AU - Aczel, Balazs

AU - Szaszi, Barnabas

AU - Nilsonne, Gustav

AU - Van Den Akker, Olmo R.

AU - Albers, Casper J.

AU - Van Assen, Marcel Alm

AU - Bastiaansen, Jojanneke A.

AU - Benjamin, Daniel

AU - Boehm, Udo

AU - Botvinik-Nezer, Rotem

AU - Bringmann, Laura F.

AU - Busch, Niko A.

AU - Caruyer, Emmanuel

AU - Cataldo, Andrea M.

AU - Cowan, Nelson

AU - Delios, Andrew

AU - Van Dongen, Noah Nn

AU - Donkin, Chris

AU - Van Doorn, Johnny B.

AU - Dreber, Anna

AU - Dutilh, Gilles

AU - Egan, Gary F.

AU - Gernsbacher, Morton Ann

AU - Hoekstra, Rink

AU - Hoffmann, Sabine

AU - Holzmeister, Felix

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Jonas, Kai J.

AU - Kindel, Alexander T.

AU - Kirchler, Michael

AU - Kunkels, Yoram K.

AU - Lindsay, D. Stephen

AU - Mangin, Jean Francois

AU - Matzke, Dora

AU - Munafò, Marcus R.

AU - Newell, Ben R.

AU - Nosek, Brian A.

AU - Poldrack, Russell A.

AU - Ravenzwaaij, Don Van

AU - Rieskamp, Jörg

AU - Salganik, Matthew J.

AU - Sarafoglou, Alexandra

AU - Schonberg, Tom

AU - Schweinsberg, Martin

AU - Shanks, David

AU - Silberzahn, Raphael

AU - Simons, Daniel J.

AU - Spellman, Barbara A.

AU - St-Jean, Samuel

AU - Starns, Jeffrey J.

AU - Uhlmann, Eric Luis

AU - Wicherts, Jelte

AU - Wagenmakers, Eric Jan

PY - 2021/11

Y1 - 2021/11

N2 - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

AB - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

UR - http://www.scopus.com/inward/record.url?scp=85120181656&partnerID=8YFLogxK

U2 - 10.7554/eLife.72185

DO - 10.7554/eLife.72185

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

C2 - 34751133

AN - SCOPUS:85120181656

SN - 2050-084X

VL - 10

JO - eLife

JF - eLife

M1 - e72185

ER -

Consensus-based guidance for conducting and reporting multi-analyst studies

Abstract

Funding

Access to Document

Other files and links

Fingerprint

Cite this