Non-standard errors

Menkveld, Albert J.; Dreber, Anna; Holzmeister, Felix; Huber, Juergen; Johannesson, Magnus; Kirchler, Michael; Neussüs, Sebastian; Razen, Michael; Weitzel, Utz; Brownlees, Christian T.; Gil-Bazo, Javier; et al.,

Non-standard errors

License

Citation

Menkveld, A, Dreber, A, Holzmeister, F, Huber, J, Johannesson, M, Kirchler, M, Neussüs, S, et al. . Non-standard errors. 2021. handle: http://hdl.handle.net/10230/68657

This citation was generated automatically.

Abstract

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.