img: enago.com

the error bar says

the journal Perspectives on Psychological Science reports a fascinating study by Rohrer & 17 colleagues. they advertised online for scientists who had lost confidence in one of their previous research papers. they interviewed them to ask why they no longer believed the results of their study & carried out a wider, anonymous, survey

13 scientists took up the challenge. they listed a variety of reasons that they no longer had confidence in their own work, categorised as 'methodological error', 'invalid inference' or 'p-hacking'

p-hacking is a kind of poor statistical practice in which researchers make a series of quite small changes to their data, each time hoping that the results become more favourable to their hypotheses

my favourite confession is that of Tal Yarkoni, who writes:

"I now think most of the conclusions drawn were absurd on their face. My understanding of statistics has improved a bit since, and it is now abundantly clear to me that I (a) p-hacked to a considerable degree and (b) because of the "winner's curse," statistically significant effect sizes from underpowered studies cannot be taken at face value. I also now think the kinds of theoretical explanations I proposed were ludicrous in their simplicity and naivete - so the results would have told us essentially nothing even if they were statistically sound."

the authors of the new study conclude by proposing that all scientists should regularly make these self-criticisms & scientific publishing needs to remove the barriers & provide venues to do it

so should we just ignore these 13 scientists' work?

no. far from it

in making their public statements of error, these 13 brave scientists, in my view, become more reliable. we should listen to them more than we listen to the N minus 13 other scientists who have not done this

we have just lived through a decade of navel-gazing by experimental psychologists. there has been what some call a 'replication crisis', where experimental results in psychology can't be repeated. we're not measuring the right things in the right way or we do so with too little data. when we get the results we test & retest & fish around until the data support the conclusions we've already made

i don't see it like this. in my view, psychology is just well ahead of many other disciplines where all of these same problems occur. psychologists are just good at studying their own behaviour as well as that of their subjects. that is why psychology is in 'crisis' - we're just the first to admit the problem

the loss-of-confidence project & this article are part of the solution & the error bar salutes them. indeed, so impressed am i that Rohrer & colleagues i can today announce, are the first winners of the error bar's golden bar of error 🏆

we now stand for the anthem of error

[🎶 ROUSING MUSIC 🎶]

here they come now, all 17 of the authors, 2 metres apart as is customary on these occasions

there's Julia Rohrer, in midnight blue, leading the researchers to receive their awards, followed by Warren Tierney on a bike, a statue of Eric Uhlmann, Lisa DeBruine with purple hair, Tom Heyman, Benedict Jones in tight white t-shirt, Stefan Schmukle, Raphael Silberzahn in a georgeous pale-blue suit, Rebecca Willén, there's Rickard Carlsson against the ice-white background typical of Sweden, Richard Lucas, Julia Strand - co-host of the Juice and the Squeeze podcast, Simine Vazire represented by a spray of sunflowers, Jessica Witt, Thomas Zentall the senior member of the team - good day to you sir, followed by the invisible gorilla Christopher Chabris, & at the rear comes Tal Yarkoni holding the two ceremonial ice creams, as tradition dictates

conclusion

thirteen bold scientists describe how they have lost confidence in some of their previous work. the method & results are inspiring & exactly what we need

rating