transcript of episode 13: LOSING CONFIDENCE, 2nd July 2021

[🎶 INTRO: "Spring Swing" by Dee Yan-Kee 🎶]

welcome to the error bar: where mindlessness becomes mindfulness

in this episode: how scientists lose confidence in their own work, the mega corporations who infiltrate our dreams & music to listen to in the gym after work, you know,

here is the brain news on the 2nd July 2021:



the journal Perspectives on Psychological Science reports a fascinating study by Rohrer & 17 colleagues. they advertised online for scientists who had lost confidence in one of their previous research papers. they interviewed them to ask why they no longer believed the results of their study & carried out a wider, anonymous, survey

13 scientists took up the challenge. they listed a variety of reasons that they no longer had confidence in their own work, categorised as 'methodological error', 'invalid inference' or 'p-hacking'

p-hacking is a kind of poor statistical practice in which researchers make a series of quite small changes to their data, each time hoping that the results become more favourable to their hypotheses

my favourite confession is that of Tal Yarkoni, who writes:

"I now think most of the conclusions drawn were absurd on their face. My understanding of statistics has improved a bit since, and it is now abundantly clear to me that I (a) p-hacked to a considerable degree and (b) because of the "winner's curse," statistically significant effect sizes from underpowered studies cannot be taken at face value. I also now think the kinds of theoretical explanations I proposed were ludicrous in their simplicity and naivete - so the results would have told us essentially nothing even if they were statistically sound."

the authors of the new study conclude by proposing that all scientists should regularly make these self-criticisms & scientific publishing needs to remove the barriers & provide venues to do it


so should we just ignore these 13 scientists' work?

no. far from it

in making their public statements of error, these 13 brave scientists, in my view, become more reliable. we should listen to them more than we listen to the N minus 13 other scientists who have not done this

we have just lived through a decade of navel-gazing by experimental psychologists. there has been what some call a 'replication crisis', where experimental results in psychology can't be repeated. we're not measuring the right things in the right way or we do so with too little data. when we get the results we test & retest & fish around until the data support the conclusions we've already made

i don't see it like this. in my view, psychology is just well ahead of many other disciplines where all of these same problems occur. psychologists are just good at studying their own behaviour as well as that of their subjects. that is why psychology is in 'crisis' - we're just the first to admit the problem

the loss-of-confidence project & this article are part of the solution & the error bar salutes them. indeed, so impressed am i that Rohrer & colleagues i can today announce, are the first winners of the error bar's golden bar of error 🏆

we now stand for the anthem of error


here they come now, all 17 of the authors, 2 metres apart as is customary on these occasions

there's Julia Rohrer, in midnight blue, leading the researchers to receive their awards, followed by Warren Tierney on a bike, a statue of Eric Uhlmann, Lisa DeBruine with purple hair, Tom Heyman, Benedict Jones in tight white t-shirt, Stefan Schmukle, Raphael Silberzahn in a georgeous pale-blue suit, Rebecca Willén, there's Rickard Carlsson against the ice-white background typical of Sweden, Richard Lucas, Julia Strand - co-host of the Juice and the Squeeze podcast, Simine Vazire represented by a spray of sunflowers, Jessica Witt, Thomas Zentall the senior member of the team - good day to you sir, followed by the invisible gorilla Christopher Chabris, & at the rear comes Tal Yarkoni holding the two ceremonial ice creams, as tradition dictates


thirteen bold scientists describe how they have lost confidence in some of their previous work. the method & results are inspiring & exactly what we need


the science was by Rohrer et al. 2021: Perspectives on Psychological Science;



in an open letter signed, according to the Daily Mail, by "nearly 40" scientists (er, it's 38), Stickgold & colleagues warn against the possibility that companies might infiltrate our dreams

the researchers argue that planting adverts for products in our dreams is unethical. they say:

"With brain imaging techniques beginning to capture the core contents of people's dreams & sleep studies establishing real-time communication between researchers and sleeping dreamers, the kind of dream incubation until recently assumed to be the pure science fiction of movies like Inception is now becoming reality"


can companies infiltrate our dreams?

no, i don't think so

beyond the hyperbole, one irony is that the 38 researchers mention the brand name of the main advertiser that they are complaining about no fewer than 16 times in their 2000 word manifesto. they link readers directly to the company's advert on you tube & their story was picked up by the Daily Mail - 36 million monthly readers - & Science Magazine - 3 million monthly readers

regular listeners at the error bar will recall that in episode #4, we discussed the claim that researchers can have two-way communication with volunteers during their sleep

the new open letter relies on this same evidence - amongst others - & involves 4 of the same researchers

if the rest of their evidence is as reliable as the earlier dream study, then we have little to fear from whichever major American alcoholic drinks company wants to infiltrate our dreams


cheers! [🎶 beer 🎶]


the science was by Moutinho 2021: Science; reported in DxE on 29/Jun/21, & The Daily Mail by Stacy Liberatore on 17/Jun/21

and the brain in brief...



the Daily Mail invokes 'The Eye of the Tiger' song from Rocky, the movie, to tell us that motivational music helps to overcome mental fatigue when exercising

in two experiments, each with nine fit young people, the researchers first found that performing a reaction time task for 30 minutes makes people feel tired

next, they tested how long they could keep running at a certain speed & how quickly they could run five kilometres while measuring mental fatigue every minute. mental fatigue makes you worse on both of these endurance tests

& this was all done both before & after the mental fatigue & both with & without the volunteers listening to their own choice of motivational music

the researchers from Edinburgh found that listening to motivational music counter-acts the effects of mental fatigue


does motivational music fight fatigue?


the question is a relatively simple one, so the authors probably did not need to present 20 graphs to answer it, but each graph is really good - all the individual data are shown & all the error bars are in place. on that basis alone, it passes

if i must quibble, the authors make a big point about how they are not doing a particular kind of statistical test - that's a null hypothesis test for error bar nerds - but then they present exactly the kind of data & error bars that allow you to do those exact same statistical tests. for the nerds listening: a 95% confidence interval is the area over which the p-value for a one-sided [ed: one-SAMPLE] t-test is bigger than .05. p-values & confidence intervals are based on the same information. as nerdy quibbles go, it's not a fatal one

the main flaw seems to be that there's some cherry-picking of results: why are only 2 of the 6 emotional rating scales presented? & why do they perform so many statistical tests without any corrections?


music fights fatigue


the science was by Lam et al. 2021: Journal of Human Sport and Exercise; reported in The Daily Mail by @jwillchad on 22/Jun/21

[🎶 OUTRO: "Cosmopolitan - Margarita - Bellini"by Dee Yan-Kee 🎶]

it's closing time at the error bar, but do drop in next time for more brain news, fact-checking & neuro-opinions. take care.

the error bar was devised & produced by Dr Nick Holmes from University of Birmingham's School of Sport, Exercise and Rehabilitation Sciences. the music by Dee Yan-Kee is available from the free music archive. find us at the error bar dot com, on twitter at bar error, or email talk at the error bar dot com