the error bar says

the New Scientist has done well this episode, finding two error bar worthy stories for me to read on a sunny Saturday morning. thank you New Scientist.

from a possibly p-hacked preprint to a massive, multiauthor mega manuscript in a glamour mag, the New Scientist tells us that "brain scanning studies are usually too small to find reliable results". oh dear.

if this is true, undergraduate students up & down the country will be rejoicing. finally - FINALLY - there is evidence that all samples are indeed 'too small' & this standard criticism can be used with abandon next exam season.

in the paper published last week in Nature magazine, 43 scientists studied around 50 thousand brains & concluded that we in fact need thousands of brains before our statistical analyses are of any use in relating the human brain to the performance of cognitive complex... [bugger]

in the paper published last week in Nature magazine, 43 scientists studied around 50 thousand brains. they concluded that we in fact need thousands of brains before our statistical analyses are of any use in relating the human brain to the performance of cogna... [ah, fucking hell].. complex cognitive tasks!

in the paper published last week in Nature magazine, 43 scientists studied around 50 thousand brains. they concluded that we in fact need thousands of brains before our statistical analyses are of any use in relating the human brain to the performance of complex cognitive tasks or mental health.

if this mega factoid is true, the error bar should be shut down. standard errors have become a standard error; the confidence interval for confidence intervals has ended.

is there any hope for brain imaging?

well yes, obviously.

this is not an open-access paper & i don't think Nature needs any more encouragement to produce sexy-sounding titles, so here is just a quick error bar opinion.

the report specifically relates to something called 'Brain Wide Association Studies' - this is a kind of approach that takes a large number of brains, looks at how they differ & tests whether these differences can be related to specific cognitive or mental health outcomes - like reaction times in a particular task, or a particular diagnosis or a score on a questionnaire.

it does *not* relate to 'all brain imaging' studies. not!

in the new study, 48,809 brains were taken from three large existing datasets & re-analysed billions of times in different ways. the goal was to see how brain structure & function is related to 41 different demographic, cognitive & mental measurements.

i've not read the details, but the gist is: even the very strongest relationships between brain & behaviour give only very weak statistical results, requiring hundreds or thousands of brains to make reliable conclusions.

so is that the end of brain imaging? no. this is a massive study looking - post-hoc - at the brains & behaviours of three other massive brain & behaviour studies. these studies all measured many different things with, presumably, the rather open-ended goal of finding something.

it would be unfair to describe these fantastic research projects as monumental fishing trips.

but as the error bar discovered last episode, just because a study is 'big' & comes to 'big' conclusions - such as that mental processing speed is high & constant until age 60 [newsflash: it really isn't], this does not mean the underlying data are good.

indeed: small, dedicated studies measuring a few aspects of brain & behaviour using high-quality & low-variability measures may well result in better data. better data gives better effect sizes & requires fewer brains.

so if this study heralds the end of 'Big MRI Data' research, that may not be a bad thing.

conclusion

that massive, unfocussed studies find few strong relationships between brain & behaviour should not be a call to increase the number of participants or studies, but rather to massively improve the quality of our hypotheses & experiments.

rating