the uk media seized on a report that rocket scientists & brain surgeons are no more intelligent than the general population. only the daily mail noted the authors' pleas for more non-white non-males to enter the professions  

original article: Usher et al., 2021 (British Medical Journal), reported in: The Independent by Vishwam Sankaran on 14th December 2021 & The Daily Mail by Shivali Best on 13th December 2021 & The Guardian by Nicola Davis on 13th December 2021

this story was in episode 23 #rocket #science #brain #surgery #christmas

the error bar says

the British Medical Journal's Christmas issue always contains some 'fun' science, resulting in two stories at the error bar.

the first is that a survey & online intelligence test of 748 aeronautical engineers (rocket scientists) & neurosurgeons (brain surgeons) has found that members of these professions are in general no better at 12 cognitive tests that a large cohort of the general British population.

the researchers recruited rocketeers & neurosurgeons through professional mailing lists, inviting them to complete 12 computer tasks over 30 minutes. these tasks were taken from a battery of tasks completed by around 250 thousand members of the Great British Public in a previous study.

after data cleaning & processing, the brain surgeons were a bit better than the rocket scientists in one cognitive domain, & the rocketeers a bit better than the surgeons in another. compared to the population, surgeons were faster on one thing & slower at another.

are rocket scientists & neurosurgeons as smart as us?

yes. probably.

with the levity of presentation, the time of year & this story's popularity with the British Press, it's fair to say that this report provides no strong evidence that these two groups - of largely white European men - did not perform better than the general public on these tests.

yes, there were some 'significant differences' between the scores of the two groups of white male Europeans, & some differences between the neurosurgeons and the British, but these small differences should be taken into account along with the study's several flaws:

first, the two groups were quite different - in size & geographic origin; there were also many more men in the groups compared to the general population.

second, the data from 748 respondents was whittled-down to just 401, after processes of 'data cleaning'. this may be typical for online research studies, but it leaves the error bar with little confidence in the quality or representativeness of the cognitive test scores & reaction time data that remains.

third, the data were processed by running a factor-analysis, which reduces lots of different kinds of data into a smaller number of simpler scores. in this study, the factor analysis was done on the 401 new datasets, reducing data from 12 tasks down to 6 different scores. the scores of the general population were then converted & compared with the rocketeers & surgeons scores. this seems like the wrong way around - why not do the factor analysis on the more reliable & less-biased whole population data, then calculate the new participants' scores?

fourth, the two groups were compared with each other using 6 statistical tests, then each group was compared to the general population again with 6 tests. 18 tests were done, four were statistically significant, & only one would have survived corrections for multiple comparisons, if they'd been done.


the error bar is not lowered at christmas or any other time of year - this study provides only weak evidence that rocket scientists & neurosurgeons have similar cognitive skills to the general british population.