in a carefully-conducted brain stimulation study comparing the movement skills of blind & sighted people, the authors excluded 93% of their data & pretended that the main purpose of their work was always to focus on the 7%. riiiiigggghhhht.  

original article: Ikegami et al., 2022 (bioRxiv), reported in: The New Scientist by Jason Arunn Murugesu on 16th March 2022

this story was in episode 27 #brain #stimulation #blind #moving #phacking

the error bar says

blindness has a massive effect on the brain. a lot of the brain is responsive to stimulation of the eyes [impression of Brian Cox] what we scientists call light [impression ends]. so when people lose their sight, these large portions of brain either end up doing nothing, or re-organising to do something else.

a long series of studies over many years has found evidence that blind people have better touch, or better hearing, or are better at particular cognitive tasks. these studies show how these blind people's brains may have reorganised so that the once-visual or potentially-visual parts of the brain now do other things.

reports that blind people have an advantage over the sighted, or that blind people's brains work differently are, understandably, exciting & interesting to the news media. & that's presumably why the New Scientist thinks that now is the time to trawl BioRXiv, the pre-print server, for an early version of just such a report.

error bar listeners may be aware that a 'pre-print' is a scientific paper that has not been peer-reviewed or has not been published in a journal. but we can still read it.

in this new report by scientists in Japan, 12 blind & 12 sighted people were asked to move their two feet rhythmically up and down while sitting in a chair. while their feet were flapping around, the researchers placed a powerful electromagnet on the back of their heads. their brains were stimulated in one of 14 different locations overlying the so-called 'visual cortex' - an important part of the brain involved in vision, at least in sighted people.

by moving the magnet around these 14 locations, the authors found that one, just one location, made the blind people worse at moving their feet rhythmically. their movements became more variable & less-tuned to the once-per-second rhythm that they were trying to keep to.

do the blind use the 'visual' brain for movement?


well, they might do. but this paper provides almost-laughably-weak evidence for it. i shall explain.

first, the positives: the experimental design, procedure & motivation for the study all look good. it seems like the authors had a clear plan & carried it out very well. but the problems started when the data appeared. most good plans do not survive contact with the enemy.

second, the negatives. there are many, but regular listeners will know what's coming.

one: the authors stimulated 14 different locations on the head, but inform us that, really, they were only interested in one of them. so the other 13 locations - 93% of the useful data - were ignored, thrown away to a supplementary file drawer. i could not access these supplementary materials, dear listener.

two: the authors did do a control condition - one in which there was no actual stimulation to the head, but these data have also been discarded because the participants realised that their head wasn't being stimulated. er, ok. so that's another one of the 16 experimental conditions discarded. the entire paper, then, depends on only 2 of the original 16 conditions being proffered to the public for scrutiny.

three: it gets worse - if you can imagine - the authors present a large number of tests - between blind & sighted, between stimulation & no stimulation, between half of the blind group who were athletes - blind footballers - & half who weren't - oh, did i, did i not mention that design factor already? - sorry - & for almost all of these statistical comparisons there was no clear differences found. except one.

four: that one statistical test, ladies & gentlemen, keeping in mind that 14 of the 16 original conditions have been thrown away, was weak.

i suspect that most of the listeners know about 'p-values' & about 'statistical significance,' but for those who want or need a reminder: there is a convention in many sciences to accept that a one-in-twenty chance of the data you've found arising by chance is about the right amount of chance occurrence before you start being interested in the results. this one-in-twenty level is the classic 'five percent level of statistical significance', in the jargon.

& this whole paper depends on the authors reporting a four point nine five nine percent chance. & they reported this number in an unnecessarily-opaque way, almost as if they were trying to hide it.

now the error bar is emphatically not accusing any particular scientist of the forbidden act of 'p-hacking'. i am simply pointing out that about 90% of the data in this study were arbitrarily thrown away & the remaining data are dangling on a statistical thread so bare it would make a spider blush.