DON'T LOOK BACK   ⏱16:21

our main brain stories this episode...

DON'T LOOK BACK

hello

this season, i'm starting a new feature on the podcast, in which i produce an audio-log, or alog, on a researchy/statisticsy topic. it's basically an essay - i'm trying out some ideas that come in & out of my head at various times, often during the peer review process or in the pub. the first essay is titled "Don't look back", & is about my new obsession with reading every single paper on a topic.

but before that, i have to make a correction. in episode 17, i raised doubts about the scientific evidence underlying the story of 'Havana syndrome' - a mysterious illness that seemed to affect American citizens working in or near the US embassy in Cuba. i was not wrong to say that there was no good published science on the topic. but that's not the same thing as saying that Havana syndrome is not real.

about a year ago, as the error bar was closing at the end of season two, i listened to two new podcast series on Havana syndrome. the reporters came at it from different angles, & i particularly enjoyed the one called the sound: mystery of havana syndrome.

by the end of both series, i was convinced that there was enough 'ear-witness,' medical & anecdotal evidence to support the conclusion that a few dozen people among the thousands of people who reported having Havana syndrome, were - in fact - the victims of a targeted sonic attack. microwave radiation is being used to take-out drones over the skies of Ukraine. i no longer think it is safe to conclude that governmental organisations do not have the technology to do this sort of thing. i still don't think we'll be reading about it in the UK newspapers.

[long pause]

don't look back

a few years ago, i promised myself - on twitter - that i would do a systematic review &/or a meta-analysis with every new experimental paper that i wrote. that wasn't a bad idea, but it has taken some real work. it has also slowed down my 'output' - how many papers i push out each year.

i'm in the middle of my next big meta-analysis now. a few weeks ago i thought i was close to reading all of the papers on a topic - 167 - when i decided to run a search on pubmed, then cross-check the results against the papers i had already found, to get an approximate final number for the review. i knew that the search already gave more than three hundred & fifty results, but most such searches are not very specific, so i guessed that maybe half would be relevant.

about 95% of the papers in the search were relevant. over a few difficult hours last week, my 174 articles turned into 390.

several months ago i told my collaborators that i expected "about 50" papers & that the review would be done by Christmas. instead, this will now take until summer. for this first statistical essay, i want to talk about looking back on the literature as a researcher.

i have a couple of phrases or quotes that come to mind & seem to contradict each other. first: "don't study the literature, study nature." second: "a month in the lab can save an hour in the library."

the first quote advises us not to get too bogged-down in what other people have done. instead, your job as a researcher is to study what's out there in the world, rather than to study what researchers have done when they said that they were studying it. you could spend your whole career or life - unsuccessfully - trying to read all of the scientific literature before running your first, final, decisive study. in general, this first quote may therefore be good advice for any researcher at any stage of their career - keep your eyes above the fog of the literature & fixate firmly on your real scientific horizon.

the second quote strikes me as requiring or encouraging the opposite approach. why waste a month of your limited time & resources in the laboratory when you can spend a relatively short time in the library checking what researchers have already done? why re-invent wheels that have long ago been carved out, rolled-off into production & caused multiple vehicle crashes on the ring-road?

going back to my recent pubmed search, these two conflicting quotes resonated as i sifted through each new page of search results, knowing that each new relevant paper will take me one or two hours to find, read, digest & extract the information i need. i've now read about 180 of the 400 relevant papers, extracted around 3000 effect-sizes, & i am not yet half way. so should i get up from my desk, go back to the lab & get on with studying nature, or should i remain seated for an almost-certainly repetitive 220 further hours in the library?

i wish i could choose otherwise, dear listener, but i've always been an 'hour-in-the-library' sort of person. i do love the lab - demonstrating & discovering effects in the flesh comprises much of the fun of research. but now that i know there are 400 relevant papers to read, how can i ever claim to be an expert on this topic if i've not read all of them?

but despite my need to read every last paper, i think there is still hope! for two reasons.

a first, distinct, benefit of reading all the papers on a topic, systematically, methodically & yes, obsessively, is that your peer reviewers can typically say nothing against any conclusions you draw from this literature. "while Reviewer 1 is correct that many researchers use method A to calibrate their stimuli, systematic review shows that the majority use method B, & a small minority use method C. further, of those who claimed to use method A, 76% of researchers did it wrong." there is nothing Reviewer 1 can say against that. nothing. in this made-up example, it's just a fact that most researchers do not use method A. so despite not collecting a single datapoint using any of these three methods, i am suddenly a world-leading expert on them. at least in theory, if not in practice.

a second benefit of being the person who spends the hour in the library, is that while sifting, skimming, reading & extracting from all these hundreds of papers - the good, the very bad, the plain ugly & the frankly unpublishable - some small corner of your brain will be thinking, processing, finding faults, making new connections & generally doing the science. your evening eyes may tire of the computer screen, but others of your organs are working through the night.

the hours in the library will mount up & the eyes will tire. but one day, a spark of inspiration might come, maybe when shopping for beer, walking around puddles, or planning who to avoid on social media: "why has no-one done it that way?" or "but weren't they doing things differently in the 90's?" or "isn't this method that everyone has been using for twenty years really limited?" or even: "but why is no-one really studying the nature of this thing?"

so my solution to the problem of when & how to look back on the literature, & when to go to the lab & when to the library, is really no solution at all - it is just to do both, as much as i can. rely on the literature as a guide to what is known about a thing, but never to forget that there must be some purpose to all of this reading & experimenting & experimenting & reading. maybe you'll discover the purpose in the lab, or maybe in the library, or maybe in the launderette. but discovering it is the goal, so just keep going.

speaking practically & personally, systematic review & meta-analysis seems a good choice to keep you busy in the middle of your scientific career. it's empirical scientific work for which you don't necessarily need a grant or a laboratory, but you do need some expertise in the method, some good statistical tools,lots of time & good access to university resources. you can work at it a few hours a day, or a few hours a week. it might take months or years. it might not produce any results at all. but it might completely overturn an established paradigm.