Fundamental Science: In how many ways is a scientific study done?

In 1747, James Lind, physician to the British Royal Navy, was stationed aboard the HMS Salisbury🇧🇷 At the time, long-haul trips had an unwanted passenger: scurvy. Today it is known that it is caused by vitamin C deficiency, but until then its cause was not known. Many people died, despite the most varied treatments – from fresh vegetables to an elixir of sulfuric acid, alcohol, sugar and spices.

On a certain crossing, having to deal with an escalation of the disease, Lind chose 12 crew members with similar symptoms and accommodated them in the same room, giving them the same diet, except for one detail. Patients were divided into pairs and each pair received a different prescription: cider, drops of said elixir, two spoons of vinegar, half a mug of sea water, laxative, and the last pair, two oranges and one lemon a day. After two weeks, only this last pair had improved. This was one of the first controlled experiments on record.

Now, what makes an investigation an experiment? It is the manipulation of the object of study. If, instead of assigning a diet to each pair, Lind had allowed the patients to eat whatever they wanted, the doctor would have done an observational study.

In an observational study, scientists, as the name says, just observe, without interfering. For example, if you want to know if coffee is good for headaches, you can compare a group of people who drink coffee every day to another group of people who never drink coffee, and then see which group reports headaches more often – this one it would be a cross-sectional study. Another alternative would be to follow the same group of people for several years, monitoring how much coffee they drink and the frequency of headaches over time – a cohort study.

You can imagine this as an endless discussion about scientific studies with a skeptic. You begin by observing that many people with a headache drink coffee and soon the symptoms disappear. Coffee cures headaches, you say. The skeptical person responds: “Maybe the pain would go away on its own, it had nothing to do with the coffee”. You persistently collect observations about several other people, some who have coffee when they have a headache, some who do not. And notice that those who drink coffee report that the pain is gone.

More confident, you go to the skeptic and announce once again that coffee cures headaches. Your interlocutor says: “It could be that the people who drink coffee are exactly the people who already know that coffee works for their headache – perhaps the pain is caused by withdrawal from coffee. But that doesn’t mean that coffee is effective for all the headaches”.

With your spirits somewhat cooled, you decide to undertake a study with experimental manipulation, so that it is easier to attribute the resolution of the headache to coffee – now approaching what Lind did on the ship. You take two people with a headache and toss a coin to choose the one who will have breakfast and the one who won’t. Half an hour later, only those who drank coffee no longer have a headache.

Reinvigorated, you claim with more conviction that coffee cures headaches. The skeptic immediately retorts that a one-person control group is unimpressive; that might just be a coincidence. Devastated, and at this point you already have a headache, you buy a coffee because you’ll have to spend the night gathering more people with this symptom…

Of course, the science is not so simplistic, but this imaginary discussion with the skeptic captures the essence of the reasoning behind good study. That is, the one that allows us to confidently extract the answers we want – in this case, whether coffee cures headaches or not. You’re always adding controls to handle the objections someone would make.

Collectively, scientists act as skeptics of one another, always looking for alternative explanations for results. An experimental manipulation diminishes the strength of one of these alternative explanations – that the result is a correlation, a coincidence, but that it does not imply causation.

Of course, a study by itself is not conclusive evidence of anything, whether experimental or observational. Science is built on consensus, with evidence accumulated over several independent studies. In some research fields, it is quite common to do systematic reviews and meta-analyses. These are studies that attempt to summarize all the accumulated evidence on a particular issue.

Nor does this mean that experimental studies are inherently superior to observational studies. The reliability of a study is a property of the study, not the type of study. The advantage of an experimental study in identifying causation is not a good reason not to be critical of that particular study and to consider any experimental study above an observational study. For example, the conclusion that smoking causes lung cancer, in the 1950s, was based on observational studies.

It is interesting to think that there is no ready-made scientific method. The very idea of ​​scientists about what makes a study good evolves over time, either because new statistical techniques appear, or because of the appearance of new ways of manipulating the objects of study or even new types of study – as when Lind implemented an experimental intervention on the ship and solved the local problem of scurvy.

🇧🇷

Kleber Neves is a neuroscientist and science manager at the Serrapilheira Institute.

Subscribe to the Serrapilheira Institute newsletter to follow more news from the institute and the Fundamental Science blog.