A doctor and a lawyer were talking at a party, but they weren’t having fun because people kept interrupting to tell the doctor about their health problems and ask for free medical advice. After a while, the annoyed doctor asked the lawyer,
“How do you stop people from asking you for legal advice when you’re at a party?”
“I give it to them,” replied the lawyer, “then I send them a bill.”
The doctor was shocked, but agreed to give it a try. The next day, still feeling slightly guilty, he prepared the bills for the people who asked him for advice the previous night. When he went to place them in his mailbox, he found a bill from the lawyer.
I’m not a medical doctor, but as a medical researcher I get a lot of questions too. What do I think about herbal medicine? Is chemotherapy worth the side effects? Can medical marijuana cure depression? What about vaccines and autism? The answer to these questions can only be found by looking at the evidence. If there is good evidence a treatment works we can probably trust it; otherwise we should be careful. Because we’ll all probably need to know about whether some treatment works at some point, we should all know what good evidence is. The problem is that media love splashing headlines about ‘magic bullet’ treatments before there is evidence proving they work, and with one or two exceptions, academics use incomprehensible mumbo jumbo to explain evidence. I think evidence is easy to understand. If you can understand a fair race you can understand the difference between good and bad evidence.
A fair start and randomized trials
I could tell you I was faster than Usain Bolt, but unless I proved it by actually racing him you wouldn’t believe me. Before a competitor starts a race they have to show up. Yet many medical studies don’t even compare treatments against anything—they don’t ‘show up’ for the race. For example a researcher—often one paid by industry— might give you vitamin C when you catch a cold. Then, if your cold went away in 5 days, he might say the cold went away because of the vitamin C. But most colds go away in 5 days without any treatment so the vitamin C probably didn’t do anything. To check whether taking vitamin C helps, you need to compare people who take vitamin C with people who don’t take vitamin C. Only if the colds in the group that got vitamin C went away faster than the other group could we say that vitamin C ‘won’. But the start of the race would have to be fair…
If I took a massive head start in a race against Usain Bolt I might cross the finish line first but that’s cheating. Yet unfair ‘head starts’ are common in medical research, because the groups that take the treatment (like vitamin C) are often different from the groups that don’t. For example a researcher might give younger, healthier people vitamin C and not give vitamin C to older less healthy people. But since young healthy people’s colds go away faster than older unhealthy people’s colds, that wouldn’t prove anything. The groups that take or don’t take vitamin C have to be similar. To create similar groups, scientists flip a coin to decide who gets vitamin C and who does not. (Actually they don’t really flip a coin, but they use a computer to achieve the same thing.) When we flip a coin to decide who gets what we have a fair start and what is called a randomized trial.
Blinding to stop cheating along the way
(the cars in the picture above are exactly the same size: humans make biased observations)
A cyclist was recently caught with a hidden motor in her bike. Even a child knows that is cheating, yet this kind of ‘cheating’ is common in medical research. If the doctor believes vitamin C works (or if they are being paid by the company who makes a drug) then they might interpret a little sniffle as a cure in the person who took the vitamin C and as failure to cure in the person who didn’t take the vitamin C. The same goes for the patients and everyone else involved in the trial. If they believe that the treatment works, they can make biased observations or cheat when they report whether the treatment works or not. The best way to prevent this bias is to design a ‘blind’ trial where patients who don’t get vitamin C get a pill that looks like vitamin C but that is just a sugar pill, and not tell anyone which is the real vitamin C and which is the sugar pill (the ‘placebo’).
Systematic reviews to make a better final judgment
In 1877 Oxford and Cambridge supposedly tied in the rowing Boatrace. However most people watching said that Oxford was ahead by over 6 feet. The problem was that the finish judge, a guy called “Honest” John Phelps, was 70 years old and blind in one eye. A fair race needs a fair finish judge, which is why they now have photo finishes at the Olympics and cameras watching the goal in football games. With medical research, a fair assessment of whether a treatment ‘wins’ has to be based on all relevant evidence. You can’t ‘turn a blind eye’ and ignore some studies. For example imagine there were 10 trials comparing vitamin C with a placebo, but vitamin C worked in only one of them. A fair judge shouldn’t just pic the single trial where vitamin C worked. He would have to look at all the trials together in what is called a ‘systematic review’, to check if there is an overall benefit.
Congratulations! In just a few minutes, you’ve learned the basics of evidence: if there is a systematic review of blinded randomized trials showing that a treatment works, then it probably does. Otherwise we should remain skeptical. Now you are probably asking whether there is good evidence showing that vitamin C can cure the cold. I just typed “systematic review vitamin C cold” into Google, and learned that yes, there is a systematic review of blinded randomized trials suggesting that taking vitamin C doesn’t make you get fewer colds, but it might make your cold go away a bit faster.
Some people will tell you that there is a lot more to learn about evidence than I’ve explained here, and they are right. You need an entirely different kind of evidence: qualitative research to learn about people’s feelings. You need long-term observational research to detect harms, and we will always need to use judgment to adapt evidence to individual patients, and lots more. So it is not surprising that people do weeklong courses, Master’s degrees, and even PhD’s about evidence. But understanding the basics I’ve explained here can get you a long way. There is just one (depressing) thing…
One last (depressing) thing
Many trials are never published, especially those with negative results, and some of the ones that are have hidden biases that experts can’t detect. This means that even systematic reviews of blinded randomized trials are often corrupt and can’t be trusted. This makes my colleagues so sad that some of the best researchers say medicine is having a crisis, others say it has been hijacked, and some are fighting back, looking for a cure to this depressing state of affairs. As a geeky philosopher I don’t really worry if I don’t know the answer because Socrates was the smartest philosopher in Ancient Greece and he thought he didn’t know anything, so he wandered around asking questions. Because of the problem with evidence, we all have to take a page out of Socrates’ book and ask: What will happen if I don’t take the treatment? What will happen if I do take the treatment?
In case you are wondering, marijuana won’t help cure depression about bad evidence. Some studies say bit might have a small effect, but too much probably makes it worse. Since there are no systematic reviews of blinded trials on the topic, even a beginner knows that we can’t trust a headline saying that marijuana cures depression.