The Relative Value of Different Types of Information
October 15, 2021
To what extent are research findings likely to have identified an actual trend? What is the likelihood that the findings are attributable to nothing but random variation?
The “null hypothesis” is the assumption that the findings do not identify a trend, that the findings are attributable to nothing but coincidence. The p value is the likelihood the null hypothesis. It is the likelihood that the findings are brought about by nothing but random chance.
A low p value means there is a high likelihood that the study has identified an actual trend. A typical scientist requires that, in order for study results to be meaningful, the p value must be .05 or less, meaning that the likelihood of results being the result of a coincidence must be 5% or less.
Almost all scientific research papers report p values. In fact, if study results are not subjected to rigorous statistical analysis with resulting p values, the study should probably be ignored.
Achieving low p values becomes possible when the study is well-designed.
The “power” of a study is the likelihood that the study will achieve meaningful results, the likelihood that the study will detect a difference if a difference exists.
One way to increase the power of a study is to increase the number of people being studied.
Imagine you run a pharmaceutical company and you have a new drug that is intended to prevent heart attacks, and you want to find out whether it works.
So you direct your employees to perform a yearlong study where 100 subjects are given a sugar pill (placebo) and 100 are given the new medication. At the end of the year, 2 people who got the placebo had a heart attack and none who got the new medication had a heart attack.
So the new medication prevents heart attacks, right? Not so fast. The study was insufficiently powered and the p value is greater than 0.1. So there is a greater than 10% chance that the medication does not actually prevent heart attacks..
Now let’s imagine there are 200,000 people in the study; 100,000 get the placebo and 100,000 get the new medication. After a year, 2000 people on placebo have a heart attack, and 100 people who took the new medication had a heart attack.
Now we have something. The p value in this study is extraordinarily low, below .0001. That means the chance the results are caused by pure coincidence are less than .01 percent which is less than 1 in 10,000.
As a physician, I would have great confidence prescribing that medication. I could rest assured that I was giving my patients a medication that has a high likelihood of saving lives.
So what about anecdotal evidence? In other words, what about stories about certain people and how those stories should influence medical decision-making?
For example, a patient says to me, “Last year, my sister, Bitsie, my uncle Fred and my cousin Billy Bob all got a flu shot, and within two months all three of them got a cold. I don’t want a cold and I’m not getting a flu shot.”
The patient is placing way too much value in data that have approximately zero actual value. If 200 people are way too few far a study to have value, then looking at only three people is that much more useless.
We humans are story-tellers and story-consumers. We use stories to make sense of our world. As a result, it is extremely common for people to place way too much value in anecdotes when making medical decisions.
In other words, basing medical decisions on stories is not just a terrible idea; it is an extraordinarily common terrible idea.
Thinking like a scientist requires us to abandon deep-seated modes of thinking and it requires a massive effort. But it is well worth the effort because it allows us to see the world with a much sharper focus.