Unfortunately, we live in a pop culture with information overload. Anyone can make some health claim and have it widely repeated by traditional media and the Internet, even though many of those claims are false. In the last few years I have personally seen about a hundred reports on how coffee is bad for you and about a hundred more why coffee is good for you. So what are you to believe? If everyone was required to learn basic scientific methods, skepticism and basic statistics, it would be easier to sort though the many claims. The following is presented as a guide on how you can decide which stories to ignore, and which you can trust. I apologize for the length and technical nature of this post, but hopefully it will prove useful.
The Gold Standard – Peer Reviewed and Duplicated Scientific Studies. The most reliable source I have found on health information is The New England Journal of Medicine. They have a strict editorial policy to only include studies that were conducted using the scientific method (to be explained below), were peer reviewed, and duplicated. A peer review means that medical professionals in the same specialty review your study to determine if all variables were properly removed and considered, that procedures were followed and that the data and results are correct. If using those same notes and techniques, the study can be duplicated and show the same results, then you are reasonably assured it is accurate.
Scientific Methods in Medical Studies – In order for a medical study to be correctly performed, you must eliminate as many extraneous factors as possible, to truly test one variable. The scientific method includes creating an hypothesis, creating a test that will use criterion to evaluate the hypothesis, a large enough group to be statistically relevant, and an unbiased review of the data.
Blind Studies – In a blind study, the patients being tested do not know if they are receiving a placebo (an inert pill with no value) or a real drug. The placebo effect is quite strong. People who believe they are being treated with medicine will often show improvement or side affects even though they are receiving no treatment. The placebo group is called the “control group” because they are a baseline to show how people react when they think they are being treated, and the “Hawthorne Affect” which is how people react differently when being observed in a study. The test group, receiving the actual drug, will also experience a placebo affect and a Hawthorne Affect even though they are actually getting a real dosage. By comparing the control group to the test group, you can see the difference actually due to the drug, not to the study itself.
Double Blind Studies – In a double blind study, neither the patients, nor the observers know who is getting the real drug or the placebo. It is well known that the researchers themselves will act differently if they know who is receiving the placebo. This can effect their observations and even their interaction with study participants. There is also tremendous pressure to show positive results, so by not knowing, researchers can gather more objective data. This data is then reviewed by others who DO know which group received the placebo.
Statistical Size Relevance – You need to include a large enough sample in order to know true results. For instance, if you toss a regular quarter in the air randomly 100,000 times, it will come up heads about half the time and tails about half the time. If you toss the coin only eight times, you might get eight heads in a row. This does not give you an accurate study, it could simply be a fluke. The more people in a study, the more significant and reliable the results.
Researcher Qualifications and Status – If a report mentions no researcher, or someone or some organization you have never heard of, be more skeptical. Many unqualified or rogue researchers have reported inaccurate or falsified data. Several researchers have said they could clone certain animals or create room temperature fusion and it is reported in the press, only to be debunked later by colleagues who find flawed data, falsified data, or are unable to duplicate the study.
Correlation Versus Causation – When I received my post graduate degree in Economics it was well known that when the NFC won the Super Bowl, the stock market would rise for three months, and when the AFC won, it would decline for three months. At the time, it was a 95% correlation. Any person knows that the winner of a football game does not really impact the entire economy for three months. It was simply a statistical coincidence. There were too few data points to statistically rely on it, just like tossing a quarter only a few times. Just because things correlate with each other, does not mean they are related.
An example of this is the often reported story that diet sodas cause people to gain weight. Researchers asked thousands of people to fill out a questionnaire about their weight, overall health conditions, eating habits, drinking habits, etc. The researchers found that those who drank diet sodas were three times as likely to be overweight. So, they report diet sodas cause people to be fat, and the press blindly reports it. The fact is that if you eat more calories than you burn, you will gain weight. If you eat less calories than you burn, you will lose weight. You cannot gain weight on a long term basis by drinking products with zero calories.
So why the correlation in the study? A much more reasonable explanation is that people who say they are fat, are more likely to purchase diet products. Thus, fat people drink more diets sodas because they are fat, they do not get fat because they drink diet sodas.
Questionnaire Data – The worst studies are usually based on questionnaires such as those above. Analysts study correlations and then make unfounded statements about why they correlate. I saw one major report that was based on a verbal phone survey of just 36 respondents. It was widely reported but any statistics major would be ashamed to report anything on such a small sample. There are many causes for correlation, sometimes, like the Super Bowl, they are completely random. One may cause the other, or they might be caused together. For instance, people with poor diets and no exercise might be obese, have diabetes, drink diet sodas, be older or work sedentary jobs. All of those correlations are related to poor diet and exercise, they are all symptoms, not causes.
Selection Bias – If you call people on their home phone between 9am and 5pm and ask if they are fully employed, you will get a much higher unemployment rate than if you call cell phones at 7pm. People at home during normal work hours are not employed as much as those who are at work. This is selection bias. When you send out a survey or call people, a certain group will not answer or hang up. Those who are willing to answer tend to have a different demographic than those that hang up. Again, your study will be invalid.
Interviewer Bias – If the interviewer is looking for a result, the tone of their voice and even the way they ask the question can tip an interviewee off to what they expect. In studies of this phenomenon, it was found that interviewees can change their answers by up to 30% if they know what the interviewer “wants to hear.”
What is in it for the Study Reporters – Are they doing real research, or are they trying to sell a book, a diet, a particular drug? Some organizations will rig data, sometimes repeating the same study until they get the result they want, then only reporting that one study. Be skeptical of any research done by that type of organization. For instance, if you are selling a milk substitute and put out an anti-milk study, I would be very skeptical.
So What Can YOU Do? – In general, ignore any health advice you hear or read about on the Internet or in mass media. Ask your doctor. Don’t bounce around from one drug or diet to another based on some story. If their is validity to a particular claim, your medical professional will know about it, or they can research it for you. Common sense dictates we eat a balanced diet, exercise regularly, reduce stress and get rest. For the most part, just ignore the pop culture.
Remember, one hundred years ago, pretty much everything reported about medicine was wrong. A hundred years from now, people will look back at what we know now, and feel the same way. If you put your health in the hands of the latest story, it is just going to cause you unneeded stress. Our average age has risen from around 45 to 75 in the last hundred years. That is attributable almost entirely to sanitation, plumbing and hygiene. Clean water to drink, unspoiled food, washing your hands, reduction of mosquitoes, screen doors to keep out flies have caused most of the increase in longevity. The decrease in infant mortality due to proper prenatal care is the next leading cause of longer average life. Interestingly, even now, medicine and healthcare have not added more than 3 to 5 years longevity, although they have increased quality of life. Worrying about studies will do you more harm from stress than ignoring them.