Want to find the flaws in a study? Ask a competitor, just like you would ask a competitor about the flaws in an iPhone or any other product.

And if you really want to see griping about the flaws in a study, ask someone who happens to believe just the opposite. 

When a paper comes out that uses the exact same terminology as studies that advocates happen to like, conservative scientific verbage like using the word "possible" is touted as a weakness and they will insist there is no "clear" correlation. When culturally contradictory papers are published, we are told cross-sectional studies can't tell us anything at all about causality, exactly the opposite of what we read about papers confirming a position.

Ground zero in this 'we only accept science that sells our world view' culture war is food.

For example, Dr. David Perlmutter, author of Grain Brain, read about a "possible association" between gluten and Alzheimer's disease using 13 subjects and declared that by changing the diet of people, he could prevent half of Alzheimer's cases. Who bought that claim without question? Every single pundit riding the anti-gluten fad. Would Perlmutter have given interviews about such a weak link, saying 'Well, maybe my book was wrong' if a "possible association" of 13 people found that bread prevented Alzheimer's? We know the answer to that.

Recently, a meta-analysis disputed that saturated fat caused heart disease. It caught mainstream media attention because decades of conventional wisdom had been overturned and journalists got to use words like "baffled" and "stunned".

Who embraced this flawed meta-analysis? The same people consumed with the naturalistic fallacy that GMOs are bad and organic food is good, that sugar is bad, that gluten is bad.  As Texas State University professor James McWilliams wrote in Pacific Standard, it was immediately embraced by people desperate to promote their "eating like grandma" agenda, the same people who are against almost all modern food and immerse themselves in a naturalistic fallacy.

I don't know how their grandmothers ate but mine wanted modern food as much as possible. My kids think it's cool that I can make butter in a mason jar in what looks like 15 minutes, but if they do it precisely one time they will want to go back to the store.

Critics jumped on the flaws they never notice about studies that affirm their beliefs - the 72 study selection for the meta-analysis was biased, the statistics were bad, the researchers used a shoddy methodology, they didn't look at the primary data.

Who didn't level any of those charges? Pundits in the war on processed food. One prominent food expert, who claims you will look like her if you follow her diet, only eats food with chemicals she can pronounce. She can presumably pronounce butter and eggs and bacon, so maybe we should eat that three times per day. 


Vani Hari, anti-GMO scare-monger. Is it any surprise she was a delegate at the Democratic National Convention? Is it any surprise she uncritically embraces butter, unless it's Monsanto Butter? Image credit and link: FoodBabe

In reality, the people embracing the study were not suddenly embracing science or evidence, they were embracing the mythology of a buy local, buy-unprocessed-food past. 

And what about meta analyses in general? It is surprising they are embraced so readily so often. Only when something really violates popular belief does it get dissected. Want to see how the strengths and weaknesses of data get washed away to make a conclusion that few researchers want to debunk for fear of a backlash? Use the same critical filter for studies on second-hand smoke.



What's left out of the modern narrative is that the science regarding butter and heart disease that got it on the 'bad food' list was never solid in the first place, it was meta-analyses and claims of "causalation". We see it with gluten today, we saw it with DDT 50 years ago. So claims that meta-analysis and correlation is a flawed approach now means we have to overturn practically every epidemiological paper.