The paper received a lot of attention after President Donald J. Trump expressed optimism that the malaria treatment might be an off-label remedy during the COVID-19 pandemic that has much of America in a lockdown. His statement came from experts who saw it in a peer-reviewed journal, the Gold Standard for science.
The public is right to ask; if the paper was retracted so quickly after it got attention due to being mentioned by a politician during a crisis, why did it pass peer review?
Peer review is not what the public thinks it is. The overwhelmingly majority of journals get paid to publish their articles while ad-driven ones are relying on a finite pool of reviewers when the number of published studies has ballooned 20X in 20 years. Catching errors is difficult, catching faked results is even more difficult.
The uncomfortable reality is that one in fifty scientists admitted faking research by fabricating or falsifying data while 33.7 percent admitted to questionable research practices. When it comes to knowledge of others doing it, that number jumped to 72 percent. Yet it’s rarely discovered. Even when other scientists blow the whistle, repercussions are few. As Ivan Oransky, who runs the website RetractionWatch, wrote in his chapter of Gaming the Metrics, “In short, the most common outcome for those who commit fraud is: a long career.” It's easy to understand why schools may want fraud swept under the rug. Universities lose prestige when it becomes known and that means less government money. If the researcher has tenure, the school faces lawsuits and will still have to pay years of salary before any punishment can be handed down. They may not even be able to fire the scientist. On the contributor side, if you are a young scientist who reveals fraud by a senior researcher, your career may be over.
This had profound implications for public policy long before 2020. For example, America has terrific air quality when it comes to actual smog that kills, particulate matter 10 microns in size (PM10), but our pollution standards this century have increasingly focused on PM2.5, invisible particles so minute you need an electron microscope to detect them. The U.S. Environmental Protection Agency began to focus on such ‘virtual’ pollution based on a statistical correlation that did not include raw data they could independently verify. New cars were forced to meet increasingly stringent standards for PM2.5 which made them more expensive, and that kept older, less efficient cars in user longer, but it has not saved a single life.
A PNAS paper claimed that a common weedkiller turned male frogs feminine, and that caused a special EPA panel to be convened to study its safety. EPA then learned that the paper had been published after hand-picked "editorial review" by an insider at the National Academy of Sciences using a courtesy procedure for a friend. It did not include raw data so results could not be reproduced. When EPA asked Berkeley Professor Tyrone Hayes for his data on frog endocrine disruption, he refused to send it. EPA wasted seven years of taxpayer money studying that weedkiller, named atrazine, because members of Congress said they remained concerned.
The paper was not retracted despite a lack of data. It had simply included pictures and that was enough to get published.
While PNAS changed that 'good old boys network' policy after I exposed it in the Wall Street Journal, not much has changed overall. Most papers are still not peer-reviewed, making the life sciences a "house built on sand." While the administration was criticized for embracing a peer-reviewed paper, that was just politics as usual. Realistically, what should government trust if not peer review?
Except peer review itself is as full of holes now as it was in 2002. Recent analyses show over 400 papers share similar title, graph, and Western blot layouts and even that obvious scam escaped notice by journals so how often is peer review really happening? Why are pictures without raw data good enough?
It’s a valid question at a time when lives are at stake, such as in wanting to rush coronavirus treatments into use.
There is an obvious way to make fraud and errors easier to detect; stop accepting papers if the authors do not include raw data. The public are often surprised when they learn that most papers published in peer-reviewed journals don't include raw data, and that government agencies accept that.There may not be fraud but there is no way for independent reviewers to determine how valid the results are.
Yet government agencies, which should be leading the nation in calling for evidence-based decision making, are least likely to support transparency. The U.S. Fish and Wildlife service is in opposition to public access to its data. A Freedom of Information Act Request I submitted to write a Supreme Cour amicus brief on an endangered species habitat decision came back in the form an Excel file with no headers attached. The numbers were meaningless and told the public nothing about why USFWS decided that the best habitat for the Mississippi Gopher Frog was a private piece of land in Louisiana where the frog had never lived.
The Supreme Court ruled against the government unanimously in that case. How many other decisions are made based on evidence no one is allowed to see but that don’t get such high-profile attention?
Comments