Do you feel good about getting that National Institutes of Health (NIH)? You should. As government has spent billions on Science, Technology, Engineering and Math (STEM) outreach promoting the idea that only government-funded science is real science, the rolls of those wanting to stay in academia have ballooned. Grants are more competitive than ever so if you got one, you beat out up to nine other people.
Does that mean that taxpayers are getting the best value for their money? Perhaps not. A new study finds that the NIH is no better at picking beneficial projects than if researchers were just picked in a lottery.
That means, according to a paper in the journal eLife by Ferric Fang, MD, at the University of Washington, Anthony Bowen, MS, at the Albert Einstein College of Medicine and Arturo Casadevall, MD, PhD, at the Johns Hopkins Bloomberg School of Public Health, that the expensive and time-consuming peer-review process can be scrapped.
Bonus: All of that saved money could fund more science, without anyone's taxes going up at all.
NIH rejects the majority of research grant proposals it receives. To decide which proposals to fund, NIH relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores and the amount of available funds. In recent years, the NIH has only funded those proposals ranked around the top 10 percent. The annual research budget for the NIH was $30.1 billion in 2015. Johns Hopkins alone gets hundreds of millions of dollars per year.
That means a lot of other schools are not getting funded much. School names count, studies have shown, and to decide which proposals to fund, NIH relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores. The annual research budget for the NIH was $30.1 billion in 2015 but with the huge influx of scientists wanting to stay in academia, it means the NIH has only funded 10 percent of proposals.
What if your idea was great but you didn't have the committee appearances or the grant-writing skill a more experienced principal investigator had? You probably lost, under the current system.
For their study, the researchers reanalyzed data on the 102,740 research project grants funded by the NIH from 1980 through 2008. Researchers who published a paper in the journal Science in 2015 had collected the data set. Their research suggested that peer review did in fact work -- that the highest ranked research projects funded by the NIH earned the most citations. The researchers in this case chose to measure the success of a research grant by determining how many papers that resulted from the work were published in scientific journals and then tracked how many times those papers are cited in future research papers.
The original researchers looked at all of the grants funded by NIH in those years and a significantly larger number of grants were funded in many of those years. The percentage of grants funded in recent years has been at historic lows because of cutbacks resulting from sequestration budget cuts stemming from the October 2013 government shutdown
For the new study, the authors decided to only look at the top 20 percent of grants awarded and found very little difference between the top-ranked projects and those projects ranked in the 20th percentile when it came to which would go on to be the most-cited research. What the peer review process can do, they determined, is discriminate between very good science and very bad science --that is, those in the top 20 percent versus those below the 50th percentile.
Peer review isn't cheap. The annual budget of the NIH Center for Scientific Review is $110 million. Individual NIH institutes and centers also spend a lot on peer review. That money could go toward more grants, the researchers say. The costs are not only financial. Writing and reviewing grants is extremely time consuming and diverts the efforts of scientists away from doing science itself.
The process also allows for substantial subjectivity. The objection of a single member of the committee can effectively kill a grant proposal, whether that objection is legitimate or not.
"When people's opinions count a lot, we may be doing worse than choosing at random," says says Casadevall, Professor and Chair of the W. Harry Feinstone Department of Molecular Microbiology and Immunology at the Bloomberg School in Johns Hopkins. "A negative word at the table can often swing the debate. And this is how we allocate research funding in this country."
To solve this, the authors suggest that the top proposals are first chosen by peer review and that those proposals then be put into a lottery, with grants awarded at random. Lotteries were used as part of the military draft during the Vietnam War and are used today to fill magnet schools with many qualified applicants and to award permanent residency applications. College student and low-income housing is often awarded by lottery. He says New Zealand has started using a lottery to make its scientific grants.
"NIH Peer Review Percentile Scores are Poorly Predictive of Grant Productivity" was written by Ferric C. Fang, Anthony Bowen and Arturo Casadevall. Source: Johns Hopkins University Bloomberg School of Public Health