Banner
    Sexist Weather And Hermaphroditic Frogs: The Problem Of Faux Peer Review
    By Hank Campbell | July 14th 2014 09:23 AM | 10 comments | Print | E-mail | Track Comments
    About Hank

    I'm the founder of Science 2.0®.

    A wise man once said Darwin had the greatest idea anyone ever had. Others may prefer Newton or Archimedes...

    View Hank's Profile


    In today's Wall Street Journal, I have an article discussing recent problems in peer review.

    It's big news when a journal busts a 'peer review ring' that had manipulated its way to 60 papers sailing through peer review that then had to be retracted. Prior to that, Nature retracted a paper on stem cells that was obviously flawed - and everyone but the peer reviewers knew it.

    The issue may be news but it's not new. Everyone in science has joked about gaming peer review, I wondered in Penny Stock Peer Review why it did not happen more and I speculated it is because most scientists are honest.

    But not everyone is playing fair - sometimes they want to Do God's Work. They believe their claims are vital to saving humanity and that if they don't rig the system, some conspiracy will block them out. The end justifies the means, and all that. The damage to policy and public trust in science can still be substantial when that happens. Why is the public going to believe peer review on climate change and evolution, or vaccines, GMOs and energy, when junk papers about Facebook emotions and female hurricane names get a faux peer review legitimacy stamp from an elite journal?

    And then there are very expensive debacles, I note. The same journal behind the genetics of why you are a lousy gambler and those two papers above, Proceedings of the National Academy of Sciences, published a paper in 2002 that got an emergency Scientific Advisory Panel called by the EPA. It claimed feminization of male frogs due to an herbicide. It turned out to be nothing, the authors refused to show their data to the EPA and the EPA ended up giving the herbicide a clean bill of health. How did all that happen? The hand-picked, pre-chosen editor for the paper inside the National Academy of Sciences happens to be a friend of the first author of that paper. Then they teamed up to do the same thing in PNAS again, about the same herbicide, in 2010. 

    The EPA spends years on issues when they are mobilized. We wasted time and money that could have been spent doing real fact-finding about products. Career scientists inside EPA were forced to stop what they were doing because environmental groups engaged in scare-mongering, all with the claim that the studies in PNAS was peer reviewed.  Sorry, but that's not peer review, it's friendly editing, and it doesn't make the public safer and it does not inspire young scientists, it makes them cynical.

    I'm not trying to pick on PNAS, they just happen to be a high-profile publication that carries a lot of weight in mainstream media, so when they drop the ball, it has expensive consequences for taxpayers and collateral damage for science acceptance. 

    Every journal has papers that slip under the radar and open access journals are actually worse, but they get a pass because it's assumed in open access that the standard is lower. 
    Not only does PNAS say it is better than open access, it says their system is better than Cell, Nature and Science.

    When they brag to a higher standard, they get held to a higher standard.

    Comments

    Hfarmer
    These are very good articles Hank. You points about these shenanigans causing public distrust and fueling anti science are well founded. There are so many examples of these incidents being used against all other science.  i.e. The links in my article about Bering Strait theory.  Inspite of bone hard rock solid evidence for OoA and the migrations it implies, past corruption of the process is used to discredit that model.  Harming the public image of science  hinders progress.  

    Career scientists inside EPA were forced to stop what they were doing because environmental groups engaged in scare-mongering, all with the claim that the studies in PNAS was peer reviewed.  Sorry, but that's not peer review, it's friendly editing, and it doesn't make the public safer and it does not inspire young scientists, it makes them cynical.


     Max Planck has a good quote about this.  


    A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.
     The old guard always protects its own.
    Science advances as much by mistakes as by plans.
    Legiononomamoi
    Dear Mr. Campbell:
    I read and enjoyed your WSJ article. Among other things, it alerted me to this site (I've been a long time follower of Retraction Watch and followed (and participated in) attempts to reform scientific research and break the stranglehold mainstream journals have both on what is considered research and how research is conducted.

    That said, I feel it is important to note a few potential issues with what you state above and in your article.

    1) The EPA didn't launch any investigation as a result of Hayes et al.'s 2002 PNAS study. Investigations began over a decade earlier (the EPA classified atrazine as a possible carinogen in 1988). The company that makes (or rather, owns the company that makes) atrazine first hired Dr. Hayes in 1997. He was then asked by Syngenta (the company that makes atrazine) to investigate the effects of atrazine on the endocrine system in amphibians due to their (Syngenta's) concern over several previous studies identifying this potential for harm. The first studies Hayes conducted were submitted to Syngenta, and were supposed to be submitted to the EPA. Eventually, as various sources note, they were. On the second day (June 18th, 2003) of the FIFRA Scientific Advisory Panel, Dr. Ashby (Senior Syngenta Fellow) acknowledges this: "Now, I mentioned earlier the initiative of the panel commissioning, Hayes and Noriega, to start looking at the frogs. There was a draft final report which was actually never issued, but which has been made available to the EPA, which was delivered to Syngenta in 2000" (FIFRA SAP transcription of the "Potential Developmental Effects of Atrazine on Amphibians".

    2) Over a dozen studies like Dr. Hayes' initial work were launched and funded by Syngenta, all of which were made available to the EPA. They can be reviewed here: http://www.epa.gov/scipoly/sap/meetings/2003/june/dataevaluationreports.htm

    3) "In 2003, EPA began to collect all scientific studies that examined the potential effects of atrazine on various species of frogs, including all of the studies published by Dr. Hayes. (As part of its efforts to understand the available data, EPA scientists visited Dr. Hayes’ lab and reviewed some of his raw data.) Altogether, EPA evaluated 17 different laboratory and field studies, including 4 studies authored by Dr. Hayes. The Agency used this information to prepare a 95 page “White Paper on Potential Developmental Effects of Atrazine on Amphibians” (White Paper) supported by over 35 references.
    EPA’s publicly available White Paper found that all of the available information was scientifically flawed. Because of these flaws, no firm conclusions could be drawn about whether atrazine affects  frogs and if so, at what levels." (p. 7 (document page 11) of http://epa.gov/quality/informationguidelines/documents/5001-response.pdf)
    The exhaustive study by the EPA amounted to dismissing basically the entirety of all scientific research, as they themselves admitted. 


    4) "The question is not whether or not Tyrone’s studies are correct (again there is no perfect study), nor is the important question whether atrazine affects frogs. The important question is whether the frog studies, despite their flaws, reflect results in other vertebrates, including humans. The important question is: “Is atrazine a potent endocrine disruptor that affects wildlife and humans?” The amphibian data simply support findings in every vertebrate class examined. Atrazine exposure is associated with demasculinization (chemical castration) and feminization in every vertebrate class examined (fish, amphibians, reptiles, birds, and mammals, including humans). Effects include reduced fertility and feminization in males and reproductive cancers in rodents and humans. The induction of aromatase (the enzyme that converts androgens into estrogens) and the decrease in testosterone and increase in estrogen has similarly been demonstrated. Even an EPA laboratory, which showed that atrazine causes a decrease in testosterone and an increase estrogens in rodents, concluded that “atrazine tested positive in the pubertal male screen that the EDSTAC (EPA Endocrine Screening and Testing Advisory Committee) is considering as an optional screen for endocrine disruptors” (parenthetical phrases mine) as early as 2000 (Stoker et al, 2000).  

    Though the regulatory branch of the EPA has NEVER asked this important question, they have denied that the evidence even exists: “It has been claimed that research on frogs shows that atrazine causes changes in the production of aromatase, an enzyme that is involved in the conversion of testosterone to estrogen. It has also been claimed that other scientists have shown similar effects in other species … There is no direct scientific information to assess this hypothesis.” (Statement of Anne Lindsay before the Agriculture and Rural Development Committee of the Minnesota House of Representatives, Feb. 16, 2005).  The “scientific information to assess this hypothesis” is represented by more than 40 publications, some of which are listed below (out of the 40+, I have only listed six of my own)."

    For the list, see the link from which the above is quoted: http://thecre.com/faca/pdf/hayes-cre-response.htm

    5) Hayes' results HAVE been reproduced. For a freely available example, see e.g.,
    Langlois, V. S., Carew, A. C., Pauli, B. D., Wade, M. G., Cooke, G. M.,&Trudeau, V. L. (2010). Low levels of the herbicide atrazine alter sex ratios and reduce metamorphic success in Rana pipiens tadpoles raised in outdoor mesocosms. Environmental health perspectives (Online), 118(4), 552.



    6) Prior to 1995, there was no direct submission method for authors of PNAS studies. The ways in which both of Hayes' studies were submitted were how virtually all PNAS studies were submitted from 1914 past the introduction of Direct Submission in 1995. The work by mathematician John Nash, made famous by the book and movie A Beautiful Mind, was published this way. The 2002 "communicated by" Dr. Wake is said to be so because of a now replaced submission method referred to as Communicated papers by PNAS. Like papers with Preferred Editors (such as Hayes' 2010 study), they are not exempt from review. Rather, they are subject to a more intense review and considered to be exceptional: "publication of papers that NAS members recognize as exceptional, but out of the mainstream, is encouraged under the Prearranged Editor system, as it has been with Communicated papers" (http://www.pnas.org/content/106/37/15518.full). Also, PNAS requires all data be made available for all studies: "To allow others to replicate and build on work published in PNAS, authors must make materials, data, and associated protocols available to readers" (http://www.pnas.org/site/authors/journal.xhtml). I've read the "Supporting Information" section of Hayes et al.'s (2010) study, so while I can't definitely say whether Hayes' 2002 study made all data available their 2010 study is pretty clear on this point.


    7) In addition to other studies that partly or wholly support Hayes' findings, Hayes' research on this matter has been published in multiple other journals over the past decade and cited thousands of times by other researchers who likewise identify harmful effects of atrazine, though only a few have focused specifically on whether it causes, as you put it, "sexual changes in frogs". It is a central part of an an extensive body of research (not all of which, by any means, supports Hayes' findings) on the effects of atrazine, which is “one of the most widely used and commonly detected herbicides in North America and receives a great deal of scientific scrutiny by both governmental and academic investigators. It is a known endocrine disruptor in mammals, birds, reptiles, fish and amphibians (Bisson and Hontela, 2002; Fan et al., 2008; Hayes et al., 2002, 2003; Holloway et al., 2008; McMullin et al., 2004; Stoker et al., 2002) affecting normal reproductive function and development in these organisms. In studies with Xenopus laevis tadpoles, changes in gonadal development were noted after exposures to low, environmentally relevant concentrations (Hayes et al., 2002, 2003; Tavera-Mendoza et al., 2001, 2002). Consequently, much research effort has been focused on studying atrazine’s reproductive effects on amphibian species at these low exposure levels (Carr et al., 2003; Hayes et al., 2002, 2003; Hecker et al., 2004; Jooste et al., 2005; Kloas et al., 2009; Langlois et al., 2010; Murphy et al., 2006; Oka et al., 2008; Rohr et al., 2008).”
    Zaya, R. M., Amini, Z., Whitaker, A. S., Kohler, S. L.,&Ide, C. F. (2011). Atrazine exposure affects growth, body condition and liver health in Xenopus laevis tadpoles. Aquatic Toxicology, 104(3), 243-253.


    I'm skeptical of any studies showing the harms of pesticides, genetically modified crops, etc., as much as I am the alleged benefits of "natural" (whatever that means) foods and "organic" products. I doubt there is cause to worry about atrazine. However, this isn't an example of a failure of the peer-review process. I had intended to write a piece on why I believe that, while your article was correct in many respects, the problems with the peer-review process are more of a symptom of bigger problems than they are the cause for central concerns with scientific research, but after spending 18 hours reading research, EPA reports, FOIA documents, court documents, etc., on a pesticide I'd had no previous knowledge of over the controversy concerning a scientist I'd never heard of, a quick and largely supportive response turned into a detailed investigation of a matter I really don't care much about. However, as you are one of the few attempting to better science reporting/journalism, I thought it important to note that your reports on this research missed quite a bit. Or maybe I just didn't want to see the time spent reading hundreds of papers I'm not particularly interested in go to waste.

    Regards,

    Andrew Messing
    Hank
    It's not peer review if your personal friend is the only one who saw the raw data and stamped it for you. It's that simple. That's not peer review and never has been, it's gaming the system.

    Why do you focus on a plea for Hayes rather than sexist hurricane names or genetic gambling or any of the other papers that had the same lack of credibility and got the same free pass? That looks odd. Anyway, the EPA disagrees with your voluminous defense of him:

    Anne Lindsay, deputy director of EPA's Office of Pesticide Programs in 2005: “Dr. Hayes claims not only that his laboratory has repeated the findings many times in experiments with thousands of frogs, but that other scientists have also replicated his results. EPA, however, has never seen either the results from any independent investigator published in peer-reviewed scientific journals or the raw data from Dr. Hayes' additional experiments."

    In 2010, Donald Brady, director of the Environmental Fate and Effects Division, said they couldn't use Hayes' work because EPA “could not properly account for the sample sizes and study design reportedly used by the Berkeley researchers. As a result, we were unable to complete any independent analysis to support the study’s conclusions.”
    Legiononomamoi

    Dear Mr. Campbell:

    First, the reason I didn't focus on e.g., the PNAS hurricane study was because I thought the study was clearly junk. Null significance hypothesis testing (NHST) is one of the most prevalent methods/research design across the sciences (medical, behavioral, cognitive, social, etc.), although it is without logical or empirical support. Not quite as prevalent (but also used in the study) is the use of statistical tests which assume the variable(s) of interest to be normally distributed over some real interval (and thus capable of taking on uncountably infinitely many values) when the variable or variables are in fact Likert-type responses that can take on (in this case) 6 values. However, this isn't (again) as much an issue of peer-review as it is rampant, pervasive, and diverse misuse of, misunderstanding of, and poor training in fairly basic statistics and data analysis across the sciences while the ability to run complex analyses with next to no knowledge increases (using software packages like SPSS or SAS). 

    Second, I focused on Hayes' work because, although I think the problems with the peer-review system are (again) more of a symptom of wider problems that have to be fixed at a different level, this was the one example in your report I thought inaccurate (and, more importantly, on just about every detail). It is quite understandably inaccurate, but as you are a promoter of and contributor to superior scientific journalism/reporting, I thought you may appreciate the "full story" (as it were). I suppose I should have been clearer that I was focusing on it because I agreed with (mostly, and certainly had no complaints with) the rest.

    1) Both PNAS articles were peer-reviewed and not by Dr. Wade. 

    You may have reason to believe that Dr. Wade and Dr. Hayes are friends, but Dr. Hayes' work was peer-reviewed and not by Dr. Wade. Dr. Wade was a “Prearranged Editor” (PE). PEs serve as member editors that are approved by the board to comment on the submission before it is even considered for review: 

    "Through a process called 'Prearranged Editor,' an author may ask an NAS member to edit a Direct Submission paper, as they would have done for a Communicated paper. When the manuscript is received by PNAS, the NAS member designated as the Prearranged Editor is asked to comment on the significance of the work and recommend a final decision on publication after peer review, which is handled—as for all Direct Submissions—by the PNAS office. Before the peer-review process is initiated, a member of the PNAS Editorial Board is asked to confirm that the Prearranged Editor has the requisite expertise and the paper is appropriate for publication in PNAS."

    (source; emphasis added) 

    PEs have to be approved and requesting one causes a delay in order to allow the PE to comment on the article before (and if) it is submitted for screening by the board: 

    "The submission will not be processed further until the PE has responded with substantive comments, or 3 days have transpired. The Board cannot guarantee that the member designated by the author will be assigned the manuscript or that it will be sent for review. If the Board decides to reject the paper without review or if the Board selects another member to handle the submission, the PE will be notified. Throughout the review process, all correspondence with an editor must be handled by the PNAS Office; authors are not permitted to contact an editor directly."

    (source)

    Finally, neither the Board nor member editors are reviewers. Peer-reviewers are qualified non-member specialists that cannot work for same institution as the author(s) whom the PNAS office contacts to review the submission. These are the actual peer-reviewers, not the PE (if a paper has one). 

    While different editorial boards differ as to how submissions are edited, reviewers chosen, etc., "peer-review" refers to the use of expert-reviewers by the editor(s), not the review of the submission by one or more editors or the decision to publish that falls to the editorial board. Thus, at best (or, rather at worst) Dr. Hayes had a friend whom he requested to have comment on, and serve as the NAS member editor of, his work rather than an NAS member editor selected by the Board (usually selected out of the recommendations authors provide for Direct Submissions anyway). NAS member editors (PE or not) cannot determine whether the paper is even approved for review rather than rejected outright, they are not the peer-reviewers, and they do not determine whether or not the submission is approved for publication. They can recommend who the peer-reviewers should be, but they can’t decide PNAS policy to have submissions sent for peer-review shouldn’t apply (same with PNAS to have all data made available).

     

    2) The EPA

     

    I’m used to the general failure of government panels when it comes to science, but usually in the opposite way (i.e., overlooking bad research and/or banning potentially harmful products based on poor evidence and sometimes at great cost). The most recent (corrected) list of studies reviewed by the EPA (Problem Formulation for the Reassessment of Ecological Risks from the Use of Atrazine: Errata Sheet for Appendix C of the White Paper - Open Literature Review of Amphibian Data) consists of a 20 page list (source) of dozens and dozens of studies published in many journals by multiple research teams over a decade. The EPA lists every, single one (with a single exception) as either “qualitative” (as opposed to “quantitative”, or literature they treat as sound) and mostly “Qualitative-Lower level”, or “Invalid”. 

    The EPA does not set, is not consulted for, and does not follow procedures of organizations/institutes that do, any authoritative guidelines for research practices. On what basis, then, does the EPA determine that scientists aren’t capable of scientific research? Because the EPA guidelines were developed mostly as government standards for corporations wishing to submit research to the EPA itself. They made this clear back in 2003 when they admitted they didn’t have the control over and access to data and protocols for peer-reviewed literature: 

    “It is important to note that the review of these registrant-sponsored studies was more detailed than for the studies obtained from open literature. For these latter studies, the reviews were less detailed because EPA did not have access through the study authors to the full range of raw data and quality control information required for registrant-submitted studies.” 

    They described it in FIFRA’s 2012 SAP: 

    “A major deficiency with the available amphibian data is the lack of an adequately standardized protocol for chronic amphibian testing, especially in regards to husbandry for native species. The available amphibian data represent a variety of methodologies that have various uncertainties, which limit the ability to discern effects from atrazine without confounding factors.”

    (p. 83).

     All fields of research use “a variety of methodologies that have various uncertainties” and scientific research practices are rarely standardized (exceptions include things like medical testing or the use of human subjects). The fact is that the EPA isn’t simply excluding two studies by Hayes and coauthors which are without replication or partial support in the scientific literature in favor of some massive body of peer-reviewed (or non-peer-reviewed) research. They are excluding the massive body of research itself. 

    Hank
    Okay, you are making PNAS actually look worse:
    You may have reason to believe that Dr. Wade and Dr. Hayes are friends, but Dr. Hayes' work was peer-reviewed and not by Dr. Wade.
    I am baffled you choose to doubt it because (a) they work in the same department and (b) Prof. Hayes uses him as a character reference. For a profile of him in The New Yorker, Hayes hand-picked his character reference who happened to be ... David Wake. Wake assured the profile writer that the concerns about Hayes' emails, 100 pages of threats and sexual harassment that got him called a "Cock-Fixated Megalomaniac Email Addict" by Gawker in 2010, were just Hayes having a laugh. This seems like an odd assurance to make for a casual acquaintance in a high-profile magazine, especially when he also says Hayes possessed "the greatest potential of anyone in the field.” His whole early history in the article is recounted by Wake.

    But they are not friends?

    If a panel of experts actually did peer-review these papers - which you unquestionably believe while unquestionably disbelieving that he and Wake are friends - it makes PNAS look even worse. Because what everyone knows is that there was no data, the methodology was suspect, even in a field where 'spray and count' is far too often acceptable and everyone could see it - except these supposed peer reviewers and the 100+ Editorial Board at PNAS? If it was one guy hand-walking it through editorial review, that is bad, but he alone is culpable and the Academy can shrug it off as a bad paper that slipped through out of 10,000 good ones that year, but if PNAS actually reviewed this, and it led to millions of wasted taxpayer dollars because no one thought to ask of Hayes and colleagues the one thing that every high school science teacher demands of students, that is truly alarming.

    They'd be in jail under Sarbanes-Oxley if they worked at a public company and signed off on that kind of fraud.

    There is no amount of quoting process - especially when you read about PNAS predetermined editor submissions and choose to believe they are actually peer reviewing 40 papers per day, even though nothing in their guidelines says they use peer-review for PE submissions at all -  that will absolve those studies. The one thing that will dispel all doubt is the one thing Hayes has said he will not do, not for the EPA or you or anyone else: show his data.
    Now wait a minute: It has to be said that PNAS has some unique procedures for reviewing papers that most scientific journals don't use - specifically allowing a certain number of papers to be reviewed by and submitted for publication by a member of the National Academy of Sciences.

    In other words, the main examples the author is complaining about of pseudo-peer review are, as far as I know, unique to that publication, not something inherent in most (let alone all) science journals.

    This having come from the Wall St Journal makes me think there is an unspoken agenda here: Cast doubt on science, since it has a way of raising inconvenient truths that might sometimes stand in the way of making as much monetary profit as possible from all things. (Republican love science when it helps build anti-missile defense systems, but not so much for most other things.)

    I doubt there is anyone who believe the peer review system is perfect, but like capitalism, it's still better than the alternatives. There is an ongoing debate in the science community about how to improve peer review, but this article adds little if anything to that debate, because the author cherry picks examples from one journal that has a unique set of review/publication options.

    Hank
    I think if you look at my body of work, rather than ironically making knee-jerk assumptions about conspiracy theories, you would find that you are as wrong as you can be.

    It's no secret that every journal has bad articles that slip through - I chided PNAS because they had a lot of them in a period of a few weeks - stupid stuff that seemed designed to get media attention but also things they let slip by that cost the EPA a lot of time and money.  But the company I applauded for for mandating open data, PLOS, I have also been critical of on many occasions. So there is no favoritism. If PNAS only has 100 bad articles a year, they might have 9900 that are okay.

    But Francis Collins has not mobilized NIH leadership to try and fix the issue because he thinks 'better than the alternatives' is good enough for the $30 billion he is accountable for. He wants the system to be 21st century. 

    So I do not share your nihilism that peer review can't be improved. I am not certain post-publication will make a difference right away, but it sure catches a lot of peer-reviewed papers that journals let pass.


    Legiononomamoi
    Dr. Mr. Campbell:
    I contacted PNAS. ALL submissions (regardless of whether the author is a member or the author has a PE) are sent out for review by non-member experts).

    I have looked at the body of your work (not all of it; you've written a lot, but I have read your book and many of your articles) which is why I said from the beginning that "you are one of the few attempting to better science reporting/journalism", that I enjoyed the article, and subsequently said the only reason I singled out this case was because I think the rest is correct (other than that I think the peer-review problems are mostly symptoms of larger issues). I don't know why you think my pointing out a mistake is means I believe there is some conspiracy at work.
    From journals like Science, Nature, & PNAS to IOSR Journal of Dental and Medical Sciences, NeuroQuantology,&Current Biology, I have repeatedly found research relying on flawed methods, citing sources to support points the sources do not, relying on rituals & conventions over and against sound practice, etc. Often, as too many peers in the field skip the technical sections, I find mistakes and/or the researchers demonstrating an inability to understand what they themselves did buried under technical jargon and hidden using deliberately indecipherable use of mathematical notation. My favorite example is
    Kanai, R., Feilden, T., Firth, C., & Rees, G. (2011). Political orientations are correlated with brain structure in young adults. Current biology, 21(8), 677-680.

    It's not just because 2 of the 4 authors weren't scientists. It's not because the entire impetus for the study was two blatantly biased BBC employees (one a pretty famous actor) seeking to "understand" what brain abnormalities must underlie right-wing views as, obviously, no sane, intelligent person could possibly disagree with left-wing ideology of the type Firth and Rees espouse (I'm being sarcastic, in case that isn't clear). It's for one, specific flaw that's actually pretty small considering the entire study is flawed from start to finish: the authors' claim to have "performed diffeomorphic anatomical registration through exponentiated lie algebra" on MRI signal data. This sounds impressive unless one realizes that it really means they used a program (SPM8) used to analyze MRI data that has a tool called DARTEL: "DARTEL stands for 'Diffeomorphic Anatomical Registration Through Exponentiated Lie algebra'. It may not use a true Lie Algebra, but the acronym is a nice one." (SPM8 manual, p. 439n1). They lied about using a lie algebra because while they decided it would sound more impressive to spell out the acronym, they didn't know what it meant.

    I am working on a piece that highlights what I think are the more serious problems, including many that are related to peer-review but can't be fixed by changing the process. If you would like, I'll share it with you first or post it here and let you know I did so that you can see what you think of my interpretations of the problems plaguing scientific research. Again, my issue wasn't that either of the two PNAS studies on atrazine are true, I don't believe that you deliberately ignored and/or lied about the problems you identified with the studies, I do believe you had reasons for writing what you did, and my purpose was only to correct what appeared to be some misunderstandings about one example you used. Initially, I was only verifying things I found curious (the first being that there must be something more to a PNAS article about female vs. male hurricanes, only to find that there wasn't- you weren't using some popular science report about a study that misrepresented it, the actual study was really that god-awful).

    I don't know about other PNAS articles, but the two you mention didn't cause the EPA to spend money on anything (the same isn't necessarily true of Dr. Hayes' public reaction to the 2003 FIFRA SAP; that gets complicated). The initial Scientific Assessment Panel was held because it was required as stipulated in the settlement of a lawsuit filed by the NRDC against the EPA on a failure "to ensure that EPA’s registration of the pesticide atrazine does not affect any of 21 named endangered or threatened species or their designated critical habitat" (see here for a quick summary and here for the settlement). By that time, in addition to potential hazards (including sex changes to frogs) already identified in studies submitted to the EPA by Novartis/Syngenta, a peer-review study pre-dating Hayes' PNAS study had already found what he would later conclude. Further, the settlement already stipulated that the EPA had to determine the effects of atrazine for each species identified (no later than 2006 for some and 2007 for others). There are now hundreds of studies on the harmful effects of atrazine on amphibians alone, most published in journals like EcotoxicologyAquatic Toxicology, Environmental Health Perspectives, NatureEnvironmental Toxicology and Chemistry, Environmental Toxicology And ChemistryEnvironmental PollutionEnvironmental science & technology, etc. Before the 2007 SAP meeting, Syngenta was being sued by Holiday Shores in a class action lawsuit over atrazine and the EPA was once again being sued by the NRDC over atrazine for failing to do what they were supposed to do when the previous suit was settled. Meanwhile, Dr. Hayes had launched a full-on public campaign railing against critics and accusing the EPA and the panel of misconduct.


    I've read something like 9,000 pages in the last 2 days on a topic I'm not really interested in, wasted the hours and hours it took to read that much (not to mention reply to your response on my comment) instead of working, just reached out to my uncle (whose worked as a agricultural consultant/farm technician in Colorado my whole life) and his daughter (whose starting her doctoral work in Colorado in chemical & biological engineering) for a second opinion, all out of some obsessive need to find answers even if I didn't really care about the question to begin with.

    However, I can say for sure what I knew already and already stated in my first response: it is not true that the papers weren't peer-reviewed, no data was hidden, the findings (right or wrong) have been replicated, the EPA didn't spend millions because of these papers, and this isn't something improving peer-review will fix. Also, none of this in any way detracts from your main point, it isn't as though you couldn't have come up with another example of this sort that was completely accurate had you known more information, even if you left this example out your points would remain valid, and there are serious problems with the quality of scientific research passing peer-review.

    -Andrew
    Hank
    I look forward to your piece. I like that literate people can sign up here and just start making everyone else smarter. But I can't let this PNAS thing go based on a claim by an editor that is in defiance of history, so I will give yet another example.

    In 2009, they published a paper by Don Williamson denying the "the Darwinian assumption that larvae and their adults evolved from a single common ancestor" - in other words, all of evolution. 

    Do you believe, based on an editor claim this week, that was peer reviewed by non-member experts in biology? Or did Academy member Lynn Marguilis approve that directly and insist it was okay? Because every evolutionary biologist and the other members who had to defend themselves over it knows this answer to that question.



    Legiononomamoi
    Dr. Mr. Campbell:
    I didn't speak to an editor, but rather to the PNAS office&Board. It's not improbable that one of the individuals I spoke with was a member, but the main conversation I had was with those who are responsible for actually inviting peer-reviewers and securing their reviews: staff (mostly non-academic employees).

    I remember the controversy over Williamson's paper (though not well; I'll have to look back at the reports and immediate literature beyond what I have on hand). This actually presents an interesting case for you and for me. The paper (according even to critics in popular science news; I still have Dolgin's Nature reports) was sent out for review. If memory serves, either the reviews were negative, the reviewers carefully chosen so that they'd be sympathetic, or both. No matter the editorial board, peer-reviewers only comment on what is sent out to them. They can't control whether or not their views are taken into account. Ideally, of course, they are. That's the point. However, their reviews can be ignored, can be communicated to the author so that the author can respond (hopefully by revision but not always), or can be used to keep good research out in the case of unpopular theories by gatekeepers trying to ensure the a certain view isn't challenged. So it is entirely possible for a submission to be universally reviewed negatively and published anyway (this doesn't happen often, but it can and does happen). This isn't just a problem of the peer-review process failing to keep bad studies out of mainstream literature; it's also a mechanism whereby dominant views can remain dominant by excluding challenges via reliance upon carefully picked reviewers and/or editorial powers. The problem is that this is the heart of peer-review: editors selecting expert specialists to write reviews they then decide how to handle. 

    Also, I personally find it important that radical views do make it into mainstream research. First, this is because peer-review isn't intended to keep out radical views so much as it is clearly and obviously (especially fraudulent) research and (rightly or wrongly) a way to reject outright submissions by those with no name, no relevant expertise, and no reason to take seriously. Second, the problem is that there is no way of determining which challenges to mainstream views that seem to be practically pseudoscience are examples of the tiny minority of studies that turn out to be game-changers.

    Do you know what is probably the most cited paper in modern physics is? EPR (named for the authors- Einstein, Podolsky,&Rosen). It was written in 1935 as one of Einstein's last, most devastating, and most desperate attempts to show that quantum mechanics was either flawed at the core or wasn't a physical theory. He and co-authors attempted to do this by showing that QM entailed phenomena that was "obviously" nonsense, in particular nonlocality (what Einstein disparagingly referred to as spukhafte Fernwirkung or "Spooky action-at-a-distance"). It is cited hundreds of times every year not because anybody accepts the argument, but because the logic Einstein used to show that QM entailed that which couldn't happen actually was correct. It's just that his assumptions about what couldn't happen were wrong. So virtually all citations of EPR are used to show something that the authors argued couldn't be shown.

    Chomskyan/generative linguistics had such an influence it was modern linguistics. Some of Chomsky's students and some independent researchers (Fillmore, Lakoff, Ross, etc.) were vilified and ostracized for challenging what was "obviously" wrong, although much of it became cognitive linguistics and mainstream linguistics, cognitive science, and neuroscience.

    Before the "many-worlds interpretation" of QM was equated with cosmology's multiverse theory, and even before either terms existed, Everett (under Wheeler's guidance) proposed it as a way to attack/solve the measurement problem. It was so critically received he quit academia (and ended up making millions). It was revived a few decades later and is now a mainstream theory as is the realization that there is a "measurement problem".

    99 out of 100 times the challenges to the mainstream view that aren't obviously wrong are still wrong. Evolutionary theory isn't singular (it's actually the basis for entire fields). My work on fitness functions, while very related to fitness functions in evolutionary biology, is notably distinct in for example how fitness is defined in computational intelligence vs. biology. There have been many wrong, discredited proposals ranging from mainstream (e.g., the entirety of eugenics) to radically marginal pseudoscience (e.g., creationism). Yet some few of these have turned out not only valuable but have changed the field. Peer-review isn't, or at least shouldn't, be a matter of keeping out challenges to the consensus. Judging papers by whether or not most experts agree ensures stagnation and prevents progress. The peer-review system isn't meant to weed out bad scientific research from good; that's what scientific literature is for. Once something is published, this means nothing more than that it is now ready for EVERY expert to tear apart rather than 2-6 chosen by some editor(s). This is the real peer-review. Unfortunately, the increasingly specialized nature of scientific fields has limited the number of experts who can be consulted to review much research (and increased the connections between authors such that they end up reviewing each other's papers). Also, more and more researchers are simply using published research uncritically, citing bad papers using poor methodology that they then repeat (and add perceived legitimacy too).

    Clearly, bad papers make it past peer-review. As far as fraud is concerned, this is DEFINITELY something that good peer-review should detect and prevent. However, when it comes to radical ideas, challenges to what most specialists believe, etc., the more rigorous peer-review becomes the more we encourage gate-keeping, not review. Despite the somewhat varied views on rather central notions within evolutionary theory, truly unsupportable ideas that have been published have been dealt with in the manner that the sciences have relied on for self-correction and progress since before Newton.

    The bulk of scientific progress existed without peer-review. We have, I think, as many examples of the failure of peer-review to keep out flawed or fraudulent papers as we do rejecting sound studies because they challenged received views/consensus. Personally, I'd err on the side of caution and let questionable papers pass peer-review in order that they be challenged not by a small number of selected peers but by the entire community of peers. The problem is that (as is noted by Collins and co-author in the Nature paper you cited) the avenues for authors seeking to challenge published studies are overly limited, while one can cite poor studies based upon reading their abstracts in order to support equally bad papers that are then accepted not on merit but on their proper support by previous work, their accordance with mainstream views, and their use of methodological ritual.

    In short, I absolutely believe the paper you mention was reviewed, and by experts. I also recognize that it doesn't necessarily matter if every review is negative, that experts can be chosen who are sympathetic, and that experts in many fields aren't sufficiently acquainted with the technical aspects of the tools (whether mechanical or mathematical) they use. Also, peer-review isn't of much use when (as is the case for certain journals) "peers" are fellow creationists, promoters of pseudoscience new age/alternative health models/methods, baseless use of poorly understood modern physics, spiritualistic "science", etc.

    Hopefully, the above made some sense!

    Andrew