Citations are a time-honored measure now used to assess scholarly standing and evaluate academic productivity by funding committees that control government research.

For that reason, citations that are critical in nature, and point out limitations, inconsistencies or flaws in previous work, can be detrimental. A new paper in Proceedings of the National Academy of Sciences finds that negative citations were more likely to criticize highly-read papers, they tended to originate from scholars who were close to the authors of the original articles in academic discipline and social distance - but at least 150 miles away geographically. 

Which lends credence to the adage 'before you criticize someone, walk a mile in their shoes - because you are a mile away and you have their shoes.'

The scholars suggest they are the first to systematically quantify and examine negative citations. They began with 15,731 full-text articles that had been published in the Journal of Immunology, the top academic journal in that field. From those articles, they extracted 762,355 citations, which referred to 146,891 unique published papers. The researchers chose immunology because they had access to the papers.  

Beginning with a manual process, they created a set of citations that was used to train a natural language processing program which classified the remainder of the citations as either objective or negative. The process identified 18,304 negative citations, about 2.4 percent of the total citations studied.

"The majority of these negative citations appear to take issue with the 'results' and 'discussion' sections of the papers, so they are really not attempting to overturn theory," said

"Given that we rely so heavily on these citation metrics as measures of quality, it's important to note that the intent of these citations isn't homogeneous," said Alexander Oettl, an assistant professor in Georgia Tech's Scheller College of Business and one of the study's three co-authors. "They can be attempting to constrain results, note inconsistencies with other research, point out statistical flaws or correct other issues."

In their study, the authors point out the relative rarity of these negative citations, and speculate that they could either demonstrate a "limited, uninfluential role" for criticism in science - or show a hesitancy to criticize.

"Making these negative citations isn't without cost," Oettl noted. "There could be reputational harm from making these negative citations, and if your criticism turns out to be false, this could heavily impact your reputation within the field. Without this cost, perhaps we might see more overt criticism appearing in the journals."

The researchers found that while negative citations tended to come from scientists close to the narrow academic topic, the criticism was more likely to involve geographic distance.

"We see that the probability of making a negative citation is much, much lower if you are co-located with the scientist whose work is being critiqued," Oettl said. "That potentially speaks to the social component, the social cost of criticism - you don't want to criticize someone you may run into on campus. Another possible interpretation is that these issues may be aired face-to-face among scientists who are located near each other geographically."

In the study, frequently-cited papers were more likely than less-read papers to get negative citations, which stands to reason, Oettl said. "These are typically more influential papers, so this may have more to do with more people reading them and more of an incentive for scholars to take issue with important papers," he explained. "Furthermore, less important papers may receive less scrutiny, as pointing out limitations and shortcomings of trivial work will not drastically shift the scientific frontier."

Replication of previous work helps ensure the accuracy of research, and criticism of published studies can lead to correction that makes science more robust. But does the current scientific publishing system encourage enough discussion?

"The pessimistic view is that only one in 50 citations is negative in nature, so possibly there isn't as much debate as we would want for a healthy discussion," Oettl said. "But on the other hand, by the time a major manuscript makes it into print, it has undergone a tremendous amount of criticism. We now have some evidence of the extent to which criticism, in the form of negative citations, occurs within scientific manuscripts."