There has been quite a lot of discussion on this site lately about peer review and how it could be improved. My wife has published a reasonable number of papers in peer reviewed journals and I have been party to discussions about how it could be done better. Over time I have had some ideas about how things could be done differently. They are presented here. Many of them have of course been independently suggested elsewhere, but not all together as far as I am aware.
Existing peer review uses social network principles
Existing peer review uses social network principles
The peer review process is similar to a kind of social network, however with “gatekeepers”. These are the editors of journals that decide what to include in their publication. Unlike on YouTube or Facebook, in order to get something published, your have to get past them first. However when something is published, the network of how people cite each other has similarities to a network of friends.
The current peer review process was started long before online social networks and the internet existed. It seems reasonable that our increased knowledge and experience of how they work can be used to improve it. The existing data can be used to start a better network.
So here is how it would start. Researchers have existing scores, citations indices etc that can be used to give them a score. This information is used to initialise the network. Not only should the number of citations be shown, but who cited them. Plotting all of this for a field would allow the visualisation of who cites whom, and would show researchers in clumps or groups of those that cited each other etc. Researchers who were cited by many different groups would show up visually and should receive higher scores than ones just cited within groups. Individual researchers would be points on a graph, with links to other researchers where they cited or were cited by that researcher.
So here is how it would start. Researchers have existing scores, citations indices etc that can be used to give them a score. This information is used to initialise the network. Not only should the number of citations be shown, but who cited them. Plotting all of this for a field would allow the visualisation of who cites whom, and would show researchers in clumps or groups of those that cited each other etc. Researchers who were cited by many different groups would show up visually and should receive higher scores than ones just cited within groups. Individual researchers would be points on a graph, with links to other researchers where they cited or were cited by that researcher.
Open peer review - No gatekeepers
In the proposed system, things would be different when a scientist submits a paper for peer review. Rather than submit it to a journal, they would submit it to some open website, online system, or group of non-affiliated websites that would store it, but not show it publicly yet. It is even technologically possible to make a fully decentralised system to do this, much like how it is done with the TOR network and other P2P networks. Invites could automatically be sent out to researchers that are linked close but not too close to them in the network described above. They could also choose to invite researchers not in their immediate network. Now when this happens, the abstract say could be submitted publicly, but the full paper sent privately to those chosen researchers. After say three accepted it for review, no other people would need to review it.
The reviewers could then offer suggestions etc like usual, but at the end of the process, they would give the paper a rating. This could be a score like 1-5 or just accept/reject. Their comments and rating would be made public so they would be accountable. (Perhaps reviewers that reject it could stay private but I think it best if not) The paper would then be published for everyone to see on the website (in the appropriate discipline) with its rating and feedback. You can of course play with the numbers and require one, two etc reviews to accept it before it is published.
Now one advantage of this is that there is no gatekeeper like the journal editor. So long as someone credible in the field saw value in your work, then it would get published in the peer reviewed section of the site. The website would be free and open with anyone able to upload papers for review. Ideas that were unpopular could not be suppressed by people that did not like them.
A more subtle endorsement of a paper than just citing it is possible
You could take the system further than this: A paper’s rating could go up after it is published if further researchers read it, critiqued it and had the option to “like” it. They wouldn’t necessarily need to cite it to indicate to the field that it had value. Journals could still have great value by sifting through the great number of papers published and bringing them to the reader’s attention, with editors of course having the option to add their own opinions and insights.
The pattern of who reviewed who’s paper, and what feedback they gave it could also make rivalries between different groups open and obvious as it could be shown visually very effectively.
The “Journal of null results”
You could take the system further than this: A paper’s rating could go up after it is published if further researchers read it, critiqued it and had the option to “like” it. They wouldn’t necessarily need to cite it to indicate to the field that it had value. Journals could still have great value by sifting through the great number of papers published and bringing them to the reader’s attention, with editors of course having the option to add their own opinions and insights.
The pattern of who reviewed who’s paper, and what feedback they gave it could also make rivalries between different groups open and obvious as it could be shown visually very effectively.
The “Journal of null results”
Many researchers from a range of fields have really wanted this kind of thing. It happens often that you spend a long time trying something and it doesn’t work for a complicated and important reason. Or you notice that many different approaches have been tried to solve a problem, but one apparently obvious one hasn’t been tried. When you try it you find it doesn’t work, and you are sure that other researchers have also tried it, and not reported the failed result. I am sure much time and effort is wasted this way. Reporting a null result in the system described here would be much easier to get accepted for wider publication. There could also be the option to just publish it without peer review for everyone to see. Once again if it gets “liked” or cited, then the researcher would get credit. There are attempts online at this already, but I think null results should be in the same place as conclusive results to make sure they are noticed.
The option of skipping the peer review process altogether
The option of skipping the peer review process altogether
As I have hinted earlier, a researcher could opt not to have their paper peer reviewed at all, but just published straight away. Now this would have to show up in a literature search as “unconfirmed” or something, but if credible researchers then subsequently read it and recommended it, then there is no reason why it shouldn’t achieve the same status as a peer reviewed paper.
Credit for finding errors in other papers
Sometimes it is more important to point out an error in a widely cited paper than to publish something original. The correction/criticism could be published in the same way, and credit given to the researcher if the criticism was accepted by a number of researchers in the field.
Introducing new people to the field
Let’s say a new student enters the field and wants to have their work reviewed. This would be simple if there was the option of their supervisor letting them use their network to request a review, and of course pointing out that this was happening. What is more difficult is for an outsider to get considered. In this case a system could be set up where an outsider gets say 3 “lives” where they can request any researcher to consider their work. They publish their paper, with an open request for someone to review it. If a number of researchers review it, and judge it to have no value, then they lose one “life”. There would probably be a few researchers in any given field who would review such articles as part on an outreach effort. If this happens 2 more times, then they can expect further papers from them to get ignored and not be included in literature searches that the online system provides.
Like before however if some people in the field “like” their work then they and their papers would gain credibility and some kind of a score.
If a group of people felt they were being unfairly treated, then they could start their own sub-field within the online system.
Rating system and better metrics
If a group of people felt they were being unfairly treated, then they could start their own sub-field within the online system.
Rating system and better metrics
It is accepted that a researcher who publishes a lot of papers that are widely cited should be valued and have some kind of a score to reflect that. The system I have described is richer in terms of the information stored in it, so it should be possible to come up with better rating systems for researchers. For example mathematical algorithms from graph theory I think could quite easily make sure a circle of researchers that just “liked” each others work all the time would not benefit from it. Also if someone “liked” everything, then each “like” could be diluted accordingly. If someone with a high score “liked” a paper, it should count much more than if someone new to the field did. There are numerous options here that have probably been discussed elsewhere.
Researchers stating their personal opinion on science issues
(Not directly related to peer review, but still relevant to science 2.0)
The online/network system I have described is a social network. Now I don’t expect a researcher to log on every day, but at least they would receive email notifications of when someone wants their work reviewed etc, and their contact details would be stored in one place. If a news media article or prominent person misrepresented their position, then they could respond quickly and effectively. For example, if someone where to say “50% of medical researchers believe that smoking is safe” then an email could quickly be sent around the appropriate field inviting them to take a survey about this. The individual researchers responses could even be kept anonymous if necessary so that what would show up would be something like:
“1500 out of 2000 people answered, 99% thought smoking was harmful”. Such a misrepresentation of the field could be quickly settled.
Summary, your thoughts?
Technological change is always faster than social change. Social networks are new and it will probably be quite a long time before their full benefit is realised, especially in situations with strong tradition such as peer review. As far as I can see the proposed system would have benefits over the current system, and less obvious drawbacks.
A null results journal has 'many researchers' desiring it? I can't find any, at least not enough to make a journal, despite saying for years I would do it. Same with preprints in fields besides physics, where arXiv spent 20 years doing it already. Researchers want other scientists to take the time to publish null results, sure, but I could start a preprint or a null results journal tomorrow and, even with the power of the Science 2.0 brand, we would hear crickets chirping waiting for submissions. What we would get are a bunch of papers unready to be published anywhere but that isn't helping anyone. 'Build it and they will come' is a myth, and that is why 200 million websites get less than 10 readers per day.