The effectiveness of peer review has been investigated in a new paper entitled Modelling the effects of subjective and objective decision making in scientific peer review that was published on 5th December, in Nature.
Researchers from the School of Economics, Finance and Management at the University of Bristol studied the system and presented a new model that improves on the current peer review system. They used a mathematical model to understand the behaviour of scientists when undertaking a review.
The following is an interview with Professor Mike Peacey, lead author of the paper.
Could you briefly describe your new model of peer review?
Our models investigate the role of reviewer subjectivity in the peer review process. We find that encouraging a moderate degree of subjectivity allows science to be self correcting. The reason for this is that if herding has occurred scientists will all submit papers which reach similar conclusions. If this herding has led to an incorrect conclusion, allowing reviewers a degree of subjectivity will increase rejection rates of these papers. This will in turn help to inform future scientists that they shouldn’t be so quick to set aside their own views.
How is it different to current models?
Some journals specifically ask reviewers to be as objective as possible, which may really put the brakes on social learning. In addition, free, open and global access to research reports have been proposed as an alternative to peer review (see http://am.ascb.org/dora). Our modelling of this process suggests that peer review (when a degree of subjectivity is permitted) can reveal more information relative to free and complete sequential publication, and therefore may be superior.
Has your study highlighted/confirmed any significant flaws in the peer review process?
The peer review system has been subject to criticism in the past. Our study highlights the natural occurrence of herding in situations where information is sequentially revealed. We show that if submissions are only judged based on objective characteristics of the paper (e.g. quality of writing, etc) then this herding will occur quickly and, crucially, will be irreversible. This means that however much research is carried out, the confidence society has that a particular hypothesis is correct is restricted.
Do you think your new method will be implemented in leading journals?
Our study brings to attention the importance of how information is transmitted through the peer review process itself. So, hopefully, editors will use this as an opportunity to carefully consider the specifics of their own field and the protocol followed by their reviewers. Whilst our work highlights the importance of allowing reviewers to express subjective opinions, we also belief that other ways in which truthful opinions are encouraged will have similar benefits. Thus our work can hopefully contribute to the debate on the role of prediction markets in science (see http://dl.dropboxusercontent.com/u/17917722/pmproject.htm) and the use of post-publication peer review (see http://pubmed.gov ).
What is the future of your research in this area?
Since peer review acts as a gatekeeper to how ideas are circulated and criticized, it is clearly an important research topic. As an economist, I’m interested in looking at questions relating to incentives present of the process itself and also the efficiency of prediction markets.