Peer review for researchers
This report investigates the pros and cons of peer reviews.
- What is peer review?
- How does it work?
- Support and criticism
- Is it effective?
- Is it fair? Subjectivity and transparency
- Is it efficient? Speeding it up and lightening the burden
- New challenges and opportunities
Peer review is a quality check used in several circumstances, such as when evaluating:
- Applications for funding, and project reports after funding ends
- Conference presentations and journal articles, both before and after publication
- Work products, to determine promotions, positions, and funding
As that list indicates, peer review is often tied to financial concerns, but the review itself is often done by unpaid volunteers who do it because they think the work is important, because they get mentioned en masse in publication “thanks” at the end of a year, or because they get some small item in trade (like a free or reduced subscription rate to the journal for which they worked).
Peer review improves quality, but perhaps at a cost:
- It takes a longer amount of time to get something published
- Reviewers may not catch lies and theft, and may be biased, subjective, and inconsistent
- It encourages conservative thinking instead of riskier innovation
- It relies too heavily in unpaid reviewers’ labor, which is an unsustainable model when there is a high and increasing number of things to review
That it encourages conservative thinking may be a natural consequence of reviewers having natural biases. If a person has an incentive to be published, then that person will write what people will agree with — to meet the incentive.
- To make a parallel with a different situation, isn’t the psychological mechanism at work here the same that’s discussed when people argue for or against likes (like on Facebook) or upvotes (like on Reddit)?
These institutions use different levels of transparency to encourage objective reviews:
- Double-blind review: Neither the reviewers nor the submitters are identified to each other
- Single-blind review: The reviewers know who the submitters are
- Open review: The reviewers and submitters are both made known to each other
Each of these methods have their pros and cons. Anonymity can hide theft and bias on the part of the reviewers, but also it makes it easier for reviewers to critique work, especially if the reviewer is in a less senior role than the submitter.
- I wonder if the stakes are too high for researchers. This report mentions actions like theft, thwarting the efforts of others, attempts to game the system, and a system that cannot scale up to accommodate an increasing volume. And to what end? Half of academic papers are never read, and 82% of humanities articles are never cited . Maybe it’s time to step back and identify the problems we are trying to solve, and then ask if there are now other ways to solve those problems.
Some people are now suggesting that reviews only be conducted after publishing, so that the system moves faster. Others argue for establishing alongside peer reviews a formal system of evaluations through comments and ratings.