Peer Review: Over-valued?
I've come across some disturbing articles lately regarding peer review. First was an article that reviewed the reviews of clinical neuroscience papers submitted to conferences and found that although editors based publication decisions on reviews, the reviews did not predict each other at all for one journal, and agreement was small for the second (Rothwell & Martin, 2000). Similar findings were reported for submitted conference abstracts.
I also found a paper reviewing the granting practices of NSERC, the main body of governmental science funding here in Canada. It found that the cost of peer review is so high that it would be cheaper to just give every scientist with the basic qualifications $30,000 per year instead of vetting and, supposedly, finding the good ones (Gordon & Poulin, 2009), which costs $40,000 per application. As you might imagine, if people didn't have to impress their peers with their research, you'd see a lot more innovation and less bread-and-butter studies. As Gordon and Poulin put it, "...control by peer review makes no sense for the allocation of scarce resources in any environment conducive of innovation..."
I am a fan of peer review, but these sobering studies have qualified my admiration for it. Peer review is good for making sure studies are methodologically acceptable, but probably has no business determining what is important.
I welcome argument to the contrary.
(If you post a comment, sign your name or I won't know who you are.)
Gordon, R. & Poulin, B. J. (2009). Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant. Accountability in Research, 16(1), 13-40.
author = "Gordon, Richard and Poulin, Bryan J.",
title = "Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant",
journal = "Accountability in Research",
year = "2009",
volume = "16",
number = "1",
pages = "13--40",
month = "January",
Rothwell, P. M. & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would expected by chance alone?. Brain, 123(9), 1964-1969.
author = "Rothwell, Peter M. and Martyn, Christopher N.",
title = "Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would expected by chance alone?",
journal = "Brain",
year = "2000",
volume = "123",
number = "9",
pages = "1964--1969",
month = "September"