Has the peer review process lost credibility?

What do the following stories have in common?

• Andrew Wakefield’s influential 1998 paper suggesting a connection between autism and the MMR vaccine is fully retracted by the journal that originally published it.

• Thousands of leaked emails appear to show climate scientists colluding to suppress the publication of research.

• 14 eminent stem cell biologists complain that an obstructive clique has been blocking the publication of research.

• The Intergovernmental Panel on Climate Change is taken to task for predicating its Fourth Assessment Report on ‘grey literature’.

• A computer-generated hoax paper from the ‘Centre for Research in Applied Phrenology’ (CRAP) is accepted for publication in the Open Information Science Journal.

For one thing, these are all controversial developments in science and medicine that have hit the headlines during the past 18 months. But more strikingly, these stories all involve some sort of dispute relating to peer review – the process whereby new work in a specialist field is subjected to scrutiny and criticism by others in the field, before being officially published.

Disgruntlement over aspects of peer review is nothing new within science and academia, where the vagaries of the review process can be challenging and frustrating. But it is unusual for disputes about the process to be aired publicly and prominently, and it’s worth asking why this should happen now.

One development that has helped to make peer review more contentious is the emphasis on ‘evidence-based policy’ in the modern world. Basing policy on scientific evidence is widely seen as a Good Thing, because it places political decisions on a more objective footing.

This has led to a blurring of the distinction between the publication of scientific findings, and what we then choose to do after taking account of those findings. The upshot is that there no longer seems to be much latitude to derive meaning from evidence after it has been published, and to make political and moral rather than scientific decisions.

Too often, it is assumed that particular policy directions follow ineluctably from published research, and that right-thinking people must formulate their policies accordingly. This assumption changes our expectations of the peer review process, and sets it up for a fall. Instead of being a useful way of promoting standards and impartiality, peer review suddenly finds itself the arbiter of what research gets to dictate how we run our affairs.

Little wonder, then, that there is now such a public outcry whenever the peer review process is criticised, or is thought to have ‘failed’. Rather than being a hurdle that researchers need to clear, peer review in today’s circumstances is seen as a sort of quarantine, which ideas must pass through before being considered fit for addition to the all-important evidence base. Such a vision of peer review has less in common with science, than it does with the imprimatur given by the church to works considered free of doctrinal error.

As someone whose work involves promoting advances in genetics, assisted conception and embryo/stem cell research, I believe that science, expertise and peer review are important and worth defending. But I also believe that the form of assurance that people have been led to expect from peer review – effectively, a guarantee of truth – is not something that the system can or should be expected to deliver.

Throughout October and November, The Independent Online is partnering with the Battle of Ideas festival to present a series of guest blogs from festival speakers on the key questions of our time.

Sandy Starr is Communications Officer at the Progress Educational Trust and Webmaster of BioNews. He is organising the debate ‘End of the peer revue: has the peer review process lost credibility?’ at the Battle of Ideas festival in London on Sunday 31 October (more information here: http://www.battleofideas.org.uk/index.php/2010/session_detail/4088/).

Recent Posts on Battle of Ideas

• 'Abortion industry': A slur on the women who support their choice
• Opera for the masses, by the masses?
• Twitter trolls and the Health Bill
• The bitter taste of sugar prohibition
• Welcome to the age of bespoke cultural identities