Nature

it just happened: post-publication peer review.

End of January, two Nature papers hit the media: Acid bath offers easy path to stem cells, explained Nature. Major Discovery, titled the BBC. A Breakthrough for Science, wrote the WSJ. A young japanese scientist, Haruko Obokata, from the Riken Centre for Developmental Biology had published results showing she could derive induced embryonic stem cells in a surprinsingly easy way.

The same day, Paul Knoefler, professor in the Department of Cell Biology at the UC Davis and one of the 50 most influential people in the stem cell field, wrote a critical post on his blog explaining the findings of Obokata. At the same time he raised six key questions, that according to him need to be answered before the importance of the study could be judged. A few days later the two publications started to face many questions, we summarized the developments during this phase in an earlier post.

Since then Knoefler run an informal poll on his blog where he invites the opinion of fellow stem cell experts to vote on the question “Do you believe in STAP stem cells?“. Participation the poll increased steadily from 400 to over 1000 in the latest edition as the topic grew more controversial over the past months. We accumulated the results from his polls in the graphic below.

Screen Shot 2014-04-26 at 17.31.37

He emphasizes that this polling is obviously not scientific but may reflect dynamic changes in the judgement of people as the discussion around the paper evolves. Without a formal post-publication peer review system in place, a sort of open peer review just happened as various research labs all around the world tried to  reproduce the findings and share their results with the community openly. The papers might be retracted in the end but the stem cell field has certainly benefited from this community driven review.

Knoepfler, in one of his latest post, wrote what can be learned from this case, not only for the stem cell field but for biomedial science in general. Cellular autofluorescence and contamination might be issues restricted to certain fields but a few points, we think, can be important lessons for peer-reviewed research in general. Below is a trimmed list of these points made by Knoepfler:

  • To be a good reviewer, data should always trump big names in importance. One of the problems exemplified by the STAP papers is that big name authors can sometimes sway reviewers inappropriately to be lenient on papers. In the end, as a good reviewer, you have to keep focused on the data, not the reputation of the authors.
  • To editors, be extra-cautious about those “sexy” papers. A paper like either of the STAP ones is certainly exciting on first read and could have big impact. […] As with the reviewer caution above, editors should not be swayed by big name authors if the story seems too good to be true and if anything, the more excited an editor is about a paper the more cautious they should be in how they handle it. Paradoxical? Perhaps, but I think it’s true.
  • To journals, give all manuscripts a thorough automated checkup. EMBO now reportedly has an automated screening process for manuscripts for image issues and EMBO editors have indicated that the STAP papers would not have passed. […] Clearly this kind of automated manuscript checkup should be standard procedure for all journals.
  • Check the hype. There is nothing wrong with being excited about a paper or its potential impact, but be cautious about crossing the line to outright hype. Not everything is a “breakthrough” and that’s OK. Good, strong science doesn’t have to be a stunning breakthrough to have a positive impact. Scientists, journals, and institutions need to walk a fine line between advocating for our work publicly (which is needed) and overstating its importance, especially to the public or reporters. Many media folks are prone to hyping science as well. I believe that STAP was hugely hyped by many of the parties involved.

 

 

Advertisements

Retractions Over Time

As the pressures on academics to publish increase it may seem logical that academics would be more inclined to ‘fudge’ the numbers in order to make the results look better. Many journalists and academics have brought up this point and are concerned that fraud and academic misconduct are becoming an increasing problem within the academic community and peer-review is failing to catch it. But are the concerns justified?

Below is a graph showing the number of retractions from PubMed. In 2011 retractions peaked at 373. Since then there has been in a decline in the number of retractions. However, the number of retractions seem to be on the rise again in 2013.

Screen Shot 2014-04-27 at 22.26.20

(more…)

Reactions to #prwdebate

Screen Shot 2014-04-09 at 21.03.36

Well, seeing as you asked so nicely…

Here is a selection of audience and Twitter discussions before, during and after last week’s debate. There were hundreds of tweets which used the #prwdebate hashtag, so if you spot something we missed, let us know in the comments!

Click here for our Storify collection of tweets from the event!

(more…)

Panel Discussion: Is Peer Review Broken?

ROOM UPDATE: BG03 in the ground floor – enter at Northampton Sq

Tomorrow we are hosting a panel discussion here at City University in London around the question ‘Is Peer Review Broken?’. We want to discuss the IF and HOW but even more so ‘How can we fix it?’

If you are actively involved in peer review, wheather you are a PhD student, postdoc, principal investigator or a publisher or if you are a journalist who wants to join the debate – come along!

Our panelists will be:

  • Tom Reller | Vice President and Head of Global Relations at Elsevier
  • Richard Van Noorden | Senior Reporter at Nature
  • Tiago Villanueva | Editorial Registrar at the BMJ
  • Maria Kowalczuk | Deputy Biology Editor at BioMed Central
  • Peter Ayton | Associate Dean of Research at City University
  • Nikolaus Kriegeskorte | Principal Investigator at the MRC, Cognition and Brain Sciences Unit

Chair: Connie St Louis | Director, City University Science Journalism MA and award winning journalist

Join the event, discuss with the panel and help to find constructive new ways for scientific peer review! There will be a panel discussion for 90 minutes and wine and nibbles after the event.

Follow the discussion before and after the event @peerrevwatch under the #prwdebate

Attendance is free but please register here at Eventbrite!

Location: City University London | EC1V0HB United Kingdom

Room BG03 [enter by Northampton Square, lower level]

Live Blogging a Panel Discussion: In Science we trust. Do we?

On March 15th the University of Cambridge held a panel discussion as part of their science festival: In science we trust – Traditional publihsing, open access, post-publication review. Panelists were:

Peer review: an author’s perspective

To get an idea of what it is like going through the peer review process as a paper’s author, I spoke to physicist Joe Goodwin, who recently had his first paper reviewed before publication in Nature Communications.

Q: How long did the reviewing process take, from submission to a published paper?

A: My paper in Nature Communications was first submitted in May, and was published in October. Half of that delay was at our end, but Nature Communications publishes so many hundreds of papers per year that everything takes a while.

(more…)

Access to Research: What’s in it for the publishers? [INTERVIEW]

Screen Shot 2014-02-16 at 13.52.07

by @jack_millner

To get a better understanding of why a publisher like Nature that relies on subscriptions would get involved with a scheme that disseminates their content for free, I interviewed Jonathan Griffin, deputy CEO and head of business development for PLS (Publishers Licensing Society) and Jessica Rutt, Rights and Licensing Manager at Nature Publishing Group. (more…)