Peer Review in the Google Age Dominy

Screencast/Podcast

Peer Review in the Google Age Dominy

Peggy Dominy:	Hi I'm Peggy Dominy, and we are going to be going through my PowerPoint that Jay and I kind of collaborated on. So, forgive me if I am a little anal here, I have to go back to the beginnings. Where did peer review begin? Some believe that it goes back as far as the seventeenth century and was known as the inquisition of the Holy Roman and Catholic Church. Scholars' works were examined for hints of heresy and we see the first victim of peer review as Galileo has to recant what he wrote. But in modern times what is it used in? Well, peer review it is used in a variety of places. We've been talking more or less about the publication process, but it is also used in the awarding of funding research and in patents and in standards. Although all of these use slightly different practices, ultimately it's colleagues evaluating each other. Maybe this is old hat for some of us, but what is the process of peer review? A paper is submitted, the editor will select one or two or three scholars from a pool of volunteers, and I want to emphasize that these are volunteers, to read and evaluate the paper. Typically it's a double blind process, the author doesn't know who the reviewers are, the reviewer doesn't know who the authors are. Although, in reality, if you have a very small universe of scientists you recognize the writing of people so you tend to know your reviewers are, you tend to know who the authors are anyway. But, supposedly it's a double blind process, that way only the merits of the paper are evaluated. Supposedly, within a reasonable amount of time the reviewers respond, it's up to the author to reply or comply with the reviewers comments. This can go back and forth adding months to a publishing process, and this is what we historically know. In the modern age, with electronic sending of text back and forth this process has certainly sped up, but the imposition on the overburdened scientist has not changed. Why do peer review? Well, it is a filter. More papers are submitted than could be printed. And I put that in quotes now. You use it to eliminate bad science, you use it to eliminate pseudo science or harmful science. It's also an aura of quality, only the best gets in. It's also a collegial stamp of approval, your colleagues have said "This is good, publish it.". Also, within a particular discipline, it's your obligation to the principles of your discipline, how you keep the high levels of integrity of your discipline. So, what's the problem? This is a very busy slide, I won't go through all of it. Essentially there are three problems to peer review. Not all papers get peer reviewed, even if they are in a journal that normally does peer review. Sometimes it's a good thing like Watson and Crick's paper on DNA structure, the editor realized right away this does not need peer review and put it right in. But then you also have a situation of Allan Sokal's paper did not get peer review it and it actually turned out to be a hoax, so not every paper gets peer reviewed. Some papers were published, passed peer review and turned out to be fraudulent. We have some examples there. Then we have famous papers that actually got rejected and then turned out to the seminal works. We have the famous example of Krebs and Johnson's paper on the role of citric acid on metabolism which turned out to be how cells convert food into energy, it's now known as the Krebs cycle if he got a Nobel prize for it. So, we do have these examples where peer review breaks down. So who's worried about peer review now? A simple search on Google, and I know this is not scientific, I went into Google Scholar, I put in peer review as a phrase. You can limit by different disciplines, naturally these are very broad disciplines, but you can get an idea about who's writing about peer review in these various broad disciplines. Again, I don't claim that this is scientific, but I think the numbers of pretty impressive. Two recent articles, why we are even here today, "Is Peer Review Broken" by Alison McCook from the Scientist and new journals are starting to take away the blind process of the reviewers, making them public. Is this a good thing or bad thing? Papers are published on the web without the constraint of peer review, you can put anything you want, it's out there. Also, papers can be published on the web with the constraints of peer review. So, what's the difference? Without constraints, it's a wild and open area, there's no gate keepers, no censors. So, if you have this wide open area and there is good stuff amongst all the bad stuff, will the good stuff percolate out? Who's got the discernment, where is the discernment, how do you know if it's good and where is the authority? These are the questions. With the constraints, the good stuff is vetted, scholarship is monitored and maintained, exposure beyond that the internet. I mean this because if you are in a peer reviewed established journal your stuff gets indexed and extracted into a lot of different databases, so when people are searching these databases your stuff is exposed, which might not happen if it's just out there on the internet. There are lots of publishers out there sharing their content with the internet, in other words, these publishers are allowing you to put up a preprint or a PDF form of your paper on your own website as long as it's not the "final version" but it will be allowed to put up as open access and accessible and this is just a short list of publishers that are allowing this. It's not perfect, there's grist for lots of mills, the web has made it less of an obstacle to access. When you are published in a peer review journal you had to be able to subscribe to it, that meant money. A lot of people weren't able to subscribe to these expensive journals that were highly peer reviewed, so there was always an obstacle to your information if it was in a peer review journal. The web has made that less of an obstacle. Defend disciplines have different perspectives, they have different issues. In some disciplines they are more concerned with funding, is peer review fair in funding. Is there sexism or prejudices that come out in peer review? Other people not getting fair treatment? But also I think at least from Jay's and my own point of view as librarians there is also a pedagogical reason for peer review. It's a yardstick for students. There's got to be something that tells a student that this is good stuff. They have not develop their skills or discerning yet to determine whether something is good or not. So we see it as a pedagogical yardstick. There is all sorts of collegial peer review on the internet and we just go through a few these, there's email, wikis, blogs that's essentially peer review right there, that's opened up. Yes Jay?
Jay Bhatt:	Can you just click on the link "Blogging as Tool"? This is an example of a post publication comment, this is a paper that I got published in the Library HiTech News and some of my colleagues were looking at this paper. When you scroll down you will see the comments posted about the paper and you can see at the top one of the comments was posted by my colleague, a library information professional, which we would value their comments very much. Not only has that person critiqued my paper, but also provided some valuable positions. This kind of dialogue is only possible because of the internet and that really enhances the quality of the paper.
Peggy Dominy:	So, the internet has changed peer review in that it's more common, it's more informal peer review then what we've always had before which was more formal peer review.
Man:	This post publication blogging is a no brainer.
Man 2:	Absolutely.
Peggy Dominy:	I mentioned this before, also with authors putting their papers on the web, even though the papers themselves have gone through the peer review process it's making their papers more accessible, so you are removing a lot of the barrier of a subscription database or a subscription journal. We're seeing more access to scientific literature as we see more and more institutions preserving their own publications. Again, we're talking about the global benefit here as scientists from less developed countries are able to put their own papers up on the web. You are seeing material and resources that you might not have otherwise had access to. Journals that come from other countries, that are in other languages in some cases, would not necessarily find their way into your library. Finally, I won't go any further here but this is just a brief bibliography. When I started looking at peer review I was overwhelmed by the amount of literature on the topic. People have been worried about this for a long time. Again, it's quite interesting to see from the different perspectives of different disciplines what about peer review they're worried. It's fascinating. This is a very short list, you can look at your leisure. Are there any questions, comments?
Man:	I have a question. You were saying that normally the peer review is double blind. I know in chemistry we have never experienced that. The reviewers always know who the paper is from. Are there some fields where that is not true?
Peggy Dominy:	Yes. In other fields that is -.
Man:	Like what?
Peggy Dominy:	Physics, anthropology I know for sure. My son, who not only is an author, but is also a reviewer, he often doesn't know who the author is. He doesn't know who the reviewers are, but by their comments he is able to...