Peer Review in the Google Age Dominy

Screencast/Podcast

Peer Review in the Google Age Dominy

Peggy Dominy:	Hi I'm Peggy Dominy, and we are going to be going through my PowerPoint that Jay and I kind of collaborated on. So, forgive me if I am a little anal here, I have to go back to the beginnings. Where did peer review begin? Some believe that it goes back as far as the seventeenth century and was known as the inquisition of the Holy Roman and Catholic Church. Scholars' works were examined for hints of heresy and we see the first victim of peer review as Galileo has to recant what he wrote. But in modern times what is it used in? Well, peer review it is used in a variety of places. We've been talking more or less about the publication process, but it is also used in the awarding of funding research and in patents and in standards. Although all of these use slightly different practices, ultimately it's colleagues evaluating each other. Maybe this is old hat for some of us, but what is the process of peer review? A paper is submitted, the editor will select one or two or three scholars from a pool of volunteers, and I want to emphasize that these are volunteers, to read and evaluate the paper. Typically it's a double blind process, the author doesn't know who the reviewers are, the reviewer doesn't know who the authors are. Although, in reality, if you have a very small universe of scientists you recognize the writing of people so you tend to know your reviewers are, you tend to know who the authors are anyway. But, supposedly it's a double blind process, that way only the merits of the paper are evaluated. Supposedly, within a reasonable amount of time the reviewers respond, it's up to the author to reply or comply with the reviewers comments. This can go back and forth adding months to a publishing process, and this is what we historically know. In the modern age, with electronic sending of text back and forth this process has certainly sped up, but the imposition on the overburdened scientist has not changed. Why do peer review? Well, it is a filter. More papers are submitted than could be printed. And I put that in quotes now. You use it to eliminate bad science, you use it to eliminate pseudo science or harmful science. It's also an aura of quality, only the best gets in. It's also a collegial stamp of approval, your colleagues have said "This is good, publish it.". Also, within a particular discipline, it's your obligation to the principles of your discipline, how you keep the high levels of integrity of your discipline. So, what's the problem? This is a very busy slide, I won't go through all of it. Essentially there are three problems to peer review. Not all papers get peer reviewed, even if they are in a journal that normally does peer review. Sometimes it's a good thing like Watson and Crick's paper on DNA structure, the editor realized right away this does not need peer review and put it right in. But then you also have a situation of Allan Sokal's paper did not get peer review it and it actually turned out to be a hoax, so not every paper gets peer reviewed. Some papers were published, passed peer review and turned out to be fraudulent. We have some examples there. Then we have famous papers that actually got rejected and then turned out to the seminal works. We have the famous example of Krebs and Johnson's paper on the role of citric acid on metabolism which turned out to be how cells convert food into energy, it's now known as the Krebs cycle if he got a Nobel prize for it. So, we do have these examples where peer review breaks down. So who's worried about peer review now? A simple search on Google, and I know this is not scientific, I went into Google Scholar, I put in peer review as a phrase. You can limit by different disciplines, naturally these are very broad disciplines, but you can get an idea about who's writing about peer review in these various broad disciplines. Again, I don't claim that this is scientific, but I think the numbers of pretty impressive. Two recent articles, why we are even here today, "Is Peer Review Broken" by Alison McCook from the Scientist and new journals are starting to take away the blind process of the reviewers, making them public. Is this a good thing or bad thing? Papers are published on the web without the constraint of peer review, you can put anything you want, it's out there. Also, papers can be published on the web with the constraints of peer review. So, what's the difference? Without constraints, it's a wild and open area, there's no gate keepers, no censors. So, if you have this wide open area and there is good stuff amongst all the bad stuff, will the good stuff percolate out? Who's got the discernment, where is the discernment, how do you know if it's good and where is the authority? These are the questions. With the constraints, the good stuff is vetted, scholarship is monitored and maintained, exposure beyond that the internet. I mean this because if you are in a peer reviewed established journal your stuff gets indexed and extracted into a lot of different databases, so when people are searching these databases your stuff is exposed, which might not happen if it's just out there on the internet. There are lots of publishers out there sharing their content with the internet, in other words, these publishers are allowing you to put up a preprint or a PDF form of your paper on your own website as long as it's not the "final version" but it will be allowed to put up as open access and accessible and this is just a short list of publishers that are allowing this. It's not perfect, there's grist for lots of mills, the web has made it less of an obstacle to access. When you are published in a peer review journal you had to be able to subscribe to it, that meant money. A lot of people weren't able to subscribe to these expensive journals that were highly peer reviewed, so there was always an obstacle to your information if it was in a peer review journal. The web has made that less of an obstacle. Defend disciplines have different perspectives, they have different issues. In some disciplines they are more concerned with funding, is peer review fair in funding. Is there sexism or prejudices that come out in peer review? Other people not getting fair treatment? But also I think at least from Jay's and my own point of view as librarians there is also a pedagogical reason for peer review. It's a yardstick for students. There's got to be something that tells a student that this is good stuff. They have not develop their skills or discerning yet to determine whether something is good or not. So we see it as a pedagogical yardstick. There is all sorts of collegial peer review on the internet and we just go through a few these, there's email, wikis, blogs that's essentially peer review right there, that's opened up. Yes Jay?
Jay Bhatt:	Can you just click on the link "Blogging as Tool"? This is an example of a post publication comment, this is a paper that I got published in the Library HiTech News and some of my colleagues were looking at this paper. When you scroll down you will see the comments posted about the paper and you can see at the top one of the comments was posted by my colleague, a library information professional, which we would value their comments very much. Not only has that person critiqued my paper, but also provided some valuable positions. This kind of dialogue is only possible because of the internet and that really enhances the quality of the paper.
Peggy Dominy:	So, the internet has changed peer review in that it's more common, it's more informal peer review then what we've always had before which was more formal peer review.
Man:	This post publication blogging is a no brainer.
Man 2:	Absolutely.
Peggy Dominy:	I mentioned this before, also with authors putting their papers on the web, even though the papers themselves have gone through the peer review process it's making their papers more accessible, so you are removing a lot of the barrier of a subscription database or a subscription journal. We're seeing more access to scientific literature as we see more and more institutions preserving their own publications. Again, we're talking about the global benefit here as scientists from less developed countries are able to put their own papers up on the web. You are seeing material and resources that you might not have otherwise had access to. Journals that come from other countries, that are in other languages in some cases, would not necessarily find their way into your library. Finally, I won't go any further here but this is just a brief bibliography. When I started looking at peer review I was overwhelmed by the amount of literature on the topic. People have been worried about this for a long time. Again, it's quite interesting to see from the different perspectives of different disciplines what about peer review they're worried. It's fascinating. This is a very short list, you can look at your leisure. Are there any questions, comments?
Man:	I have a question. You were saying that normally the peer review is double blind. I know in chemistry we have never experienced that. The reviewers always know who the paper is from. Are there some fields where that is not true?
Peggy Dominy:	Yes. In other fields that is -.
Man:	Like what?
Peggy Dominy:	Physics, anthropology I know for sure. My son, who not only is an author, but is also a reviewer, he often doesn't know who the author is. He doesn't know who the reviewers are, but by their comments he is able to...

Peer Review in the Google Age Bradley

Screencast/Podcast

Peer Review in the Google Age - JC Bradley

JC Bradley:	Ok, so this is our 22nd RSS club meeting, it's a special meeting that's going to be on peer review in the Google age. This follows up on a blog post that I put in, having to do with how being able to search on Google and these kinds of search engines actually changes or might change in the future the way that scientists use peer review. I've invited a couple of librarians, we've got Jay Bhatt and Peggy Dominy from our Drexel Haggerty Library. I have Heather Morrison from Simon Fraser University that is connected to us via NetMeeting. Can you still hear us Heather?
Heather:	Yes I can.
JC Bradley:	Ok, so she'll be giving a talk after we do here, and hopefully all the technology will work and this will be recorded and everybody who was not able to attend will be able to benefit from it. So I want to keep this fairly short. I'm basically going to give you the perspective of the scientists, because I think the librarians will give you their viewpoint of what's happening here, but for me the important thing with peer review is, before the Internet and after the Internet, how that's actually changed it. I remember as a Grad student, using the books, there was no way to electronically search, or if there were we couldn't afford it. You know, to search if other people had been working on your area, and it was a very, very time consuming process, because you'd have to go into these big books, and once you found a reference you had to look in another book, and then you had to find out if the library actually carried that, and if it didn't, what are you going to do? So, peer review, actually, the way I used it as a scientist was to make a decision as to how much time I should spend trying to hunt down a particular reference. So if I was looking for a synthetic strategy, and I found a Journal of Organic Chemistry article, I knew that would be worth getting. Well, first of all, I knew the library had it, and secondly I knew that you could follow their experimental section pretty well, and same thing with the Journal of the American Chemical Society and all of these things. If there was something you didn't recognize, now unless I was really desperate, I would just skip it, because it was so time consuming. But what's happened now is we're no longer in that situation, at least we shouldn't be, and there's no reason to be anymore. Now search is actually extremely fast and extremely easy and there are open tools that Jay's actually talked about, Google Scholar and these different free services, there's also SciFinder that is a pay service and will get you, usually, better information. But there's actually quite a bit on the Google Scholar, on the research we've been doing lately. What's interesting here is I'm not really that interested in whether the article is peer reviewed or not. I'm more interested in, do they have the experimental conditions for the compound that I'm trying to make, and I can judge whether their description of the experimental is actually valid, or how likely is it to be good, just based on the way in which they describe it. So it's kind of an interesting situation, because, I'm not using peer review in the way that I used to use it, to protect my time. So now it's something completely different that determines the value of an article. It's whether or not I can get it online immediately, and if I can't I usually don't bother. Again, unless I'm very desperate, and then I'll try to hunt it down. But honestly there's so much repetition now in the scientific literature that you can usually find what you're looking for online directly, or at least know that it hasn't been done, that's the other way to look at it. So, here peer review is actually changing, for one more reason, because in the past there were far fewer publications and those publications had a pretty long life before you got to them. So you could really understand and get to know these various publications. Today, really anyone can set up a journal online and can call it peer reviewed and you don't even who the peers are, because almost by definition they're anonymous. So, I could decide that you guys are all peers in something, and you would validate or not, and so I think that simply stamping something being peer reviewed or not, today, doesn't make much sense. I think if we say that something is in the Journal of Organic Chemistry and it was peer reviewed, that still has meaning, because the editors are going to use the same set of reviewers, and they have a whole system in place to maintain their quality and their trend, or what they're trying to focus on. So I think this has definitely changed. Again, this whole idea of what does peer review mean today, from the scientist's perspective, it has to do with when you come up for tenure or promotion, justifying the value of your scholarship. When you report the papers that you have, you have the peer reviewed and the non-peer reviewed. Then again, if you have a paper that you don't even know who the reviewers are, it's a new journal that just came up, what does it really mean? Or if you do a conference proceedings, that's supposedly peer reviewed, but we all know how closely the reviewers usually look at that. I think it's a term that we might have to refine a little bit more, just exactly what we mean. I think that, in terms of peer review we have to remember there are two separate problems here. One of them is communicating science; how do I communicate my science to another scientist, which is completely different from justifying myself once a year, as to what I've done. I'll show you some alternate ways of communicating science that don't include peer review and that might actually be useful. In the journal article, The Best Way of Communicating Science, here's an example of where we've recently been using a public blog, a simple Blogger account, and just using it as a research notebook. It's been really interesting to see how people around the world have been responding to that. Here's an example of an experiment, which is a fairly complex experiment. It's called a UG reaction, and it involves mixing four components together, and making this compound. As we report this, this experiment is not actually completed yet. It's still going on and we're still waiting to collect the data. But what's interesting is that people have been coming in and putting comments. So here we've got, Matt Todd, who's actually an Australian chemist, and he made a comment that, "You know, a quick comment on this reaction. You're trying to do a condensation, so a high concentration of reagents would be favorable. I think your current level of milligram per milliliter is quite dilute, try it again in five mils of methanol". So this actually has no analogue in the normal print. This is an experiment that not only hasn't been published formally, but hasn't even been finished. We're in the process of doing it, and already people are responding to us. That's why I see the power of these new technologies in doing science. It's that you can actually do something at a different level of immediacy. So there's a lot of these examples, I just wanted to highlight a couple of things. The other way in which we can have connections is through looking at how people are finding your blog. There's a little tool called Sitemeter, that is free, that you can put at the bottom of your blog and find out how people are finding it. Here's an interesting thing, when I looked at, recently, the Sitemeter listings, one of them was for adrenaline decomposed, and here is the actual entry inside of our lab notebook, where we actually used adrenaline, and the contents of the reaction lies decomposed, and we aborted this experiment. So this is an experiment that would never make it in a normal publication, because it was aborted, something went wrong. But the person who was looking for adrenaline decomposed, which is interesting because they're searching for English and Japanese and they're actually in Jersey, so who knows who these people are that are looking for this. But the point is, if they were looking for a way that adrenaline can decompose, they found it, and this would never make it a normal journal. So those are some of the elements that are coming together. I don't want to talk long here, because a lot of this I've actually written in the blog and I can give you links to go to that. But just to put it out on the table, there are lots of these little information aggregators, or people trying to do this kind of thing that are kicking around, and it's a completely decentralized operation. You know, I started this, using Blogger to report our lab results. There are other people, the Synaptic Leap, these are people who do Open Source biomedical research. They, so far, have not actually put experimental results but they've actually been talking about, you know, "What do we need to do to solve things like Malaria and other tropical diseases?" So these are connected, all these arrows show that there's some kind of link, some kind of Internet link between each of these. The one that we were looking at is the Useful Chem Experiments blog, where that's actually the hardcore experimental details, every last detail of how to do stuff in the lab. This is connected to the Useful Chem Wiki that has a high level of organization, so if you wanted to see the big picture, you go to the Wiki. The Useful Chem blog is where we discuss synthesis, synthetic strategies, what's important, what's not important. The Useful Chem Molecules actually is a blog that has each of the molecules we're working with in a separate post. So all of these are completely interconnected and you can follow from one to the other. The point I wanted to make on this slide, is we're doing all this, and we're doing this in the open, and then the Synaptic Leap basically said that they were looking for an alternate synthesis for a drug that's currently effective in one of the tropical diseases. And at the same time, well, very shortly after, that was put in the Useful Chem blog, this company, Chemrefer, these are people that search the literature for free for you, and answer chemical questions, and their plan is to make money by advertising. But they have an RSS feed to our Useful Chem blog, so when they saw that people were looking for that, they came in and found a thesis that was not already in the database that everybody was looking at. So I thought that was kind of cool, because that's actually a commercial sector that is connecting, doing Open Source science for free, and trying to find a way to make money at it. This is a very, very small snapshot of everything that's happening in Open Source right now, but I think that it gives you a good perspective on how things can go very quickly in the future. So that's pretty much all I wanted to do on my side. On the last slide here, there is a link to the Useful Chem Wiki, Usefulchem.wikispaces.com, and we've got a summary of everything that I just told you here, and I also have some links to some other people who are doing Open Source science. So Open Wetware, Bioroot archive, NABOJ, Physics Comment, all of these are sites that actually have experimental details, and then you have Open Source coordinating sites, Synaptic Leap we were just looking at, Chemists without Borders, World Community Grid, Bioforge, CODATA, and the ACS actually just started a chemical biology Wiki, which I'm not sure if you're aware of that, but it's kind of interesting. So these are actually, there's no experimental details here yet, and there may not be, but that's fine, because these people are coordinating one group with another and that's just as useful as putting primary data. At the very bottom here I gave you some links to some general information on Open Source science. So there's a presentation there and a couple of things. So that's pretty much all I wanted to talk about. We've got a couple of minutes, do you have any comments?
Woman1:	You mentioned the chemist from Australia suggested a higher concentration for your experiment. So you're going to get comments and statements from all over the place, when you put your experiment out there for the world to see. In the traditional sense of, you've got the interaction with the colleague next door, in the office down the hallway or something like that, and this has opened up your connections, but then how do you in your paper, how do you acknowledge contribution like that?
JC Bradley:	The contributions, you mean in the final paper?
Woman1:	Yes.
JC Bradley:	Well, most of this stuff will never be published in a formal paper, because I can't publish a failed experiment. If you try, you'll see it's very, very difficult. A paper has a small fraction of all the experiments that you ever did, and it has to tell a story. It can't just be random experiments put together. Of course, if I do end up using them, I'll put them in the acknowledgments, but it's pretty unlikely that I would use those experiments ultimately.
Woman2:	How do you work that in to the typical paper?
JC Bradley:	Well, you know it's also possible that someone would really get involved, and we'd have a true collaboration. That could happen also. I hope that happens at some point. Heather Morrison: This is Heather, I'm wondering if one possibility there, to have that acknowledgment, rather than it be in the formal paper, would be to have the paper have a link to the blog with the comment on it.
JC Bradley:	Yes, Heather, absolutely, when we publish this stuff, there would definitely be a link as a reference, because this is the first official publication. Much in the same way that if you go to a conference and you give a talk and you disclose things, that's a publication, it's a disclosure in terms of patent rights and all that stuff. So it's nothing new in that sense, it's just a different modality of doing it.
Jay Bhatt:	What I would like to add is, these new means of communication like Wiki and blogs have resulted in an enormous amount of opportunities in terms of communicating with other fields which I believe that these other peers are going to give you feedback on your own research ideas and topics, and that will really help to go forth the research, and I think the new Internet has created this enormous potential, and people and scientists all over the world have started to take advantage of these opportunities. [xx] [xx] [xx] [xx] Bloglines, if people subscribe to these, they can get RSS feeds, can monitor thousands of feeds in a very short amount of time, and also be able to give positive feedback in the process.
JC Bradley:	Yeah, Jay makes a good point. One of the things that maybe I didn't make very explicit, is the reason that I'm using Blogger or one big reason, is that I can easily put RSS feeds. So this Praziquantel, that was the drug that I was talking about earlier, and you see here that I have the standard RSS feed, so Chemrefer was probably subscribed through RSS. The idea is that if you have a bunch of scientists working together, nothing is faster than RSS. As soon as it's put in the blog, everybody in the group knows that we've done another experiment.