Peer Review in the Google Age Bradley

Screencast/Podcast

Peer Review in the Google Age - JC Bradley

JC Bradley:	Ok, so this is our 22nd RSS club meeting, it's a special meeting that's going to be on peer review in the Google age. This follows up on a blog post that I put in, having to do with how being able to search on Google and these kinds of search engines actually changes or might change in the future the way that scientists use peer review. I've invited a couple of librarians, we've got Jay Bhatt and Peggy Dominy from our Drexel Haggerty Library. I have Heather Morrison from Simon Fraser University that is connected to us via NetMeeting. Can you still hear us Heather?
Heather:	Yes I can.
JC Bradley:	Ok, so she'll be giving a talk after we do here, and hopefully all the technology will work and this will be recorded and everybody who was not able to attend will be able to benefit from it. So I want to keep this fairly short. I'm basically going to give you the perspective of the scientists, because I think the librarians will give you their viewpoint of what's happening here, but for me the important thing with peer review is, before the Internet and after the Internet, how that's actually changed it. I remember as a Grad student, using the books, there was no way to electronically search, or if there were we couldn't afford it. You know, to search if other people had been working on your area, and it was a very, very time consuming process, because you'd have to go into these big books, and once you found a reference you had to look in another book, and then you had to find out if the library actually carried that, and if it didn't, what are you going to do? So, peer review, actually, the way I used it as a scientist was to make a decision as to how much time I should spend trying to hunt down a particular reference. So if I was looking for a synthetic strategy, and I found a Journal of Organic Chemistry article, I knew that would be worth getting. Well, first of all, I knew the library had it, and secondly I knew that you could follow their experimental section pretty well, and same thing with the Journal of the American Chemical Society and all of these things. If there was something you didn't recognize, now unless I was really desperate, I would just skip it, because it was so time consuming. But what's happened now is we're no longer in that situation, at least we shouldn't be, and there's no reason to be anymore. Now search is actually extremely fast and extremely easy and there are open tools that Jay's actually talked about, Google Scholar and these different free services, there's also SciFinder that is a pay service and will get you, usually, better information. But there's actually quite a bit on the Google Scholar, on the research we've been doing lately. What's interesting here is I'm not really that interested in whether the article is peer reviewed or not. I'm more interested in, do they have the experimental conditions for the compound that I'm trying to make, and I can judge whether their description of the experimental is actually valid, or how likely is it to be good, just based on the way in which they describe it. So it's kind of an interesting situation, because, I'm not using peer review in the way that I used to use it, to protect my time. So now it's something completely different that determines the value of an article. It's whether or not I can get it online immediately, and if I can't I usually don't bother. Again, unless I'm very desperate, and then I'll try to hunt it down. But honestly there's so much repetition now in the scientific literature that you can usually find what you're looking for online directly, or at least know that it hasn't been done, that's the other way to look at it. So, here peer review is actually changing, for one more reason, because in the past there were far fewer publications and those publications had a pretty long life before you got to them. So you could really understand and get to know these various publications. Today, really anyone can set up a journal online and can call it peer reviewed and you don't even who the peers are, because almost by definition they're anonymous. So, I could decide that you guys are all peers in something, and you would validate or not, and so I think that simply stamping something being peer reviewed or not, today, doesn't make much sense. I think if we say that something is in the Journal of Organic Chemistry and it was peer reviewed, that still has meaning, because the editors are going to use the same set of reviewers, and they have a whole system in place to maintain their quality and their trend, or what they're trying to focus on. So I think this has definitely changed. Again, this whole idea of what does peer review mean today, from the scientist's perspective, it has to do with when you come up for tenure or promotion, justifying the value of your scholarship. When you report the papers that you have, you have the peer reviewed and the non-peer reviewed. Then again, if you have a paper that you don't even know who the reviewers are, it's a new journal that just came up, what does it really mean? Or if you do a conference proceedings, that's supposedly peer reviewed, but we all know how closely the reviewers usually look at that. I think it's a term that we might have to refine a little bit more, just exactly what we mean. I think that, in terms of peer review we have to remember there are two separate problems here. One of them is communicating science; how do I communicate my science to another scientist, which is completely different from justifying myself once a year, as to what I've done. I'll show you some alternate ways of communicating science that don't include peer review and that might actually be useful. In the journal article, The Best Way of Communicating Science, here's an example of where we've recently been using a public blog, a simple Blogger account, and just using it as a research notebook. It's been really interesting to see how people around the world have been responding to that. Here's an example of an experiment, which is a fairly complex experiment. It's called a UG reaction, and it involves mixing four components together, and making this compound. As we report this, this experiment is not actually completed yet. It's still going on and we're still waiting to collect the data. But what's interesting is that people have been coming in and putting comments. So here we've got, Matt Todd, who's actually an Australian chemist, and he made a comment that, "You know, a quick comment on this reaction. You're trying to do a condensation, so a high concentration of reagents would be favorable. I think your current level of milligram per milliliter is quite dilute, try it again in five mils of methanol". So this actually has no analogue in the normal print. This is an experiment that not only hasn't been published formally, but hasn't even been finished. We're in the process of doing it, and already people are responding to us. That's why I see the power of these new technologies in doing science. It's that you can actually do something at a different level of immediacy. So there's a lot of these examples, I just wanted to highlight a couple of things. The other way in which we can have connections is through looking at how people are finding your blog. There's a little tool called Sitemeter, that is free, that you can put at the bottom of your blog and find out how people are finding it. Here's an interesting thing, when I looked at, recently, the Sitemeter listings, one of them was for adrenaline decomposed, and here is the actual entry inside of our lab notebook, where we actually used adrenaline, and the contents of the reaction lies decomposed, and we aborted this experiment. So this is an experiment that would never make it in a normal publication, because it was aborted, something went wrong. But the person who was looking for adrenaline decomposed, which is interesting because they're searching for English and Japanese and they're actually in Jersey, so who knows who these people are that are looking for this. But the point is, if they were looking for a way that adrenaline can decompose, they found it, and this would never make it a normal journal. So those are some of the elements that are coming together. I don't want to talk long here, because a lot of this I've actually written in the blog and I can give you links to go to that. But just to put it out on the table, there are lots of these little information aggregators, or people trying to do this kind of thing that are kicking around, and it's a completely decentralized operation. You know, I started this, using Blogger to report our lab results. There are other people, the Synaptic Leap, these are people who do Open Source biomedical research. They, so far, have not actually put experimental results but they've actually been talking about, you know, "What do we need to do to solve things like Malaria and other tropical diseases?" So these are connected, all these arrows show that there's some kind of link, some kind of Internet link between each of these. The one that we were looking at is the Useful Chem Experiments blog, where that's actually the hardcore experimental details, every last detail of how to do stuff in the lab. This is connected to the Useful Chem Wiki that has a high level of organization, so if you wanted to see the big picture, you go to the Wiki. The Useful Chem blog is where we discuss synthesis, synthetic strategies, what's important, what's not important. The Useful Chem Molecules actually is a blog that has each of the molecules we're working with in a separate post. So all of these are completely interconnected and you can follow from one to the other. The point I wanted to make on this slide, is we're doing all this, and we're doing this in the open, and then the Synaptic Leap basically said that they were looking for an alternate synthesis for a drug that's currently effective in one of the tropical diseases. And at the same time, well, very shortly after, that was put in the Useful Chem blog, this company, Chemrefer, these are people that search the literature for free for you, and answer chemical questions, and their plan is to make money by advertising. But they have an RSS feed to our Useful Chem blog, so when they saw that people were looking for that, they came in and found a thesis that was not already in the database that everybody was looking at. So I thought that was kind of cool, because that's actually a commercial sector that is connecting, doing Open Source science for free, and trying to find a way to make money at it. This is a very, very small snapshot of everything that's happening in Open Source right now, but I think that it gives you a good perspective on how things can go very quickly in the future. So that's pretty much all I wanted to do on my side. On the last slide here, there is a link to the Useful Chem Wiki, Usefulchem.wikispaces.com, and we've got a summary of everything that I just told you here, and I also have some links to some other people who are doing Open Source science. So Open Wetware, Bioroot archive, NABOJ, Physics Comment, all of these are sites that actually have experimental details, and then you have Open Source coordinating sites, Synaptic Leap we were just looking at, Chemists without Borders, World Community Grid, Bioforge, CODATA, and the ACS actually just started a chemical biology Wiki, which I'm not sure if you're aware of that, but it's kind of interesting. So these are actually, there's no experimental details here yet, and there may not be, but that's fine, because these people are coordinating one group with another and that's just as useful as putting primary data. At the very bottom here I gave you some links to some general information on Open Source science. So there's a presentation there and a couple of things. So that's pretty much all I wanted to talk about. We've got a couple of minutes, do you have any comments?
Woman1:	You mentioned the chemist from Australia suggested a higher concentration for your experiment. So you're going to get comments and statements from all over the place, when you put your experiment out there for the world to see. In the traditional sense of, you've got the interaction with the colleague next door, in the office down the hallway or something like that, and this has opened up your connections, but then how do you in your paper, how do you acknowledge contribution like that?
JC Bradley:	The contributions, you mean in the final paper?
Woman1:	Yes.
JC Bradley:	Well, most of this stuff will never be published in a formal paper, because I can't publish a failed experiment. If you try, you'll see it's very, very difficult. A paper has a small fraction of all the experiments that you ever did, and it has to tell a story. It can't just be random experiments put together. Of course, if I do end up using them, I'll put them in the acknowledgments, but it's pretty unlikely that I would use those experiments ultimately.
Woman2:	How do you work that in to the typical paper?
JC Bradley:	Well, you know it's also possible that someone would really get involved, and we'd have a true collaboration. That could happen also. I hope that happens at some point. Heather Morrison: This is Heather, I'm wondering if one possibility there, to have that acknowledgment, rather than it be in the formal paper, would be to have the paper have a link to the blog with the comment on it.
JC Bradley:	Yes, Heather, absolutely, when we publish this stuff, there would definitely be a link as a reference, because this is the first official publication. Much in the same way that if you go to a conference and you give a talk and you disclose things, that's a publication, it's a disclosure in terms of patent rights and all that stuff. So it's nothing new in that sense, it's just a different modality of doing it.
Jay Bhatt:	What I would like to add is, these new means of communication like Wiki and blogs have resulted in an enormous amount of opportunities in terms of communicating with other fields which I believe that these other peers are going to give you feedback on your own research ideas and topics, and that will really help to go forth the research, and I think the new Internet has created this enormous potential, and people and scientists all over the world have started to take advantage of these opportunities. [xx] [xx] [xx] [xx] Bloglines, if people subscribe to these, they can get RSS feeds, can monitor thousands of feeds in a very short amount of time, and also be able to give positive feedback in the process.
JC Bradley:	Yeah, Jay makes a good point. One of the things that maybe I didn't make very explicit, is the reason that I'm using Blogger or one big reason, is that I can easily put RSS feeds. So this Praziquantel, that was the drug that I was talking about earlier, and you see here that I have the standard RSS feed, so Chemrefer was probably subscribed through RSS. The idea is that if you have a bunch of scientists working together, nothing is faster than RSS. As soon as it's put in the blog, everybody in the group knows that we've done another experiment.