Academic peer review has historically been framed as a strict and rigid process by which manuscripts are submitted, vetted, and finally published. In reality, the practice of academic peer review is organic, and happens informally on a daily basis throughout the Astronomical (indeed, every) community. Through discussions over coffee or tea or malted beverages, presentations at journal clubs and our own readings, the merits of manuscripts are discussed, debated, questioned, and acclaimed with the insight of domain experts. Many of these points, were they to make their way back to the authors, referees, or editors in a timely manner, would yield marked improvements of clarity and completeness in the final published work. Unfortunately, the product of these many disparate conversations is almost never archived in any formal way, and is thus lost. It is this resource we wish to capture here at CPR : the collected peer review of our community.
Modern open access websites like arXiv.org allow for authors to submit their manuscripts for public dissemination before (or even instead of) acceptance and publication in Journals and Proceedings. While the merits of this practice may be debated (freedom from the tyranny of hostile referees and Journal page charges vs. unleashing unvetted science on the public), what is not debatable is that a significant number of authors choose to exercise this freedom, and it has become accepted practice in the field of Astronomy. This opens a window of opportunity for the community to impact the final published work via feedback and critique before the work is published (and therefore immutable). We envision this site as a first attempt to facilitate the interaction of the community as a whole with individual authors and referees. We hope that this site can help capture, in an open resource, the informal discussions held at institutions everywhere. Here the community can leave comments summarizing their own informal discussions on a paper; authors can subscribe to comments on their own papers, clarify points and respond to criticisms; and referees can refer to it as an additional resource when providing their review of a submission.
Blogs are a proven environment for the posting of information and the free trade of ideas on that information. They are also notorious for allowing spammers and flamers to disrupt the community. At CPR we’ve set a moderate bar of requiring signed posts, with the hope that we’ll be able to recognize friends and colleagues, and ignore the noise. The success of this idea is now on your shoulders. Take your laptop to journal club and astro coffee. Jot down comments, questions, and issues using the CPR blogs. This is all extremely important information. Let’s share it! We encourage you to refer this site to friends and colleagues, so that we can establish a thriving ecosystem, and your own posts do not disappear into the void. Finally, we encourage you to post to the main CPR site your ideas, comments, and suggestions on how to make this implementation more useful.
Great idea! Are you going to expand to other fields, like cond-mat? Can I help?
ReplyDeleteI just posted in my blog about the need to go beyond the standard model of publication... ;)
http://physicsnapkins.wordpress.com/2012/01/16/occupy_scientific_journals/
This should be easily do-able, since all articles from arXiv are available using the same API. We'll work on putting this together in the next week or so.
ReplyDeleteGreat! I'll spread the voice, it is very important to make this site (and others like it) work. BTW, how to start a new discussion thread, on a new paper?
ReplyDeleteHi Yvi - Thanks for your help! The idea is to use the Comments section for each paper, just like here, to capture the discussion thread. Its pretty simple at this stage; there are all sorts of ways one might consider improving this (e.g. references to sections of the paper, equations, or paragraphs of text) that would require tighter integration with the public document. Definitely out-of-scope for now, but on our minds in case this idea gains traction.
ReplyDeleteCheers!
Hi C.P.R,
ReplyDeletewe share the same motivation. I'm absolutely certain that the scientific community has a tremendous amount of collective wisdom about the papers that appear on a daily basis. It would be to the benefit of everybody involved if we could bundle this knowledge and feed it back to the readers and authors. Journal clubs are the canonical way of doing this, but they can only be as good as the most experienced people in the room. By sharing our knowledge worldwide, we can overcome this limitation.
I've create the website http://paperrater.org for this reason. It automatically reads all arXiv submissions and performs authors searches on ADS (which provides access to all published literature in astronomy and physics). Accessing additional databases is possible and will be implemented on demand. Anybody can sign up and rate or comment on any paper. Ratings are anonymous, comments can be. Comments allow LaTex and are subject to criticism by fellow users, too.
I wonder if we could collaborate somehow to give this idea a wider spread.
Hi,
ReplyDeleteIf you want to rescue old content from physcomments.org (from archive.org, most surely), you are welcome
Dear CPR team,
ReplyDeletethis is a great attempt!
Actually, during the past few months I had been discussing the merits of such a scheme with several colleagues, without being aware of CPR. Fortunately, one of my PhD students recently pointed me to your site.
Have you ever discussed with the arXiv people of incorporating it there? Or are you aware of any discussions about trying something similar on arXiv? I'm asking because the whole thing is obviously a highly nonlinear process. So if the visibility is larger (which it would be in arXiv by a great deal), then more comments roll in, which attract even more comments, and we have an exponential explosion of impact and usefulness... One of the good things about arXiv is that all the scientists already are registered there with their full names, which clearly also lowers the threshold of submitting (signed) comments.
Alternatively, do you know whether it would be technically possible to feed the comments from this blog directly through to arXiv and then automatically display them there (below the abstract, say; maybe in a part of the panel that can be hidden) ? That would then represent a very easy start and get rid of the burden of implementing it from scratch in the arXiv system. People could then click on some link if they want to add a comment or reply.
Do you think there's some way of implementing the arXiv
trackback feature (http://arxiv.org/help/trackback) on CPR?
I understand that it would probably a bad idea to link back
every article you post, since many will not have any comments,
and so the arXiv people might get a little upset, if you just
do this automatically.
Dreaming ahead, in the long run, this might even turn into the prevalent model of publishing (without the need for extra peer-reviewed journals). Comments would take the place of peer reviews. To this end, one could then possibly also implement the possibility of submitting anonymous comments, which would be sent out to other registered users for moderation (to prevent polemics and flame wars). And then there's the whole issue of also ranking the level of experience of commenters (like in physics.stackexchange.com).
Virtually all of the colleagues I've talked to agree that they would much prefer their paper being refereed and ranked by many of their knowledgeable peers from the community --- rather than two somewhat randomly picked referees (I know it's pretty hard for editors to pick good referees, there are so many topics; I am an editor myself). What makes matters worse currently is that the referee decision, with all its random factors, ultimately is a digital black/white decision that makes your paper get much attention (like in Nature or Physical Review Letters) or much less attention if it's rejected there.
Best regards,
Florian Marquardt
(Florian.Marquardt@physik.uni-erlangen.de)
Florian - Thank you for your support, and you make some very good points! We have indeed debated about how to get the ball rolling with CPR. We have frequently heard that "this is a good idea", but, it hasn't yet translated into usage. Some communication with arXiv is warranted in this regard, including use of their Trackback feature (which I had not noticed before). We also share your hope that this becomes a prevalent model of publishing; the sentiment is there and growing.
ReplyDeleteI am actually on the board of editors with the electronic-only New Journal of Physics (NJP). In principle, they might be open to introducing comments on papers. But understandably they also want to avoid pitfalls like flame wars etc. So you want to have named comments (real names). However, that creates a hurdle which is simply that everyone needs an account. And it would probably a real pain if you need to get new accounts for every journal. So it would be great to have a kind of integrated platform where people just use their arXiv accounts (e.g.) to post comments, and possibly where the comments that had been posted to the arXiv paper can then also be linked to the final paper version (like NJP).
ReplyDeleteBut the first thing would be to get arXiv involved. Do you have any contact there?
Florian - We have heard from Don Beyer at arXiv that they're not willing to back any particular implementation of this idea. So, it sounds like we're left to develop these tools on our own.
ReplyDelete