Reviewing Process

(Related: Open Access.)

Consider a continuous review process by a broad community of experts, where PC meeting has much more material to work with and is mostly driven by the analysis of existing reviews (of course, PC members would also participate in writing those reviews). Moreover, the reviews remain visible as part of the ongoing community discussion process that can continue beyond paper publication.

In 2007, William Thies and Viktor Kuncak explored some of these topics and launched a web site called “PaperCritic” with the description below (slightly edited). We were of course not the first ones to consider this (see the related links below). It would be nice to revive this process.

PaperCritic

:PaperCritic is a web site designed to promote the sharing of research paper reviews. The goal of the site is to promote rapid evaluation of new research results and thereby increase the pace of research.  The ideal would be to become a centralized repository for evaluation of time-stamped papers, including papers published in conferences and journals, as well as technical reports and http://arxiv.org submissions. We address some anticipated questions about :PaperCritic below. Please feel to contact us using the menu on the left if you have any further questions.

Content: Where Reviews Come From

The primary content of :PaperCritic are reviews, associated with bibliographical references for the reviewed papers and links to any publically available online versions. The reviews may contain numerical scores and author confidence. We plan to add more functionality that aggregates and filters these values.

The reviews can be posted by the users of the system. There are two basic forms of these reviews:

A user may post the review that they wrote themselves, after reading a paper. This provides a written record of feedback from a potentially large part of the community that (for technical or social reasons) decide that they would like to contribute a review. As a special case, if you are organizing a reading group that discusses a particular paper, we encourage you to post a summary of your discussion as a review and make it available to a wider audience.
A user may post the review that received from for their own papers from the standard conference or journal reviewing systems, if they believe that this is appropriate. We suggest that the users contact the program committee to verify whether posting such reviews is acceptable. This may further encourage responsibility of reviewers and make them more comfortable with posting such reviews voluntarily in the future.

In the above two cases, users are fully responsible for the content they post.

Reviews can be forwarded from committees and journals:

We expect to also directly store public review sections of conferences that adopt the idea of public reviews. Currently, in many conferences in computer science there exist a section of the review sent to the Authors and a section of the review sent to the Program Committee only. We suggest a natural extension of this structure with a Public section for each review. Public sections can then be automatically forwarded to :PaperCritic or similar systems. As a result, the reviewers can conveniently express their positive and negative criticisms of a paper to the entire community. If you are in a position to influence the process used by journals and conferences and would like them to introduce public review sections we strongly encourage you to consider the benefits of this option and we would be interested in talking to you.

Privacy and Visibility of Reviews and Reviewers

The idea of sharing reviews is useful whenever there are multiple members of the community that would benefit from the reviews. This idea reaches its full potential when both the review content and the author of the review are public. That said, there may be reasons to keep the reviews private to certain groups. In any case, we suggest that you write reviews that you are comfortable sharing with the rest of the community: if the review is accurate and competent, why not do so?

Target Audience

:PaperCritic site is primarily targetted to researchers that write and review the papers. We are open to all responsible users that would like to contribute. The filtering system (not implemented yet) will enable ranking of reviewers themselves and filtering reviews according to the desired class of reviewers. Reading of the site is meant to be free and accessible to everyone.

Relationship to other systems

We are very interested in making it easy to link :PaperCritic with web sites such as http://arxiv.org/, http://dspace.org/, http://dspace.org, citeseer, http://portal.acm.org, http://www.springerlink.com.

Nevertheless, we believe that :PaperCritic should be usable even without access to these sites.

LARA