Talk:Two-phase Reviewing

From Health of Conferences Committee

To add your comment to this discussion, please click the + sign tab above. Like an email message, you can then contribute:

• a subject (use subject Re: FOO to continue a discussion of FOO)
• message body
• (optionally) your name.

Starting Comments

This is more negatively, called quick rejection. Papers must pass an initial review by two or three reviewers before being circulated for a full complement of five or six reviews.

Some conferences (e.g., SIGCOMM, SIGMOBILE,SIGMOD) have introduced a 2-phase review process where papers that are clearly not going to make it into the conference due to a serious flaw (out of scope, lacking an evaluation, clearly non-novel) are rejected with a less rigorous review than those that are competitive. This is a compromise that allows the maximum reviewing resources to be devoted to those papers that are in serious contention. For SIGCOMM, about 10-20% of papers are rejected at an early stage based on one PC member's view (the lead reviewer) and a double-check by the PC chairs. SIGMOD assigns all papers to two reviewers in Phase 1, and only papers with at least one positive reviewer are assigned a third reviewer.

SIGITE

The only change we have made, starting with SIGITE 06, is to drop the requirement that authors first submit abstracts. Instead, we will be reviewing full papers only. The primary reason for this change has not been a problem with reviewer load. Rather, there is a feeling that potentially good papers were rejected because of poor abstracts.

SIGGRAPH, CHI, DAC, OOPSLA, SIGCSE

None of these large conferences reported using or considering two-phase reviewing at this time.

ICSE

ICSE has been using two-phase reviewing since 2004. The details of their review process are described in the survey.

SIGMETRICS

In reviewing, we seem particularly careful to make sure that the paper has at least one reviewer who knows the area very well. If it turns out at the PC meeting that somehow that was not the case, we always get an additional review from someone directly in the area.

Discussion Begins

I'm not sure this would be of any help, and it would complicate things. I actually *like* getting out of area papers and obvious duds to review, since I either quickly point out they are submitted to the wrong conference or write out one or two fatal flaws and call it a wrap. These papers end up at the bottom of the rankings in the PC meeting, and are usually flushed en masse if nobody objects. Norm Jouppi (SIGARCH)