Talk:Tracking Reviews

From Health of Conferences Committee

(Difference between revisions)
Revision as of 16:38, 8 March 2006
MarkDHill (Talk | contribs)
Added ICSE and OOPSLA response - JM.
← Previous diff
Revision as of 16:57, 9 March 2006
MarkDHill (Talk | contribs)
Starting Comments
Next diff →
Line 5: Line 5:
== Starting Comments == == Starting Comments ==
 +The most direct approach to tracking reviews has been undertaken by SIGMOD, where they have created a "pipeline" with another major database conference, VLDB, where it is common for some papers rejected at one conference are sent to the next, with their reviews carried over. The original reviewers continue to be involved. This is being done on a limited basis, only for borderline papers where it is felt that a round of author revision could lead to a solid contribution. If this is successful, SIGMOD might extend the pipeline to include IEEE's ICDE conference as well. The idea is that this will reduce repeated submission of borderline papers. More importantly, SIGMOD anticipates this will help good papers with specific problems that can be fixed, much as the journal reviewing process helps in such situations.
 +
 +Often, such tracking is done informally in small communities. For example, SIGACT has at least one member for FOCS usually on STOC, so that a paper that it is submitted to both can be compared.
 +
 +Another interesting idea is to track reviewers, to detect chronically irresponsible, weak or unfair reviewers. SIGIR is beginning to track reviewer quality, and is so far just in the data gathering stage.
 +
 +Note that paper tracking is made more difficult by double blind reviewing.
'''SIGGRAPH''' '''SIGGRAPH'''

Revision as of 16:57, 9 March 2006

To add your comment to this discussion, please click the + sign tab above. Like an email message, you can then contribute:

  • a subject (use subject Re: FOO to continue a discussion of FOO)
  • message body
  • (optionally) your name.

Starting Comments

The most direct approach to tracking reviews has been undertaken by SIGMOD, where they have created a "pipeline" with another major database conference, VLDB, where it is common for some papers rejected at one conference are sent to the next, with their reviews carried over. The original reviewers continue to be involved. This is being done on a limited basis, only for borderline papers where it is felt that a round of author revision could lead to a solid contribution. If this is successful, SIGMOD might extend the pipeline to include IEEE's ICDE conference as well. The idea is that this will reduce repeated submission of borderline papers. More importantly, SIGMOD anticipates this will help good papers with specific problems that can be fixed, much as the journal reviewing process helps in such situations.

Often, such tracking is done informally in small communities. For example, SIGACT has at least one member for FOCS usually on STOC, so that a paper that it is submitted to both can be compared.

Another interesting idea is to track reviewers, to detect chronically irresponsible, weak or unfair reviewers. SIGIR is beginning to track reviewer quality, and is so far just in the data gathering stage.

Note that paper tracking is made more difficult by double blind reviewing.

SIGGRAPH

SIGGRAPH uses a form of voluntary review tracking. Authors can voluntarily request that their reviews of a rejected paper be forwarded to other venues, including next year's conference. They can then write a cover letter indicating how they addressed the reviewers' concerns. The hope is that this makes subsequent reviewing much easier. The scheme is voluntary so as to give authors the chance to escape from reviewers they may judge as biased or uninformed.
In a variant of this scheme, some authors are given a conditional accept to ACM TOG based on their conference-submission reviews. This is typically done for those small number of papers that are of high quality, but not quite complete. In effect, it means that the conference reviewing is the first round of a standard journal review, hence it's a form of review tracking from the conference to the journal.

SIGUCCS

tracking reviews of rejected papers from conference to conference as is done in journal reviewing - We don't have a formal mechanism for this (which we think is a good idea!) but since several folks serve on the program committee year after year there is probably some informal check on this. Also, acceptance of a paper is tied to presentation at the conference.

SIGMOD

SIGMOD has done the following:
(1) Created a "pipeline" with another major DB conference, VLDB, where some papers rejected at one conference are sent to the next, with their reviews carried over. The original reviewers continue to be involved. This is being done on a limited basis, only for borderline papers where it is felt that a round of author revision could lead to a solid contribution. If this is successful, we might extend the pipeline to include IEEE's ICDE conference as well. The idea is that this will reduce repeated submission of borderline papers. More importantly, we hope this will help good papers with specific problems that can be fixed, much as the journal reviewing process helps in such situations. ...

SIGDA

Yes, but everything is done manually and ad-hoc (i.e., it happens that some TPC member in one conference saw this paper in another conference). That's why I think support from ACM Pubs Dept. in being able to query existing submissions (not only just rejects) across conferences would be good. ...

SIGSIM

Programme committee sizes are growing. We tend to track rejected papers by knowledge sharing between common PC members.

ICSE

The idea of tracking reviews has been discussed, but nothing has yet been implemented.

OOPSLA

No formal process in place.

Discussion Begins