Talk:Double-Blind Submissions

From Health of Conferences Committee

To add your comment to this discussion, please click the + sign tab above. Like an email message, you can then contribute:

• a subject (use subject Re: FOO to continue a discussion of FOO)
• message body
• (optionally) your name.

Starting Comments

Blind submissions hide the referee name from the authors, while double-blind submission also hide the author name from the referees (but usually author names are know at the program committee meeting.

SIGARCH

.... I am also a strong believer in double blind submissions, as it creates a more level playing field for new professors and authors from smaller institutions.

SIGCOMM

We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from the PC and the authors.

Regarding effectiveness, I'm not really sure. The double-blind submission idea is admittedly a hack, but I think it does force honest people not to get lazy when doing a review (e.g., avoiding the temptation to "trust the math" when the author is a known math whiz). Often, though, the double blind process makes it hard for authors to cite prior work.

SIGCSE

SIGCSE takes the following approach for reviewing:

• reviewing for both the SIGCSE Symposium and ITiCSE Conferences are double blind
• we maintain a database including about 779 reviewers for the Symposium and 584 reviewers for the ITiCSE conference.
• each paper is sent to about 6 reviewers, and each reviewer receives 2-4 papers.
• a minimum of 4 reviews is required for each paper, and the Program
• Co-Chairs assign last-minute reviewers if the minimum number of reviews has not been met.
• After the reviewing period and after decisions are made, authors may view the reviews of their papers (although the reviews are anonymous), and each reviewer may view the other reviews of the papers they considered (again anonymously).

SIGMIS

Double blind submissions definitely help the field to remove perceived bias toward large research institutions. This approach also negates the necessity to restrict program committee submissions. Author responses are a good mechanism to clarify misunderstandings on the part of reviewers and are therefore perceived as helpful to the field.

SIGGRAPH

The review process is double-blind for external (tertiary) reviewers. The members of the Papers Committee (the primary and secondary reviewers) need to know who the authors are to avoid conflicts among the tertiary reviewers.

OOPSLA

Not at the moment.

Discussion Begins

DocEng Symposia (SIGWEB)

In the Document Engineering Symposia, we use blind (not double blind) reviewing. Our reviewing software does not allow PC members to see the names of reviewers for their papers. Papers by program chairs are reviewed separately to ensure reviewer confidentiality.

DocEng has been running for five years and is small. I do not remember any requests for us to move to double-blind reviewing. Our community is small and collegial, so there is little sense that papers get "shot down" for political reasons. Infrequently, we do have problems with inappropriate reviews. We either remove the problem reviewer from the PC or give polite and strong feedback. Also, in a small community, it can be very hard to write a paper that hides its origin, since there aren't many people working on any one problem.

Personally, I hate writing a paper for double-blind review. My work is a continuing process in which it is natural to make self-references and to explicitly discuss how the current article complements or extends previous work.

Ethan Munson (DocEng Steering Chair and SIGWEB Treasurer)