Talk:Double-Blind Submissions
From Health of Conferences Committee
To add your comment to this discussion, please click the + sign tab above. Like an email message, you can then contribute:
- a subject (use subject Re: FOO to continue a discussion of FOO)
- message body
- (optionally) your name.
Contents |
Starting Comments
Blind submissions hide the referee name from the authors, while double-blind submission also hide the author name from the referees (but usually author names are know at the program committee meeting.
SIGARCH
- .... I am also a strong believer in double blind submissions, as it creates a more level playing field for new professors and authors from smaller institutions.
SIGCOMM
- We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from the PC and the authors.
- Regarding effectiveness, I'm not really sure. The double-blind submission idea is admittedly a hack, but I think it does force honest people not to get lazy when doing a review (e.g., avoiding the temptation to "trust the math" when the author is a known math whiz). Often, though, the double blind process makes it hard for authors to cite prior work.
SIGCSE
- SIGCSE takes the following approach for reviewing:
- reviewing for both the SIGCSE Symposium and ITiCSE Conferences are double blind
- we maintain a database including about 779 reviewers for the Symposium and 584 reviewers for the ITiCSE conference.
- each paper is sent to about 6 reviewers, and each reviewer receives 2-4 papers.
- a minimum of 4 reviews is required for each paper, and the Program
- Co-Chairs assign last-minute reviewers if the minimum number of reviews has not been met.
- After the reviewing period and after decisions are made, authors may view the reviews of their papers (although the reviews are anonymous), and each reviewer may view the other reviews of the papers they considered (again anonymously).
SIGMIS
- Double blind submissions definitely help the field to remove perceived bias toward large research institutions. This approach also negates the necessity to restrict program committee submissions. Author responses are a good mechanism to clarify misunderstandings on the part of reviewers and are therefore perceived as helpful to the field.
SIGGRAPH
- The review process is double-blind for external (tertiary) reviewers. The members of the Papers Committee (the primary and secondary reviewers) need to know who the authors are to avoid conflicts among the tertiary reviewers.
OOPSLA
- Not at the moment.
Discussion Begins
DocEng Symposia (SIGWEB)
In the Document Engineering Symposia, we use blind (not double blind) reviewing. Our reviewing software does not allow PC members to see the names of reviewers for their papers. Papers by program chairs are reviewed separately to ensure reviewer confidentiality.
DocEng has been running for five years and is small. I do not remember any requests for us to move to double-blind reviewing. Our community is small and collegial, so there is little sense that papers get "shot down" for political reasons. Infrequently, we do have problems with inappropriate reviews. We either remove the problem reviewer from the PC or give polite and strong feedback. Also, in a small community, it can be very hard to write a paper that hides its origin, since there aren't many people working on any one problem.
Personally, I hate writing a paper for double-blind review. My work is a continuing process in which it is natural to make self-references and to explicitly discuss how the current article complements or extends previous work.
Ethan Munson (DocEng Steering Chair and SIGWEB Treasurer)
SIGAPP
At the Symposium on Applied Computing (SAC, SIGAPP's flagship conference), we use double-blind reviewing. My opinion is mixed. As an author, it is inconvenient to write papers without referring to previous work and/or a website describing the project. The flip side of this is that a reviewer rejected a paper by an author X who had not cited his own work because "this paper does not give proper credit to author X!" On the other hand, some very well known people have been rejected for submitting a substandard paper, where their name might have been enough to carry it without the blind review. --Barrett Bryant
Lightweight approach
My personal opinion is that I feel I'm fairer and more careful when I am reviewing a paper if I don't know who the authors are immediately. In many cases of course by the end of the paper you can figure out who the authors are, but that's a different story: at least you have read the main ideas in a more open state of mind.
However, when the rules are too strict and threaten to reject the paper if you mistakenly provide information about who you are, the cost seems to be too high for authors. I would prefer a less draconian approach in which authors should make reasonable efforts to conceal their identities (e.g.: by not putting their names in the paper and citing their own work in neutral form) but are not required to distort the paper too much nor to spend too much time on this. --Carlos Castillo
