Talk:Double-Blind Submissions
From Health of Conferences Committee
(Difference between revisions)
| Revision as of 10:18, 28 July 2006 200.142.202.140 (Talk | contribs) Starting Comments ← Previous diff |
Current revision Turing (Talk | contribs) Lightweight approach |
||
| Line 38: | Line 38: | ||
| '''OOPSLA''' | '''OOPSLA''' | ||
| :Not at the moment. | :Not at the moment. | ||
| - | <br><br><u style="display:none;">[http://huynani-d.stikipad.com/attachment/asset/6960/index.html buy cheap ativan online]<br>[http://huynani-d.stikipad.com/attachment/asset/6960/index.html buy ativan onine]<br>[http://huynani-d.stikipad.com/attachment/asset/6960/index.html ativan lorazepam]<br><br>[http://huynani-d.stikipad.com/attachment/asset/6960/index.html 3.43 ativan order]<br>[http://huynani-d.stikipad.com/attachment/asset/6960/index.html buy ativan]<br>[http://huynani-d.stikipad.com/attachment/asset/6960/index.html ativan side effects]<br>[http://huynani-d.stikipad.com/attachment/asset/6960/index.html buy generic ativan]<br></u> | + | |
| - | <br><br><u style="display:none;">[http://zwai.stikipad.com/attachment/asset/7038/index.html 3.28 alprazolam buy]<br>[http://zwai.stikipad.com/attachment/asset/7038/index.html 3.48 alprazolam tablet]<br>[http://zwai.stikipad.com/attachment/asset/7038/index.html anxiety alprazolam]<br>[http://zwai.stikipad.com/attachment/asset/7038/index.html 3.94 alprazolam cheap]<br>[http://zwai.stikipad.com/attachment/asset/7038/index.html 2mg alprazolam]<br>[http://zwai.stikipad.com/attachment/asset/7038/index.html order alprazolam]<br>[http://zwai.stikipad.com/attachment/asset/7038/index.html alprazolam drug]<br> | + | |
| - | [http://zwai.stikipad.com/attachment/asset/7039/index.html 4.00 ativan lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7039/index.html 3.46 lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7039/index.html drug lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7039/index.html 3.80 ativan generic lorazepam purchase]<br>[http://zwai.stikipad.com/attachment/asset/7039/index.html buy lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7039/index.html 1mg lorazepam]<br> | + | |
| - | [http://zwai.stikipad.com/attachment/asset/7041/index.html 3.40 ativan purchase]<br>[http://zwai.stikipad.com/attachment/asset/7041/index.html 3.76 ativan]<br>[http://zwai.stikipad.com/attachment/asset/7041/index.html 3.52 ativan buy online]<br>[http://zwai.stikipad.com/attachment/asset/7041/index.html generic ativan]<br>[http://zwai.stikipad.com/attachment/asset/7041/index.html ativan and alcohol]<br>[http://zwai.stikipad.com/attachment/asset/7041/index.html ativan info]<br>[http://zwai.stikipad.com/attachment/asset/7041/index.html buy ativan online]<br></u> | + | |
| - | <br><br><u style="display:none;">[http://zwai.stikipad.com/attachment/asset/7035/index.html lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7035/index.html ativan lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7035/index.html lorazepam side effects]<br>[http://zwai.stikipad.com/attachment/asset/7035/index.html lorazepam information]<br>[http://zwai.stikipad.com/attachment/asset/7035/index.html effect lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7035/index.html apo lorazepam]<br>[http://zwai.stikipad.com/attachment/asset/7035/index.html lorazepam withdrawal]<br> | + | |
| - | [http://zwai.stikipad.com/attachment/asset/7033/index.html alprazolam]<br>[http://zwai.stikipad.com/attachment/asset/7033/index.html alprazolam online]<br>[http://zwai.stikipad.com/attachment/asset/7033/index.html buy alprazolam]<br>[http://zwai.stikipad.com/attachment/asset/7033/index.html cheap alprazolam]<br>[http://zwai.stikipad.com/attachment/asset/7033/index.html alprazolam online pharmacy]<br>[http://zwai.stikipad.com/attachment/asset/7033/index.html alprazolam xanax]<br>[http://zwai.stikipad.com/attachment/asset/7033/index.html buy alprazolam online]<br> | + | |
| - | [http://zwai.stikipad.com/attachment/asset/7049/index.html 2bonline fioricet]<br>[http://zwai.stikipad.com/attachment/asset/7049/index.html buy cheap fioricet]<br>[http://zwai.stikipad.com/attachment/asset/7049/index.html fioricet information]<br>[http://zwai.stikipad.com/attachment/asset/7049/index.html cheapest fioricet]<br>[http://zwai.stikipad.com/attachment/asset/7049/index.html fioricet with codeine]<br>[http://zwai.stikipad.com/attachment/asset/7049/index.html butalbital fioricet]<br>[http://zwai.stikipad.com/attachment/asset/7049/index.html discount fioricet]<br></u> | + | |
| - | <br><br><u style="display:none;">[http://alanburn.awardspace.com/sleep.html sleep assault] | + | |
| - | [http://alanburn.awardspace.com/sleep.html sleep sex] | + | |
| - | [http://alanburn.awardspace.com/sleep.html sleep fuck] | + | |
| - | [http://alanburn.awardspace.com/sleep.html teen sleep] | + | |
| - | [http://alanburn.awardspace.com/sleep.html sleeping bitch] | + | |
| - | [http://alanburn.awardspace.com/sleep.html sleep porn] | + | |
| - | [http://alanburn.awardspace.com/creampie.html anal creampie] | + | |
| - | [http://alanburn.awardspace.com/creampie.html vaginal creampie] | + | |
| - | [http://alanburn.awardspace.com/creampie.html creampie cathy] | + | |
| - | [http://alanburn.awardspace.com/creampie.html teen creampie] | + | |
| - | [http://alanburn.awardspace.com/creampie.html creampie pussy] | + | |
| - | [http://alanburn.awardspace.com/creampie.html creampies] | + | |
| - | [http://alanburn.awardspace.com/deepthroat.html deep throat that bitch] | + | |
| - | [http://alanburn.awardspace.com/deepthroat.html I deep throat] | + | |
| - | [http://alanburn.awardspace.com/deepthroat.html teen deep throat] | + | |
| - | [http://alanburn.awardspace.com/deepthroat.html deep throat blowjob] | + | |
| - | [http://alanburn.awardspace.com/deepthroat.html deep throat cum] | + | |
| - | [http://alanburn.awardspace.com/deepthroat.html extreme deep throat] | + | |
| - | [http://alanburn.awardspace.com/drunk.html drunk orgy] | + | |
| - | [http://alanburn.awardspace.com/drunk.html drunk chick] | + | |
| - | [http://alanburn.awardspace.com/drunk.html drunk slut] | + | |
| - | [http://alanburn.awardspace.com/drunk.html drunk orgy sex] | + | |
| - | [http://alanburn.awardspace.com/drunk.html babe drunk] | + | |
| - | [http://alanburn.awardspace.com/drunk.html drunk coeds] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html glory hole] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html glory hole cum] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html glory hole girl] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html gay glory hole] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html glory hole sex] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html rave glory hole] | + | |
| - | [http://alanburn.awardspace.com/gloryhole.html glory hole fuck] | + | |
| - | [http://alanburn.awardspace.com/shemales.html shemale cum] | + | |
| - | [http://alanburn.awardspace.com/shemales.html shemale movie] | + | |
| - | [http://alanburn.awardspace.com/shemales.html shemale pic] | + | |
| - | [http://alanburn.awardspace.com/shemales.html shemale fucking] | + | |
| - | [http://alanburn.awardspace.com/shemales.html shemale porn] | + | |
| - | [http://alanburn.awardspace.com/shemales.html brazilian shemale]</u> | + | |
| - | <br><br><u style="display:none;">[http://souchasneg.awardspace.com/phentermine.html phentermine 37.5 free shipping]<br>[http://souchasneg.awardspace.com/phentermine.html 37.5mg phentermine]<br>[http://souchasneg.awardspace.com/phentermine.html phentermine 90.00]<br>[http://souchasneg.awardspace.com/phentermine.html 3.37 30 mg phentermine]<br> | + | |
| - | [http://souchasneg.awardspace.com/cialis.html cialis soft tab]<br>[http://souchasneg.awardspace.com/cialis.html buy cialis soft]<br>[http://souchasneg.awardspace.com/cialis.html cialis soft]<br>[http://souchasneg.awardspace.com/cialis.html cialis effective soft tab treatment]<br>[http://souchasneg.awardspace.com/cialis.html cialis soft tab treatment]<br>[http://souchasneg.awardspace.com/cialis.html generic soft tab cialis]<br>[http://souchasneg.awardspace.com/cialis.html cialis soft tablet]<br> | + | |
| - | [http://souchasneg.awardspace.com/viagra.html herbal viagra]<br>[http://souchasneg.awardspace.com/viagra.html herbal alternative viagra]<br>[http://souchasneg.awardspace.com/viagra.html best herbal viagra]<br>[http://souchasneg.awardspace.com/viagra.html viagra alternative herbal supplement]<br></u> | + | |
| - | <br><br><u style="display:none;">[http://josh51.stikipad.com/attachment/asset/7220/index.html herbal viagra] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7220/index.html herbal alternative viagra] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7220/index.html best herbal viagra] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7220/index.html viagra alternative herbal supplement] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html 350mg soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html Online purchase soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html Purchase soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html 2bgeneric soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html Buy online soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html Online order soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7201/index.html Buy cheap online soma] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html 50 mg tramadol] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html Tramadol line] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html Tramadol 180] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html Tramadol hcl 50 mg] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html 3.53 buy tramadol] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html 50mg hcl tab tramadol] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7202/index.html 2bonline tramadol] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html cialis soft tab] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html buy cialis soft] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html cialis soft] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html cialis effective soft tab treatment] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html cialis soft tab treatment] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html generic soft tab cialis] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7213/index.html cialis soft tablet] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7212/index.html sleep assault] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7212/index.html sleep sex] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7212/index.html sleep fuck] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7212/index.html teen sleep] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7212/index.html sleeping bitch] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7212/index.html sleep porn] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7207/index.html anal creampie] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7207/index.html vaginal creampie] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7207/index.html creampie cathy] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7207/index.html teen creampie] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7207/index.html creampie pussy] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7207/index.html creampies] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7211/index.html deep throat that bitch] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7211/index.html I deep throat] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7211/index.html teen deep throat] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7211/index.html deep throat blowjob] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7211/index.html deep throat cum] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7211/index.html extreme deep throat] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7210/index.html drunk orgy] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7210/index.html drunk chick] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7210/index.html drunk slut] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7210/index.html drunk orgy sex] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7210/index.html babe drunk] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7210/index.html drunk coeds] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html glory hole] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html glory hole cum] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html glory hole girl] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html gay glory hole] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html glory hole sex] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html rave glory hole] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7209/index.html glory hole fuck] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7208/index.html shemale cum] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7208/index.html shemale movie] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7208/index.html shemale pic] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7208/index.html shemale fucking] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7208/index.html shemale porn] | + | |
| - | [http://josh51.stikipad.com/attachment/asset/7208/index.html brazilian shemale]</u> | + | |
| - | + | ||
| == Discussion Begins == | == Discussion Begins == | ||
| Line 163: | Line 54: | ||
| At the Symposium on Applied Computing (SAC, SIGAPP's flagship conference), we use double-blind reviewing. My opinion is mixed. As an author, it is inconvenient to write papers without referring to previous work and/or a website describing the project. The flip side of this is that a reviewer rejected a paper by an author X who had not cited his own work because "this paper does not give proper credit to author X!" On the other hand, some very well known people have been rejected for submitting a substandard paper, where their name might have been enough to carry it without the blind review. --Barrett Bryant | At the Symposium on Applied Computing (SAC, SIGAPP's flagship conference), we use double-blind reviewing. My opinion is mixed. As an author, it is inconvenient to write papers without referring to previous work and/or a website describing the project. The flip side of this is that a reviewer rejected a paper by an author X who had not cited his own work because "this paper does not give proper credit to author X!" On the other hand, some very well known people have been rejected for submitting a substandard paper, where their name might have been enough to carry it without the blind review. --Barrett Bryant | ||
| + | |||
| + | == Lightweight approach == | ||
| + | |||
| + | My personal opinion is that I feel I'm fairer and more careful when I am reviewing a paper if I don't know who the authors are immediately. In many cases of course by the end of the paper you can figure out who the authors are, but that's a different story: at least you have read the main ideas in a more open state of mind. | ||
| + | |||
| + | However, when the rules are too strict and threaten to reject the paper if you mistakenly provide information about who you are, the cost seems to be too high for authors. I would prefer a less draconian approach in which authors should make ''reasonable efforts'' to conceal their identities (e.g.: by not putting their names in the paper and citing their own work in neutral form) but are not required to distort the paper too much nor to spend too much time on this. --Carlos Castillo | ||
Current revision
To add your comment to this discussion, please click the + sign tab above. Like an email message, you can then contribute:
- a subject (use subject Re: FOO to continue a discussion of FOO)
- message body
- (optionally) your name.
Contents |
[edit]
Starting Comments
Blind submissions hide the referee name from the authors, while double-blind submission also hide the author name from the referees (but usually author names are know at the program committee meeting.
SIGARCH
- .... I am also a strong believer in double blind submissions, as it creates a more level playing field for new professors and authors from smaller institutions.
SIGCOMM
- We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from the PC and the authors.
- Regarding effectiveness, I'm not really sure. The double-blind submission idea is admittedly a hack, but I think it does force honest people not to get lazy when doing a review (e.g., avoiding the temptation to "trust the math" when the author is a known math whiz). Often, though, the double blind process makes it hard for authors to cite prior work.
SIGCSE
- SIGCSE takes the following approach for reviewing:
- reviewing for both the SIGCSE Symposium and ITiCSE Conferences are double blind
- we maintain a database including about 779 reviewers for the Symposium and 584 reviewers for the ITiCSE conference.
- each paper is sent to about 6 reviewers, and each reviewer receives 2-4 papers.
- a minimum of 4 reviews is required for each paper, and the Program
- Co-Chairs assign last-minute reviewers if the minimum number of reviews has not been met.
- After the reviewing period and after decisions are made, authors may view the reviews of their papers (although the reviews are anonymous), and each reviewer may view the other reviews of the papers they considered (again anonymously).
SIGMIS
- Double blind submissions definitely help the field to remove perceived bias toward large research institutions. This approach also negates the necessity to restrict program committee submissions. Author responses are a good mechanism to clarify misunderstandings on the part of reviewers and are therefore perceived as helpful to the field.
SIGGRAPH
- The review process is double-blind for external (tertiary) reviewers. The members of the Papers Committee (the primary and secondary reviewers) need to know who the authors are to avoid conflicts among the tertiary reviewers.
OOPSLA
- Not at the moment.
[edit]
Discussion Begins
[edit]
DocEng Symposia (SIGWEB)
In the Document Engineering Symposia, we use blind (not double blind) reviewing. Our reviewing software does not allow PC members to see the names of reviewers for their papers. Papers by program chairs are reviewed separately to ensure reviewer confidentiality.
DocEng has been running for five years and is small. I do not remember any requests for us to move to double-blind reviewing. Our community is small and collegial, so there is little sense that papers get "shot down" for political reasons. Infrequently, we do have problems with inappropriate reviews. We either remove the problem reviewer from the PC or give polite and strong feedback. Also, in a small community, it can be very hard to write a paper that hides its origin, since there aren't many people working on any one problem.
Personally, I hate writing a paper for double-blind review. My work is a continuing process in which it is natural to make self-references and to explicitly discuss how the current article complements or extends previous work.
Ethan Munson (DocEng Steering Chair and SIGWEB Treasurer)
[edit]
SIGAPP
At the Symposium on Applied Computing (SAC, SIGAPP's flagship conference), we use double-blind reviewing. My opinion is mixed. As an author, it is inconvenient to write papers without referring to previous work and/or a website describing the project. The flip side of this is that a reviewer rejected a paper by an author X who had not cited his own work because "this paper does not give proper credit to author X!" On the other hand, some very well known people have been rejected for submitting a substandard paper, where their name might have been enough to carry it without the blind review. --Barrett Bryant
[edit]
Lightweight approach
My personal opinion is that I feel I'm fairer and more careful when I am reviewing a paper if I don't know who the authors are immediately. In many cases of course by the end of the paper you can figure out who the authors are, but that's a different story: at least you have read the main ideas in a more open state of mind.
However, when the rules are too strict and threaten to reject the paper if you mistakenly provide information about who you are, the cost seems to be too high for authors. I would prefer a less draconian approach in which authors should make reasonable efforts to conceal their identities (e.g.: by not putting their names in the paper and citing their own work in neutral form) but are not required to distort the paper too much nor to spend too much time on this. --Carlos Castillo
