Q3: Medium Conferences

From Health of Conferences Committee

Question 3: PROGRAM COMMITTEES

Does your community practice:

• double blind submissions
• program committee submission restrictions
• rebuttals (author responses)
• large program committees
• program subcommittees
• others?

Do these practices seem to help or hurt promoting your field?

SIGART

The PC is large, and we use a two-level committee structure (PC and SPC).

SIGARCH

double blind submissions

Yes

program committee submission restrictions

Usually not, although acceptances may be held to a slightly higher standard (e.g., not eligible for conditional acceptance with shepherding.)

rebuttals (author responses)

Yes.

large program committees Usually.

program subcommittees

No.

others?

No.

SIGCHI

double blind submissions

IN SOME CONFERENCES, NOT IN OTHERS. UNCLEAR WHETHER THERE IS MUCH BENEFIT.

program committee submission restrictions

GENERALLY NOT.

rebuttals (author responses)

WE'VE TRIED THIS IN ONE SMALLER (250-PERSON) CONFERENCE WITH SIGNIFICANT SUCCESS. WE'VE ALSO STARTED TO USE IT IN A LARGE (2000+ PERSON) CONFERENCE WITH POSITIVE INITIAL FEEDBACK. THIS IS A MECHANISM NOT ONLY FOR AVOIDING THE COMPOUNDING OF SMALL MISUNDERSTANDINGS, BUT ALSO FOR MAKING PEOPLE FEEL THEY HAD A CHANCE TO MAKE THEIR CASE AND FOR PRESSURING REVIEWERS TO BE ON TIME. HIGHLY RECOMMENDED.

large program committees

THE CONFERENCES WE HAVE THAT WOULD HAVE VERY LARGE PROGRAM COMMITTEES INSTEAD HAVE MOST REVIEWING DONE BY INDIVIDUAL REVIEWERS OUTSIDE THE PC.

program subcommittees

others?

Do these practices seem to help or hurt promoting your field?

SIGART

I think they help. We have placed rules on the number of consecutive years anyone can serve as SPC, to ensure a broader participation and representation from the community.

SIGARCH

Yes. I think the author rebuttals are very important, since the ISCA acceptance ratio is less than that of many journals. I am also a strong believer in double blind submissions, as it creates a more level playing field for new professors and authors from smaller institutions.

SIGIR

We have blind submissions. There are no PC submission restrictions; our PC is huge, so this would eliminate a large fraction of our active researchers. We have debated rebuttals, but haven't yet adopted them. As I've mentioned, our PC is large. I don't know how large without checking, but I'd guess O(150) members, plus O(25) Area Coordinators. Sue Dumais (current Program Chair, past SIGIR Chair) could give you current numbers. As mentioned above, we use a two-tier structure.

For the most part, this seems to be working. We're comfortable with our quality and workload. The current approach makes it easy to introduce and mentor younger members of the community.

SIGACT

Program committee members are not allowed to submit to the major conferences.

Program committee members are expected to be knowledgable about the review they make or solicit from subreviewer.

There are no double blind submissions nor rebuttal.

Program committees are on the order or 20

There are enough theory conferences that not having the ability to submit to a conference is not very damaging. You can always submit to the next one.

SIGPLAN

We will be trying out "double blind submissions" for PLDi 2007.

POPL and PLDI have prevented program committee members from submitting for many years. POPL is just lifting this restriction for next year.

SIGPLAN has been using author responses recently, with generally very positive reactions. The major kink in the system seems to be that authors want responses to their responses, and there is not time in the reviewing process to accommodate this request.

SIGPLAN program committees tend to have between 15 and 20 people. They do not have subcommittees. Major conferences have in person meetings.

My sense is that people like author response and are happy with the size and in-personness of the committees. The double blind reviewing is an experiment; there is considerable controversy about its usefulness.

SIGMOD

Double-Blind: The SIGMOD conference does. Others don't. We're studying the effectiveness ...

PC submission restrictions: None

Rebuttals: The SIGMOD conference is trying these out. Seems to be of limited value, but too early to draw definitive conclusions.

Subcommittees: Yes, e.g., ICDE, VLDB.

SIGCOMM

We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from the PC and the authors.

Regarding effectiveness, I'm not really sure. The double-blind submission idea is admittedly a hack, but I think it does force honest people not to get lazy when doing a review (e.g., avoiding the temptation to "trust the math" when the author is a known math whiz). Often, though, the double blind process makes it hard for authors to cite prior work.

SIGKDD

double blind submissions

not using currently, but under consideration

program committee submission restrictions

rebuttals (author responses)

large program committees

YES

program subcommittees

Under consideration

others?

KDD does use large program committees and a separate industrial subcommittee. The program committee has gotten so large that it is being restructured this year to introduce more hierarchical structure. ICML does use double blind reviewing and rebuttals. But I am unsure how effective this is.

SIGOS

SOSP uses double blind, and will probably abandon it. Indeed, at the last SIGOPS business meeting, there was strong support for nonblind reviews (reveal the identity of the reviewer). I think that this is because SOSP has a large number of students who attend the conference and who didn't get papers in.

I've brought up rebuttals as an idea several times (I liked it in INFOCOM), but it's not gained traction.

The issue of PC submission restrictions is very problematic for the flagship conferences. This is especially true for SOSP which occurs every other year. For PhD students trying to publish their work in SOSP, having their advisor be the SOSP chair - and unable to submit a paper - is too high a price to pay.

Program subcommittees are almost always a terrible idea; the quality of the conference becomes quite uneven.

SIGMOBILE

SIGMOBILE for most conferences has always done double-bind reviews. On the whole, it works well, although in some cases, it is hard to write a paper that cites your own work. The author has to be careful, and sometimes it is nearly impossible to cite your own work without giving away the author's identity. We also have a lot of trouble with authors who don't follow the rules at all and just put their names on the paper. We have to either get them to change and resubmit the paper, or directly edit the names out ourselves.

I have not found (in other conferences) rebuttals to work well at all. It always has seemed as an author that, no matter what you say in your rebuttal, the end result is the same as the original reviews and doesn't take the rebuttal into account at all. Allowing time for rebuttals also lengthens the whole review process, requiring papers to be submitted longer before the conference.