Q3: Medium Conferences

From Health of Conferences Committee

(Difference between revisions)

Current revision

[edit]

Question 3: PROGRAM COMMITTEES

Does your community practice:

double blind submissions
program committee submission restrictions
rebuttals (author responses)
large program committees
program subcommittees
others?

Do these practices seem to help or hurt promoting your field?

SIGART

The PC is large, and we use a two-level committee structure (PC and SPC).

SIGARCH

double blind submissions

Yes

program committee submission restrictions

Usually not, although acceptances may be held to a slightly higher standard (e.g., not eligible for conditional acceptance with shepherding.)

rebuttals (author responses)

Yes.

large program committees Usually.

program subcommittees

No.

others?

No.

SIGCHI

double blind submissions

IN SOME CONFERENCES, NOT IN OTHERS. UNCLEAR WHETHER THERE IS MUCH BENEFIT.

program committee submission restrictions

GENERALLY NOT.

rebuttals (author responses)

WE'VE TRIED THIS IN ONE SMALLER (250-PERSON) CONFERENCE WITH SIGNIFICANT SUCCESS. WE'VE ALSO STARTED TO USE IT IN A LARGE (2000+ PERSON) CONFERENCE WITH POSITIVE INITIAL FEEDBACK. THIS IS A MECHANISM NOT ONLY FOR AVOIDING THE COMPOUNDING OF SMALL MISUNDERSTANDINGS, BUT ALSO FOR MAKING PEOPLE FEEL THEY HAD A CHANCE TO MAKE THEIR CASE AND FOR PRESSURING REVIEWERS TO BE ON TIME. HIGHLY RECOMMENDED.

large program committees

THE CONFERENCES WE HAVE THAT WOULD HAVE VERY LARGE PROGRAM COMMITTEES INSTEAD HAVE MOST REVIEWING DONE BY INDIVIDUAL REVIEWERS OUTSIDE THE PC.

program subcommittees

others?

Do these practices seem to help or hurt promoting your field?

SIGART

I think they help. We have placed rules on the number of consecutive years anyone can serve as SPC, to ensure a broader participation and representation from the community.

SIGARCH

Yes. I think the author rebuttals are very important, since the ISCA acceptance ratio is less than that of many journals. I am also a strong believer in double blind submissions, as it creates a more level playing field for new professors and authors from smaller institutions.

SIGIR

We have blind submissions. There are no PC submission restrictions; our PC is huge, so this would eliminate a large fraction of our active researchers. We have debated rebuttals, but haven't yet adopted them. As I've mentioned, our PC is large. I don't know how large without checking, but I'd guess O(150) members, plus O(25) Area Coordinators. Sue Dumais (current Program Chair, past SIGIR Chair) could give you current numbers. As mentioned above, we use a two-tier structure.

For the most part, this seems to be working. We're comfortable with our quality and workload. The current approach makes it easy to introduce and mentor younger members of the community.

SIGACT

Program committee members are not allowed to submit to the major conferences.

Program committee members are expected to be knowledgable about the review they make or solicit from subreviewer.

There are no double blind submissions nor rebuttal.

Program committees are on the order or 20

There are enough theory conferences that not having the ability to submit to a conference is not very damaging. You can always submit to the next one.

SIGPLAN

We will be trying out "double blind submissions" for PLDi 2007.

POPL and PLDI have prevented program committee members from submitting for many years. POPL is just lifting this restriction for next year.

SIGPLAN has been using author responses recently, with generally very positive reactions. The major kink in the system seems to be that authors want responses to their responses, and there is not time in the reviewing process to accommodate this request.

SIGPLAN program committees tend to have between 15 and 20 people. They do not have subcommittees. Major conferences have in person meetings.

My sense is that people like author response and are happy with the size and in-personness of the committees. The double blind reviewing is an experiment; there is considerable controversy about its usefulness.

SIGMOD

Double-Blind: The SIGMOD conference does. Others don't. We're studying the effectiveness ...

PC submission restrictions: None

Rebuttals: The SIGMOD conference is trying these out. Seems to be of limited value, but too early to draw definitive conclusions.

Subcommittees: Yes, e.g., ICDE, VLDB.

SIGCOMM

We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from the PC and the authors.

Regarding effectiveness, I'm not really sure. The double-blind submission idea is admittedly a hack, but I think it does force honest people not to get lazy when doing a review (e.g., avoiding the temptation to "trust the math" when the author is a known math whiz). Often, though, the double blind process makes it hard for authors to cite prior work.

SIGKDD

double blind submissions

not using currently, but under consideration

program committee submission restrictions

rebuttals (author responses)

large program committees

YES

program subcommittees

Under consideration

others?

KDD does use large program committees and a separate industrial subcommittee. The program committee has gotten so large that it is being restructured this year to introduce more hierarchical structure. ICML does use double blind reviewing and rebuttals. But I am unsure how effective this is.

SIGOS

SOSP uses double blind, and will probably abandon it. Indeed, at the last SIGOPS business meeting, there was strong support for nonblind reviews (reveal the identity of the reviewer). I think that this is because SOSP has a large number of students who attend the conference and who didn't get papers in.

I've brought up rebuttals as an idea several times (I liked it in INFOCOM), but it's not gained traction.

The issue of PC submission restrictions is very problematic for the flagship conferences. This is especially true for SOSP which occurs every other year. For PhD students trying to publish their work in SOSP, having their advisor be the SOSP chair - and unable to submit a paper - is too high a price to pay.

Program subcommittees are almost always a terrible idea; the quality of the conference becomes quite uneven.

SIGMOBILE

SIGMOBILE for most conferences has always done double-bind reviews. On the whole, it works well, although in some cases, it is hard to write a paper that cites your own work. The author has to be careful, and sometimes it is nearly impossible to cite your own work without giving away the author's identity. We also have a lot of trouble with authors who don't follow the rules at all and just put their names on the paper. We have to either get them to change and resubmit the paper, or directly edit the names out ourselves.

I have not found (in other conferences) rebuttals to work well at all. It always has seemed as an author that, no matter what you say in your rebuttal, the end result is the same as the original reviews and doesn't take the rebuttal into account at all. Allowing time for rebuttals also lengthens the whole review process, requiring papers to be submitted longer before the conference.

Retrieved from "http://wiki.acm.org/healthcc/index.php?title=Q3:_Medium_Conferences"

Q3: Medium Conferences

From Health of Conferences Committee

Current revision

Question 3: PROGRAM COMMITTEES

Views

Personal tools

Navigation

Search

Toolbox

 :Yes
 :program committee submission restrictions
-:Usually not, although acceptances may be held to a slightly higher standard (e.g., not eligible for conditional acceptance with
+:Usually not, although acceptances may be held to a slightly higher standard (e.g., not eligible for conditional acceptance with shepherding.)
-shepherding.)
 :rebuttals (author responses)
 :Yes.
 '''SIGCOMM'''
-:We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members
+:We do double-blind submissions at our main SIGCOMM conference but not at most (any?) of our other events. Restrictions on PC members tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from the PC and the authors.
-tend not to happen, though occassionally the PC chairs will decide to impose (say) a two-paper limit, or hold PC papers to a higher
-bar. We tried rebuttals one time at SIGCOMM about 5-6 years ago, and it wasn't worth the substantial time and energy it required from
-the PC and the authors.
 :Regarding effectiveness, I'm not really sure. The double-blind submission idea is admittedly a hack, but I think it does force honest people not to get lazy when doing a review (e.g., avoiding the temptation to "trust the math" when the author is a known math whiz). Often, though, the double blind process makes it hard for authors to cite prior work.
 :others?
 :KDD does use large program committees and a separate industrial subcommittee. The program committee has gotten so large that it is being restructured this year to introduce more hierarchical structure. ICML does use double blind reviewing and rebuttals. But I am unsure how effective this is.
-----
-'''SIGART'''
+'''SIGOS'''
-:Yes, for the large conferences we have workshops co-located with the conference.
+:SOSP uses double blind, and will probably abandon it. Indeed, at the last SIGOPS business meeting, there was strong support for nonblind reviews (reveal the identity of the reviewer). I think that this is because SOSP has a large number of students who attend the conference and who didn't get papers in.
+:I've brought up rebuttals as an idea several times (I liked it in INFOCOM), but it's not gained traction.
-'''SIGARCH'''
+:The issue of PC submission restrictions is very problematic for the flagship conferences. This is especially true for SOSP which occurs every other year. For PhD students trying to publish their work in SOSP, having their advisor be the SOSP chair - and unable to submit a paper - is too high a price to pay.
-:workshop co-located at conferences
-:Yes
-:stand-alone workshops
-:No
-:panels
-:Often (But just one per conference.)
-:crazy idea sessions
-:Not at ISCA.
+:Program subcommittees are almost always a terrible idea; the quality of the conference becomes quite uneven.
-'''SIGCHI'''
-:YES, MANY OF THEM, AT CONFERENCES OF ALL SIZES. THESE ARE A GREAT BENEFIT TO BOTH THE COMMUNITY AND THE CONFERENCE. WE BOTH HAVE "MAJOR" ONES THAT ARE SEPARATELY ORGANIZED EVENTS AND MINOR ONES THAT ARE PART OF THE CONFERENCE PROGRAM (BUT WITH SEPARATE SUBMISSIONS/REVIEW).
-:stand-alone workshops
-:YES, WE HAVE MANY OF THESE, THOUGH MANY OF THEM ARE NOT ACM-SPONSORED BECAUSE OF THE COSTS INVOLVED. IN SOME CASES WE SPONSOR OR CO-SPONSOR THEM, IN OTHER CASES THEY ARE HOSTED BY UNIVERSITIES OR CORPORATE LABS.
-:panels
-:SOMETIMES. SOME CONFERENCE HAVE A TRADITION OF PANELS. OTHERS HAVE ELIMINATED THEM.
-:crazy idea sessions
-:On balance, are these other venues effective for advancing your field?
-'''SIGART'''
-:Workshops help broadening the community while keeping a low acceptance rate for the conference.
-'''SIGARCH'''
-:Yes, the workshops (and tutorials) are a great way to broaden the active participation in the conference.
-'''SIGIR'''
-:We have about 10 workshops per year co-located with the conference (on the "workshop day", the day after the main conference ends). Turnout is very good, at all levels of the community (junior to senior).
-:The IR community has a few standalone workshops each year, but not many. Occasionally, e.g., every 3-4 years, there might be an invitational workshop that allows the senior members of the community to meet to discuss the direction of the field.
-:We used to do panels, but not in recent years. Many felt that they were not helpful - just an opportunity to pontificate without any
-real debate emerging.
-:SIGIR has a 2-hour Business Meeting at each conference. This is an opportunity for the Executive Committee to report to the community
-about what it's doing, to solicit opinions on major changes to the conference (e.g., to do away with paper proceedings, or to make major changes in reviewing structure), and for the community to sound off about what it feels is important. Attendance is usually very good.
-'''SIGACT'''
-:Some of our conference do have workshops and tutorials colocated. There are few stand-alone workshops. Some conferences have panels but only at the business meeting.
-'''SIGPLAN'''
-:SIGPLAN conferences tend to have many workshops co-located with conferences, often as many as six or seven.
-:We tend not to have panels or crazy idea sessions.
-:There are some number of stand alone workshops, but not so many.
-'''SIGMOD'''
-:Yes, to all of the above. And I think they are effective.
-'''SIGCOMM'''
-:Starting with SIGCOMM'03, the SIGCOMM conference has been a five-day event, with three-days of conference (T-W-Th) and two days of tutorials and workshops (M and F). The workshops vary from year to year and are chosen by review of proposals. We also sponsor a number of stand-alone worksohps, including HotNets and the SIGCOMM Asia Workshop, and conferences (IMC, ANCS, etc.). SIGCOMM sometimes has a work-in-progress session, and also has a poster session. We also have an outrageous opinion session.
-:The workshops -- both stand-alone and co-located -- are effective. I have mixed feelings about panels and work-in-progress talks, as
-sometimes they can be rather useless but, just every so often, they are really good. The outrageous opinion session often degenerates
-into stand-up comedy but it is loads of fun and really lightens the mood and brings the every-larger community closer together.
-What mechanisms, if any, do you use to allow good papers from these venues to later achieve wider dissemination?
-'''SIGART'''
-:Nothing specific, it is up to the workshop organizers.
-'''SIGARCH'''
-:Some workshops have their papers published as special issues in SIGARCH Computer Architecture News or other venues. But this is based more on the energy of the workshop organizers than the quality of the workshop.
-'''SIGCHI'''
-:DEPENDS ON THE VENUE. PANELS MAY BE WRITTEN UP AS MAGAZINE ARTICLES (FOR INTERACTIONS MAGAZINE) IF WARRANTED. SMALL WORKSHOP PAPERS OFTEN ARE REVISED AND EXTENDED INTO CONFERENCE PAPERS. OUR "MAJOR" WORKSHOPS OFTEN PUBLISH THEIR PROCEEDINGS IN THE ACM DL.
-'''SIGIR'''
-:Extended / revised versions of workshop papers may be submitted to an IR conference or journal.
-'''SIGACT'''
-:Authors of conference papers are expected to submit full papers to journals. Most do just that because the conference paper has severe page limits. There are quite a few very strong theory journals. Most conferences have a best papers solicitation for special issues of journals. When a new result comes out there is often a lot of buzz on blogs and around the community. Very often preprints of these hot results circulate before the conference.
-'''SIGPLAN'''
-:Good papers from such workshops are often later published in conferences. Occasionally there are special issues of journals
-devoted to them. We often publish the abstracts of such workshops in SIGPLAN Notices.
-'''SIGMOD'''
-:None, really, that I'm aware of.
-'''SIGCOMM'''
-:The top few papers from SIGCOMM (and from IMC) get forwarded to IEEE/ACM Transactions on Networking for fast-track consideration.
-Several of the conferences (including the main SIGCOMM conference) give a best student paper award. We are in the process of instituting the SIGCOMM Test of Time award to recognize an influential paper published 10-12 years ago in a SIGCOMM-sponsored venue. We also give a SIGCOMM Award for lifetime achievement each year, and are active in nominating networking researchers for ACM Fellow and other ACM awards.
-'''SIGKDD'''
-:workshop co-located at conferences
-:YES
-:stand-alone workshops
-:panels
-:YES
-:crazy idea sessions
-:we try to encourage novel ideas
-:These approaches are effective.
-:Regarding publication, frequently they have follow on special issue of journals or books.
-----
-'''SIGART'''
-:Not that I can think of.
-'''SIGARCH'''
-:We have shepherding for papers that make a valuable technical contribution but cannot be accepted as is. With shepherding a program
-committee member works with the authors to fix a problem (or problems) with the paper. Usually these problems are more an issue of
-presentation or writing, since there is not enough time to get new results (and the significance of new results cannot be known in
-advance). If the authors do not make the changes requested by the reviewers, then the shepherd can recommend rejection of the paper before the final camera-ready copy date. In order to be effective, the threat of rejection has to be real (I ultimately rejected a paper with uncooperative authors once.) Usually, about 10% of the accepted papers are conditionally accepted with shepherding. Although some people do not like shepherding, I am a strong believer in it. Given that the ISCA proceedings is more selective than many journals, I think it makes sense to have the more active editorship that shepherding can provide. It is also a way of accepting newer and bigger ideas when the language or presentation may have otherwise not initially been up to the standards of the conference.
-'''SIGCHI'''
-:DON'T FORGET THE IMPORTANCE OF VARIOUS TYPES OF INVITED SESSION. INVITED "TUTORIALS" CAN BE VALUABLE, AS CAN INVITED SPEAKERS ON TOPICS THAT CAN HELP PUSH THE COMMUNITY FORWARD IN WAYS A PAPER CANNOT.
-'''SIGIR'''
-:I think the distinguishing feature of the IR community is the extremely broad base of volunteers that it draws upon. Junior people
-get into the system quickly, via reviewing. If they're any good, they get tapped to do other things, e.g., organize workshops, chair the review process for workshops or tutorials or posters, manage the SIGIR web site, etc. This gets them into the system and gives them
-visibility, which eventually helps them move on to positions of higher responsibility. When we run elections, we are able to offer 3 strong candidates for each of the 4 elected positions. We then appoint several additional representatives to the EC to cover regions that didn't win any seats via the election. The net effect is that we have reasonably good turnover in positions of responsibility (Executive Committee, Area Coordinators, Program Chairs, General Chairs) without a real dip in quality. We don't have strict rules about turnover, but it's part of the culture of the community that there be some, to make room for others.
-:We also plan conferences further in advance than most SIGs, e.g., right now we are doing site selection for Summer 2009. This allows
-a long cycle for mentoring General Chairs, and it allows us to insert the Program Chairs for year N+1 as Areas Chairs for year N. This
-provides continuity, and gives them a chance to see what worked well and what didn't the year before.
-:Like many SIGs, our Past Chair is an unofficial member of the Executive Committee. She (in this case) sees all of the email, but
-tends to reserve her comments for the current Chair, essentially serving as a mentor for the current Chair's full term. Again, this
-provides continuity while allowing turnover.
-'''SIGPLAN'''
-:Paper submission software
-:SIGPLAN provides "supported" conference management software for our major conferences (although the PC chair is free to use other
-software if they desire). As someone who has seen some of the homegrown systems fail in various ways (e.g., the conference
-submission software lost submitted papers, the software was not configured properly to block PC submitters of papers the reviews of
-their paper resulting in the use of an "honor" system was used, etc.), I think such software is critical to the smooth running of a conference.
-:Improved latex template for conference papers.
-:To provide a more uniform look to our conference proceedings and to relieve authors the burden of struggling with a poorly designed paper template, SIGPLAN commissioned the design of a new Latex template. The template has been very well received by the SIGPLAN community. I believe that it has been used by conferences not directly affiliated with SIGPLAN.
-'''SIGCOMM'''
-:We've struggled somewhat with our efforts to reach the larger international community. We've tried a few things, including
-regional workshops (first in Latin America, then in Asia) and regional travel grants (to junior faculty in Latin America),
-with some success but by no means a slam dunk. We're trying harder to rotate our main conferences outside our usual
-geographies (e.g., SIGCOMM'07 will be in Kyoto, Japan -- our first time outside of North America and Europe).
-'''SIGKDD'''
-:We have 2-minute poster session previews, which seem to work very well.
-:KDD Cup has also been very successful
+'''SIGMOBILE'''
+:SIGMOBILE for most conferences has always done double-bind reviews.  On the whole, it works well, although in some cases, it is hard to write a paper that cites your own work.  The author has to be careful, and sometimes it is nearly impossible to cite your own work without giving away the author's identity.  We also have a lot of trouble with authors who don't follow the rules at all and just put their names on the paper.  We have to either get them to change and resubmit the paper, or directly edit the names out ourselves.
-:"I don't know what we have also tried but I do know what other communities do. I'm also involved in the Operation Research (OR) community. The KDD and Machine Learning have highly competitive refereed conferences. The number of these is proliferating and the review burden is ever increasing. In OR they have big conferences where almost anyone can participate with minimal or no refereeing. The quality is more of a function of who is organizing the sub-tracks and thus quality can be spotty but it is also somewhat predictable. So perhaps there is room for more of such conferences in computer science. That way we can have more novel ideas more rapidly without increasing the amount of reviewing.
+:I have not found (in other conferences) rebuttals to work well at all.  It always has seemed as an author that, no matter what you say in your rebuttal, the end result is the same as the original reviews and doesn't take the rebuttal into account at all.  Allowing time for rebuttals also lengthens the whole review process, requiring papers to be submitted longer before the conference.