Talk:Hierarchical Program Committees
From Health of Conferences Committee
(Difference between revisions)
| Revision as of 20:16, 28 February 2006 MarkDHill (Talk | contribs) ← Previous diff |
Current revision 199.222.74.48 (Talk | contribs) |
||
| Line 6: | Line 6: | ||
| == Starting Comments == | == Starting Comments == | ||
| - | As some conference get more papers, they add more members to the program committee. At some point this ''scaling'' does work and other option should be considered. | + | As some conference get more papers, they add more members to the program committee. At some point this ''scaling'' does work and other options should be considered. |
| Line 19: | Line 19: | ||
| '''SIGIR''' | '''SIGIR''' | ||
| - | :I think the distinguishing feature of the IR community is the extremely broad base of volunteers that it draws upon. Junior people get into the system quickly, via reviewing. If they're any good, they get tapped to do other things, e.g., organize workshops, chair the review process for workshops or tutorials or posters, manage the SIGIR web site, etc. This gets them into the system and gives them visibility, which eventually helps them move on to positions of higher responsibility. When we run elections, we are able to offer 3 strong candidates for each of the 4 elected positions. We then appoint several additional representatives to the EC to cover regions that didn't win any seats via the election. The net effect is that we have reasonably good turnover in positions of responsibility (Executive Committee, Area Coordinators, Program Chairs, General Chairs) without a real dip in quality. We don't have strict rules about turnover, but it's part of the culture of the community that there be some, to make room for others. | + | :I think the distinguishing feature of the IR community is the extremely broad base of volunteers that it draws upon. Junior people get into the system quickly, via reviewing. If they're any good, they get tapped to do other things, e.g., organize workshops, chair the review process for workshops or tutorials or posters, manage the SIGIR web site, etc. This gets them into the system and gives them visibility, which eventually helps them move on to positions of higher responsibility. When we run elections, we are able to offer 3 strong candidates for each of the 4 elected positions. We then appoint several additional representatives to the EC to cover regions that didn't win any seats via the election. The net effect is that we have reasonably good turnover in positions of responsibility (Executive Committee, Area Coordinators, Program Chairs, General Chairs) without a real dip in quality. We don't have strict rules about turnover, but it's part of the culture of the community that there be some, to make room for others. |
| + | '''SIGecom ''' | ||
| + | |||
| + | :We increased the program committee size from 18 last year to 41 this year. We also allowed PC members to delegate the reviews to external reviewers. Part of the reason was to reduce the reviewer load. Part of the reason was to signal the broad scope of the conference and to attract submissions from different communities. | ||
| + | |||
| + | '''SIGCSE ''' | ||
| + | :...large program committees - we encourage reasonably large program committees to include many members in meaningful ways. We do not use this approach to replace or duplicate the regular reviewers. | ||
| + | :program subcommittees - program committees have members focused on various aspects of the event (e.g, papers, panels, workshops, local arrangements, ...). We do not use subcommittees to subdivide the reviewing process. | ||
| + | :Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance. | ||
| + | |||
| + | '''SIGDA''' | ||
| + | :large program committees – Yes | ||
| + | :program subcommittees - Yes, in all (I think) conferences/symposia we have subcomms. | ||
| + | :Others - ICCAD and DAC have rotation rules of TPC members (3 yrs. in, 3yrs. out), except for subcomm chairs who can stay an addl. year. ISLPED doesn't have it, but it is done at the discretion of the TPC chairs. I think such rules are good. | ||
| + | :There were times in the past where people had spent 10 years or more on the same TPC. This should be unacceptable. | ||
| + | |||
| + | '''SIGACCESS''' | ||
| + | :The PC does not meet in person, but does all reviewing through an online system. For '05, a three-step process was used that seemed to work well. That is, each paper got three reviews. When this cycle was completed, all papers were opened up to the entire committee for online voting on which to accept and which to reject. Reviewers could read manuscripts they hadn't previously read and also had available the reviews of the other committee members. This seemed to get consensus. Finally, after acceptances were decided, there was voting for the Best Paper awards. | ||
| + | |||
| + | '''SIGMICRO''' | ||
| + | :The discussion order of papers at the PC meeting is very important. Also we have experimented with having the program committee "grade" papers before the PC meeting. This sets up a review order for the papers. Unfortunately, the paper grade scores can disenfranchise external (non-PC member) reviewers who may actually, in some cases, know the work in question better than the PC members. This past year, we have started to set the discussion order as the max(PC grades, review score average) so that no PC grading can 'sink' a paper lower in the order, but it can 'raise up' a paper that may have gotten worse than expected scores. This seems to be working well. | ||
| + | |||
| + | '''OOPSLA''' | ||
| + | :Not officially, but the assumption is that every PC member will reply on others that assist him/her in reviewing the paper assigned to them. However, ultimately the members of the PC are responsible (which it the common practice). However, with some of the newly introduced categories of papers (such as Essays and the selected Onward! papers) subcommittees are formed that are responsible for the selection of these papers. | ||
| + | |||
| + | '''SIGGRAPH''' | ||
| + | :All papers submitted to SIGGRAPH (except for a tiny number of desk rejects at the Chair's discretion) are reviewed by one primary, one secondary, and at least three tertiary reviewers. The primary and secondary reviewers comprise the Papers Committee. Tertiary reviewers are external to the Papers Committee and are selected for each paper by the primary and secondary reviewers. | ||
| + | |||
| + | '''ICSE''' | ||
| + | :Not presently, although a hierarchical scheme was used some years ago that did not go well -- see the survey for full details. | ||
| + | |||
| == Discussion Begins == | == Discussion Begins == | ||
| + | |||
| + | == SAC == | ||
| + | |||
| + | SIGAPP's Symposium on Applied Computing is organized as a set of tracks, each with its own program committee and program chair. The track program chairs make recommendations to the general program chairs who decide the final paper allocations per track. This system is generally working well although there is a tendency for larger tracks to split into multiple smaller tracks. --Barrett Bryant | ||
Current revision
To add your comment to this discussion, please click the + sign tab above. Like an email message, you can then contribute:
- a subject (use subject Re: FOO to continue a discussion of FOO)
- message body
- (optionally) your name.
[edit]
Starting Comments
As some conference get more papers, they add more members to the program committee. At some point this scaling does work and other options should be considered.
SIGKDD
- We did increase program committee size. But management and evaluation of so many papers and 1000+ reviews was difficult because it is harder to ensure that the papers are appropriately assigned and more difficult to ensure quality and consistancy of all reviews with a larger program committee.
- We recommended that this years Program Chairs use a more hierarchical programs stucture so that there can be better oversight of the reviewing and discussion processes.
SIGART
- Increased the size of the PC. Used a 2 layer approach, with Senior Program Committee (SPC) members supervising the work of PC members and coordinating the discussion among the reviewers of each paper to attempt to reach consensus.
SIGIR
- I think the distinguishing feature of the IR community is the extremely broad base of volunteers that it draws upon. Junior people get into the system quickly, via reviewing. If they're any good, they get tapped to do other things, e.g., organize workshops, chair the review process for workshops or tutorials or posters, manage the SIGIR web site, etc. This gets them into the system and gives them visibility, which eventually helps them move on to positions of higher responsibility. When we run elections, we are able to offer 3 strong candidates for each of the 4 elected positions. We then appoint several additional representatives to the EC to cover regions that didn't win any seats via the election. The net effect is that we have reasonably good turnover in positions of responsibility (Executive Committee, Area Coordinators, Program Chairs, General Chairs) without a real dip in quality. We don't have strict rules about turnover, but it's part of the culture of the community that there be some, to make room for others.
SIGecom
- We increased the program committee size from 18 last year to 41 this year. We also allowed PC members to delegate the reviews to external reviewers. Part of the reason was to reduce the reviewer load. Part of the reason was to signal the broad scope of the conference and to attract submissions from different communities.
SIGCSE
- ...large program committees - we encourage reasonably large program committees to include many members in meaningful ways. We do not use this approach to replace or duplicate the regular reviewers.
- program subcommittees - program committees have members focused on various aspects of the event (e.g, papers, panels, workshops, local arrangements, ...). We do not use subcommittees to subdivide the reviewing process.
- Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance.
SIGDA
- large program committees – Yes
- program subcommittees - Yes, in all (I think) conferences/symposia we have subcomms.
- Others - ICCAD and DAC have rotation rules of TPC members (3 yrs. in, 3yrs. out), except for subcomm chairs who can stay an addl. year. ISLPED doesn't have it, but it is done at the discretion of the TPC chairs. I think such rules are good.
- There were times in the past where people had spent 10 years or more on the same TPC. This should be unacceptable.
SIGACCESS
- The PC does not meet in person, but does all reviewing through an online system. For '05, a three-step process was used that seemed to work well. That is, each paper got three reviews. When this cycle was completed, all papers were opened up to the entire committee for online voting on which to accept and which to reject. Reviewers could read manuscripts they hadn't previously read and also had available the reviews of the other committee members. This seemed to get consensus. Finally, after acceptances were decided, there was voting for the Best Paper awards.
SIGMICRO
- The discussion order of papers at the PC meeting is very important. Also we have experimented with having the program committee "grade" papers before the PC meeting. This sets up a review order for the papers. Unfortunately, the paper grade scores can disenfranchise external (non-PC member) reviewers who may actually, in some cases, know the work in question better than the PC members. This past year, we have started to set the discussion order as the max(PC grades, review score average) so that no PC grading can 'sink' a paper lower in the order, but it can 'raise up' a paper that may have gotten worse than expected scores. This seems to be working well.
OOPSLA
- Not officially, but the assumption is that every PC member will reply on others that assist him/her in reviewing the paper assigned to them. However, ultimately the members of the PC are responsible (which it the common practice). However, with some of the newly introduced categories of papers (such as Essays and the selected Onward! papers) subcommittees are formed that are responsible for the selection of these papers.
SIGGRAPH
- All papers submitted to SIGGRAPH (except for a tiny number of desk rejects at the Chair's discretion) are reviewed by one primary, one secondary, and at least three tertiary reviewers. The primary and secondary reviewers comprise the Papers Committee. Tertiary reviewers are external to the Papers Committee and are selected for each paper by the primary and secondary reviewers.
ICSE
- Not presently, although a hierarchical scheme was used some years ago that did not go well -- see the survey for full details.
[edit]
Discussion Begins
[edit]
SAC
SIGAPP's Symposium on Applied Computing is organized as a set of tracks, each with its own program committee and program chair. The track program chairs make recommendations to the general program chairs who decide the final paper allocations per track. This system is generally working well although there is a tendency for larger tracks to split into multiple smaller tracks. --Barrett Bryant
