Q3: Large Conferences

From Health of Conferences Committee

(Difference between revisions)
Revision as of 21:32, 22 February 2006
MarkDHill (Talk | contribs)

← Previous diff		 Current revision
MarkDHill (Talk | contribs)

Line 33: Line 33:
'''SIGGRAPH''' '''SIGGRAPH'''
:double blind submissions :double blind submissions
-only with external reviewers of papers. Although I have been told that the papers community know who is doing what research and the blind review process isn't always possible. Other SIGGRAPH programs have not exercised the blind process.+:only with external reviewers of papers. Although I have been told that the papers community know who is doing what research and the blind review process isn't always possible. Other SIGGRAPH programs have not exercised the blind process.
:program committee submission restrictions :program committee submission restrictions
Line 47: Line 47:
:others? :others?
-:SIGGRAPH has spent many years fine-tuning the process of content from submission to presentation. It is a never-ending process that continuously changes as the needs arise. Most of our practices work well. Because SIGGRAPH has an average of 12-18 programs each year, program reviews are done on a rotational basis (average of 3-4 per year) by the Conference Advisory Group.+:SIGGRAPH has spent many years fine-tuning the process of content from submission to presentation. It is a never-ending process that continuously changes as the needs arise. Most of our practices work well. Because SIGGRAPH has an average of 12-18 programs each year, program reviews are done on a rotational basis (average of 3-4 per year) by the Conference Advisory Group.
'''SIGCSE''' '''SIGCSE'''
-:stand-alone workshops +:double blind submissions
-in computer science education, CCSC and other groups sponsor quite a number of regional conferences. SIGCSE is in cooperation with these. Since these have a strong following, SIGCSE has not seen any reason to try to duplicate them.+:Yes
-:panels +:large program committees
-:each conference has a range of panels on new or emerging ideas.+:we encourage reasonably large program committees to include many members in meaningful ways.
 +:We do not use this approach to replace or duplicate the regular reviewers.
 + 
 +:program subcommittees
 +:program committees have members focused on various aspects of the event e.g, papers, panels, workshops, local arrangements, ...). We do not use subcommittees to subdivide the reviewing process.
 + 
 +:Do these practices seem to help or hurt promoting your field?
 +:Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance.
 + 
 + 
 +'''ICSE'''
 +:Past instances of the Int'l Symposium on the Foundations of Software Engineering (FSE) had an informal evening 'fun flames' session over beer and snacks, giving interested people a low-stress way of getting feedback on very early and immature ideas. This was quite an informal gathering that was intended more for *airing* new ideas rather than *promoting* them. This year the Int'l Symposium on Software Testing and Analysis (ISSTA) are soliciting submission for a more formal new ideas track, but it is too early to tell how successful this will be.
 + 
 +:large program committees
 +:Yes (as many as 45 on past PCs of ICSE).
 + 
 +:program subcommittees
 +:Not presently (although this was used many years ago in ICSE).
 + 
 +:Do these practices seem to help or hurt promoting your field?
 +:Past experiences with a subcommittee structure for the ICSE PC were reportedly disastrous. In addition, there is a feeling among some that scientific and scholarly standards in software engineering are not sufficiently uniform to justify author rebuttals, which because of the lack of uniformity would end up being broad advocacy statements that all authors would submit rather than narrow rebuttals of misunderstood points.
 +:From ICSE 2005 PC co-Chair: A note on managing large submissions and subcommittees. At ICSE 2005 (I was co-PC chair of the Research Track), the General Chair decided to split out the Education Track and Experience Track, which in previous years had been reviewed by the same PC as the research track. Not only had the single PC model failed to scale, but the PC members had troubles keeping the unique acceptance criteria of each track in focus (e.g., lots of Edu and Experience submissions rejected for lack of novelty). And perhaps research-PC types weren't best suited for judging those categories anyway.
 +:The result was that although the research PC was at least 10% smaller than the previous year, we didn't need a 2-stage review process and the number of reviews per PC member averaged 22.5. That was lower than the 28 reviews from 2004, which had a two-stage process. The two-stage process is problematic because of the higher level of management and interaction, and the need to compress the first reviewing phase to make time for the second phase.
 +:I'll note that this fragmentation is like establishing subcommittees, but it is not like dividing a discipline into subjects. Each track is really different. A problem with topical subcommittees in SE is that many papers are multi-topic: "An AI approach to IDE automation of refactoring to support software evolution - an empirical evaluation".
 +:I think the number of submissions has leveled off for the time-being (for us), so further efforts are not really necessary at this time.
 + 
 + 
 +'''Super Computing'''
 +:We try to have 3 to 4 reviews, with a minimum of 2. The reviews and papers are arranged in sub areas and we have a data base system to handle the reviews, ratings and scheduling. Each reviewer handles about 3-4 papers, although load balancing sometimes means more. The lead of each of our six areas my end up reading most in their area, but that is not required.
 + 
 +:program committee submission restrictions
 +:No, but we are very careful about declaring and hopefully avoiding conflicts of interest. We have enough reviewers.
 + 
 +:rebuttals (author responses)
 +:Yes we have two or three every year. This year we reconsidered and accepted two papers based on not just the author comments but also supporting notes from other senior community leaders.
 + 
 +:large program committees
 +:It has stayed about the same.
-:crazy idea sessions +:program subcommittees
-:The SIGCSE Symposium provides an opportunity for "Special Sessions" and Birds-of-a-Feather than can promote a range of "crazy" ideas.+:As I mentioned, we have used 4 to 6 focus areas in the technical review committee. In general they rate they papers in their areas and are allocated some number of sessions (we have ~20 with 3 papers per session). They then schedule the accepted papers in their sessions as a draft. Each area also proposes papers for some sessions we hold in reserve. The tech program chair(s), in collaboration with the area chairs, decide which areas get the extra sessions.
-:On balance, are these other venues effect for advancing your field? +:others?
-:What mechanisms, if any, do you use allow good papers from these venues +:Do these practices seem to help or hurt promoting your field?
-to later achieve wider dissemination?+:In general we are getting excellent response from the community. You can see the number of papers is increasing, and our acceptance rate is dropping. Our Quality Control surveys as well as word of mouth, indicate the community is happy with the program. Since SC is a general conference covering a broad range (applications, systems, architectures, High Performance computing, High performance networking, high performance storage and analytics) we often get comments that people would like to see more papers in their area of emphasis.
-:The SIGCSE Bulletin and conferences are the primary mechanisms for communication within the computer science education community -- especially at the college level. We have tried to expand this to other levels (with special emphasis on two-year colleges and high schools), within our resources. For example, we have had special conference rates for high school teachers.+

Current revision

[edit]

Question 3: PROGRAM COMMITTEES

Does your community practice:

  • double blind submissions
  • program committee submission restrictions
  • rebuttals (author responses)
  • large program committees
  • program subcommittees
  • others?

Do these practices seem to help or hurt promoting your field?


OOPSLA

double blind submissions
Not for the moment
program committee submission restrictions
No restrictions but stricter review for program committee papers
rebuttals (author responses)
No, though every now and then a complaint is registered and the Program Chair as well as the Conference Chair respond appropriately.
large program committees
Not sure what qualifies as large and I assume it is in relationship to the number of submitted papers. For OOPSLA the submission usually range from 160-190 papers. Program Committee ranges from 22 - 28. I will leave it up to you to determine if this is a large committee or not.
program subcommittees
Not officially, but the assumption is that every PC member will reply on others that assist him/her in reviewing the paper assigned to them. However, ultimately the members of the PC are responsible. (which it the common practice). However, with some of the introduced new categories of papers (such as Essays and the selected Onward! subcommittees are formed and are responsible for the selection of their papers.
Do these practices seem to help or hurt promoting your field?
Some of these are new and the impact is not completely determined, however, we noticed a very positive impact for introducing the Onward! track.


SIGGRAPH

double blind submissions
only with external reviewers of papers. Although I have been told that the papers community know who is doing what research and the blind review process isn't always possible. Other SIGGRAPH programs have not exercised the blind process.
program committee submission restrictions
SIGGRAPH allows for committee members to submit to any program. However, there is a process mechanism for each that precludes review/participation in discussion/voting on the part of the committee member for their submission.
There is no rebuttal process in any SIGGRAPH program that alters the decision of the committee.
large program committees
SIGGRAPH supports whatever program committee size is necessary in order to meet the needs of each program's submissions and implementation of the presentations at the conference.
program subcommittees
SIGGRAPH programs requiring a lot of on site preparation/coordination have subcommittees that assist in these areas.
others?
SIGGRAPH has spent many years fine-tuning the process of content from submission to presentation. It is a never-ending process that continuously changes as the needs arise. Most of our practices work well. Because SIGGRAPH has an average of 12-18 programs each year, program reviews are done on a rotational basis (average of 3-4 per year) by the Conference Advisory Group.


SIGCSE

double blind submissions
Yes
large program committees
we encourage reasonably large program committees to include many members in meaningful ways.
We do not use this approach to replace or duplicate the regular reviewers.
program subcommittees
program committees have members focused on various aspects of the event e.g, papers, panels, workshops, local arrangements, ...). We do not use subcommittees to subdivide the reviewing process.
Do these practices seem to help or hurt promoting your field?
Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance.


ICSE

Past instances of the Int'l Symposium on the Foundations of Software Engineering (FSE) had an informal evening 'fun flames' session over beer and snacks, giving interested people a low-stress way of getting feedback on very early and immature ideas. This was quite an informal gathering that was intended more for *airing* new ideas rather than *promoting* them. This year the Int'l Symposium on Software Testing and Analysis (ISSTA) are soliciting submission for a more formal new ideas track, but it is too early to tell how successful this will be.
large program committees
Yes (as many as 45 on past PCs of ICSE).
program subcommittees
Not presently (although this was used many years ago in ICSE).
Do these practices seem to help or hurt promoting your field?
Past experiences with a subcommittee structure for the ICSE PC were reportedly disastrous. In addition, there is a feeling among some that scientific and scholarly standards in software engineering are not sufficiently uniform to justify author rebuttals, which because of the lack of uniformity would end up being broad advocacy statements that all authors would submit rather than narrow rebuttals of misunderstood points.
From ICSE 2005 PC co-Chair: A note on managing large submissions and subcommittees. At ICSE 2005 (I was co-PC chair of the Research Track), the General Chair decided to split out the Education Track and Experience Track, which in previous years had been reviewed by the same PC as the research track. Not only had the single PC model failed to scale, but the PC members had troubles keeping the unique acceptance criteria of each track in focus (e.g., lots of Edu and Experience submissions rejected for lack of novelty). And perhaps research-PC types weren't best suited for judging those categories anyway.
The result was that although the research PC was at least 10% smaller than the previous year, we didn't need a 2-stage review process and the number of reviews per PC member averaged 22.5. That was lower than the 28 reviews from 2004, which had a two-stage process. The two-stage process is problematic because of the higher level of management and interaction, and the need to compress the first reviewing phase to make time for the second phase.
I'll note that this fragmentation is like establishing subcommittees, but it is not like dividing a discipline into subjects. Each track is really different. A problem with topical subcommittees in SE is that many papers are multi-topic: "An AI approach to IDE automation of refactoring to support software evolution - an empirical evaluation".
I think the number of submissions has leveled off for the time-being (for us), so further efforts are not really necessary at this time.


Super Computing

We try to have 3 to 4 reviews, with a minimum of 2. The reviews and papers are arranged in sub areas and we have a data base system to handle the reviews, ratings and scheduling. Each reviewer handles about 3-4 papers, although load balancing sometimes means more. The lead of each of our six areas my end up reading most in their area, but that is not required.
program committee submission restrictions
No, but we are very careful about declaring and hopefully avoiding conflicts of interest. We have enough reviewers.
rebuttals (author responses)
Yes we have two or three every year. This year we reconsidered and accepted two papers based on not just the author comments but also supporting notes from other senior community leaders.
large program committees
It has stayed about the same.
program subcommittees
As I mentioned, we have used 4 to 6 focus areas in the technical review committee. In general they rate they papers in their areas and are allocated some number of sessions (we have ~20 with 3 papers per session). They then schedule the accepted papers in their sessions as a draft. Each area also proposes papers for some sessions we hold in reserve. The tech program chair(s), in collaboration with the area chairs, decide which areas get the extra sessions.
others?
Do these practices seem to help or hurt promoting your field?
In general we are getting excellent response from the community. You can see the number of papers is increasing, and our acceptance rate is dropping. Our Quality Control surveys as well as word of mouth, indicate the community is happy with the program. Since SC is a general conference covering a broad range (applications, systems, architectures, High Performance computing, High performance networking, high performance storage and analytics) we often get comments that people would like to see more papers in their area of emphasis.
Retrieved from "http://wiki.acm.org/healthcc/index.php?title=Q3:_Large_Conferences"