Q1: Large Conferences

From Health of Conferences Committee

(Difference between revisions)

Revision as of 20:34, 22 February 2006

Question 1: REVIEWER LOAD.

Has your community recently adopted new practices to deal with growing reviewer load, such as:

tracking reviews of rejected papers from conference to conference as is done in journal reviewing
increasing program committee size
charging a review fee
others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

tracking reviews of rejected papers from conference to conference as is done in journal reviewing OOPSLA: not formally

increasing program committee size OOPSLA: Program committee didn't increase dramatically though it was increased by few from 5 years ago

tracking reviews of rejected papers from conference to conference as is done in journal reviewing ICSE: There has been some talk of doing this, but it hasn't been implemented yet.

increasing program committee size ICSE: Most PCs are at a maximum feasible size now.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. ICSE: For the 2004 Int'l Conference on Software Engineering (ICSE), we used a two-phase review process with a large PC. In the first phase each submission was reviewed by just two PC members (or three in the case of submissions from PC members), with clearly low-scoring papers summarily rejected at the end of the first phase. In the second phase, the surviving papers were each assigned an additional reviewer. The highest-scoring papers coming out of the second phase were discussed at the PC meeting, and the rest were summarily rejected. This seemed to work quite well, as it reduced the amount of effort spent reviewing low-quality submissions, and reduced the total number of papers each PC member reviewed as compared to the traditional one-phase process in which each submission receives three reviews (or four in the case of submissions from PC members).

For ICSE 2006, the PC co-chairs used a smaller PC and a large population of external reviewers. I don't have first-hand experience with this process, but I believe each paper was reviewed initially by two PC members plus an external reviewer, and then high-scoring papers went through a second round of reviewing by an additional reviewer. From second-hand accounts I have received, this process seems to have been unsuccessful, for a number of reasons. First, the external reviewers tended to be quite negative (and several were reportedly from outside the community). Furthermore, because of the large percentage of low-quality submissions, it was felt that exposing so many people to the pool of submissions in this way damaged the reputation of ICSE. Second, the time budgeted for the two parts of the process didn't align with the reviewing load for the two parts, with the first part involving a very heavy load and the second part involving a very light load.

tracking reviews of rejected papers from conference to conference as is done in journal reviewing SIGGRAPH: I believe that our papers committees do keep track of rejected papers, but it is unclear of this process is chronicled in any fashion from year-to-year. There has been a mechanism within the papers community to provide feedback. Other SIGGRAPH programs (courses, panels, educators) that also have a reviewer/jury process have begun using our online review system to track responses of all reviewers that can also be seen by all. These responses are then used by the program chair to resond to submittors with constructive feedback as to why their submissions were not accepted.

increasing program committee size

SIGGRAPH: In order to handle the increasing number of submissions of several SIGGRAPH programs, the committee size has been allowed to increase where necessary.  This is of concern, however, because of the related increases in costs for travel and contributor recognition.

charging a review fee SIGGRAPH: Charge a fee has been discussed, but has met with an allergic reaction for papers submissions. However, there is growing interest with the Computer Animation Festival (CAF) and the Art Gallery. Even with these programs, an actual implemenation process is probably two years away.

others? SIGGRAPH: I believe that there was a revised review process put into place by the Papers committee in recent years, refining/increaseing the tiered review process. Joe, I think you are probably better equipped to detail this.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. SIGGRAPH: Again, I don't think I am clear enough on the papers community process to answer this one. Because of the growing concern of increased submissions in the CAF and Art Gallery, it is extremely likely that at least an experimental process will be implemented because of the growing challenges of our current practices. The success of these experiments could result in moving more programs to a fee process.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. DAC: Yes, we have had to work on creative ways to reduce reviewer load as our number of papers has grown. We have increased committee size, and asked each committee member to only review a subset of papers. This has worked fine.

SIGCSE has not engaged in any of the above.

 * reviewing for both the SIGCSE Symposium and ITiCSE Conferences are double blind
 * we maintain a database including about 779 reviewers for the Symposium and 584 reviewers for the ITiCSE conference.
 * each paper is sent to about 6 reviewers, and each reviewer receives 2-4 papers.
 * a minimum of 4 reviews is required for each paper, and the Program

Co-Chairs assign last-minute reviewers if the minimum number of reviews has not been met.

   Most papers have 5 or 6 reviews.
 * SIGCSE members typically view their involvement in reviewing as a contribution to the community, and they are enthusiastic in this work.
 * After the reviewing period and after decisions are made, authors may view the reviews of their papers (although the reviews are anonymous), and each reviewer may view the other reviews of the papers they considered (again anonymously).

Question 2 NON-INCREMENTAL. Has your community recently adopted new practices to promote non-incremental new ideas? OOPSLA: If I understand it correctly, then the answer is yes. OOPSLA has been very active in introducing new venues for a long time (Educators Symposium, Doctoral Symposium, Practitioners Reports, Demos, of course in additon to the traditional other venues such as workshops, and tutorials. I apologize if I didn't understand what is meant by non-incremental.

big ideas sessions

OOPSLA: Yes, it has been a tradition for OOPSLA to host specific tracks devoted to specific issue with related tutorials, workshops and panels, and some times with its own invited speaker (e.g., THe OnWard! track)

more papers OOPSLA: Introduced recently different categories to the papers track (e.g., Essays, selected papers from the OnWard! track that pass the same technical papers review criteria)

shorter papers OOPSLA: Yes, we introduced Practioner Reports which doesn't have the same acceptance criteria as the technical papers but they are not citable either. They server the practioners community rather than academia though we see more and more interst from Academia in what the practioners are doing.

deemphasizing detailed evaluation OOPSLA: Only for non technical papers

SIGGRAPH: SIGGRAPH already has a special sessions program where broader ideas/interest areas are presented each year. It is a very successful program that draws anywhere from several hundred to a couple of thousand attendees. There is also the Sketches program that may not focus on "big ideas" but on new and innovative ideas that are on the horizon of interest amongst attendees.

more papers SIGGRAPH: While SIGGRAPH is concerned about the growing number of submissions, there is also concern for bringing in appropriate content with the changes in research areas. There is increasing pursuit of these new and less represented areas so that the scope of content provided at the conference is comprehensive.

shorter papers SIGGRAPH: I am unclear about the papers committee's concern about papers length. However, the Sketches program seems to be filling a need for shorter papers presentation.

deemphasizing detailed evaluation SIGGRAPH: Again, I think that this depends on the program chair and the committee process and it changes from year to year.

others? SIGCSE: I believe that we always have looked for a balance of papers on a variety of subjects.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. SIGCSE: This is not done formally, although each committee looks for balance and interesting new thoughts. Also, we have added several keynotes to our conferences to solicit new or different perspectives.

double blind submissions SIGCSE: Yes

large program committees SIGCSE: we encourage reasonably large program committees to include many members in meaningful ways. We do not use this approach to replace or duplicate the regular reviewers.

program subcommittees SIGCSE: program committees have members focused on various aspects of the event e.g, papers, panels, workshops, local arrangements, ...). We do not use subcommittees to subdivide the reviewing process.

Do these practices seem to help or hurt promoting your field? SIGCSE: Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance.

For OOPSLA Practitioner Reports and Onward! proved very valuable to our community.

double blind submissions OOPSLA: Not for the moment

program committee submission restrictions OOPSLA: No restrictions but stricter review for program committee papers

rebuttals (author responses) OOPSLA: No, though every now and then a complaint is registered and the Program Chair as well as the Conference Chair respond appropriately.

large program committees OOPSLA: Not sure what qualifies as large and I assume it is in relationship to the number of submitted papers. For OOPSLA the submission usually range from 160-190 papers. Program Committee ranges from 22 - 28. I will leave it up to you to determine if this is a large committee or not.

program subcommittees OOPSLA: Not officially, but the assumption is that every PC member will reply on others that assist him/her in reviewing the paper assigned to them. However, ultimately the members of the PC are responsible. (which it the common practice). However, with some of the introduced new categories of papers (such as Essays and the selected Onward! subcommittees are formed and are responsible for the selection of their papers.

Do these practices seem to help or hurt promoting your field? OOPSLA: Some of these are new and the impact is not completely determined, however, we noticed a very positive impact for introducing the Onward! track.

ICSE: Past instances of the Int'l Symposium on the Foundations of Software Engineering (FSE) had an informal evening 'fun flames' session over beer and snacks, giving interested people a low-stress way of getting feedback on very early and immature ideas. This was quite an informal gathering that was intended more for *airing* new ideas rather than *promoting* them. This year the Int'l Symposium on Software Testing and Analysis (ISSTA) are soliciting submission for a more formal new ideas track, but it is too early to tell how successful this will be.

large program committees ICSE: Yes (as many as 45 on past PCs of ICSE).

program subcommittees ICSE: Not presently (although this was used many years ago in ICSE).

Do these practices seem to help or hurt promoting your field? ICSE: Past experiences with a subcommittee structure for the ICSE PC were reportedly disastrous. In addition, there is a feeling among some that scientific and scholarly standards in software engineering are not sufficiently uniform to justify author rebuttals, which because of the lack of uniformity would end up being broad advocacy statements that all authors would submit rather than narrow rebuttals of misunderstood points. From ICSE 2005 PC co-Chair: A note on managing large submissions and subcommittees. At ICSE 2005 (I was co-PC chair of the Research Track), the General Chair decided to split out the Education Track and Experience Track, which in previous years had been reviewed by the same PC as the research track. Not only had the single PC model failed to scale, but the PC members had troubles keeping the unique acceptance criteria of each track in focus (e.g., lots of Edu and Experience submissions rejected for lack of novelty). And perhaps research-PC types weren't best suited for judging those categories anyway. The result was that although the research PC was at least 10% smaller than the previous year, we didn't need a 2-stage review process and the number of reviews per PC member averaged 22.5. That was lower than the 28 reviews from 2004, which had a two-stage process. The two-stage process is problematic because of the higher level of management and interaction, and the need to compress the first reviewing phase to make time for the second phase. I'll note that this fragmentation is like establishing subcommittees, but it is not like dividing a discipline into subjects. Each track is really different. A problem with topical subcommittees in SE is that many papers are multi-topic: "An AI approach to IDE automation of refactoring to support software evolution - an empirical evaluation". ;) I think the number of submissions has leveled off for the time-being (for us), so further efforts are not really necessary at this time.

DAC: We do have short papers, and we do try to keep the submission rate above 20%. However, we still require rigorous review of all papers.

Do these practices seem to help or hurt promoting your field? DAC: We practice all of the above, and feel that they are all necessary. They seem to help.

double blind submissions SIGGRAPH: only with external reviewers of papers. Although I have been told that the papers community know who is doing what research and the blind review process isn't always possible. Other SIGGRAPH programs have not exercised the blind process.

program committee submission restrictions SIGGRAPH: SIGGRAPH allows for committee members to submit to any program. However, there is a process mechanism for each that precludes review/participation in discussion/voting on the part of the committee member for their submission.

rebuttals (author responses) SIGGRAPH: There is no rebuttal process in any SIGGRAPH program that alters the decision of the committee.

large program committees SIGGRAPH supports whatever program committee size is necessary in order to meet the needs of each program's submissions and implementation of the presentations at the conference.

program subcommittees SIGGRAPH programs requiring a lot of on site preparation/coordination have subcommittees that assist in these areas.

others? SIGGRAPH has spent many years fine-tuning the process of content from submission to presentation. It is a never-ending process that continuously changes as the needs arise. Most of our practices work well. Because SIGGRAPH has an average of 12-18 programs each year, program reviews are done on a rotational basis (average of 3-4 per year) by the Conference Advisory Group.

Question 3 WORKSHOPS, ETC. Does your community provide venue for work not mature enough for your major conferences, such as:

workshop co-located at conferences

OOPSLA: Workshops have been always an impportant part of OOPSLA. For the last 4 years OOPSLA has been co-located with other conferences.

stand-alone workshops OOPSLA: We didn't call them Workshops but we have tracks such as the Educators' and Doctoral Symposia that are stand alone. For few years OOPSLA sponsored mid-year series of Workshop as a separate meetings from OOPSLA but achieved moderate success and the idea died when its champion stepped down.

panels OOPSLA: Always as part of the core OOPSLA conference

crazy idea sessions OOPSLA: Last year we introduced the Lightning Talks (strictly 5 minutes talks for any idea an author would like to share with the audience). It is too early to assess its success, but the concept was borrowed from other conferences where such talks were a great success.

On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination? OOPSLA: I believe OOPSLA was instrumental in introducing many of such venues that other conference adopted and such venues proved very valuable for the community. OOPSLA publishes a companion to the proceedings which includes the workshops conclusion, panels, etc. Also, Onward! presentations have been published in separate publications.

stand-alone workshops SIGCSE: in computer science education, CCSC and other groups sponsor quite a number of regional conferences. SIGCSE is in cooperation with these. Since these have a strong following, SIGCSE has not seen any reason to try to duplicate them.

panels SIGCSE: each conference has a range of panels on new or emerging ideas.

crazy idea sessions The SIGCSE Symposium provides an opportunity for "Special Sessions" and Birds-of-a-Feather than can promote a range of "crazy" ideas.

On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination? The SIGCSE Bulletin and conferences are the primary mechanisms for communication within the computer science education community -- especially at the college level. We have tried to expand this to other levels (with special emphasis on two-year colleges and high schools), within our resources. For example, we have had special conference rates for high school teachers.

crazy idea sessions SIGGRAPH does all of these things annually as well as implement new programs (panels is a permanent program) . For example, Web Graphics was funded for a three-year period to assess the need for permanent implementation as a separate program or blending this content into other existing programs. Other experimental programs have included SIGGRAPH TV, Online Services, Community Outreach, SIGKids, to name few.

On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination? SIGGRAPH: Most of SIGGRAPH's experimental programs have been successful and have been necessary in promoting the different areas of visual fields and education. The technology and content of some of these programs live on in the more established programs after the end of the experimental runs.

crazy idea sessions DAC: We have all of the above except crazy idea sessions. They all work well.

On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination? DAC: We also have a student design contest, and some of the top ones get in our technical program. This gives them a wider dissemination.

On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination? ICSE: Crazy ideas sessions were discussed above under NON-INCREMENTAL. There are numerous co-located and stand-alone workshops in software engineering. Regrettably many of these are becoming more like conferences, with solicitation and publication of conference-length papers, and serial presentation of conference-length talks. Thus it is difficult for papers from these workshops to receive any wider dissemination than they receive as papers in the workshop proceedings, since any attempt to submit similar ideas and text to a conference would be viewed as a re-submission of published work.

Question 5 CATCH-ALL. Are there other approaches your community has tried or abandoned that the rest of us can learn from? OOPSLA: We established a wiki for the conference committee to use while planning for the conference as well as a wiki to use during the conference by the attendees. We are also investigating new ideas in the next couple of years to improve some of the traditional venues such as introducing classes for certification purposes as part of the tutorial program and others but nothing has been implemented yet so it is not appropriate to talk about them yet.

SIGGRAPH: Yes. When an experimental program is offered for multiple years, don't be afraid to let it go if it doesn't continue to fill a substantial need. Sometimes SIGGRAPH has continued some new programs beyond their useful life because of the dedication, commitment, and passion for it by hardworking volunteers. It is important to retain an objective point of view.

DAC: We are in the process of trying theme days -- there is one day of a conference dedicated to it, and a different theme is chosen each year. People seem generally very interested but it is hard to get a hold in areas that are not traditional for the conference. A must is getting some people on the program committee that are in that area.

Retrieved from "http://wiki.acm.org/healthcc/index.php?title=Q1:_Large_Conferences"

Q1: Large Conferences

From Health of Conferences Committee

Revision as of 20:34, 22 February 2006

Question 1: REVIEWER LOAD.

Views

Personal tools

Navigation

Search

Toolbox

 For each practice you are using, what is your view of how well it is working within your community?  Please comment on the merit of the other strategies as applies to your community.
+tracking reviews of rejected papers from conference to conference as
+is done in journal reviewing
+OOPSLA: not formally
-'''ICSE: There has been some talk of doing this, but it hasn't been implemented yet.
+increasing program committee size
+OOPSLA: Program committee didn't increase dramatically though it was increased by few from 5 years ago
+------------------------
+tracking reviews of rejected papers from conference to conference as
+is done in journal reviewing
+ICSE: There has been some talk of doing this, but it hasn't been implemented yet.
-'''SIGGRAPH: I believe that our papers committees do keep track of rejected papers, but it is unclear of this process is chronicled in any fashion from year-to-year.  There has been a mechanism within the papers community to provide feedback.  Other SIGGRAPH programs (courses, panels, educators) that also have a reviewer/jury process have begun using our online review system to track responses of all reviewers that can also be seen by all.  These responses are then used by the program chair to resond to submittors with constructive feedback as to why their submissions were not accepted.
+increasing program committee size
+ICSE: Most PCs are at a maximum feasible size now.
-* increasing program committee size
+For each practice you are using, what is your view of how well it is
-'''OOPSLA: Program committee didn't increase dramatically though it was increased by few from 5 years ago
+working within your community?  Please comment on the merit of the other
+strategies as applies to your community.
+ICSE: For the 2004 Int'l Conference on Software Engineering (ICSE), we used  a two-phase review process with a large PC.  In the first phase each  submission was reviewed by just two PC members (or three in the case  of submissions from PC members), with clearly low-scoring papers  summarily rejected at the end of the first phase.  In the second  phase, the surviving papers were each assigned an additional  reviewer.  The highest-scoring papers coming out of the second phase  were discussed at the PC meeting, and the rest were summarily  rejected.  This seemed to work quite well, as it reduced the amount  of effort spent reviewing low-quality submissions, and reduced the  total number of papers each PC member reviewed as compared to the  traditional one-phase process in which each submission receives three  reviews (or four in the case of submissions from PC members).
-ICSE: Most PCs are at a maximum feasible size now.
+For ICSE 2006, the PC co-chairs used a smaller PC and a large population of external reviewers.  I don't have first-hand experience with this process, but I believe each paper was reviewed initially by two PC members plus an external reviewer, and then high-scoring  papers went through a second round of reviewing by an additional  reviewer.  From second-hand accounts I have received, this process seems to have been unsuccessful, for a number of reasons.  First, the external reviewers tended to be quite negative (and several were reportedly from outside the community).  Furthermore, because of the large percentage of low-quality submissions, it was felt that exposing so many people to the pool of submissions in this way damaged the reputation of ICSE.  Second, the time budgeted for the  two parts of the process didn't align with the reviewing load for the  two parts, with the first part involving a very heavy load and the  second part involving a very light load.
-SIGGRAPH: In order to handle the increasing number of submissions of several SIGGRAPH programs, the committee size has been allowed to increase where necessary.  This is of concern, however, because of the related increases in costs for travel and contributor recognition.
+----------------------------
+tracking reviews of rejected papers from conference to conference as
+is done in journal reviewing
+SIGGRAPH: I believe that our papers committees do keep track of rejected papers, but it is unclear of this process is chronicled in any fashion from year-to-year.  There has been a mechanism within the papers community to provide feedback.  Other SIGGRAPH programs (courses, panels, educators) that also have a reviewer/jury process have begun using our online review system to track responses of all reviewers that can also be seen by all.  These responses are then used by the program chair to resond to submittors with constructive feedback as to why their submissions were not accepted.
-* charging a review fee
+increasing program committee size
+ SIGGRAPH: In order to handle the increasing number of submissions of several SIGGRAPH programs, the committee size has been allowed to increase where necessary.  This is of concern, however, because of the related increases in costs for travel and contributor recognition.
+charging a review fee
 SIGGRAPH: Charge a fee has been discussed, but has met with an allergic reaction for papers submissions.  However, there is growing interest with the Computer Animation Festival (CAF) and the Art Gallery.  Even with these programs, an actual implemenation process is probably two years away.
-* others?
+others?
-SIGCSE has not engaged in any of the above.
 SIGGRAPH: I believe that there was a revised review process put into place by the Papers committee in recent years, refining/increaseing the tiered review process.  Joe, I think you are probably better equipped to detail this.
+For each practice you are using, what is your view of how well it is
+working within your community?  Please comment on the merit of the other
+strategies as applies to your community.
+SIGGRAPH: Again, I don't think I am clear enough on the papers community process to answer this one.  Because of the growing concern of increased submissions in the CAF and Art Gallery, it is extremely likely that at least an experimental process will be implemented because of the growing challenges of our current practices.  The success of these experiments could result in moving more programs to a fee process.
+For each practice you are using, what is your view of how well it is
+working within your community?  Please comment on the merit of the other
+strategies as applies to your community.
+DAC: Yes, we have had to work on creative ways to reduce reviewer load as our number of papers has grown.  We have increased committee size, and asked each committee member to only review a subset of papers.  This has worked fine.
+--------------
+SIGCSE has not engaged in any of the above.
 For each practice you are using, what is your view of how well it is
   * After the reviewing period and after decisions are made, authors may view the reviews of their papers (although the reviews are anonymous), and each reviewer may view the other reviews of the papers they considered (again anonymously).
-ICSE: For the 2004 Int'l Conference on Software Engineering (ICSE), we used  a two-phase review process with a large PC.  In the first phase each  submission was reviewed by just two PC members (or three in the case  of submissions from PC members), with clearly low-scoring papers  summarily rejected at the end of the first phase.  In the second  phase, the surviving papers were each assigned an additional  reviewer.  The highest-scoring papers coming out of the second phase  were discussed at the PC meeting, and the rest were summarily  rejected.  This seemed to work quite well, as it reduced the amount  of effort spent reviewing low-quality submissions, and reduced the  total number of papers each PC member reviewed as compared to the  traditional one-phase process in which each submission receives three  reviews (or four in the case of submissions from PC members).
-For ICSE 2006, the PC co-chairs used a smaller PC and a large population of external reviewers.  I don't have first-hand experience with this process, but I believe each paper was reviewed initially by two PC members plus an external reviewer, and then high-scoring  papers went through a second round of reviewing by an additional  reviewer.  From second-hand accounts I have received, this process seems to have been unsuccessful, for a number of reasons.  First, the external reviewers tended to be quite negative (and several were reportedly from outside the community).  Furthermore, because of the large percentage of low-quality submissions, it was felt that exposing so many people to the pool of submissions in this way damaged the reputation of ICSE.  Second, the time budgeted for the  two parts of the process didn't align with the reviewing load for the  two parts, with the first part involving a very heavy load and the  second part involving a very light load.
-SIGGRAPH: Again, I don't think I am clear enough on the papers community process to answer this one.  Because of the growing concern of increased submissions in the CAF and Art Gallery, it is extremely likely that at least an experimental process will be implemented because of the growing challenges of our current practices.  The success of these experiments could result in moving more programs to a fee process.
-DAC: Yes, we have had to work on creative ways to reduce reviewer load as our number of papers has grown.  We have increased committee size, and asked each committee member to only review a subset of papers.  This has worked fine.
+Question 2
 NON-INCREMENTAL. Has your community recently adopted new practices to promote non-incremental new ideas?
 OOPSLA: If I understand it correctly, then the answer is yes. OOPSLA has been very active in introducing new venues for a long time (Educators Symposium, Doctoral Symposium, Practitioners Reports, Demos, of course in additon to the traditional other venues such as workshops, and tutorials.  I apologize if I didn't understand what is meant by non-incremental.
 OOPSLA: Yes, it has been a tradition for OOPSLA to host specific tracks devoted to specific issue with related tutorials, workshops and panels, and some times with its own invited speaker (e.g., THe OnWard! track)
-SIGGRAPH: SIGGRAPH already has a special sessions program where broader ideas/interest areas are presented each year.  It is a very successful program that draws anywhere from several hundred to a couple of thousand attendees.  There is also the Sketches program that may not focus on "big ideas" but on new and innovative ideas that are on the horizon of interest amongst attendees.
+more papers
-* more papers
 OOPSLA: Introduced recently different categories to the papers track (e.g., Essays, selected papers from the OnWard! track that pass the same technical papers review criteria)
-SIGGRAPH: While SIGGRAPH is concerned about the growing number of submissions, there is also concern for bringing in appropriate content with the changes in research areas.  There is increasing pursuit of these new and less represented areas so that the scope of content provided at the conference is comprehensive.
+shorter papers
-* shorter papers
 OOPSLA: Yes, we introduced Practioner Reports which doesn't have the same acceptance criteria as the technical papers but they are not citable either.  They server the practioners community rather than academia though we see more and more interst from Academia in what the practioners are doing.
-SIGGRAPH: I am unclear about the papers committee's concern about papers length.  However, the Sketches program seems to be filling a need for shorter papers presentation.
+deemphasizing detailed evaluation
-* deemphasizing detailed evaluation
 OOPSLA: Only for non technical papers
+---------------
+SIGGRAPH: SIGGRAPH already has a special sessions program where broader ideas/interest areas are presented each year.  It is a very successful program that draws anywhere from several hundred to a couple of thousand attendees.  There is also the Sketches program that may not focus on "big ideas" but on new and innovative ideas that are on the horizon of interest amongst attendees.
+more papers
+SIGGRAPH: While SIGGRAPH is concerned about the growing number of submissions, there is also concern for bringing in appropriate content with the changes in research areas.  There is increasing pursuit of these new and less represented areas so that the scope of content provided at the conference is comprehensive.
+shorter papers
+SIGGRAPH: I am unclear about the papers committee's concern about papers length.  However, the Sketches program seems to be filling a need for shorter papers presentation.
+deemphasizing detailed evaluation
 SIGGRAPH: Again, I think that this depends on the program chair and the
 committee process and it changes from year to year.
+---------------------
-* others?
+others?
 SIGCSE: I believe that we always have looked for a balance of papers on a variety of subjects.
 SIGCSE: This is not done formally, although each committee looks for balance and interesting new thoughts.  Also, we have added several keynotes to our conferences to solicit new or different perspectives.
-For OOPSLA Practitioner Reports and Onward! proved very valuable to our community.
+double blind submissions
+SIGCSE: Yes
-ICSE: Past instances of the Int'l Symposium on the Foundations of Software  Engineering (FSE) had an informal evening 'fun flames' session over beer and snacks, giving interested people a  low-stress way of getting feedback on very early and immature ideas.   This was quite an informal gathering that was intended more for *airing* new ideas rather than *promoting* them.  This year the Int'l Symposium on Software Testing and Analysis (ISSTA) are soliciting submission for a more formal new ideas track, but it is too early to  tell how successful this will be.
+large program committees
+SIGCSE: we encourage reasonably large program committees to include many members in meaningful ways.  We do not use this approach to replace or duplicate the regular reviewers.
-DAC: We do have short papers, and we do try to keep the submission rate above 20%.  However, we still require rigorous review of all papers.
+program subcommittees
+SIGCSE: program committees have members focused on various aspects of the event e.g, papers, panels, workshops, local arrangements, ...).  We do not use subcommittees to subdivide the reviewing process.
-PROGRAM COMMITTEES.  Does your community practice:
+Do these practices seem to help or hurt promoting your field?
-* double blind submissions
+SIGCSE: Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance.
-SIGCSE: Yes
-OOPSLA: Not for the moment
-SIGGRAPH: only with external reviewers of papers. Although I have been told that the papers community know who is doing what research and the blind review process isn't always possible.  Other SIGGRAPH programs have not exercised the blind process.
+-------------------------------
+For OOPSLA Practitioner Reports and Onward! proved very valuable to our community.
-* program committee submission restrictions
+double blind submissions
+OOPSLA: Not for the moment
+program committee submission restrictions
 OOPSLA: No restrictions but stricter review for program committee papers
-SIGGRAPH: SIGGRAPH allows for committee members to submit to any program. However, there is a process mechanism for each that precludes review/participation in discussion/voting on the part of the committee member for their submission.
+rebuttals (author responses)
-* rebuttals (author responses)
 OOPSLA: No, though every now and then a complaint is registered and the Program Chair as well as the Conference Chair respond appropriately.
-SIGGRAPH: There is no rebuttal process in any SIGGRAPH program that alters the decision of the committee.
+large program committees
-* large program committees
-SIGCSE: we encourage reasonably large program committees to include many members in meaningful ways.  We do not use this approach to replace or duplicate the regular reviewers.
 OOPSLA: Not sure what qualifies as large and I assume it is in relationship to the number of submitted papers. For OOPSLA the submission usually range from 160-190 papers.  Program Committee ranges from 22 - 28.  I will leave it up to you to determine if this is a large committee or not.
-ICSE: Yes (as many as 45 on past PCs of ICSE).
+program subcommittees
-SIGGRAPH supports whatever program committee size is necessary in order to meet the needs of each program's submissions and implementation of the presentations at the conference.
-* program subcommittees
-SIGCSE: program committees have members focused on various aspects of the event e.g, papers, panels, workshops, local arrangements, ...).  We do not use subcommittees to subdivide the reviewing process.
 OOPSLA: Not officially, but the assumption is that every PC member will reply on others that assist him/her in reviewing the paper assigned to them. However, ultimately the members of the PC are responsible. (which it the common practice). However, with some of the introduced new categories of papers (such as Essays and the selected Onward! subcommittees are formed and are responsible for the selection of their papers.
-ICSE: Not presently (although this was used many years ago in ICSE).
+Do these practices seem to help or hurt promoting your field?
+OOPSLA: Some of these are new and the impact is not completely determined, however, we noticed a very positive impact for introducing the Onward! track.
-SIGGRAPH programs requiring a lot of on site preparation/coordination have subcommittees that assist in these areas.
-* others?
+------------------
-SIGGRAPH has spent many years fine-tuning the process of content from submission to presentation.  It is a never-ending process that continuously changes as the needs arise.  Most of our practices work well.  Because SIGGRAPH has an average of 12-18 programs each year, program reviews are done on a rotational basis (average of 3-4 per year) by the Conference Advisory Group.
+ICSE: Past instances of the Int'l Symposium on the Foundations of Software  Engineering (FSE) had an informal evening 'fun flames' session over beer and snacks, giving interested people a  low-stress way of getting feedback on very early and immature ideas.   This was quite an informal gathering that was intended more for *airing* new ideas rather than *promoting* them.  This year the Int'l Symposium on Software Testing and Analysis (ISSTA) are soliciting submission for a more formal new ideas track, but it is too early to  tell how successful this will be.
-Do these practices seem to help or hurt promoting your field?
+large program committees
-SIGCSE: Having many reviewers and utilizing many people on a program committee has been a great help in giving the community a better sense of identify and connection. It also seems to have had a significant effect in encouraging increased conference attendance.
+ICSE: Yes (as many as 45 on past PCs of ICSE).
-OOPSLA: Some of these are new and the impact is not completely determined, however, we noticed a very positive impact for introducing the Onward! track.
+program subcommittees
+ICSE: Not presently (although this was used many years ago in ICSE).
+Do these practices seem to help or hurt promoting your field?
 ICSE: Past experiences with a subcommittee structure for the ICSE PC were  reportedly disastrous.  In addition, there is a feeling among some  that scientific and scholarly standards in software engineering are  not sufficiently uniform to justify author rebuttals, which because  of the lack of uniformity would end up being broad advocacy  statements that all authors would submit rather than narrow rebuttals  of misunderstood points.
 From ICSE 2005 PC co-Chair: A note on managing large submissions and subcommittees.
 I think the number of submissions has leveled off for the time-being (for us), so further efforts are not really necessary at this time.
+----------------
+DAC: We do have short papers, and we do try to keep the submission rate above 20%.  However, we still require rigorous review of all papers.
+Do these practices seem to help or hurt promoting your field?
 DAC: We practice all of the above, and feel that they are all necessary.  They seem to help.
+----------------
+double blind submissions
+SIGGRAPH: only with external reviewers of papers. Although I have been told that the papers community know who is doing what research and the blind review process isn't always possible.  Other SIGGRAPH programs have not exercised the blind process.
+program committee submission restrictions
+SIGGRAPH: SIGGRAPH allows for committee members to submit to any program. However, there is a process mechanism for each that precludes review/participation in discussion/voting on the part of the committee member for their submission.
+rebuttals (author responses)
+SIGGRAPH: There is no rebuttal process in any SIGGRAPH program that alters the decision of the committee.
+large program committees
+SIGGRAPH supports whatever program committee size is necessary in order to meet the needs of each program's submissions and implementation of the presentations at the conference.
+program subcommittees
+SIGGRAPH programs requiring a lot of on site preparation/coordination have subcommittees that assist in these areas.
+others?
+SIGGRAPH has spent many years fine-tuning the process of content from submission to presentation.  It is a never-ending process that continuously changes as the needs arise.  Most of our practices work well.  Because SIGGRAPH has an average of 12-18 programs each year, program reviews are done on a rotational basis (average of 3-4 per year) by the Conference Advisory Group.
+Question 3
 WORKSHOPS, ETC.  Does your community provide venue for work not mature enough for your major conferences, such as:
 * workshop co-located at conferences
 OOPSLA: Workshops have been always an impportant part of OOPSLA.  For the last 4 years OOPSLA has been co-located with other conferences.
-* stand-alone workshops
+stand-alone workshops
-SIGCSE: in computer science education, CCSC and other groups sponsor quite a number of regional conferences.  SIGCSE is in cooperation with these.  Since these have a strong following, SIGCSE has not seen any reason to try to duplicate them.
 OOPSLA: We didn't call them Workshops but we have tracks such as the Educators' and Doctoral Symposia that are stand alone.  For few years OOPSLA sponsored mid-year series of Workshop as a separate meetings from OOPSLA but achieved moderate success and the idea died when its champion stepped down.
-* panels
+panels
-SIGCSE: each conference has a range of panels on new or emerging ideas.
 OOPSLA: Always as part of the core OOPSLA conference
-* crazy idea sessions
-The SIGCSE Symposium provides an opportunity for "Special Sessions" and
-Birds-of-a-Feather than can promote a range of "crazy" ideas.
+crazy idea sessions
 OOPSLA: Last year we introduced the Lightning Talks (strictly 5 minutes talks for any idea an author would like to share with the audience).  It is too early to assess its success, but the concept was borrowed from other conferences where such talks were a great success.
-SIGGRAPH does all of these things annually as well as implement new programs (panels is a permanent program) .  For example, Web Graphics was funded for a three-year period to assess the need for permanent implementation as a separate program or blending this content into other existing programs.  Other experimental programs have included SIGGRAPH TV, Online Services, Community Outreach, SIGKids, to name few.
+On balance, are these other venues effect for advancing your field?
+What mechanisms, if any, do you use allow good papers from these venues
+to later achieve wider dissemination?
+OOPSLA: I believe OOPSLA was instrumental in introducing many of such venues that other conference adopted and such venues proved very valuable for the community. OOPSLA publishes a companion to the proceedings which includes the workshops conclusion, panels, etc. Also, Onward! presentations have been published in separate publications.
-DAC: We have all of the above except crazy idea sessions.  They all work well.
+------------------------
+stand-alone workshops
+SIGCSE: in computer science education, CCSC and other groups sponsor quite a number of regional conferences.  SIGCSE is in cooperation with these.  Since these have a strong following, SIGCSE has not seen any reason to try to duplicate them.
+panels
+SIGCSE: each conference has a range of panels on new or emerging ideas.
+crazy idea sessions
+The SIGCSE Symposium provides an opportunity for "Special Sessions" and
+Birds-of-a-Feather than can promote a range of "crazy" ideas.
 On balance, are these other venues effect for advancing your field?
 The SIGCSE Bulletin and conferences are the primary mechanisms for communication within the computer science education community -- especially at the college level. We have tried to expand this to other levels (with special emphasis on two-year colleges and high schools), within our resources.  For example, we have had special conference rates for high school teachers.
-OOPSLA: I believe OOPSLA was instrumental in introducing many of such venues that other conference adopted and such venues proved very valuable for the community. OOPSLA publishes a companion to the proceedings which includes the workshops conclusion, panels, etc. Also, Onward! presentations have been published in separate publications.
-ICSE: Crazy ideas sessions were discussed above under NON-INCREMENTAL.   There are numerous co-located and stand-alone workshops in software  engineering.  Regrettably many of these are becoming more like  conferences, with solicitation and publication of conference-length  papers, and serial presentation of conference-length talks.  Thus it  is difficult for papers from these workshops to receive any wider  dissemination than they receive as papers in the workshop  proceedings, since any attempt to submit similar ideas and text to a  conference would be viewed as a re-submission of published work.
+crazy idea sessions
+SIGGRAPH does all of these things annually as well as implement new programs (panels is a permanent program) .  For example, Web Graphics was funded for a three-year period to assess the need for permanent implementation as a separate program or blending this content into other existing programs.  Other experimental programs have included SIGGRAPH TV, Online Services, Community Outreach, SIGKids, to name few.
+On balance, are these other venues effect for advancing your field?
+What mechanisms, if any, do you use allow good papers from these venues
+to later achieve wider dissemination?
 SIGGRAPH: Most of SIGGRAPH's experimental programs have been successful and have been necessary in promoting the different areas of visual fields and education.  The technology and content of some of these programs live on in the more established programs after the end of the experimental runs.
+crazy idea sessions
+DAC: We have all of the above except crazy idea sessions.  They all work well.
+On balance, are these other venues effect for advancing your field?
+What mechanisms, if any, do you use allow good papers from these venues
+to later achieve wider dissemination?
 DAC: We also have a student design contest, and some of the top ones get in our technical program.  This gives them a wider dissemination.
+On balance, are these other venues effect for advancing your field?
+What mechanisms, if any, do you use allow good papers from these venues
+to later achieve wider dissemination?
+ICSE: Crazy ideas sessions were discussed above under NON-INCREMENTAL.   There are numerous co-located and stand-alone workshops in software  engineering.  Regrettably many of these are becoming more like  conferences, with solicitation and publication of conference-length  papers, and serial presentation of conference-length talks.  Thus it  is difficult for papers from these workshops to receive any wider  dissemination than they receive as papers in the workshop  proceedings, since any attempt to submit similar ideas and text to a  conference would be viewed as a re-submission of published work.
+Question 5
 CATCH-ALL.  Are there other approaches your community has tried or abandoned that the rest of us can learn from?
 OOPSLA: We established a wiki for the conference committee to use while planning for the conference as well as a wiki to use during the conference by the attendees.  We are also investigating new ideas in the next couple of years to improve some of the traditional venues such as introducing classes for certification purposes as part of the tutorial program and others but nothing has been implemented yet so it is not appropriate to talk about them yet.