Q1: Large Conferences

From Health of Conferences Committee

Question 1: REVIEWER LOAD.

Has your community recently adopted new practices to deal with growing reviewer load, such as:

• tracking reviews of rejected papers from conference to conference as is done in journal reviewing
• increasing program committee size
• charging a review fee
• others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

OOPSLA

not formally

increasing program committee size

Program committee didn't increase dramatically though it was increased by few from 5 years ago

ICSE

There has been some talk of doing this, but it hasn't been implemented yet.

increasing program committee size

Most PCs are at a maximum feasible size now.

For the 2004 Int'l Conference on Software Engineering (ICSE), we used a two-phase review process with a large PC. In the first phase each submission was reviewed by just two PC members (or three in the case of submissions from PC members), with clearly low-scoring papers summarily rejected at the end of the first phase. In the second phase, the surviving papers were each assigned an additional reviewer. The highest-scoring papers coming out of the second phase were discussed at the PC meeting, and the rest were summarily rejected. This seemed to work quite well, as it reduced the amount of effort spent reviewing low-quality submissions, and reduced the total number of papers each PC member reviewed as compared to the traditional one-phase process in which each submission receives three reviews (or four in the case of submissions from PC members).

For ICSE 2006, the PC co-chairs used a smaller PC and a large population of external reviewers. I don't have first-hand experience with this process, but I believe each paper was reviewed initially by two PC members plus an external reviewer, and then high-scoring papers went through a second round of reviewing by an additional reviewer. From second-hand accounts I have received, this process seems to have been unsuccessful, for a number of reasons. First, the external reviewers tended to be quite negative (and several were reportedly from outside the community). Furthermore, because of the large percentage of low-quality submissions, it was felt that exposing so many people to the pool of submissions in this way damaged the reputation of ICSE. Second, the time budgeted for the two parts of the process didn't align with the reviewing load for the two parts, with the first part involving a very heavy load and the second part involving a very light load.

SIGGRAPH

I believe that our papers committees do keep track of rejected papers, but it is unclear of this process is chronicled in any fashion from year-to-year. There has been a mechanism within the papers community to provide feedback. Other SIGGRAPH programs (courses, panels, educators) that also have a reviewer/jury process have begun using our online review system to track responses of all reviewers that can also be seen by all. These responses are then used by the program chair to resond to submittors with constructive feedback as to why their submissions were not accepted.

increasing program committee size

In order to handle the increasing number of submissions of several SIGGRAPH programs, the committee size has been allowed to increase where necessary. This is of concern, however, because of the related increases in costs for travel and contributor recognition.

charging a review fee

Charge a fee has been discussed, but has met with an allergic reaction for papers submissions. However, there is growing interest with the Computer Animation Festival (CAF) and the Art Gallery. Even with these programs, an actual implemenation process is probably two years away.

others?

I believe that there was a revised review process put into place by the Papers committee in recent years, refining/increaseing the tiered review process. Joe, I think you are probably better equipped to detail this.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

Again, I don't think I am clear enough on the papers community process to answer this one. Because of the growing concern of increased submissions in the CAF and Art Gallery, it is extremely likely that at least an experimental process will be implemented because of the growing challenges of our current practices. The success of these experiments could result in moving more programs to a fee process.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

DAC

Yes, we have had to work on creative ways to reduce reviewer load as our number of papers has grown. We have increased committee size, and asked each committee member to only review a subset of papers. This has worked fine.

SIGCSE

SIGCSE takes the following approach for reviewing

reviewing for both the SIGCSE Symposium and ITiCSE Conferences are double blind we maintain a database including about 779 reviewers for the Symposium and 584 reviewers for the ITiCSE conference.

each paper is sent to about 6 reviewers, and each reviewer receives 2-4 papers. a minimum of 4 reviews is required for each paper, and the Program Co-Chairs assign last-minute reviewers if the minimum number of reviews has not been met.

Most papers have 5 or 6 reviews.

SIGCSE members typically view their involvement in reviewing as a contribution to the community, and they are enthusiastic in this work.

After the reviewing period and after decisions are made, authors may view the reviews of their papers (although the reviews are anonymous), and each reviewer may view the other reviews of the papers they considered (again anonymously).

Super Computing

We have not implemented this yet. We do notice, based on reviewers involvement with multiple conferences, that we are seeing a increased number of papers being submitted to mulitple conferences simultaneously

increasing program committee size

You can see that the number of reviewers has increased and then remained constant. 2001 was an exceptionally small committee however. A significant reason for the increase is also we have extended the areas the conference supports.

charging a review fee

No we have no plans to do this and would probably resist it.

others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.