Small Conferences

From Health of Conferences Committee

Question 1: REVIEWER LOAD.

Has your community recently adopted new practices to deal with growing reviewer load, such as:

• tracking reviews of rejected papers from conference to conference as is done in journal reviewing
• increasing program committee size
• charging a review fee
• others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

SIGITE

The only change we have made, starting with SIGITE 06, is to drop the requirement that authors first submit abstracts. Instead, we will be reviewing full papers only. The primary reason for this change has not been a problem with reviewer load. Rather, there is a feeling that potentially good papers were rejected because of poor abstracts.

SIGCSE SIGCSE

has not engaged in any of the above.

SIGAda

We have not had any problems with reviewer load and continue our practice of the Program Chair assigning 3-4 papers per Program Committee Member.

SIGDA

Yes, but everything is done manually and ad-hoc (i.e., it happens that some TPC member in one conference saw this paper in another conference). That's why I think support from ACM Pubs Dept. in being able to query existing submissions (not only just rejects) across conferences would be good.

increasing program committee size - Yes

charging a review fee - No (I'd like to hear more details about this - who is charged and/or who charges?)

others - External reviewers are assigned for some of the larger conferences, but sometimes their expertise is not what is expected. :In cases where external reviewing or delegation is discouraged the load is very large (25-30 papers per TPC member)

SIGAPP

We continually increase our program committee size to accommodate increasing paper submissions. This strategy is working well. I do not believe that any other strategy is needed.

SIGUCCS

tracking reviews of rejected papers from conference to conference as is done in journal reviewing - We don't have a formal mechanism for this (which we think is a good idea!) but since several folks serve on the program committee year after year there is probably some informal check on this. Also, acceptance of a paper is tied to presentation at the conference.

increasing program committee size - Yes, the size of the program committee has grown over the years as more reviewers have been added

charging a review fee - No fee is charged, although as I said above, paper acceptance is tied to conference presentation and there are conference fees.

SIGACCESS

For the '05 conference, there were more submissions than expected and the number of manuscripts assigned to each reviewer was very large (about 7). Not surprisingly, this created unhappy members of the Program Committee who had not expected so many manuscripts to review. To deal with this, the PC Chair indicated that committee members could ask colleagues to review some of the papers in areas in which their colleagues were particularly qualified. These people who were asked for special reviews were listed separately in the conference Proceedings.

While this system worked post hoc for the situation, it is not ideal. For '06, a larger PC has been created. The conference has not resorted to charging a reviewing fee or tracking reviews from conference to conference. We have, however, worked to better describe criteria for acceptable papers in the hopes of reducing the number of unqualified submissions. For example, in the past there has always been a tendency for authors to submit papers about proposals they have that have not been tested sufficiently. The CFP for '06 makes clear that testing with user populations will be important in the evaluation of submitted manuscripts.

SIGMIS

No. However, charging a submission fee has been discussed, but the conclusion is that increasing program committee size is a workable strategy if needed.

SIGMICRO

We have increased the committee size. Also recently the program chairs have taken on the task of assigning some papers to external reviewers to reduce the already heavy load on the program committee members. In general though, these practices have had only minimal impact. Increasing the committee size has side effects that are undesireable, for example it reduces the likelihood that all the committee can meet in one place to debate the papers.

SIGDOC

We’ve generally kept reviewer load to 5-10 proposals per reviewer and have not, to my awareness, received complains about that number.

SIGSAC

We've done as much tracking as possible, though it is hard to do it without a central database of submitted papers to various conferences. Almost invariably we find a couple of double submissions per major conference or "recycled" papers (papers rejected from one conference submitted to another).

Yes, we've increased program committee sizes in an attempt to hold review load to about 10-12 papers per reviewer. We have not always succeded, though.

We coupled this practice with substantial care in reviewing potential conflicts of interest for each program committee member.

charging a review fee - No, we have not done that and have no plans to do it.

others - We attempt to draft outside reviewers whose area of research fits specific topics covered in papers. These are not program committee members and generally do not get more than two papers to review. This has worked well and I believe contributed to maintaining high quality. The work of these external reviewers is acknowledged in the conference proceedings.

Tracking has worked moderately well. Increasing the program committee sizes has worked well Conflict of interest assessments have worked well. Using outside reviewers has worked very well.

SIGecom

We increased the program committee size from 18 last year to 41 this year. We also allowed PC members to delegate the reviews to external reviewers. Part of the reason was to reduce the reviewer load. Part of the reason was to signal the broad scope of the conference and to attract submissions from different communities.

SIGBED

This is, of course, a concern for SIGBED meetings. Our program committees are typically large, and we try to keep the number of papers per PC member to be around 15 -- 20. This seems to be working ok. I am not aware of any other special steps.

SIGMETRICS

Sigmetrics has increased the committee size (since we are suppose to review all the papers ourself) and increased a bit the number of accepted papers (one year we had 2 parallel sessions and we've had longer days), so basically we can do a bit over 30 papers these days (rather than say around 20-25).

In reviewing, we seem particularly careful to make sure that the paper has at least one reviewer who knows the area very well. If it turns out at the PC meeting that somehow that was not the case, we always get an additional review from someone directly in the area.

SIGSIM

Programme committee sizes are growing. We tend to track rejected papers by knowledge sharing between common PC members.

Our conferences tend to have two to three reviewers per paper. Reviewer expectations depend on the conference. Some of our conferences are "practitioner" led and expect real world experiences. Others are academically led and expect theory. Balancing between the two is difficult.

Increasing papers from China. This is an excellent development.

Not sure about review charging! Might bar some poorer contributors from submitting.