Q1: IEEE Conferences

From Health of Conferences Committee

Question 1: REVIEWER LOAD.

Has your community recently adopted new practices to deal with growing reviewer load, such as:

• tracking reviews of rejected papers from conference to conference as is done in journal reviewing
• increasing program committee size
• charging a review fee
• others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

ACM/IEEE

Increase PC size works but not ideally. A first round of screening to remove obviously lousy papers (e.g. too many or too few pages) would also work.

-K.Z.

Increasing PC size appears to be effective as each member sees the work involved in reviewing for one conference as less. However, one tends to be involved in more PCs.

-D. W.

My favorite suggestion is to use text mining to automatically identify 3-5 published papers that are closest to the current submission (over the last 5 years of published papers). These can be forwarded to reviewers along with the reviewed paper.

-I.M.

At point of invitation, it is now common to negotiate how many papers you can commit to review. I do this all the time now and it works. I know others who do the same. By limiting the # of reviews upfront, the process becomes doable and expectations are not mismatched.

-S.H.

Over the past decade we have increased program committee size. We try to reduce reviewer load by recruiting new reviewers, and by emphasizing that each program committee member must get a reviewer's consent before sending a review. We have just implemented a mechanism for reviewers to reject reviewers, with a reason. We hope this will reduce the number of unreturned reviews, and allow us to monitor committee members who are not getting in touch with reviewers in advance.

-S.D.

Reviewer load with IEEE CBMS increases each year indeed due to the increasing number of submissions. We increase Programme Committee size each year, inviting new members well-recognized in the field - it helps to keep the reviewer load reasonable. I believe it is the only solution to the problem.

-A.T.

Improve the quality and efficiency of web services, using work group practices.

-S.J.

I think the Software Engineering (IEEE Computer Society) workshops and conferences are still doing fairly conventional reviewing, so maybe our program committees are larger - but that's about it.

-C.B.

We do notice, based on reviewer’s involvement with multiple conferences, that we are seeing an increased number of papers being submitted to multiple conferences simultaneously. You can see that the number of reviewers has increased and then remained constant. 2001 was an exceptionally small committee however. A significant reason for the increase is also we have extended the areas the conference supports.

-B.K.

We distributed papers among PC members. No special measures taken.

-V.C.

We already have a large program committee for most of our events that I'm associated with - in many cases I think it is too large and is a practice that only serves to attract papers from the program committee.

-A.A.

We have increased PC size to keep up with submissions (at least to some extent), and are -- given the fact that information security /information assurance is a relatively small research community -- there is informal tracking of rejected publications in instances when PC members have come across similar (or even identical) submissions before. Review fees are highly impractical in my personal opinion since this raises an additional barrier for international submissions, particularly from countries with limited financial resources for supporting research.

-S.W.

We had tried to increase the size of our program committee, but it is not easy. Many of the most desirable PC members serve on several committees, so these reviewers are already severely overloaded. On the plus side, we do allow our PC members to recruit colleagues to take on individual papers, so that most PC members do not personally review all the papers which are assigned to them.

-V.M.

Review is an honorary responsibility of every scientist. It is done for free. We do tracking of rejected papers. For our 3 year history we have about 10% of rejected papers.

-V.H.

USE PHD STUDENTS. INCREASE PC SIZE. It helped reduce average review load. It helped increase conference attendance. It helped build the sense of community. It looked funny because of the size of PC vs. the size of attendance, except for a few major conferences.

-C.K.K.

Program committees increase every year. The increase of the PC can work well, keeping manageable, if members are grouped around a topic that has a particular chair.

-M.L.

As I understand, they have increased PC size. Review fee has been discussed once in one conference, but I have not seen it implemented anywhere.

-D.S.

Increase in committee size has helped reduce load which has enabled continued high standards in reviewing. There are plenty of qualified people who are willing to serve on the PCs so this has worked out well.

-R.G.

IPDPS, SC, HiPC, ICPP, SPAA

The most common practice has been to increase the PC size. However, the PC is normally formed before the paper submission deadline, so it is problematic when an unusually high number of submissions are received that surpass historic records. For instance, IPDPS 2006 received approximately 200 more submissions than the previous year. Other practices were started, such as in HiPC, where an initial pass through the papers is taken to make a "quick reject" decision on any paper that is clearly A) outside of scope, B) plagiarized, C) contains no scholarly content. This reduces the number of papers in the "regular review" and the reviewers' load. However, care must be taken to reject only those papers that are "obvious" rejects for administrative reasons, and not for ones where a single person is evaluating the novelty/originality/significance/quality of the technical contribution.

-D.B.

ASE

The size of the PC for ASE was increased recently to reflect the large number of submission. For ASE'05 PC members had 18-20 papers to review. Otherwise, no specific actions have been taken to address reviewer load. A couple of venues have taken steps to assure workload is between 6 and 10 paper to review. So the size of the PC is determined by the number of submission expected (from historical data). A review fee may be an interesting option - but in general I've not run into serious problems with this issue. If PC members are allowed to bid on papers to review this normally gets solved. That is, if I review a paper in one conference and reject it, than see it again submitted in another venue I bid on it (easy review) and get to see if it was improved.

-J.M.