Q3: IEEE Conferences

From Health of Conferences Committee

Question 3: PROGRAM COMMITTEES

Does your community practice:

• double blind submissions
• program committee submission restrictions
• rebuttals (author responses)
• large program committees
• program subcommittees
• others?

Do these practices seem to help or hurt promoting your field?

ACM/IEEE

The committee is split into areas, and the program chair allocates papers to area topic coordinators who allocate to topic coordinators. Sessions are put together from the papers in the area, not from one topic coordinator.

- S.D.

We use blind review at IEEE CBMS. We also use the idea of programme subcommittees - each special track has its own programme committee. It helps for each sub-PC to concentrate on relevant papers.

- A.T.

We try to have 3 to 4 reviews, with a minimum of 2. The reviews and papers are arranged in sub areas and we have a data base system to handle the reviews, ratings and scheduling. Each reviewer handles about 3-4 papers, although load balancing sometimes means more. The lead of each of our six areas my end up reading most in their area, but that is not required. In general we are getting excellent response from the community. You can see the number of papers is increasing, and our acceptance rate is dropping. Our Quality Control surveys as well as word of mouth indicate the community is happy with the program. Since SC is a general conference covering a broad range(applications, systems, architectures, High Performance computing, High performance networking, high performance storage and analytics) we often get comments that people would like to see more papers in their area of emphasis.

- B.K.

There is a big necessity to think outside the box for the survival of a lot of events - clearly, also, there are too many events being maintained purely because they've been around for many years and there are ego's to maintain - let's cut some out and start again.

- T.A.

Software engineering does NOT do double blind. I don't think this would benefit the community - it's pretty easy to guess the research group from the paper. It may help junior reviewers rate poor papers by well know folks lower! But this is really a job for senior reviewers.

- J.M.

Double-blind reviews are standard, as are fairly large committees to permit the inclusion of specialized reviewers (also external reviewers). Program committee submission restrictions are usually handled by declaring potential conflicts of interest (on the part of the authors and/or the program chair). We also practice paper shepherding (which can be difficult to arrange given the additional workload this imposes on a PC member) to ensure that reviewer concerns are addressed by the authors in a constructive dialog. However, there is clearly a limit to how frequently this instrument can be used.

- S.W.

We have not practiced double blind submissions, although I personally think it would be a good idea. We also permit PC members to submit papers, and it falls to the PC chair to insure that there is no conflict of interest in assigning reviewers for these papers. When there are widely differing evaluations of a given paper, we send all reviews of those papers to the cohort of reviewers, and instigate a discussion between the reviewers to see if a closer consensus can be found. This has worked in some cases, but there have been some glaring failures. As mentioned earlier, we have been trying to increase our PC size, but this is difficult to do. We also like to change at least 1/3 of the PC members from year to year, in order to avoid becoming too narrowly focused.

- V.M.

We form PC from the people who are active scientists, are active professors, are heads of scientific schools, are representatives of the industry, which delivers innovations, inviting their colleges to the conference, submit own papers to the conference, and review papers.

- V.H.

Automation. Use manuscript submission and review systems that are not too expensive. Some charges are very high. Philosophy and practice of organization evolve all the time from conference to conference. I don't see any particular thing hurting the community definitively or intentionally. We are all volunteers operating within constraints of time and energy. QUESTION: Shall we look into the emerging problems with some low-end (I mean: very low) conferences that are positioned well down the food chain and seem to compete with main stream conferences (or not competitive at all), many even carrying identical titles with existing ones? Have we looked into these new kids on the block? How well are they actually doing? (Of course we won't spend time or make a trip to see the events.)

- C.K.C.

I start thinking that some restrictions to program committee submissions could also be helpful in medium and big size conferences.

- M.L.

Double blind submissions may work better than other measures. The same goes for PC submission restrictions. However, in most conferences I am familiar with or involved, there are no such restrictions. Rebuttals were used once in a big conference that I know (Infocom), but they were not used again (I was not on the PC, so I do not know the exact reasons why). In general, I believe that a large size of the PC committee is not an advantage. However, a small PC does not help either, because it leads to high load of PC members who have to distribute the reviews to a large number of reviewers finally. Overall, I believe that large PC's hurt more than they benefit quality. I believe that PC submission restrictions and double blind reviews will help, because they will motivate change of PC committees every year for good conferences and enable more objective review of papers.

- D.S.

They certainly help. It gives me greater confidence in the whole process. Quality of reviewing is good (less load due to larger committees)and more fair (double blind submissions, rebuttals).

- R.G.

WCRE / ASE

Of the main SE venues I'm involved in there are no PC submission restrictions or rebuttals. There are subcommittees for demo submissions, working sessions, doctoral sessions etc. But not subcommittees for areas/topics.

- J.M.

ICSM/ICPC

There has been a general move to only allow PC members to server for 3 years in a row. They must take at least one year off after three. Also, these venues require some new faces on the PC along with a majority of previous members. That is reassure continuity and still bring in new researchers.

- J.M.

ITC

The number of papers track the health of the industry more than anything. ITC is far more than a technical conference, since we are part of what we call Test Week, with tutorials before, exhibits during and workshops after, all coordinated by the ITC Steering Committee. This coordination has helped. However, industry economics has by far the biggest role.

- S.D.

IPDPS, SC, HiPC, ICPP, SPAA

In the parallel and distributed computing community, almost always the reviewing process is single-blind (reviewers are anonymous to the authors). Program committee members are encouraged to submit papers; however, the threshold for acceptance for a PC member is elevated so that it must be a clear and strong accept, rather than a borderline paper. The PC member may not enter into deliberations on his/her own paper. We have not practiced rebuttals, but this is in active discussion with IPDPS. Also, our larger conferences (IPDPS, SC, HiPC) have had tracked program subcommittees for a number of years, mostly due to the broad range of areas. Increasingly so, these tracks are harder to define as silos, and many papers span 2, 3, 4, or more tracks in topic!

- D.B.