Q2: IEEE Conferences

From Health of Conferences Committee

Revision as of 18:20, 9 March 2006 MarkDHill (Talk | contribs) ← Previous diff		Current revision 199.222.74.48 (Talk | contribs)
Line 1:		Line 1:
-	===Question 1: REVIEWER LOAD.===	+	===Question 2: NON-INCREMENTAL===
-	Has your community recently adopted new practices to deal with growing reviewer load, such as:	+	Has your community recently adopted new practices to promote non-incremental new ideas?
-	*tracking reviews of rejected papers from conference to conference as is done in journal reviewing	+	*big ideas sessions
-	*increasing program committee size	+	*more papers
-	*charging a review fee	+	*shorter papers
		+	*deemphasizing detailed evaluation
	*others?		*others?
Line 11:		Line 12:
	'''ACM/IEEE'''		'''ACM/IEEE'''
-	:Increase PC size works but not ideally. A first round of screening to remove obviously lousy papers (e.g. too many or too few pages) would also work.	+	:Panels of discussion, that only publishes summaries, are a useful practice.
-	:K.Z.	+	:- K.Z.
-	:Increasing PC size appears to be effective as each member sees the work involved in reviewing for one conference as less. However, one tends to be involved in more PCs.
-	:D. W.
-	:My favorite suggestion is to use text mining to automatically identify 3-5 published papers that are closest to the current submission (over the last 5 years of published papers). These can be forwarded to reviewers along with the reviewed paper.	+	:Not really, that's what the workshops are good for. Short papers will only undermine the quality of submissions and make the review process more arbitrary.
-	:I.M.	+	:- I.M.
-	:At point of invitation, it is now common to negotiate how many papers you can commit to review. I do this all the time now and it works. I know others who do the same. By limiting the # of reviews upfront, the process becomes doable and expectations are not mismatched.
-	:S.H.
-	:Over the past decade we have increased program committee size. We try to reduce reviewer load by recruiting new reviewers, and by emphasizing that each program committee member must get a reviewer's consent before sending a review. We have just implemented a mechanism for reviewers to reject reviewers, with a reason. We hope this will reduce the number of unreturned reviews, and allow us to monitor committee members who are not getting in touch with reviewers in advance.	+	:Rejecting papers with good ideas based on one negative review hurt the community. Also rejecting papers from Asia because they are not well written (English problem) is also hurting the community. I have seen so many good idea papers that are not well written. The benefit of presenting the idea to the community is lost by rejecting the paper on presentation ground or by rejecting the paper based on one negative. This is something I would lie to see change.
-	:S.D.	+	:-S.H.
		+
		+
		+	:We are not doing any of these things. We introduce new topics to the Conference through a panel to increase interest, and then, possibly, a special session. We do have a lecture and application series, where the reviewing is less formal, to provide specific knowledge to our attendees. However, we've never had an issue refusing "big ideas" - many best papers in the past have been in this category.
		+	:- S.D.
-	:Reviewer load with IEEE CBMS increases each year indeed due to the increasing number of submissions. We increase Programme Committee size each year, inviting new members well-recognized in the field - it helps to keep the reviewer load reasonable. I believe it is the only solution to the problem.
-	:A.T.
-	:Improve the quality and efficiency of web services, using work group practices.	+	:We use the practice of Special Tracks. Each year we have a Call for Special Tracks - so that experts representing new emerging areas could organize special sessions on some particular topics besides the usual regular sessions. Besides having a separate programme and review committee, Special Tracks have no difference from the regular sessions (the presentation time is the same, and the publication length in the proceedings is the same). I believe Special Tracks significantly help to promote new emerging sub-areas and ideas at the conference.
-	:S.J.	+	:- A.T.
		+
		+
		+	:Some workshops/conferences have a 6 page limit; but shorter than that most people will not want to come to the conference to give a short paper.
		+	:- C.B.
		+
		+
		+	:In addition to technical papers, the conference include about 40-50 tutorials, 4 invited speakers for Plenary sessions, one Keynote, and 12 invited "Masterworks" presentations. The masterworks sessions and invited speakers are intended to provided targeted "big issues" and also bring to the conference highly regarded people who are leaders of the different communities. This works very well.
		+	:- B.K.
		+
-	:I think the Software Engineering (IEEE Computer Society) workshops and conferences are still doing fairly conventional reviewing, so maybe our program committees are larger - but that's about it.	+	:Have a special session dedicated to new ideas that are untested and futuristic.
-	:C.B.	+	:- V.C.
		+
-	:We do notice, based on reviewer’s involvement with multiple conferences, that we are seeing an increased number of papers being submitted to multiple conferences simultaneously. You can see that the number of reviewers has increased and then remained constant. 2001 was an exceptionally small committee however. A significant reason for the increase is also we have extended the areas the conference supports.	+	:We are consistently trying to adopt this practice to make the conference(s) much more meaningful to attendees. Sessions of only submitted papers are not enough any more - this way we can ensure the input from experienced people who might not otherwise have submitted papers. We can add very controversial topics that no-one else would have considered adopting. Already the conferences seem to be too big. Let's maintain standards, but also go back to considering poster sessions.
-	:B.K.	+	:- A.A.
-	:We distributed papers among PC members. No special measures taken.
-	:V.C.
-	:We already have a large program committee for most of our events that I'm associated with - in many cases I think it is too large and is a practice that only serves to attract papers from the program committee.	+	:The security community has had the NSPW (new security paradigms workshop, this used to be an ACM conference and is now sponsored by ACSA) for more than a decade, and there has been a proliferation of papers, often also scattered across several conferences, which blunted the impact. CRA recently hosted a `Grand Challenges' workshop in the field, but that turned out to be more of a consensus group. We (IEEE TFIA) try to encourage this type of work by organizing smaller max. 60 participants) single-track workshops which are to provide a richer and more focused forum for interaction and debate among participants, but on balance I'm not convinced that the present approach is truly enjoying encouraging success in supporting or generating truly disruptive ideas.
-	:A.A.	+	:- S.W.
-		+
-	:We have increased PC size to keep up with submissions (at least to some extent), and are -- given the fact that information security /information assurance is a relatively small research community -- there is informal tracking of rejected publications in instances when PC members have come across similar (or even identical) submissions before. Review fees are highly impractical in my personal opinion since this raises an additional barrier for international submissions, particularly from countries with limited financial resources for supporting research.	+
-	:S.W.	+
-	:We had tried to increase the size of our program committee, but it is not easy. Many of the most desirable PC members serve on several committees, so these reviewers are already severely overloaded. On the plus side, we do allow our PC members to recruit colleagues to take on individual papers, so that most PC members do not personally review all the papers which are assigned to them.
-	:V.M.
-	:Review is an honorary responsibility of every scientist. It is done for free. We do tracking of rejected papers. For our 3 year history we have about 10% of rejected papers.	+	:We adopted accepting short papers (4 pages or less) in 2005. It seems to be working well. We still accept full papers (limited to 12 pages single-spaced). About 20% of the submissions have been short papers, and, while we have not set any quotas for acceptance, about 20% of the accepted papers for our 2005 conference were short papers.
-	:V.H.	+	:- V.M.
-	:USE PHD STUDENTS. INCREASE PC SIZE. It helped reduce average review load. It helped increase conference attendance. It helped build the sense of community. It looked funny because of the size of PC vs. the size of attendance, except for a few major conferences.
-	:C.K.K.
-	:Program committees increase every year. The increase of the PC can work well, keeping manageable, if members are grouped around a topic that has a particular chair.	+	:We are successfully using practice of having approximately equal(about 30%) groups of papers/tutorials from scientists from East (EU,USA, …), from scientist from West (Ukraine, Russia, Armenia, Belorus,…), and from students (PhD’s are in this group too).
-	:M.L.	+	:- V.H.
		+
		+
		+	:Co-locating; no fixed partners. Yet to see the long-term effectiveness.
		+	:- C.K.K.
		+
		+
		+	:Short papers and posters are becoming more popular. Those are no more considered "devaluated" publications. I’m aware of just one big idea session, held along with VTS and called Elevator Talks. Too early to say if this is really a good idea. Seems to be, but I’m not sure that the number of submissions is high enough for this sort of session.
		+	:- M.L.
		+
		+
		+	:In very few conferences there are "new ideas" sessions. I don't think they have worked as expected though.
		+	:- D.S.
		+
		+
		+	:The big idea sessions have just been introduced in CFPs this year. We will have to wait and see if the published papers actually contain big ideas or simply poorly evaluated small ideas. Some conferences have accepted increased number of papers. However, this is a modest increase (e.g., 32 instead of 28). While this is a good idea, this will have only small benefit.
		+	:- R.G.
		+
		+
		+	'''OOPSLA'''
		+
		+	:OOPSLA has a "big idea" session. It's called Onward! This has a subcommittee (I was on it once) to review the submissions. The accepted papers get a one hour presentation slot. Many software engineering (SE) venues have promoted workshops that have newer ideas - things less mature or not completely evaluated. Also most SE venues have short papers.
		+	:- J.M.
		+
-	:As I understand, they have increased PC size. Review fee has been discussed once in one conference, but I have not seen it implemented anywhere.	+	'''SC 05'''
-	:D.S.	+
-	:Increase in committee size has helped reduce load which has enabled continued high standards in reviewing. There are plenty of qualified people who are willing to serve on the PCs so this has worked out well.	+	:In SC 05 we went to 5 parallel tracks for the technical papers. This was done not to increase the technical papers but to expand other aspects of the conference, including an additional award sessions, special sessions associated with the themes of the conference, etc. The chart shows that actually the number of technical papers declined slightly and then stayed about the same. Several years ago, we tried the concept of "extended abstracts" for selection and the having the full papers done for only the accepted papers. This was done mostly to help authors - but it was found that it causes several problems. First, there was not enough detail to fully evaluate some papers. Second, we felt getting the final papers in on time is more challenging. In SC05 we had full papers submitted and it seemed to work well. This appears to work well.
-	:R.G.	+	:- B.K.
-	'''IPDPS, SC, HiPC, ICPP, SPAA'''
-	:The most common practice has been to increase the PC size. However, the PC is normally formed before the paper submission deadline, so it is problematic when an unusually high number of submissions are received that surpass historic records. For instance, IPDPS 2006 received approximately 200 more submissions than the previous year. Other practices were started, such as in HiPC, where an initial pass through the papers is taken to make a "quick reject" decision on any paper that is clearly A) outside of scope, B) plagiarized, C) contains no scholarly content. This reduces the number of papers in the "regular review" and the reviewers' load. However, care must be taken to reject only those papers that are "obvious" rejects for administrative reasons, and not for ones where a single person is evaluating the novelty/originality/significance/quality of the technical contribution.
-	:D.B.
		+	'''IPDPS, SC, HiPC, ICPP, SPAA'''
-	'''ASE'''	+	:I have not seen this practice yet in the parallel and distributed computing community.
-	:The size of the PC for ASE was increased recently to reflect the large number of submission. For ASE'05 PC members had 18-20 papers to review. Otherwise, no specific actions have been taken to address reviewer load. A couple of venues have taken steps to assure workload is between 6 and 10 paper to review. So the size of the PC is determined by the number of submission expected (from historical data). A review fee may be an interesting option - but in general I've not run into serious problems with this issue. If PC members are allowed to bid on papers to review this normally gets solved. That is, if I review a paper in one conference and reject it, than see it again submitted in another venue I bid on it (easy review) and get to see if it was improved.	+	:- D.B.
-	:J.M.	+

---

Current revision

Question 2: NON-INCREMENTAL

Has your community recently adopted new practices to promote non-incremental new ideas?

• big ideas sessions
• more papers
• shorter papers
• deemphasizing detailed evaluation
• others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

ACM/IEEE

Panels of discussion, that only publishes summaries, are a useful practice.

- K.Z.

Not really, that's what the workshops are good for. Short papers will only undermine the quality of submissions and make the review process more arbitrary.

- I.M.

Rejecting papers with good ideas based on one negative review hurt the community. Also rejecting papers from Asia because they are not well written (English problem) is also hurting the community. I have seen so many good idea papers that are not well written. The benefit of presenting the idea to the community is lost by rejecting the paper on presentation ground or by rejecting the paper based on one negative. This is something I would lie to see change.

-S.H.

We are not doing any of these things. We introduce new topics to the Conference through a panel to increase interest, and then, possibly, a special session. We do have a lecture and application series, where the reviewing is less formal, to provide specific knowledge to our attendees. However, we've never had an issue refusing "big ideas" - many best papers in the past have been in this category.

- S.D.

We use the practice of Special Tracks. Each year we have a Call for Special Tracks - so that experts representing new emerging areas could organize special sessions on some particular topics besides the usual regular sessions. Besides having a separate programme and review committee, Special Tracks have no difference from the regular sessions (the presentation time is the same, and the publication length in the proceedings is the same). I believe Special Tracks significantly help to promote new emerging sub-areas and ideas at the conference.

- A.T.

Some workshops/conferences have a 6 page limit; but shorter than that most people will not want to come to the conference to give a short paper.

- C.B.

In addition to technical papers, the conference include about 40-50 tutorials, 4 invited speakers for Plenary sessions, one Keynote, and 12 invited "Masterworks" presentations. The masterworks sessions and invited speakers are intended to provided targeted "big issues" and also bring to the conference highly regarded people who are leaders of the different communities. This works very well.

- B.K.

Have a special session dedicated to new ideas that are untested and futuristic.

- V.C.

We are consistently trying to adopt this practice to make the conference(s) much more meaningful to attendees. Sessions of only submitted papers are not enough any more - this way we can ensure the input from experienced people who might not otherwise have submitted papers. We can add very controversial topics that no-one else would have considered adopting. Already the conferences seem to be too big. Let's maintain standards, but also go back to considering poster sessions.

- A.A.

The security community has had the NSPW (new security paradigms workshop, this used to be an ACM conference and is now sponsored by ACSA) for more than a decade, and there has been a proliferation of papers, often also scattered across several conferences, which blunted the impact. CRA recently hosted a `Grand Challenges' workshop in the field, but that turned out to be more of a consensus group. We (IEEE TFIA) try to encourage this type of work by organizing smaller max. 60 participants) single-track workshops which are to provide a richer and more focused forum for interaction and debate among participants, but on balance I'm not convinced that the present approach is truly enjoying encouraging success in supporting or generating truly disruptive ideas.

- S.W.

We adopted accepting short papers (4 pages or less) in 2005. It seems to be working well. We still accept full papers (limited to 12 pages single-spaced). About 20% of the submissions have been short papers, and, while we have not set any quotas for acceptance, about 20% of the accepted papers for our 2005 conference were short papers.

- V.M.

We are successfully using practice of having approximately equal(about 30%) groups of papers/tutorials from scientists from East (EU,USA, …), from scientist from West (Ukraine, Russia, Armenia, Belorus,…), and from students (PhD’s are in this group too).

- V.H.

Co-locating; no fixed partners. Yet to see the long-term effectiveness.

- C.K.K.

Short papers and posters are becoming more popular. Those are no more considered "devaluated" publications. I’m aware of just one big idea session, held along with VTS and called Elevator Talks. Too early to say if this is really a good idea. Seems to be, but I’m not sure that the number of submissions is high enough for this sort of session.

- M.L.

In very few conferences there are "new ideas" sessions. I don't think they have worked as expected though.

- D.S.

The big idea sessions have just been introduced in CFPs this year. We will have to wait and see if the published papers actually contain big ideas or simply poorly evaluated small ideas. Some conferences have accepted increased number of papers. However, this is a modest increase (e.g., 32 instead of 28). While this is a good idea, this will have only small benefit.

- R.G.

OOPSLA

OOPSLA has a "big idea" session. It's called Onward! This has a subcommittee (I was on it once) to review the submissions. The accepted papers get a one hour presentation slot. Many software engineering (SE) venues have promoted workshops that have newer ideas - things less mature or not completely evaluated. Also most SE venues have short papers.

- J.M.

SC 05

In SC 05 we went to 5 parallel tracks for the technical papers. This was done not to increase the technical papers but to expand other aspects of the conference, including an additional award sessions, special sessions associated with the themes of the conference, etc. The chart shows that actually the number of technical papers declined slightly and then stayed about the same. Several years ago, we tried the concept of "extended abstracts" for selection and the having the full papers done for only the accepted papers. This was done mostly to help authors - but it was found that it causes several problems. First, there was not enough detail to fully evaluate some papers. Second, we felt getting the final papers in on time is more challenging. In SC05 we had full papers submitted and it seemed to work well. This appears to work well.

- B.K.

IPDPS, SC, HiPC, ICPP, SPAA

I have not seen this practice yet in the parallel and distributed computing community.

- D.B.