Health of Conferences Committee - New pages [en] http://wiki.acm.org/healthcc/index.php?title=Special:Newpages From Health of Conferences Committee en MediaWiki 1.5.1 Fri, 29 Mar 2024 00:11:57 GMT Q5: IEEE Conferences http://wiki.acm.org/healthcc/index.php?title=Q5:_IEEE_Conferences <p>Summary: </p> <hr /> <div>===Question 5: CATCH-ALL.===<br /> Are there other approaches your community has tried or abandoned that the rest of us can learn from?<br /> <br /> <br /> '''ACM/IEEE'''<br /> :Another thing we have is SC Global which uses Access Grid Nodes to provide a separate program of sessions. The content is presented not just at the conference but at from remote sites. We typically have 30-40 sites on 6 (and in SC 04 seven) continents. In SC 05, we introduced SC Desktop that broadcast the conference session (all the ones discussed above) to anyone. We charged a nominal fee - about 10% of on-site fees and it was highly successful. We also archived the session. The target audience is people who can not come to the conference, such as graduate students, people who can't get visa and people who are ill. This appears to be very effective.<br /> :- B.K.</div> Thu, 09 Mar 2006 18:32:36 GMT MarkDHill http://wiki.acm.org/healthcc/index.php?title=Talk:Q5:_IEEE_Conferences Q4: IEEE Conferences http://wiki.acm.org/healthcc/index.php?title=Q4:_IEEE_Conferences <p>Summary: </p> <hr /> <div>===Question 4: WORKSHOPS, ETC.=== <br /> Does your community provide venue for work not mature <br /> enough for your major conferences, such as:<br /> <br /> *workshop co-located at conferences<br /> *stand-alone workshops<br /> *panels<br /> *crazy idea sessions<br /> <br /> On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination?<br /> <br /> <br /> '''ACM/IEEE'''<br /> :We have 8-10 workshops associated with the conference. One has grown quite large (the Grid Workshop) and we are &quot;graduating&quot; it over the next few years. Not all of these are immature work. Indeed the Grid Workshop has an acceptance below the conference. We have about 2-4 standalone workshops - some with separate registration. Yes we use panels and they are sometimes our more popular sessions. Typically we have about 6 sessions. We tried two &quot;gather and scatter&quot; sessions this year - which were a success. These sessions allow people to sign up on site for 15 mini-sessions to discuss late breaking/evolving work. We do other things. We have poster sessions. A small number of rejected papers are referred to the poser sessions, but the general rule is the posters are separate submissions. We have an Exhibitor Forum - where vendors can come and talk about the technical aspects of their products. They are supposed to be non-marketing and are reviewed before selection. <br /> :- B.K.</div> Thu, 09 Mar 2006 18:31:46 GMT MarkDHill http://wiki.acm.org/healthcc/index.php?title=Talk:Q4:_IEEE_Conferences Q3: IEEE Conferences http://wiki.acm.org/healthcc/index.php?title=Q3:_IEEE_Conferences <p>Summary: </p> <hr /> <div>===Question 3: PROGRAM COMMITTEES===<br /> Does your community practice:<br /> *double blind submissions<br /> *program committee submission restrictions<br /> *rebuttals (author responses)<br /> *large program committees<br /> *program subcommittees<br /> *others?<br /> <br /> Do these practices seem to help or hurt promoting your field?<br /> <br /> <br /> <br /> '''ACM/IEEE'''<br /> :The committee is split into areas, and the program chair allocates papers to area topic coordinators who allocate to topic coordinators. Sessions are put together from the papers in the area, not from one topic coordinator.<br /> :- S.D.<br /> <br /> <br /> :We use blind review at IEEE CBMS. We also use the idea of programme subcommittees - each special track has its own programme committee. It helps for each sub-PC to concentrate on relevant papers.<br /> :- A.T.<br /> <br /> <br /> :We try to have 3 to 4 reviews, with a minimum of 2. The reviews and papers are arranged in sub areas and we have a data base system to handle the reviews, ratings and scheduling. Each reviewer handles about 3-4 papers, although load balancing sometimes means more. The lead of each of our six areas my end up reading most in their area, but that is not required. In general we are getting excellent response from the community. You can see the number of papers is increasing, and our acceptance rate is dropping. Our Quality Control surveys as well as word of mouth indicate the community is happy with the program. Since SC is a general conference covering a broad range(applications, systems, architectures, High Performance computing, High performance networking, high performance storage and analytics) we often get comments that people would like to see more papers in their area of emphasis.<br /> :- B.K.<br /> <br /> <br /> :There is a big necessity to think outside the box for the survival of a lot of events - clearly, also, there are too many events being maintained purely because they've been around for many years and there are ego's to maintain - let's cut some out and start again.<br /> :- T.A.<br /> <br /> <br /> :Software engineering does NOT do double blind. I don't think this would benefit the community - it's pretty easy to guess the research group from the paper. It may help junior reviewers rate poor papers by well know folks lower! But this is really a job for senior reviewers.<br /> :- J.M.<br /> <br /> <br /> :Double-blind reviews are standard, as are fairly large committees to permit the inclusion of specialized reviewers (also external reviewers). Program committee submission restrictions are usually handled by declaring potential conflicts of interest (on the part of the authors and/or the program chair). We also practice paper shepherding (which can be difficult to arrange given the additional workload this imposes on a PC member) to ensure that reviewer concerns are addressed by the authors in a constructive dialog. However, there is clearly a limit to how frequently this instrument can be used.<br /> :- S.W.<br /> <br /> <br /> :We have not practiced double blind submissions, although I personally think it would be a good idea. We also permit PC members to submit papers, and it falls to the PC chair to insure that there is no conflict of interest in assigning reviewers for these papers. When there are widely differing evaluations of a given paper, we send all reviews of those papers to the cohort of reviewers, and instigate a discussion between the reviewers to see if a closer consensus can be found. This has worked in some cases, but there have been some glaring failures. As mentioned earlier, we have been trying to increase our PC size, but this is difficult to do. We also like to change at least 1/3 of the PC members from year to year, in order to avoid becoming too narrowly focused.<br /> :- V.M.<br /> <br /> <br /> :We form PC from the people who are active scientists, are active professors, are heads of scientific schools, are representatives of the industry, which delivers innovations, inviting their colleges to the conference, submit own papers to the conference, and review papers.<br /> :- V.H.<br /> <br /> <br /> :Automation. Use manuscript submission and review systems that are not too expensive. Some charges are very high. Philosophy and practice of organization evolve all the time from conference to conference. I don't see any particular thing hurting the community definitively or intentionally. We are all volunteers operating within constraints of time and energy. QUESTION: Shall we look into the emerging problems with some low-end (I mean: very low) conferences that are positioned well down the food chain and seem to compete with main stream conferences (or not competitive at all), many even carrying identical titles with existing ones? Have we looked into these new kids on the block? How well are they actually doing? (Of course we won't spend time or make a trip to see the events.)<br /> :- C.K.C.<br /> <br /> <br /> :I start thinking that some restrictions to program committee submissions could also be helpful in medium and big size conferences.<br /> :- M.L.<br /> <br /> <br /> :Double blind submissions may work better than other measures. The same goes for PC submission restrictions. However, in most conferences I am familiar with or involved, there are no such restrictions. Rebuttals were used once in a big conference that I know (Infocom), but they were not used again (I was not on the PC, so I do not know the exact reasons why). In general, I believe that a large size of the PC committee is not an advantage. However, a small PC does not help either, because it leads to high load of PC members who have to distribute the reviews to a large number of reviewers finally. Overall, I believe that large PC's hurt more than they benefit quality. I believe that PC submission restrictions and double blind reviews will help, because they will motivate change of PC committees every year for good conferences and enable more objective review of papers.<br /> :- D.S.<br /> <br /> <br /> :They certainly help. It gives me greater confidence in the whole process. Quality of reviewing is good (less load due to larger committees)and more fair (double blind submissions, rebuttals).<br /> :- R.G.<br /> <br /> <br /> '''WCRE / ASE'''<br /> :Of the main SE venues I'm involved in there are no PC submission restrictions or rebuttals. There are subcommittees for demo submissions, working sessions, doctoral sessions etc. But not subcommittees for areas/topics.<br /> :- J.M.<br /> <br /> <br /> '''ICSM/ICPC'''<br /> :There has been a general move to only allow PC members to server for 3 years in a row. They must take at least one year off after three. Also, these venues require some new faces on the PC along with a majority of previous members. That is reassure continuity and still bring in new researchers.<br /> :- J.M.<br /> <br /> <br /> '''ITC''' <br /> :The number of papers track the health of the industry more than anything. ITC is far more than a technical conference, since we are part of what we call Test Week, with tutorials before, exhibits during and workshops after, all coordinated by the ITC Steering Committee. This coordination has helped. However, industry economics has by far the biggest role.<br /> :- S.D.<br /> <br /> <br /> '''IPDPS, SC, HiPC, ICPP, SPAA''' <br /> :In the parallel and distributed computing community, almost always the reviewing process is single-blind (reviewers are anonymous to the authors). Program committee members are encouraged to submit papers; however, the threshold for acceptance for a PC member is elevated so that it must be a clear and strong accept, rather than a borderline paper. The PC member may not enter into deliberations on his/her own paper. We have not practiced rebuttals, but this is in active discussion with IPDPS. Also, our larger conferences (IPDPS, SC, HiPC) have had tracked program subcommittees for a number of years, mostly due to the broad range of areas. Increasingly so, these tracks are harder to define as silos, and many papers span 2, 3, 4, or more tracks in topic!<br /> :- D.B.</div> Thu, 09 Mar 2006 18:30:48 GMT MarkDHill http://wiki.acm.org/healthcc/index.php?title=Talk:Q3:_IEEE_Conferences Q1: IEEE Conferences http://wiki.acm.org/healthcc/index.php?title=Q1:_IEEE_Conferences <p>Summary: </p> <hr /> <div>===Question 1: REVIEWER LOAD.=== <br /> Has your community recently adopted new practices to deal with growing reviewer load, such as:<br /> <br /> *tracking reviews of rejected papers from conference to conference as is done in journal reviewing<br /> *increasing program committee size<br /> *charging a review fee<br /> *others?<br /> <br /> For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.<br /> <br /> <br /> '''ACM/IEEE'''<br /> :Increase PC size works but not ideally. A first round of screening to remove obviously lousy papers (e.g. too many or too few pages) would also work.<br /> :-K.Z.<br /> <br /> <br /> :Increasing PC size appears to be effective as each member sees the work involved in reviewing for one conference as less. However, one tends to be involved in more PCs.<br /> :-D. W.<br /> <br /> <br /> :My favorite suggestion is to use text mining to automatically identify 3-5 published papers that are closest to the current submission (over the last 5 years of published papers). These can be forwarded to reviewers along with the reviewed paper.<br /> :-I.M.<br /> <br /> <br /> :At point of invitation, it is now common to negotiate how many papers you can commit to review. I do this all the time now and it works. I know others who do the same. By limiting the # of reviews upfront, the process becomes doable and expectations are not mismatched.<br /> :-S.H.<br /> <br /> <br /> :Over the past decade we have increased program committee size. We try to reduce reviewer load by recruiting new reviewers, and by emphasizing that each program committee member must get a reviewer's consent before sending a review. We have just implemented a mechanism for reviewers to reject reviewers, with a reason. We hope this will reduce the number of unreturned reviews, and allow us to monitor committee members who are not getting in touch with reviewers in advance.<br /> :-S.D.<br /> <br /> <br /> :Reviewer load with IEEE CBMS increases each year indeed due to the increasing number of submissions. We increase Programme Committee size each year, inviting new members well-recognized in the field - it helps to keep the reviewer load reasonable. I believe it is the only solution to the problem.<br /> :-A.T.<br /> <br /> <br /> :Improve the quality and efficiency of web services, using work group practices. <br /> :-S.J.<br /> <br /> <br /> :I think the Software Engineering (IEEE Computer Society) workshops and conferences are still doing fairly conventional reviewing, so maybe our program committees are larger - but that's about it.<br /> :-C.B.<br /> <br /> <br /> :We do notice, based on reviewer’s involvement with multiple conferences, that we are seeing an increased number of papers being submitted to multiple conferences simultaneously. You can see that the number of reviewers has increased and then remained constant. 2001 was an exceptionally small committee however. A significant reason for the increase is also we have extended the areas the conference supports. <br /> :-B.K.<br /> <br /> <br /> :We distributed papers among PC members. No special measures taken.<br /> :-V.C.<br /> <br /> <br /> :We already have a large program committee for most of our events that I'm associated with - in many cases I think it is too large and is a practice that only serves to attract papers from the program committee.<br /> :-A.A.<br /> <br /> <br /> :We have increased PC size to keep up with submissions (at least to some extent), and are -- given the fact that information security /information assurance is a relatively small research community -- there is informal tracking of rejected publications in instances when PC members have come across similar (or even identical) submissions before. Review fees are highly impractical in my personal opinion since this raises an additional barrier for international submissions, particularly from countries with limited financial resources for supporting research.<br /> :-S.W.<br /> <br /> <br /> :We had tried to increase the size of our program committee, but it is not easy. Many of the most desirable PC members serve on several committees, so these reviewers are already severely overloaded. On the plus side, we do allow our PC members to recruit colleagues to take on individual papers, so that most PC members do not personally review all the papers which are assigned to them. <br /> :-V.M.<br /> <br /> <br /> :Review is an honorary responsibility of every scientist. It is done for free. We do tracking of rejected papers. For our 3 year history we have about 10% of rejected papers. <br /> :-V.H.<br /> <br /> <br /> :USE PHD STUDENTS. INCREASE PC SIZE. It helped reduce average review load. It helped increase conference attendance. It helped build the sense of community. It looked funny because of the size of PC vs. the size of attendance, except for a few major conferences.<br /> :-C.K.K.<br /> <br /> <br /> :Program committees increase every year. The increase of the PC can work well, keeping manageable, if members are grouped around a topic that has a particular chair. <br /> :-M.L.<br /> <br /> <br /> :As I understand, they have increased PC size. Review fee has been discussed once in one conference, but I have not seen it implemented anywhere.<br /> :-D.S.<br /> <br /> <br /> :Increase in committee size has helped reduce load which has enabled continued high standards in reviewing. There are plenty of qualified people who are willing to serve on the PCs so this has worked out well.<br /> :-R.G.<br /> <br /> <br /> '''IPDPS, SC, HiPC, ICPP, SPAA'''<br /> :The most common practice has been to increase the PC size. However, the PC is normally formed before the paper submission deadline, so it is problematic when an unusually high number of submissions are received that surpass historic records. For instance, IPDPS 2006 received approximately 200 more submissions than the previous year. Other practices were started, such as in HiPC, where an initial pass through the papers is taken to make a &quot;quick reject&quot; decision on any paper that is clearly A) outside of scope, B) plagiarized, C) contains no scholarly content. This reduces the number of papers in the &quot;regular review&quot; and the reviewers' load. However, care must be taken to reject only those papers that are &quot;obvious&quot; rejects for administrative reasons, and not for ones where a single person is evaluating the novelty/originality/significance/quality of the technical contribution.<br /> :-D.B.<br /> <br /> <br /> '''ASE'''<br /> :The size of the PC for ASE was increased recently to reflect the large number of submission. For ASE'05 PC members had 18-20 papers to review. Otherwise, no specific actions have been taken to address reviewer load. A couple of venues have taken steps to assure workload is between 6 and 10 paper to review. So the size of the PC is determined by the number of submission expected (from historical data). A review fee may be an interesting option - but in general I've not run into serious problems with this issue. If PC members are allowed to bid on papers to review this normally gets solved. That is, if I review a paper in one conference and reject it, than see it again submitted in another venue I bid on it (easy review) and get to see if it was improved.<br /> :-J.M.</div> Thu, 09 Mar 2006 18:23:16 GMT MarkDHill http://wiki.acm.org/healthcc/index.php?title=Talk:Q1:_IEEE_Conferences Q2: IEEE Conferences http://wiki.acm.org/healthcc/index.php?title=Q2:_IEEE_Conferences <p>Summary: </p> <hr /> <div>===Question 2: NON-INCREMENTAL===<br /> Has your community recently adopted new practices to promote non-incremental new ideas?<br /> <br /> *big ideas sessions<br /> *more papers<br /> *shorter papers<br /> *deemphasizing detailed evaluation<br /> *others?<br /> <br /> For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.<br /> <br /> <br /> '''ACM/IEEE'''<br /> :Panels of discussion, that only publishes summaries, are a useful practice.<br /> :- K.Z.<br /> <br /> <br /> :Not really, that's what the workshops are good for. Short papers will only undermine the quality of submissions and make the review process more arbitrary. <br /> :- I.M.<br /> <br /> <br /> :Rejecting papers with good ideas based on one negative review hurt the community. Also rejecting papers from Asia because they are not well written (English problem) is also hurting the community. I have seen so many good idea papers that are not well written. The benefit of presenting the idea to the community is lost by rejecting the paper on presentation ground or by rejecting the paper based on one negative. This is something I would lie to see change.<br /> :-S.H.<br /> <br /> <br /> :We are not doing any of these things. We introduce new topics to the Conference through a panel to increase interest, and then, possibly, a special session. We do have a lecture and application series, where the reviewing is less formal, to provide specific knowledge to our attendees. However, we've never had an issue refusing &quot;big ideas&quot; - many best papers in the past have been in this category.<br /> :- S.D.<br /> <br /> <br /> :We use the practice of Special Tracks. Each year we have a Call for Special Tracks - so that experts representing new emerging areas could organize special sessions on some particular topics besides the usual regular sessions. Besides having a separate programme and review committee, Special Tracks have no difference from the regular sessions (the presentation time is the same, and the publication length in the proceedings is the same). I believe Special Tracks significantly help to promote new emerging sub-areas and ideas at the conference.<br /> :- A.T.<br /> <br /> <br /> :Some workshops/conferences have a 6 page limit; but shorter than that most people will not want to come to the conference to give a short paper.<br /> :- C.B.<br /> <br /> <br /> :In addition to technical papers, the conference include about 40-50 tutorials, 4 invited speakers for Plenary sessions, one Keynote, and 12 invited &quot;Masterworks&quot; presentations. The masterworks sessions and invited speakers are intended to provided targeted &quot;big issues&quot; and also bring to the conference highly regarded people who are leaders of the different communities. This works very well. <br /> :- B.K.<br /> <br /> <br /> :Have a special session dedicated to new ideas that are untested and futuristic.<br /> :- V.C.<br /> <br /> <br /> :We are consistently trying to adopt this practice to make the conference(s) much more meaningful to attendees. Sessions of only submitted papers are not enough any more - this way we can ensure the input from experienced people who might not otherwise have submitted papers. We can add very controversial topics that no-one else would have considered adopting. Already the conferences seem to be too big. Let's maintain standards, but also go back to considering poster sessions.<br /> :- A.A.<br /> <br /> <br /> :The security community has had the NSPW (new security paradigms workshop, this used to be an ACM conference and is now sponsored by ACSA) for more than a decade, and there has been a proliferation of papers, often also scattered across several conferences, which blunted the impact. CRA recently hosted a `Grand Challenges' workshop in the field, but that turned out to be more of a consensus group. We (IEEE TFIA) try to encourage this type of work by organizing smaller max. 60 participants) single-track workshops which are to provide a richer and more focused forum for interaction and debate among participants, but on balance I'm not convinced that the present approach is truly enjoying encouraging success in supporting or generating truly disruptive ideas.<br /> :- S.W.<br /> <br /> <br /> :We adopted accepting short papers (4 pages or less) in 2005. It seems to be working well. We still accept full papers (limited to 12 pages single-spaced). About 20% of the submissions have been short papers, and, while we have not set any quotas for acceptance, about 20% of the accepted papers for our 2005 conference were short papers. <br /> :- V.M.<br /> <br /> <br /> :We are successfully using practice of having approximately equal(about 30%) groups of papers/tutorials from scientists from East (EU,USA, …), from scientist from West (Ukraine, Russia, Armenia, Belorus,…), and from students (PhD’s are in this group too).<br /> :- V.H.<br /> <br /> <br /> :Co-locating; no fixed partners. Yet to see the long-term effectiveness.<br /> :- C.K.K.<br /> <br /> <br /> :Short papers and posters are becoming more popular. Those are no more considered &quot;devaluated&quot; publications. I’m aware of just one big idea session, held along with VTS and called Elevator Talks. Too early to say if this is really a good idea. Seems to be, but I’m not sure that the number of submissions is high enough for this sort of session.<br /> :- M.L.<br /> <br /> <br /> :In very few conferences there are &quot;new ideas&quot; sessions. I don't think they have worked as expected though.<br /> :- D.S.<br /> <br /> <br /> :The big idea sessions have just been introduced in CFPs this year. We will have to wait and see if the published papers actually contain big ideas or simply poorly evaluated small ideas. Some conferences have accepted increased number of papers. However, this is a modest increase (e.g., 32 instead of 28). While this is a good idea, this will have only small benefit.<br /> :- R.G.<br /> <br /> <br /> '''OOPSLA''' <br /> <br /> :OOPSLA has a &quot;big idea&quot; session. It's called Onward! This has a subcommittee (I was on it once) to review the submissions. The accepted papers get a one hour presentation slot. Many software engineering (SE) venues have promoted workshops that have newer ideas - things less mature or not completely evaluated. Also most SE venues have short papers.<br /> :- J.M.<br /> <br /> <br /> '''SC 05'''<br /> <br /> :In SC 05 we went to 5 parallel tracks for the technical papers. This was done not to increase the technical papers but to expand other aspects of the conference, including an additional award sessions, special sessions associated with the themes of the conference, etc. The chart shows that actually the number of technical papers declined slightly and then stayed about the same. Several years ago, we tried the concept of &quot;extended abstracts&quot; for selection and the having the full papers done for only the accepted papers. This was done mostly to help authors - but it was found that it causes several problems. First, there was not enough detail to fully evaluate some papers. Second, we felt getting the final papers in on time is more challenging. In SC05 we had full papers submitted and it seemed to work well. This appears to work well.<br /> :- B.K.<br /> <br /> <br /> '''IPDPS, SC, HiPC, ICPP, SPAA''' <br /> <br /> :I have not seen this practice yet in the parallel and distributed computing community.<br /> :- D.B.</div> Thu, 09 Mar 2006 18:20:45 GMT MarkDHill http://wiki.acm.org/healthcc/index.php?title=Talk:Q2:_IEEE_Conferences