Q2: Medium Conferences

From Health of Conferences Committee

Question 2: NON-INCREMENTAL

Has your community recently adopted new practices to promote non-incremental new ideas?

• big ideas sessions
• more papers
• shorter papers
• deemphasizing detailed evaluation
• others?

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

SIGART

We have increased the number of papers accepted by adding parallel sessions.

SIGARCH

big ideas sessions

No

more papers

Yes

shorter papers

No

deemphasizing detailed evaluation

No (Some calls for papers and PC directives may say so, but in practice not.)

others?

None. (But ASPLOS has a wild and crazy ideas session.)

SIGKDD

big ideas sessions

Tried with moderate success, but we keep trying

more papers

NO

shorter papers

We have both a long and short papers

deemphasizing detailed evaluation

others?

We introduced KDD Cup whose goal was to provide competition which would practically evaluate many approaches to the same problem

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

SIGARCH

I adding more tracks to conferences (to get more good papers accepted) is a reasonable approach. This lets conference attendees select the papers most relevant to their interests. ISCA moved from 1.5-track to 2-track in 2005, and although there are inevitably some conflicts in terms of desired session attendance for some attendees, the field is diverse enough that it usually works out pretty well.

SIGCHI

SEVERAL OF OUR CONFERENCES HAVE INTRODUCED A "NOTES" FORMAT -- A SHORT PAPER THAT IS DESIGNED FOR SMALLER BUT INNOVATIVE CONTRIBUTIONS (SINCE SUCH SMALLER IDEAS OFTEN LACK THE FULL CONTEXT AND EVALUATION THAT IS EXPECTED IN FULL PAPERS). SOME CONFERENCES HAVE DIFFERENT VENUES FOR EMERGING WORK (POSTERS, DEMOS) INCLUDING BRIEF TIME IN PLENARY SESSIONS (OR PARALLEL SESSIONS) FOR AN OVERVIEW OF THAT WORK (E.G., ONE-MINUTE POSTER MADNESS). THE DESIGN-ORIENTED CONFERENCES WE SPONSOR HAVE BEEN CREATIVE IN CREATING NEW VENUES FOR DESIGN CASES (COMPETITIONS, CASE STUDIES, ETC.).

SIGIR

We are accepting mroe papers. Our acceptance rates are holding right around 20%. So far we have been able to manage this by just reminding Area Coordinators that we're aiming for consistent quality. We don't have an official target number - some years it may be a little higher or lower - but we don't want the conference to drop below 15% without having a very clear understanding of why (e.g., maybe we have a large influx of weak or off-topic papers).

We have discussed big ideas sessions, etc. SIGIR is skeptical about anybody's ability to distinguish between big ideas that work and big ideas that don't without proper experiments. 9 out of 10 of my good ideas are wrong - that's why we do the experiments. SIGIR's culture is rooted very strongly in experimental validation - I don't see that changing anytime soon.

We run a Posters track. It has about a 40% acceptance rate, and each poster gets 2 pages in the proceedings. It used to be considered low value, but has gained in prestige in recent years; some feel that it is the most interesting part of the program. There is some discussion about migrating these to short papers without presentation.

SIGACT

There have been very few changes to STOC over the years. We increased the number of papers some time ago by introducing parallel sessions. We have tutorials and plenary speakers. STOC is a very prestigious conference, so there is little incentive to "lower the quality" by having anything other than technical talks.

SIGPLAN

There has been a trend to try to accept more papers, aiming for acceptance rates more than 20%. Presentations have occasionally been shortened from 30 to 25 minutes to accommodate the larger number of accepted papers.

There have been proposals for more categories of accepted papers, ie, short and long papers. This approach has not yet been tried in any of our mainline conferences.

People were happy with the larger number of papers at the most recent POPL and didn't seem to mind the slightly shorter presentations. There is not a consensus that we need to accept significantly more papers, although senior people in the field tend to think we should be accepting more "wild-idea" papers and fewer "well-polished bricks."

There also seems to be a sense that the conference/journal system is broken. At least a vocal minority think that our community place too much importance on conference papers. This group thinks we need to improve the journal response rate, make journal publication meaningful, and increase the acceptance rate at conferences significantly.

SIGMOD

The VLDB conference has a special class of papers called "vision papers" for this purpose. The CIDR conference was explicitly organized to help disseminate half-baked big ideas, rather than fully worked out (big or small) ideas. ICDE and others accept a new category of short papers; but this is not limited to "big ideas". Surajit has been explicitly asking SIGMOD PC members to push for papers with non-incremental ideas even if they are borderline because of technical limitations (in contrast to polished papers that make incremental contributions).

Frankly, how well these work in unearthing big ideas is a question I don't have a good feel for. Nonetheless, I think it is important for the research community to recognize that non-incremental ideas are valued, and I think we're accomplishing this.

SIGCOMM

We experimented with having position papers at our main conference, with limited success; we ultimately abandoned that idea in favor of the co-located workshops. Plus, we have a workshop on Hot Topics in Networking that emphasize "idea" papers over evaluation. This year at SIGCOMM'06 we plan to accept more papers (and have correspondingly shorter talks). Our Internet Measurement Conference has a mix of short (6-page) and long (12-14 page) papers. Our SIG newsletter also has a mix of long and short (i.e., editorial-zone) papers.

For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

The short papers at IMC were a big success -- and especially good for measurement work, where sometimes folks have a cute little idea or measurement result that doesn't warrant a full-length paper but would be nice to get out to the community. The recent changes in our SIG newsletter (to include an "editorial zone") has been a big energizer. Loads of fun to read, with opinions, recommended reading, background on research institutes/labs, comparisons of past work, gossip column, etc. HotNets last year started having "public reviews" for each paper, something that our newsletter has recently adopted for the long papers, and SIGCOMM'06 will do this year. These are also loads of fun, and help in providing broader context and values; many people have mentioned to me that they use these public reviews in their classes.

Our community has struggled recently with whether events (especially the HotNets workshop) can have limited attendance. See the discourse online in the archive at http://www.postel.org/pipermail/sigcomm/ or join the list at http://www.postel.org/sigcomm/

This list, I think, has been useful for getting the community more engaged, and having follow-up discussions for topics raised at our SIGCOMM business meeting.

SIGKDD

Workshops help to promote new ideas quickly. Frequently the workshops are the most exciting parts of the conferences. Best papers awards highlight new ideas. The tradition review format can penalize new ideas that are controversial and this problem is perhaps exacerbated with the larger volume of papers. Poster presentations allow more papers to be accepted and gain exposure. Poster do seem to work well.

SIGOS

We've been using big idea sessions (and big idea workshops) for some time. They are popular.

If anything, there is pressure to improve the quality of the papers rather than trying to increase non-incremental ideas in the flagship conferences.

We've tried to broaden some conferences (PODC and practical papers, SOSP and more formal papers), with mixed results.

In distributed computing, we've held a specific conference in Italy (FuDiCo) to look specifically at new ideas. The goal was to generate ideas that could be used to spur future research programs. There have been two, both popular.

SIGMOBILE

Since MobiCom 1999 (and each year since), we have included in the CFP a call for special "challenges" papers "that challenge the research community with revolutionary new approaches, technologies, or applications in the field of mobile computing and wireless networking. These 'challenge papers' should provide stimulating ideas or visions that may open up exciting avenues of far-reaching future research." This has been successful in some years, but in other years we don't get enough good submissions for these papers.

We also almost always have a poster session at our conferences, where the criteria for accepting a poster is different (generally) than for a full paper. For posters, we look more for new ideas that may not be finished pieces of work (which is generally needed for a paper). The poster sessions have been very successful and popular.

I'm not sure how more papers or shorter papers would address the problem of promoting non-incremental work, other than lessening the pressure on the whole paper acceptance process. We keep looking for ways to do this, but it is hard to fit more papers in the time for presentations.