Q2: Medium Conferences
From Health of Conferences Committee
(Difference between revisions)
| Revision as of 20:17, 20 February 2006 MarkDHill (Talk | contribs) ← Previous diff |
Revision as of 15:40, 21 February 2006 MarkDHill (Talk | contribs) Next diff → |
||
| Line 9: | Line 9: | ||
| For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. | For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. | ||
| + | |||
| + | |||
| + | '''SIGART''' | ||
| + | :We have increased the number of papers accepted by adding parallel sessions. | ||
| + | |||
| + | |||
| + | '''SIGARCH''' | ||
| + | :big ideas sessions | ||
| + | :No | ||
| + | :more papers | ||
| + | :Yes | ||
| + | :shorter papers | ||
| + | :No | ||
| + | :deemphasizing detailed evaluation | ||
| + | :No (Some calls for papers and PC directives may say so, but in practice not.) | ||
| + | :others? | ||
| + | :None. (But ASPLOS has a wild and crazy ideas session.) | ||
| + | |||
| + | |||
| + | '''SIGKDD''' | ||
| + | :big ideas sessions | ||
| + | :Tried with moderate success, but we keep trying | ||
| + | :more papers | ||
| + | :NO | ||
| + | :shorter papers | ||
| + | :We have both a long and short papers | ||
| + | :deemphasizing detailed evaluation | ||
| + | :others? | ||
| + | :We introduced KDD Cup whose goal was to provide competition which would practically evaluate many approaches to the same problem | ||
| + | :For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. | ||
| + | |||
| + | |||
| + | '''SIGARCH''' | ||
| + | :I adding more tracks to conferences (to get more good papers accepted) is a reasonable approach. This lets conference attendees select the papers most relevant to their interests. ISCA moved from 1.5-track to 2-track in 2005, and although there are inevitably some conflicts in terms of desired session attendance for some attendees, the field is diverse enough that it usually works out pretty well. | ||
| + | |||
| + | |||
| + | '''SIGCHI''' | ||
| + | :SEVERAL OF OUR CONFERENCES HAVE INTRODUCED A "NOTES" FORMAT -- A SHORT PAPER THAT IS DESIGNED FOR SMALLER BUT INNOVATIVE CONTRIBUTIONS (SINCE SUCH SMALLER IDEAS OFTEN LACK THE FULL CONTEXT AND EVALUATION THAT IS EXPECTED IN FULL PAPERS). SOME CONFERENCES HAVE DIFFERENT VENUES FOR EMERGING WORK (POSTERS, DEMOS) INCLUDING BRIEF TIME IN PLENARY SESSIONS (OR PARALLEL SESSIONS) FOR AN OVERVIEW OF THAT WORK (E.G., ONE-MINUTE POSTER MADNESS). THE DESIGN-ORIENTED CONFERENCES WE SPONSOR HAVE BEEN CREATIVE IN CREATING NEW VENUES FOR DESIGN CASES (COMPETITIONS, CASE STUDIES, ETC.). | ||
| + | |||
| + | |||
| + | '''SIGIR''' | ||
| + | :We are accepting mroe papers. Our acceptance rates are holding right around 20%. So far we have been able to manage this by just reminding Area Coordinators that we're aiming for consistent quality. We don't have an official target number - some years it may be a little higher or lower - but we don't want the conference to drop below 15% without having a very clear understanding of why (e.g., maybe we have a large influx of weak or off-topic papers). | ||
| + | |||
| + | :We have discussed big ideas sessions, etc. SIGIR is skeptical about anybody's ability to distinguish between big ideas that work and big ideas that don't without proper experiments. 9 out of 10 of my good ideas are wrong - that's why we do the experiments. SIGIR's culture is rooted very strongly in experimental validation - I don't see that changing anytime soon. | ||
| + | |||
| + | :We run a Posters track. It has about a 40% acceptance rate, and each poster gets 2 pages in the proceedings. It used to be considered low value, but has gained in prestige in recent years; some feel that it is the most interesting part of the program. There is some discussion about migrating these to short papers without presentation. | ||
| + | |||
| + | |||
| + | '''SIGACT''' | ||
| + | :There have been very few changes to STOC over the years. We increased the number of papers some time ago by introducing parallel sessions. We have tutorials and plenary speakers. STOC is a very prestigious conference, so there is little incentive to "lower the quality" by having anything other than technical talks. | ||
| + | |||
| + | |||
| + | '''SIGPLAN''' | ||
| + | :There has been a trend to try to accept more papers, aiming for acceptance rates more than 20%. Presentations have occasionally been shortened from 30 to 25 minutes to accommodate the larger number of accepted papers. | ||
| + | |||
| + | :There have been proposals for more categories of accepted papers, ie, short and long papers. This approach has not yet been tried in any of our mainline conferences. | ||
| + | |||
| + | :People were happy with the larger number of papers at the most recent POPL and didn't seem to mind the slightly shorter presentations. There is not a consensus that we need to accept significantly more papers, although senior people in the field tend to think we should be accepting more "wild-idea" papers and fewer "well-polished bricks." | ||
| + | |||
| + | :There also seems to be a sense that the conference/journal system is broken. At least a vocal minority think that our community place too much importance on conference papers. This group thinks we need to improve the journal response rate, make journal publication meaningful, and increase the acceptance rate at conferences significantly. | ||
| + | |||
| + | |||
| + | '''SIGMOD''' | ||
| + | :The VLDB conference has a special class of papers called "vision papers" for this purpose. The CIDR conference was explicitly organized to help disseminate half-baked big ideas, rather than fully worked out (big or small) ideas. ICDE and others accept a new category of short papers; but this is not limited to "big ideas". Surajit has been explicitly asking SIGMOD PC members to push for papers with non-incremental ideas even if they are borderline because of technical limitations (in contrast to polished papers that make incremental contributions). | ||
| + | |||
| + | :Frankly, how well these work in unearthing big ideas is a question I don't have a good feel for. Nonetheless, I think it is important for the research community to recognize that non-incremental ideas are valued, and I think we're accomplishing this. | ||
| + | |||
| + | |||
| + | '''SIGCOMM''' | ||
| + | :We experimented with having position papers at our main conference, with limited success; we ultimately abandoned that idea in favor of the co-located workshops. Plus, we have a workshop on Hot Topics in Networking that emphasize "idea" papers over evaluation. This year at SIGCOMM'06 we plan to accept more papers (and have correspondingly shorter talks). Our Internet Measurement Conference has a mix of short (6-page) and long (12-14 page) papers. Our SIG newsletter also has a mix of long and short (i.e., editorial-zone) papers. | ||
| + | |||
| + | :For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community. | ||
| + | |||
| + | :The short papers at IMC were a big success -- and especially good for measurement work, where sometimes folks have a cute little idea or measurement result that doesn't warrant a full-length paper but would be nice to get out to the community. The recent changes in our SIG newsletter (to include an "editorial zone") has been a big energizer. Loads of fun to read, with opinions, recommended reading, background on research institutes/labs, comparisons of past work, gossip column, etc. HotNets last year started having "public reviews" for each paper, something that our newsletter has recently adopted for the long papers, and SIGCOMM'06 will do this year. These are also loads of fun, and help in providing broader context and values; many people have mentioned to me that they use these public reviews in their classes. | ||
| + | |||
| + | :Our community has struggled recently with whether events (especially the HotNets workshop) can have limited attendance. See the discourse online in the archive at http://www.postel.org/pipermail/sigcomm/ or join the list at http://www.postel.org/sigcomm/ | ||
| + | |||
| + | :This list, I think, has been useful for getting the community more engaged, and having follow-up discussions for topics raised at our SIGCOMM business meeting. | ||
| + | |||
| + | |||
| + | '''SIGKDD''' | ||
| + | :Workshops help to promote new ideas quickly. Frequently the workshops are the most exciting parts of the conferences. Best papers awards highlight new ideas. The tradition review format can penalize new ideas that are controversial and this problem is perhaps exacerbated with the larger volume of papers. Poster presentations allow more papers to be accepted and gain exposure. Poster do seem to work well. | ||
Revision as of 15:40, 21 February 2006
Question 2: NON-INCREMENTAL
Has your community recently adopted new practices to promote non-incremental new ideas?
- big ideas sessions
- more papers
- shorter papers
- deemphasizing detailed evaluation
- others?
For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.
SIGART
- We have increased the number of papers accepted by adding parallel sessions.
SIGARCH
- big ideas sessions
- No
- more papers
- Yes
- shorter papers
- No
- deemphasizing detailed evaluation
- No (Some calls for papers and PC directives may say so, but in practice not.)
- others?
- None. (But ASPLOS has a wild and crazy ideas session.)
SIGKDD
- big ideas sessions
- Tried with moderate success, but we keep trying
- more papers
- NO
- shorter papers
- We have both a long and short papers
- deemphasizing detailed evaluation
- others?
- We introduced KDD Cup whose goal was to provide competition which would practically evaluate many approaches to the same problem
- For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.
SIGARCH
- I adding more tracks to conferences (to get more good papers accepted) is a reasonable approach. This lets conference attendees select the papers most relevant to their interests. ISCA moved from 1.5-track to 2-track in 2005, and although there are inevitably some conflicts in terms of desired session attendance for some attendees, the field is diverse enough that it usually works out pretty well.
SIGCHI
- SEVERAL OF OUR CONFERENCES HAVE INTRODUCED A "NOTES" FORMAT -- A SHORT PAPER THAT IS DESIGNED FOR SMALLER BUT INNOVATIVE CONTRIBUTIONS (SINCE SUCH SMALLER IDEAS OFTEN LACK THE FULL CONTEXT AND EVALUATION THAT IS EXPECTED IN FULL PAPERS). SOME CONFERENCES HAVE DIFFERENT VENUES FOR EMERGING WORK (POSTERS, DEMOS) INCLUDING BRIEF TIME IN PLENARY SESSIONS (OR PARALLEL SESSIONS) FOR AN OVERVIEW OF THAT WORK (E.G., ONE-MINUTE POSTER MADNESS). THE DESIGN-ORIENTED CONFERENCES WE SPONSOR HAVE BEEN CREATIVE IN CREATING NEW VENUES FOR DESIGN CASES (COMPETITIONS, CASE STUDIES, ETC.).
SIGIR
- We are accepting mroe papers. Our acceptance rates are holding right around 20%. So far we have been able to manage this by just reminding Area Coordinators that we're aiming for consistent quality. We don't have an official target number - some years it may be a little higher or lower - but we don't want the conference to drop below 15% without having a very clear understanding of why (e.g., maybe we have a large influx of weak or off-topic papers).
- We have discussed big ideas sessions, etc. SIGIR is skeptical about anybody's ability to distinguish between big ideas that work and big ideas that don't without proper experiments. 9 out of 10 of my good ideas are wrong - that's why we do the experiments. SIGIR's culture is rooted very strongly in experimental validation - I don't see that changing anytime soon.
- We run a Posters track. It has about a 40% acceptance rate, and each poster gets 2 pages in the proceedings. It used to be considered low value, but has gained in prestige in recent years; some feel that it is the most interesting part of the program. There is some discussion about migrating these to short papers without presentation.
SIGACT
- There have been very few changes to STOC over the years. We increased the number of papers some time ago by introducing parallel sessions. We have tutorials and plenary speakers. STOC is a very prestigious conference, so there is little incentive to "lower the quality" by having anything other than technical talks.
SIGPLAN
- There has been a trend to try to accept more papers, aiming for acceptance rates more than 20%. Presentations have occasionally been shortened from 30 to 25 minutes to accommodate the larger number of accepted papers.
- There have been proposals for more categories of accepted papers, ie, short and long papers. This approach has not yet been tried in any of our mainline conferences.
- People were happy with the larger number of papers at the most recent POPL and didn't seem to mind the slightly shorter presentations. There is not a consensus that we need to accept significantly more papers, although senior people in the field tend to think we should be accepting more "wild-idea" papers and fewer "well-polished bricks."
- There also seems to be a sense that the conference/journal system is broken. At least a vocal minority think that our community place too much importance on conference papers. This group thinks we need to improve the journal response rate, make journal publication meaningful, and increase the acceptance rate at conferences significantly.
SIGMOD
- The VLDB conference has a special class of papers called "vision papers" for this purpose. The CIDR conference was explicitly organized to help disseminate half-baked big ideas, rather than fully worked out (big or small) ideas. ICDE and others accept a new category of short papers; but this is not limited to "big ideas". Surajit has been explicitly asking SIGMOD PC members to push for papers with non-incremental ideas even if they are borderline because of technical limitations (in contrast to polished papers that make incremental contributions).
- Frankly, how well these work in unearthing big ideas is a question I don't have a good feel for. Nonetheless, I think it is important for the research community to recognize that non-incremental ideas are valued, and I think we're accomplishing this.
SIGCOMM
- We experimented with having position papers at our main conference, with limited success; we ultimately abandoned that idea in favor of the co-located workshops. Plus, we have a workshop on Hot Topics in Networking that emphasize "idea" papers over evaluation. This year at SIGCOMM'06 we plan to accept more papers (and have correspondingly shorter talks). Our Internet Measurement Conference has a mix of short (6-page) and long (12-14 page) papers. Our SIG newsletter also has a mix of long and short (i.e., editorial-zone) papers.
- For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.
- The short papers at IMC were a big success -- and especially good for measurement work, where sometimes folks have a cute little idea or measurement result that doesn't warrant a full-length paper but would be nice to get out to the community. The recent changes in our SIG newsletter (to include an "editorial zone") has been a big energizer. Loads of fun to read, with opinions, recommended reading, background on research institutes/labs, comparisons of past work, gossip column, etc. HotNets last year started having "public reviews" for each paper, something that our newsletter has recently adopted for the long papers, and SIGCOMM'06 will do this year. These are also loads of fun, and help in providing broader context and values; many people have mentioned to me that they use these public reviews in their classes.
- Our community has struggled recently with whether events (especially the HotNets workshop) can have limited attendance. See the discourse online in the archive at http://www.postel.org/pipermail/sigcomm/ or join the list at http://www.postel.org/sigcomm/
- This list, I think, has been useful for getting the community more engaged, and having follow-up discussions for topics raised at our SIGCOMM business meeting.
SIGKDD
- Workshops help to promote new ideas quickly. Frequently the workshops are the most exciting parts of the conferences. Best papers awards highlight new ideas. The tradition review format can penalize new ideas that are controversial and this problem is perhaps exacerbated with the larger volume of papers. Poster presentations allow more papers to be accepted and gain exposure. Poster do seem to work well.
