Main Page
From Health of Conferences Committee
| Revision as of 14:26, 9 March 2006 MarkDHill (Talk | contribs) Talk: Two-phase Reviewing ← Previous diff |
Current revision Turing (Talk | contribs) References |
||
| Line 22: | Line 22: | ||
| Filed: //wiki.acm.org/healthcc | Filed: //wiki.acm.org/healthcc | ||
| - | Release 1.0 on XXX, 2006 | + | Release 1.0 on 31 May 2006 |
| Line 31: | Line 31: | ||
| ===Process=== | ===Process=== | ||
| - | The committee--with the members listed above--was formed in December 2005. It "met" for several conference calls to create this Wiki, the initial version of which was released in XXX 2006. Key steps were formulating questions to ask ACM SIG chairs and IEEE XXXs, mailing out the questionnaires, and compiling a short list of ideas to be recommended or avoided. | + | The committee--with the members listed above--was formed in December 2005. It "met" for several conference calls to create this Wiki, the initial version of which was released in May 2006. Key steps were formulating questions to ask ACM SIG chairs and IEEE TCs, mailing out the questionnaires, and compiling a short list of ideas to be recommended or avoided. |
| The goals are to unearth actionable ideas and reveal failed ideas. It is NOT our goal to produce summary statistics because we expect our audience is more interested in groups in situations similar to their own than in some average. | The goals are to unearth actionable ideas and reveal failed ideas. It is NOT our goal to produce summary statistics because we expect our audience is more interested in groups in situations similar to their own than in some average. | ||
| The next two sections present (a) selected ideas that the committee decided to highlight and (b) the complete survey results. Large conferences have more than one-thousand attendees, small conferences have less than 100, while medium conferences are in the middle. | The next two sections present (a) selected ideas that the committee decided to highlight and (b) the complete survey results. Large conferences have more than one-thousand attendees, small conferences have less than 100, while medium conferences are in the middle. | ||
| + | |||
| + | Since this page has been '''vandalized''', editing now requires a login and password that should be easy for computer scientists to figure out, but should, at least, stop bots: | ||
| + | |||
| + | * Click on "Create an account or log in" in upper left-hand corner of this page. | ||
| + | |||
| + | * Login: ****** (Highest award in computer science; first letter capitalized; remaining letters lower case). | ||
| + | |||
| + | * Password: ***** (Last name of the author of "The Art of Computer Programming"; first letter capitalized; remaining letters lower case). | ||
| ==Let's Talk About Selected Ideas== | ==Let's Talk About Selected Ideas== | ||
| - | This section lists several ideas that the committee decided to highlight. Please click on those of interest to see: | + | This section lists several ideas that the committee decided to highlight. Please click on topic of interest (e.g., ''Talk: Tracking Reviews'', but NOT the ''edit'' button the right) to see: |
| * Comments from survey or the committee | * Comments from survey or the committee | ||
| * Comments by others | * Comments by others | ||
| * Add a comment yourself! | * Add a comment yourself! | ||
| - | |||
| ===[[Talk: Tracking Reviews]]=== | ===[[Talk: Tracking Reviews]]=== | ||
| - | The most direct approach to tracking reviews has been undertaken by SIGMOD, where they have created a "pipeline" with another major database conference, VLDB, where it is common for some papers rejected at one conference are sent to the next, with their reviews carried over. The original reviewers continue to be involved. This is being done on a limited basis, only for borderline papers where it is felt that a round of author revision could lead to a solid contribution. If this is successful, SIGMOD might extend the pipeline to include IEEE's ICDE conference as well. The idea is that this will reduce repeated submission of borderline papers. More importantly, SIGMOD anticipates this will help good papers with specific problems that can be fixed, much as the journal reviewing process helps in such situations. | ||
| - | |||
| - | Often, such tracking is done informally in small communities. For example, SIGACT has at least one member for FOCS usually on STOC, so that a paper that it is submitted to both can be compared. | ||
| - | |||
| - | Another interesting idea is to track reviewers, to detect chronically irresponsible, weak or unfair reviewers. SIGIR is beginning to track reviewer quality, and is so far just in the data gathering stage. | ||
| - | |||
| - | Note that paper tracking is made more difficult by double blind reviewing. | ||
| - | |||
| ===[[Talk: Two-phase Reviewing]]=== | ===[[Talk: Two-phase Reviewing]]=== | ||
| - | Some conferences (e.g., SIGCOMM, SIGMOBILE,SIGMOD) have introduced a 2-phase review process where papers that are clearly not going to make it into the conference due to a serious flaw (out of scope, lacking an evaluation, clearly non-novel) are rejected with a less rigorous review than those that are competitive. This is a compromise that allows the maximum reviewing resources to be devoted to those papers that are in serious contention. For SIGCOMM, about 10-20% of papers are rejected at an early stage based on one PC member's view (the lead reviewer) and a double-check by the PC chairs. SIGMOD assigns all papers to two reviewers in Phase 1, and only papers with at least one positive reviewer are assigned a third reviewer. | ||
| - | |||
| ===[[Talk: Author Responses (Rebuttals)]]=== | ===[[Talk: Author Responses (Rebuttals)]]=== | ||
| ===[[Talk: Double-Blind Submissions]]=== | ===[[Talk: Double-Blind Submissions]]=== | ||
| Line 64: | Line 61: | ||
| ===[[Talk: Co-Located Workshops]]=== | ===[[Talk: Co-Located Workshops]]=== | ||
| ===[[Talk: Competitions]]=== | ===[[Talk: Competitions]]=== | ||
| + | |||
| + | |||
| ===[[Talk: Catch All]]=== | ===[[Talk: Catch All]]=== | ||
| + | |||
| Line 90: | Line 90: | ||
| [[Q1: Small Conferences]] | [[Q1: Small Conferences]] | ||
| + | |||
| + | [[Q1: IEEE Conferences]] | ||
| Line 108: | Line 110: | ||
| [[Q2: Small Conferences]] | [[Q2: Small Conferences]] | ||
| + | |||
| + | [[Q2: IEEE Conferences]] | ||
| Line 126: | Line 130: | ||
| [[Q3: Small Conferences]] | [[Q3: Small Conferences]] | ||
| + | |||
| + | [[Q3: IEEE Conferences]] | ||
| Line 144: | Line 150: | ||
| [[Q4: Small Conferences]] | [[Q4: Small Conferences]] | ||
| + | |||
| + | [[Q4: IEEE Conferences]] | ||
| Line 154: | Line 162: | ||
| [[Q5: Small Conferences]] | [[Q5: Small Conferences]] | ||
| + | |||
| + | [[Q5: IEEE Conferences]] | ||
| ==Acknowledgments== | ==Acknowledgments== | ||
| - | The committee acknowledges the constructive advice of David A. Patterson, MORE NAMES HERE. | + | The committee acknowledges the constructive advice of David A. Patterson. |
| - | + | ||
| - | The committee acknowledges the excellent work of Mike Marty, Caitlin Scopel, MORE NAMES HERE. | + | |
| + | The committee acknowledges the excellent support work of Adam Greenberg, Mike Marty, and Caitlin Scopel. | ||
| ==References== | ==References== | ||
| + | |||
| + | Mark D. Hill, Jean-Luc Gaudiot, Mary Hall, Joe Marks, Paolo Prinetto, and Donna Baglio, A Wiki for Discussing and Promoting Best Practices in Research, Communications of the ACM, Vol. 49, No. 9, pp. 63-64, September 2006. Points readers to this Wiki. | ||
| Mark D. Hill, Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience, April 2005, http://www.cs.wisc.edu/~markhill/pcadvice2005.html. | Mark D. Hill, Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience, April 2005, http://www.cs.wisc.edu/~markhill/pcadvice2005.html. | ||
Current revision
ACM/IEEE Health of Conferences Committee
MARK D. HILL (Chair), University of Wisconsin, ACM/SIGARCH & IEEE
JEAN-LUC GAUDIOT, UC Irvine, IEEE
MARY HALL, USC/ISI, ACM/SIGPLAN
JOE MARKS, Mitsubishi Electric Research Labs, ACM/SIGGRAPH
PAOLO PRINETTO, Politecnico di Torino, IEEE
DONNA BAGLIO, ACM Headquarters
Filed: //wiki.acm.org/healthcc
Release 1.0 on 31 May 2006
Charge from ACM President David Patterson
The idea is to collect the best practices onto a web page so that conference organizers can see innovative ways to cope with the demands of paper submissions, refereeing, and presentations, as the number of papers increase. The hope is that organizers will either try good new ideas or at least avoid the mistakes of others.
Process
The committee--with the members listed above--was formed in December 2005. It "met" for several conference calls to create this Wiki, the initial version of which was released in May 2006. Key steps were formulating questions to ask ACM SIG chairs and IEEE TCs, mailing out the questionnaires, and compiling a short list of ideas to be recommended or avoided.
The goals are to unearth actionable ideas and reveal failed ideas. It is NOT our goal to produce summary statistics because we expect our audience is more interested in groups in situations similar to their own than in some average.
The next two sections present (a) selected ideas that the committee decided to highlight and (b) the complete survey results. Large conferences have more than one-thousand attendees, small conferences have less than 100, while medium conferences are in the middle.
Since this page has been vandalized, editing now requires a login and password that should be easy for computer scientists to figure out, but should, at least, stop bots:
- Click on "Create an account or log in" in upper left-hand corner of this page.
- Login: ****** (Highest award in computer science; first letter capitalized; remaining letters lower case).
- Password: ***** (Last name of the author of "The Art of Computer Programming"; first letter capitalized; remaining letters lower case).
Let's Talk About Selected Ideas
This section lists several ideas that the committee decided to highlight. Please click on topic of interest (e.g., Talk: Tracking Reviews, but NOT the edit button the right) to see:
- Comments from survey or the committee
- Comments by others
- Add a comment yourself!
Talk: Tracking Reviews
Talk: Two-phase Reviewing
Talk: Author Responses (Rebuttals)
Talk: Double-Blind Submissions
Talk: Accepting More Papers
Talk: Hierarchical Program Committees
Talk: Visionary Venues
Talk: Co-Located Workshops
Talk: Competitions
Talk: Catch All
Complete Survey Results
This section after this one presents the raw data so you can mine it yourself.
- Large conferences: >1000 attendees
- Medium conferences: 100-1000 attendees
- Small conferences: <100 attendees
Question 1: REVIEWER LOAD
Has your community recently adopted new practices to deal with growing reviewer load, such as:
- tracking reviews of rejected papers from conference to conference as is done in journal reviewing
- increasing program committee size
- charging a review fee
- others?
For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.
Question 2: NON-INCREMENTAL
Has your community recently adopted new practices to promote non-incremental new ideas?
- big ideas sessions
- more papers
- shorter papers
- deemphasizing detailed evaluation
- others?
For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.
Question 3: PROGRAM COMMITTEES
Does your community practice:
- double blind submissions
- program committee submission restrictions
- rebuttals (author responses)
- large program committees
- program subcommittees
- others?
Do these practices seem to help or hurt promoting your field?
Question 4: WORKSHOPS, ETC.
Does your community provide venue for work not mature enough for your major conferences, such as:
- workshop co-located at conferences
- stand-alone workshops
- panels
- crazy idea sessions
On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination?
Question 5: CATCH-ALL
Are there other approaches your community has tried or abandoned that the rest of us can learn from?
Acknowledgments
The committee acknowledges the constructive advice of David A. Patterson.
The committee acknowledges the excellent support work of Adam Greenberg, Mike Marty, and Caitlin Scopel.
References
Mark D. Hill, Jean-Luc Gaudiot, Mary Hall, Joe Marks, Paolo Prinetto, and Donna Baglio, A Wiki for Discussing and Promoting Best Practices in Research, Communications of the ACM, Vol. 49, No. 9, pp. 63-64, September 2006. Points readers to this Wiki.
Mark D. Hill, Some Advice for Program Committee Chairs Based on my ISCA 2005 Experience, April 2005, http://www.cs.wisc.edu/~markhill/pcadvice2005.html.
Mark D. Hill, Interim Talk to the ACM Special Interest Group Governing Board, February 24, 2006, http://www.cs.wisc.edu/~markhill/acm_ieee_heath_conf_2006_02.ppt.
Kathyryn D. McKinley, Notes of Chairing Program Committees, May 2005, http://www.cs.utexas.edu/users/mckinley/notes/PC.html.
David A. Patterson, President's Letter: The Health of Research Conferences and the Dearth of Big Idea Papers, Communications of the ACM, December 2004, pp. 23-24.
Please see documentation on customizing the interface + Health of Conferences Committee
- and the User's Guide for usage and configuration help.
