As I have written previously, academic computer science differs from other scientific disciplines in its heavy use of peer-reviewed conference publications.
Since other disciplines’ conferences typically do not employ peer review, results published at highly selective computer science conferences may not be given the credit they deserve, i.e., the same credit they would receive if published in a similarly selective journal.
The main remedy has simply been to explain the situation to the possibly confused party, be it a dean or provost or a colleague from another department. But this remedy is sometimes ineffective: At some institutions, scientists risk a poor evaluation if they publish too few journal articles, but they risk muting the influence of their work in their own community if they publish too few articles at top conferences.
The ACM publications board has recently put forth a proposal that takes this problem head on by formally recognizing conference publications as equal in quality to journal publications. How? By collecting them in a special journal series called the Proceedings of the ACM (PACM).
In this post I briefly summarize the motivation and substance of the ACM proposal and provide some thoughts about it. In the end, I support it, but with some caveats. You have the opportunity to voice your own opinion via survey. You can also read other opinions for (by Kathryn McKinley) and against (by David S. Rosenblum) the proposal (if you can get past the ACM paywall, but that’s a topic for another day…).
Update: PACM has been approved, as has a new journal series called PACM PL that will collect papers accepted by major SIGPLAN Conferences. It will debut during late 2017.
Comparing Conferences and Journals
In disciplines like Physics, conference presentations are typically not peer reviewed — the conference is about reporting on previously published results (in a peer reviewed journal) and germinating new ideas among colleagues.
In computer science, conferences serve a similar purpose, but they are also the publication vehicle: To present at a major conference, you submit a paper that describes a substantial result, e.g., an innovation in compiler design along with an evaluation of it on a set of benchmarks. A program committee consisting of area experts chosen a the program chair reviews the paper for quality (including correctness), and only if there is sufficient consensus is the paper accepted. All of the accepted papers are published in the conference “proceedings.”
The process for top journals is similar. When a paper is submitted to a journal, an editor asks 2-3 (sometimes more) experts to review the paper for quality, and based on their reviews decides to accept the paper or not. There may be additional revisions and re-review required before finally accepting the paper.
We can see that the program chair of a conference acts like an editor at a journal, and the program committee serves as the expert reviewers. 1 The conference review process sometimes allows reviewed revisions, too (e.g., OOPSLA’s two-phase process, or “shepherding” by PLDI), or such revisions may occur “between conferences” — i.e., a paper is rejected with the expectation it will be submitted to another conference in short order, and then re-reviewed.
It is easy to see, just by looking at the papers themselves, that the process for top conferences engenders high quality results. Quantitatively, simply reformatting a POPL or PLDI paper into ACM “Transactions” format (the format used by ACM journals) often turns a 10-12 page conference paper into 25-30 pages. Compare this to the relatively shorter journal format used in other disciplines; e.g., a paper in Nature is 5 pages long.
The Proceedings of the ACM
So, top conference papers in computer science are of the same ilk as journal papers, and should be viewed as such. How to convince the uninitiated of this? The ACM publication board’s proposal is to start collecting conference proceedings in a journal called PACM, for Proceedings of the ACM (or something else if we don’t like that name). Different communities would have their own “series” of PACM, e.g., PACM for Programming Languages. How a SIG would set up its series is somewhat up to them, but the conferences included have a sound review process, i.e., one of the sort I described above.
I should add that this proposal is particularly impactful for the PL community because of all the sub-areas of CS, programming languages is perhaps the most reliant on conference publications. Our flagship journal, TOPLAS, publishes no more than a couple of dozen papers per year (it has published 13 so far this year), while hundreds of papers appear at top conferences.
My opinion: Supportive, with caveats
In general I’m in favor of this proposal because it solves a problem for many people. But to really work it should be as streamlined as possible — conference papers already are of high quality, so why add extra hoops to fix what amounts to a marketing problem?
Here are a three elements of the current proposal that I think should be relaxed:
- This process would require written reviews by a minimum number of three qualified reviewers. While three is a fine and common number, I don’t think the bar needs to be that high. Highly regarded journals like Science and Nature require two, not three, reviewers (see Nature‘s process, and Science‘s process).
- The review process must permit at least one substantive revision by the authors and review of that revision by the reviewers to determine whether their concerns are adequately met. I also disagree with this requirement. While rounds of review might be useful for authors, they are not necessary for ensuring quality, which is the key motivator here. If the PC doesn’t think the paper is strong enough it can just reject it, and the paper can go to the next conference for re-review. That said, the (selective) use of shepherding ought to be sufficient to meet this requirement; as with journals, some papers have too much to fix to be shepherded, and can be rejected, while others can be changed (with vetting) in sufficient time.
- Papers may not have artificial limits on length that would prevent full disclosure of references, documentation of methods, and so on. Again: Too strong. Look at Nature: it has both page limits on material, and a limit on the total number of references. I’m not against relaxing page limits or allowing more references. I’m against the claim that journal papers must not have page limits.
Here’s another element that is important to get right. At the recent SIGB meeting, Kathryn McKinley pointed out that getting the right name is crucial. PLDI, POPL, OOPSLA, ICFP, etc. are all substantial brands. Over the years, many people have come to understand that a paper from one of these conferences is likely to be very good. Therefore, we want to keep that brand as part of the name of the PACM series, so that its effect is retained.
My last point, which is not really about PACM itself but what happens if we adopt it, is that the existence of PACM should not preclude other conference/journal combinations. One attractive model is “submit to a journal, present at a conference.” As examples: You can submit a paper to the ACM TACO journal, and once accepted it can be presented at the HiPEAC conference. You can do the same with PVLDB (journal) and VLDB (conference). I think this model has a lot to speak for it, and we should explore using it for our conferences. For example, one benefit is that you have the option to present your paper at the conference, it’s not a requirement; if you’d prefer not to travel, you don’t have to. You also get rolling submission deadlines (rather than having to rewrite your paper for the slightly, or vastly, different audience of the next conference). Agreeing to PACM should not be the end of the story (and I don’t think it the proposal says that it should be).
I support this new ACM proposal, with caveats that I think are easily addressed. I think the proposed journal series solves a real problem, and if done right should be little to no additional work for authors, conference PC members, etc., once the ball is rolling.
Please share your thoughts in the comments below, and in the survey that ACM is taking about the idea. As ACM SIGPLAN Chair, I will take your feedback up the chain. Things are moving fast, and changes could start happening by the end of the year.
- Conference committees often solicit outside experts to fill gaps in the committee’s expertise. ↩