Recently I wrote a document for my students giving them advice on reviewing scientific papers, particularly those in programming languages. John Regehr recently blogged about reviewing papers efficiently (and Shriram Krishnamurthi before him), and it reminded me that I had this document, so I decided I would post it here, following up on my recent post on the importance of peer review. I hope that students and those interested in peer reviewing will find it useful.
The ideas in this post come from my experiences as a journal reviewer and editor, and as a program committee member and Chair. I outline how I believe that papers should be judged, and how to write a review to express that judgment. Judgments should involve usefulness/appeal, novelty, correctness, and exposition. Reviews should aim to be self-contained, clearly expressing support for their recommendation; constructive, providing feedback for improving the work; and respectful of the authors who put a lot of time into their research. In general, think of the kind of review you’d like to receive, and act accordingly.