Category Archives: Process

Advice on reviewing papers

Recently I wrote a document for my students giving them advice on reviewing scientific papers, particularly those in programming languages. John Regehr recently blogged about reviewing papers efficiently (and Shriram Krishnamurthi before him), and it reminded me that I had this document, so I decided I would post it here, following up on my recent post on the importance of peer review. I hope that students and those interested in peer reviewing will find it useful.

The ideas in this post come from my experiences as a journal reviewer and editor, and as a program committee member and Chair. I outline how I believe that papers should be judged, and how to write a review to express that judgment. Judgments should involve usefulness/appeal, novelty, correctness, and exposition. Reviews should aim to be self-contained, clearly expressing support for their recommendation; constructive, providing feedback for improving the work; and respectful of the authors who put a lot of time into their research. In general, think of the kind of review you’d like to receive, and act accordingly.

Continue reading


Filed under Process, Science

Peer review, and why it matters

Scientific results intersect with all aspects of the modern world, and they underpin many important decisions made at the level of governments, corporations, and individuals. When you read about a scientific result—like the correlation of a particular gene’s mutation with a certain disease, or that it’s better for children to engage in unstructured play rather than structured learning at early ages—why should you believe it?

If you were an expert, you might be able to (with time) capably judge the work itself. But what if you are a non-expert, which is increasingly likely as branches of study become hyper-specialized? In this case, you are left to trust the scientific process, i.e., the way in which scientific work is judged to be true and important. At the heart of this process is peer review.

In this post I describe elements of the peer review process we use in scientific research about programming languages.

Continue reading


Filed under Process, Science