The POPL’15 author response period just ended a few days ago, and now submitted papers are in their final stages of review. Author response, a.k.a. author rebuttal, is a feature of conference review processes that allows authors to respond to claims made by reviewers prior to the final decision being made. The idea goes back at least as far as ISMM’06 in the PL community, but perhaps even further.
Following my previous post on advice for writing reviews, a commenter wondered whether I might also have advice for writing rebuttals. Indeed I do, and this post contains it. Your thoughts welcome!
What is author response?
Nothing is more irritating than a reviewer badly misunderstanding your paper and rejecting it on that false basis. Author response was originally designed to redress this wrong by allowing authors to point out factual inaccuracies in reviews. A mistaken reviewer, or his fellow reviewers, could take the author’s response into account and perhaps change their minds and end up accepting the paper rather than rejecting it (or sometimes go the other way!).
Some forms of author response try to limit it to “factual inaccuracies.” Authors should not make impassioned pleas, full of persuasive language but low on factual content, for reviewers to reconsider their opinions. Authors should also not introduce new results, like theorems or experiments, in their responses that would address weaknesses in the original submission. And authors should aim to be brief, with the conference review system limiting a response to 1000 or 500 words.
Over time, some have realized that this restricted form of response is both unenforceable and ineffective, and so have relaxed the restrictions. 500 words is awfully little to work with, and authors can be forced into being so brief that reviewers are unmoved for lack of context, making the whole exercise a waste of time. Moreover, it’s hard to enforce the “no new results” restriction: research is not just the idea, but also how it’s presented, and as such a rebuttal that explains something better to correct a mistaken impression is arguably introducing new material. Finally, while some notion of fairness suggests rebuttals should contain no new results, if the goal of the conference is to take the best papers, and a new result turns a good submission into a great paper, some reviewers may be inclined to consider it.
As such, author response processes I’m aware of now are more open, oftentimes having suggested rather than enforced limits, with fewer stated restrictions on what can be said. Basically, authors should make the best argument they can in light of the reviews.
The goal of your response
So, you have your reviews, now what should you write? I suggest that the rebuttal should not aim to get every reviewer to change his/her mind. Indeed, if most of the reviewers are in favor of rejection, you have little hope.
Instead, you want to convince at least one reviewer that your paper is sufficiently above the bar that he/she will argue to accept your paper. In other words, you want to embolden a possible advocate for your paper that will argue on your behalf and as such you want to help that person make this argument. This can be done by responding to his/her concerns, of course, but also requires good responses to the most important of the other reviewers’ concerns. You have to decide who you think your champion will be, and which of the other reviewers’ concerns the champion can already handle and which need extra help from you.
General themes for writing the response
Be polite and appreciative. Even if the reviewers didn’t like your paper as much as you think they should, they volunteered to review it and gave you feedback. Respect that and be thankful for what you got. More pragmatically, telling a reviewer that he is a rank amateur and can’t tell an alpha from an apple is not going to get him to change his mind.
Do not address every concern. The goal of finding a champion implies you shouldn’t feel the need to address every concern. You don’t have the space and even if you did, lots of text will only dilute your message. Instead, simply point out to the reviewers that you have *considered* all of their concerns and largely agree with the points you don’t specifically address; you will address those points in the next paper revision. (Yes, you should say this and burn a few words, IMO, rather than assume the reviewers know this. And it’s polite.)
Address the main concerns first. The high-level goal is to make it as easy as possible for the reviewer to understand how you have addressed their (most important) concerns. The most important concerns, shared by most of the reviewers, should be addressed right away, and decisively.
Reference the key concerns in a manner close to how the reviewers phrased them, so they are easy to find. Quote them verbatim if you have space. Do not write several paragraphs of text that respond to the ideas the reviewers raised without direct reference to them (and I don’t mean referencing the reviewer, I mean referencing the remark). This is a temptation due to limited space, but it will be ineffective if the reviewers cannot tell you are responding to their issues. Remember that a reviewer probably read your paper and wrote the review a month before reading your response to the review. The reviewer doesn’t exactly remember what he/she wrote. He/she will go back and look at the review and figure out what he didn’t like about the paper, and then figure out if he agrees with your response. But if he can’t figure out what you are responding to, he’s just going to give up.
Reference your own paper for answers to reviewer concerns. Doing so demonstrates to a reviewer that what you are talking about was, to some extent, already in the paper, and it allows you to be more brief, since the paper can do the talking for you.
Address smaller concerns, and place elaborated responses, last. If you do not have a strictly enforced word limit, you can go over any suggested word limit. Extremely long responses (I saw a 6600 word response when POPL’12 PC Chair!) will simply turn off the reviewers, but with care a longer response can be effective; it has worked for me. I suggest hitting the main points within the stated limit, and potentially going over, but not too far, to address lower priority points, perhaps those specific to just one of the reviewers. If you have a hard limit, you might write a longer response and post it somewhere else (e.g., your own web page), referencing it by URL in your submitted response. But don’t do this unless the PC Chair gives you permission.
Here is a response we recently wrote for our paper, Quantifying Information Flow for Dynamic Secrets, which appeared at the 2014 IEEE Symposium on Security and Privacy. Prior to the rebuttal, our scores were:
Reviewer A: Weak accept
Reviewer B: Accept
Reviewer C: Weak reject
Reviewer D: Reject
Obviously the outcome of the paper was in doubt. Reviewers A and B were mostly on board, while Reviewer C was interested, but had a lengthy list of concerns, many of which were about the presentation. The PC Chairs suggested keeping to a 500 word limit, but did not restrict the length, so we addressed the main concerns of all reviewers in the first 500 words, and then addressed Reviewer C’s concerns point by point afterward. We hoped that we could convince reviewer C to move up to a weak accept, and to convince reviewer B to champion the paper, with the first three overcoming the relatively weak complaints made by reviewer D (which in our view were the least substantive). In the end, reviewer C changed his/her score to a weak accept and the paper was accepted on the condition we made the writing changes we promised.
This response has most of the main elements I recommend above. I hope you find it useful!
We thank the reviewers for their thoughtful comments! We address the most important/common questions in the first 500 words; other answers below the line can be viewed as optional/supplemental. **Reviewers A and D: Is your model fundamental, or can be it be encoded in existing batch-oriented models (like )? Our model is fundamental. Existing batch-oriented models (like ) can be encoded in our model, but batch-oriented models cannot encode our model. That's because our model permits feedback: past outputs may affect future inputs. So the traditional information-theoretic approach to quantifying leakage fails. You might at first suspect that batch-oriented models could emulate our time-varying model by viewing a run with multiple time steps as a single batch-execution of the system. But there is no information-theoretic channel representing that run that is invariant with respect to the probability distribution on secrets (see [22, Example 1]), so information-theoretic measures can't be used with a batch-model encoding. By contrast, information-theoretic measures can be used with our time-varying model. The result in III-E shows our model encodes a joint probability distribution, which can be decomposed into a product of conditional probabilities, to which measures can be applied. We will expand/clarify this result (including moving material from the appendix into the main text as suggested by Reviewer C) to also address this question of encoding. **Reviewer D: Is the complication of the model warranted since the examples remove model features? Each example removes some model features for simplicity, but every model feature is required by at least one example; removing features would thus rule out some examples. **Reviewers C and D: The experiments are not very well explained. We agree that adding further explanation and intuition would improve the paper. We will include the exact parameters for each experiment. We will also sketch the behavior of the optimal adversary in each case, to give intuition as to the shape of the graphs. For most of the experiments we show, the behavior of the optimal adversary is to make observations within the period where the secret doesn't change and attack right before the secret changes based on what they have observed in this static period. For some parameters (perfect observations), the attacker doesn't necessarily need to wait to right before the change to be optimal. Answers to reviewer C's particular questions are given in an optional/supplemental portion to the rebuttal below. **Reviewer D: How do you compare to attack graphs, game-theoretic models, etc.? We will add text to compare. In short: we can view our table-based optimal adversary as implementing an attack graph (computed to optimize the adversary's expected gain). We can view our approach as a Bayesian game, but game-theoretic notions are not relevant for us as only one party (the adversary) has choices. ---------------------------------------------------------------------- We address some particular questions from reviewer C in the remainder of the rebuttal. --How did you implement the optimization procedure? By exact simulation; more efficient means of analysis (e.g., via abstract interpretation) we leave to interesting future work. --Example V-F: The role of the change function is confusing since it could be more directly modeled as another part of the secret. The reviewer is right and so we should at least clarify the example. In principle, separating the secret from the change function in our model is done for two reasons: 1) the function cannot be observed directly, only inferred, and 2) it cannot change itself. As a result, this special part of the secret behaves most similarly to how secrets behave in existing work on quantified information flow: uncertainty of the change function decreases with additional observations through imperfect channels and this uncertainty cannot be recovered. Conversely, uncertainty in the dynamic parts of the secret can be recovered, as shown by the periodicity in the figures. The example in V-F was designed to show a counterintuitive situation in which applying the change function more often results in higher vulnerability of the secret. To show this we made the example have a very high correlation between the change function and another part of the secret (the floor). This has an unfortunate side effect of trivializing the distinction between the change functions and other parts of the secret. We conjecture that the correlation is a necessary feature of scenarios that have the counterintuitive property; we will attempt a proof and expand the discussion in any case. --Why do the periods in the examples differ in the graphs? This is sloppiness on our part: the change_freq parameter differed with different experiments. We will unify it as much as possible, make it explicit in the text, and rerun the experiments. --What is the point of figures 7, 8? Figures 7 and 8 illustrate existing min-vulnerability and guessing entropy measures in our model, respectively, compared against a measure that accounts for an adaptive adversary, which is a signature feature of our approach. The figures show that adaptivity significantly increases a secret's vulnerability. --How to interpret the bottom of figure 9? Figure 9 illustrates the impact of memory limitation on the adversary. The bottom portion can be explained partially from the periodic effect that also occurs in the top portion of the figure. The advantage of more memory only matters as long as there are more things to remember. At the start, or right after the secret has changed, there is nothing to remember hence the various limited adversaries have the same expected raid gain. Their odds diverge further into the "epoch" of the current secret as there are more and more relevant observations to remember. The gains are most diverged right before the secret changes. The bottom figure illustrates the situation for an adaptive adversary, for which the periodic effect also occurs. There, however, the gain increases monotonically with time towards 1 (except for the adversary with no memory at all). This occurs as every adversary with at least one observation's worth of memory has ever-increasing odds of luckily observing a single stakeout that pinpoints the stash location exactly. --Why is the parameter raid_odds included? Doesn't it just scale the gain? This aspect of our stakeout/raid examples is mostly used that we can generalize beyond min-vulnerability (which would be equivalent to raids without raid_odds) and show how such a parameter manifests itself in the overall vulnerability of a secret. The reviewer is right that the parameter serves to scale vulnerability and does not impact the behavior of the adversary for most stakeout/raid examples. The situation is more interesting when there are penalties for making stakeouts (Section V.E). There the adversary cannot just decide to stakeout until right before the stash location changes as each stakeout incurs a cost; the attacker must weigh their chances of a successful raid given what they know now vs. the prospect of learning more about the stash location. The first quantity is scaled with raid_odds, whereas the second depends more on the parameter obs_odds. These parameters thus impact the adaptive decisions the adversary makes before the attack occurs.