Advice on writing an author response

The POPL’15 author response period just ended a few days ago, and now submitted papers are in their final stages of review. Author response, a.k.a. author rebuttal, is a feature of conference review processes that allows authors to respond to claims made by reviewers prior to the final decision being made. The idea goes back at least as far as ISMM’06 in the PL community, but perhaps even further.

Following my previous post on advice for writing reviews, a commenter wondered whether I might also have advice for writing rebuttals. Indeed I do, and this post contains it. Your thoughts welcome!

What is author response?

Nothing is more irritating than a reviewer badly misunderstanding your paper and rejecting it on that false basis. Author response was originally designed to redress this wrong by allowing authors to point out factual inaccuracies in reviews. A mistaken reviewer, or his fellow reviewers, could take the author’s response into account and perhaps change their minds and end up accepting the paper rather than rejecting it (or sometimes go the other way!).

Some forms of author response try to limit it to “factual inaccuracies.” Authors should not make impassioned pleas, full of persuasive language but low on factual content, for reviewers to reconsider their opinions. Authors should also not introduce new results, like theorems or experiments, in their responses that would address weaknesses in the original submission. And authors should aim to be brief, with the conference review system limiting a response to 1000 or 500 words.

Over time, some have realized that this restricted form of response is both unenforceable and ineffective, and so have relaxed the restrictions. 500 words is awfully little to work with, and authors can be forced into being so brief that reviewers are unmoved for lack of context, making the whole exercise a waste of time. Moreover, it’s hard to enforce the “no new results” restriction: research is not just the idea, but also how it’s presented, and as such a rebuttal that explains something better to correct a mistaken impression is arguably introducing new material. Finally, while some notion of fairness suggests rebuttals should contain no new results, if the goal of the conference is to take the best papers, and a new result turns a good submission into a great paper, some reviewers may be inclined to consider it.

As such, author response processes I’m aware of now are more open, oftentimes having suggested rather than enforced limits, with fewer stated restrictions on what can be said. Basically, authors should make the best argument they can in light of the reviews.

The goal of your response

So, you have your reviews, now what should you write? I suggest that the rebuttal should not aim to get every reviewer to change his/her mind. Indeed, if most of the reviewers are in favor of rejection, you have little hope.

Instead, you want to convince at least one reviewer that your paper is sufficiently above the bar that he/she will argue to accept your paper. In other words, you want to embolden a possible advocate for your paper that will argue on your behalf and as such you want to help that person make this argument. This can be done by responding to his/her concerns, of course, but also requires good responses to the most important of the other reviewers’ concerns. You have to decide who you think your champion will be, and which of the other reviewers’ concerns the champion can already handle and which need extra help from you.

General themes for writing the response

Be polite and appreciative. Even if the reviewers didn’t like your paper as much as you think they should, they volunteered to review it and gave you feedback. Respect that and be thankful for what you got. More pragmatically, telling a reviewer that he is a rank amateur and can’t tell an alpha from an apple is not going to get him to change his mind.

Do not address every concern. The goal of finding a champion implies you shouldn’t feel the need to address every concern. You don’t have the space and even if you did, lots of text will only dilute your message. Instead, simply point out to the reviewers that you have *considered* all of their concerns and largely agree with the points you don’t specifically address; you will address those points in the next paper revision. (Yes, you should say this and burn a few words, IMO, rather than assume the reviewers know this. And it’s polite.)

Address the main concerns first. The high-level goal is to make it as easy as possible for the reviewer to understand how you have addressed their (most important) concerns. The most important concerns, shared by most of the reviewers, should be addressed right away, and decisively.

Reference the key concerns in a manner close to how the reviewers phrased them, so they are easy to find. Quote them verbatim if you have space. Do not write several paragraphs of text that respond to the ideas the reviewers raised without direct reference to them (and I don’t mean referencing the reviewer, I mean referencing the remark). This is a temptation due to limited space, but it will be ineffective if the reviewers cannot tell you are responding to their issues. Remember that a reviewer probably read your paper and wrote the review a month before reading your response to the review. The reviewer doesn’t exactly remember what he/she wrote. He/she will go back and look at the review and figure out what he didn’t like about the paper, and then figure out if he agrees with your response. But if he can’t figure out what you are responding to, he’s just going to give up.

Reference your own paper for answers to reviewer concerns. Doing so demonstrates to a reviewer that what you are talking about was, to some extent, already in the paper, and it allows you to be more brief, since the paper can do the talking for you.

Address smaller concerns, and place elaborated responses, last. If you do not have a strictly enforced word limit, you can go over any suggested word limit. Extremely long responses (I saw a 6600 word response when POPL’12 PC Chair!) will simply turn off the reviewers, but with care a longer response can be effective; it has worked for me. I suggest hitting the main points within the stated limit, and potentially going over, but not too far, to address lower priority points, perhaps those specific to just one of the reviewers. If you have a hard limit, you might write a longer response and post it somewhere else (e.g., your own web page), referencing it by URL in your submitted response. But don’t do this unless the PC Chair gives you permission.

Example response

Here is a response we recently wrote for our paper, Quantifying Information Flow for Dynamic Secrets, which appeared at the 2014 IEEE Symposium on Security and Privacy. Prior to the rebuttal, our scores were:

Reviewer A: Weak accept
Reviewer B: Accept
Reviewer C: Weak reject
Reviewer D: Reject

Obviously the outcome of the paper was in doubt. Reviewers A and B were mostly on board, while Reviewer C was interested, but had a lengthy list of concerns, many of which were about the presentation. The PC Chairs suggested keeping to a 500 word limit, but did not restrict the length, so we addressed the main concerns of all reviewers in the first 500 words, and then addressed Reviewer C’s concerns point by point afterward. We hoped that we could convince reviewer C to move up to a weak accept, and to convince reviewer B to champion the paper, with the first three overcoming the relatively weak complaints made by reviewer D (which in our view were the least substantive). In the end, reviewer C changed his/her score to a weak accept and the paper was accepted on the condition we made the writing changes we promised.

This response has most of the main elements I recommend above. I hope you find it useful!

We thank the reviewers for their thoughtful comments! We address the 
most important/common questions in the first 500 words; other 
answers below the line can be viewed as optional/supplemental. 

**Reviewers A and D: Is your model fundamental, or can be it be 
encoded in existing batch-oriented models (like [4])? 

Our model is fundamental. Existing batch-oriented models (like [4]) 
can be encoded in our model, but batch-oriented models cannot encode 
our model. That's because our model permits feedback: past outputs 
may affect future inputs. So the traditional information-theoretic 
approach to quantifying leakage fails. 

You might at first suspect that batch-oriented models could emulate 
our time-varying model by viewing a run with multiple time steps as a 
single batch-execution of the system. But there is no 
information-theoretic channel representing that run 
that is invariant with respect to the 
probability distribution on secrets (see [22, Example 1]), 
so information-theoretic measures can't be used with a batch-model 
encoding. 

By contrast, information-theoretic measures can be used 
with our time-varying model. The result in III-E shows our model 
encodes a joint probability distribution, which can be decomposed 
into a product of conditional probabilities, to which measures can 
be applied. We will expand/clarify this result 
(including moving material from the appendix into the main text as 
suggested by Reviewer C) to also address this question of encoding. 

**Reviewer D: Is the complication of the model warranted since the 
examples remove model features? 

Each example removes some model features for simplicity, but every 
model feature is required by at least one example; removing features 
would thus rule out some examples. 

**Reviewers C and D: The experiments are not very well explained. 

We agree that adding further explanation and intuition would improve 
the paper. We will include the exact parameters for each 
experiment. We will also sketch the behavior of the optimal adversary 
in each case, to give intuition as to the shape of the graphs. For 
most of the experiments we show, the behavior of the optimal adversary 
is to make observations within the period where the secret doesn't 
change and attack right before the secret changes based on what they 
have observed in this static period. For some parameters (perfect 
observations), the attacker doesn't necessarily need to wait to right 
before the change to be optimal. Answers to reviewer C's particular 
questions are given in an optional/supplemental portion to the 
rebuttal below. 

**Reviewer D: How do you compare to attack graphs, game-theoretic 
models, etc.? 

We will add text to compare. In short: we can view our table-based 
optimal adversary as implementing an attack graph (computed to 
optimize the adversary's expected gain). We can view our approach as a 
Bayesian game, but game-theoretic notions are not relevant for us as 
only one party (the adversary) has choices. 

---------------------------------------------------------------------- 

We address some particular questions from reviewer C in the remainder 
of the rebuttal. 

--How did you implement the optimization procedure? 

By exact simulation; more efficient means of analysis (e.g., via 
abstract interpretation) we leave to interesting future work. 

--Example V-F: The role of the change function is confusing since it 
  could be more directly modeled as another part of the secret. 

The reviewer is right and so we should at least clarify the example. 
In principle, separating the secret from the change function in our 
model is done for two reasons: 1) the function cannot be observed 
directly, only inferred, and 2) it cannot change itself. As a result, 
this special part of the secret behaves most similarly to how secrets 
behave in existing work on quantified information flow: uncertainty of 
the change function decreases with additional observations through 
imperfect channels and this uncertainty cannot be 
recovered. Conversely, uncertainty in the dynamic parts of the secret 
can be recovered, as shown by the periodicity in the figures. The 
example in V-F was designed to show a counterintuitive situation in 
which applying the change function more often results in higher 
vulnerability of the secret. To show this we made the example have a 
very high correlation between the change function and another part of 
the secret (the floor). This has an unfortunate side effect of 
trivializing the distinction between the change functions and other 
parts of the secret. We conjecture that the correlation is a necessary 
feature of scenarios that have the counterintuitive property; we will 
attempt a proof and expand the discussion in any case. 
  
--Why do the periods in the examples differ in the graphs? 

This is sloppiness on our part: the change_freq parameter differed 
with different experiments. We will unify it as much as possible, make 
it explicit in the text, and rerun the experiments. 

--What is the point of figures 7, 8? 

Figures 7 and 8 illustrate existing min-vulnerability and guessing 
entropy measures in our model, respectively, compared against a 
measure that accounts for an adaptive adversary, which is a signature 
feature of our approach. The figures show that adaptivity 
significantly increases a secret's vulnerability. 

--How to interpret the bottom of figure 9? 

Figure 9 illustrates the impact of memory limitation on the 
adversary. The bottom portion can be explained partially from the 
periodic effect that also occurs in the top portion of the figure. The 
advantage of more memory only matters as long as there are more things 
to remember. At the start, or right after the secret has changed, 
there is nothing to remember hence the various limited adversaries 
have the same expected raid gain. Their odds diverge further into the 
"epoch" of the current secret as there are more and more relevant 
observations to remember. The gains are most diverged right before the 
secret changes. The bottom figure illustrates the situation for an 
adaptive adversary, for which the periodic effect also occurs. There, 
however, the gain increases monotonically with time towards 1 (except 
for the adversary with no memory at all). This occurs as every 
adversary with at least one observation's worth of memory has 
ever-increasing odds of luckily observing a single stakeout that 
pinpoints the stash location exactly. 

--Why is the parameter raid_odds included? Doesn't it just scale the 
  gain? 

This aspect of our stakeout/raid examples is mostly used that we can 
generalize beyond min-vulnerability (which would be equivalent to 
raids without raid_odds) and show how such a parameter manifests 
itself in the overall vulnerability of a secret. The reviewer is right 
that the parameter serves to scale vulnerability and does not impact 
the behavior of the adversary for most stakeout/raid examples. The 
situation is more interesting when there are penalties for making 
stakeouts (Section V.E). There the adversary cannot just decide to 
stakeout until right before the stash location changes as each 
stakeout incurs a cost; the attacker must weigh their chances of a 
successful raid given what they know now vs. the prospect of learning 
more about the stash location. The first quantity is scaled with 
raid_odds, whereas the second depends more on the parameter 
obs_odds. These parameters thus impact the adaptive decisions the 
adversary makes before the attack occurs.