Confessions of a researchaholic

March 5, 2010

Publishing reviews

Filed under: Real — liyiwei @ 5:43 pm

This is one of the mysteries that I have yet to figure out. Obviously, publishing is very important, and reviewing is the main factor deciding whether a submitted paper will be published. But somehow the reviewers receive almost no credit (except for token ones like conference mugs or listed names on the proceedings).

As a result, it should not be a surprise that many people spend much less efforts in reviewing than in publishing. And it is hard for me to imagine this asymmetry in efforts would not cause any problems, or least would not prevent the peer review systems from being better or more effective.

A simple solution is to publish good reviews, *under the reviewers consensus*, for both positive and negative opinions towards the relevant papers. The reviews could appear alongside the published papers in the proceedings, or at least in the digital library.

I could see several advantages of such an approach. First of all, it would make the recognition between papers and reviews more symmetric, and thus gives the reviewers more incentives to do a good job. Since the reviewers could opt for not having the reviews published, there should be no concern about integrity of the review process. The reviews could also provide more useful information for the papers, in the form of digests or opinions. Since reviewers are often among those who read the papers most carefully and thoroughly, their comments could provide additional angles for the relevant papers, which by nature reflect the authors (mostly) one sided opinions. This also makes the review process more transparent, producing a better sense of fairness and letting out some hints on why the papers are accepted.

Of course, reviews for rejected papers would not be published, but the symmetry between publishing/reviewing remains so I do not see any major problems here. One side effect would be that people might steer away from reviewing apparently bad papers even more so than they would now. But I would say that if the committee finds it hard to recruit reviewers for a particular submission, it might as well be a good reason to reject the paper outright; if a paper cannot even attract reviewers, how likely is that it would attract readers once published?

March 3, 2010

How to do rebuttal

Filed under: Real — liyiwei @ 6:21 pm
Tags: ,

(This is mainly for SIGGRAPH, but can be adapted to other scenarios.)

Check out Aaron Hertzmann’s blog about rebuttal as well. In short, our goal of the rebuttal is to convince the reviewers to accept the paper, regardless of the official rules.

First of all, understand the rebuttal process by reading the relevant information, e.g. the email from paper chair as well as the SIGGRAPH website. These already provide useful hints. Below are my additional suggestions.

The basics
Rebuttal is a scripted process, and it is not about the scores. The content does depend on the reviews, but the process should remain the same if you are rational.

Rebuttal does make a difference, not always, but frequent enough to worth the efforts. I have had a paper accepted with average score $<$ 2.8, and a paper rejected with average score $>$ 4.0 (out of a 5.0 scale).

The psychology
It is human nature to rate their own works more highly than others would. So you will likely get disappointed, especially for venues with high quality bar like SIGGRAPH. Lowering your expectation can help. (I usually assume the worst case scenario.) As you become more experienced you might be able to stand in the reviewers shoes and judge your own work more objectively with less emotional attachment. But before that, I suggest following a pre-scripted procedure.

Feel free to design your own procedure that works best for you, but here is mine for your reference. I designed it in a way so that in no stage would your emotion be easily slipped in.

. Save the review files somewhere (svn check-in the plain text .htm files if you are collaborating with me).

. Read the reviews once, grouped by reviewers instead of questions. This will give you a more coherent impression of each reviewer’s general stance. It also allows you a chance to vent (in a non-harmful way); if you feel angry now, go punch the wall or something. Do not proceed to the following stages until you are sufficiently relieved.

. Prepare a blank document for the rebuttal text. If you are collaborating with me via svn, check in the document first.

. Read the reviews again, and write down the questions in the rebuttal document that remotely seem to need answering. (Yes, we should keep the rebuttal document reasonably short and try not to answer every single question, but we need to identify the important ones first. This is why I prefer to be more conservative in listing questions in this early stage.) Focus on the questions now and do not worry about answers in this stage.

. It is likely that different reviewers might ask the same or similar questions. If so, consolidate the questions and mark the corresponding reviewer ids.

. Sort the questions in a roughly prioritized order, put in front these questions that are more important (e.g. factual misunderstanding/misinterpretation or specific questions asked by reviewers that they wish you to address in the rebuttal). Do not remove any question at this stage.

. Start adding answers to the questions. In the process, we might need to reorganize the questions and their orders. This is a natural process of writing rebuttal, just like writing the paper. We will iterate multiple rounds until every co-author is satisfied with the document.

. The official suggestion is to keep the rebuttal document short, but I would prefer to make it longer than necessary instead of risking omit important questions. (It is not always easy to correctly identify which questions are really important.) Of course the rebuttal text cannot run over the size limit (e.g. 2000 words), but to avoid confusing the reviewers, I usually separate the rebuttal into two parts: the main part for major and common questions, and the detailed part for individual reviewers. My personal experience as a reviewer (both primary/secondary and tertiary) is that I would not mind seeing a rebuttal running up to the length as long as the main part is clearly marked and shown up front. This allows me the flexibility to skip the more detailed parts if necessary. I have also observed that the reviewers tend to feel respected if the authors answer their questions, even for relatively minor ones.

. Before sending out the rebuttal text, add a short paragraph in the very beginning to thank all the reviewers for their effort in reading your paper and all the wonderful comments they have made.

. Submit the rebuttal. Do it early instead of waiting for the last minute, for obvious reasons. If the mechanism allows you to overwrite the previous uploads, definitely do so early. You can update the document later.

Theme: Rubric. Get a free blog at