Confessions of a researchaholic

(If you are doing graphics research, you probably know that this is about the time every year for people to feel frustrated about rejections. So allow me to share my personal thoughts on this in the hope to make this world better.)

Theorem: life is a stochastic sampling process

Corollary 1: keep on trying

The whole paper submission and reviewing process is also stochastic. Out of the whole world of about 6 to 7 billion people, only 5 review your paper, and they, along with about 50 other committee members, decide the fate of your paper. For these papers that are either very good or very bad, the outcome would probably be the same regardless of who are the reviewers. However, since most papers are in the borderline gray area, they can easily end up in very different outcomes depending on who review your papers.

So, in a sense, getting upset about paper rejections is about as useful as getting upset at slot machines in a casino.

Now, assuming the review process is reasonably independent and unbiased (issues mostly beyond your control anyway), then the best way for variance reduction is to generate more samples. So, if you are tough and persistent enough and keep on submitting, in the long run your aggregate acceptance rate should approach your “intrinsic” value. In my personal case, I have huge yearly variations on acceptance rates, but my life time acceptance rate (so far) for SIGGRAPH is about 1/3, which I heard is about right for a reasonably good graphics researcher.

Corollary 2: do what you love

I hope it is not too late, but it is very important that you do what you absolutely love (instead of for any other reasons). This will give you more buffers to live through rejections. For me, I will not regret doing research even assuming all my future submissions are rejected.

Corollary 3: look for good things at other peoples papers (and look for bad things at yours)

I have seen multiple occasions where someone with rejected papers looked at the accepted ones and said: “These papers are not better than mine! How come they got accepted while mine got rejected?”

First of all, according to Corollary 1, this might actually be true due to the stochastic nature of the review process.

However, this might be (and likely is) untrue as well. It is human nature to evaluate their own work more highly than that of the others, so such a comparison is inherently biased.

More importantly, I do not see how that will help. Very few papers are perfect, so if one is accepted, it might as well have some merits, despite its apparent flaws. If you keep on looking for bad things instead of good things of the accepted papers, you will never learn why they are accepted, and thus keep on getting yours rejected.

So, I always try my best to look for good things in others papers. This not only motivates me to learn new things but also keeps me positive.
And when I look at my own papers, I always try my best to look for flaws and defects. Obviously, it is much better for me to spot and fix these before the reviewers have a chance to do so.

Corollary 4: quit whining

According to http://www.100people.org/, among the world population, 50% live in poverty, 1% have a college education, and 1% own computers. So if you could manage to get papers rejected, you are in the lucky 1% minority. That is, you are 99% likely to have ended up in a situation where you will be worrying about other stuff rather than paper rejections. So quit whining.

(This is mainly for SIGGRAPH, but can be adapted to other scenarios.)

Check out Aaron Hertzmann’s blog about rebuttal as well. In short, our goal of the rebuttal is to convince the reviewers to accept the paper, regardless of the official rules.

First of all, understand the rebuttal process by reading the relevant information, e.g. the email from paper chair as well as the SIGGRAPH website. These already provide useful hints. Below are my additional suggestions.

The basics
Rebuttal is a scripted process, and it is not about the scores. The content does depend on the reviews, but the process should remain the same if you are rational.

Rebuttal does make a difference, not always, but frequent enough to worth the efforts. I have had a paper accepted with average score $<$ 2.8, and a paper rejected with average score $>$ 4.0 (out of a 5.0 scale).

The psychology
It is human nature to rate their own works more highly than others would. So you will likely get disappointed, especially for venues with high quality bar like SIGGRAPH. Lowering your expectation can help. (I usually assume the worst case scenario.) As you become more experienced you might be able to stand in the reviewers shoes and judge your own work more objectively with less emotional attachment. But before that, I suggest following a pre-scripted procedure.

Procedure
Feel free to design your own procedure that works best for you, but here is mine for your reference. I designed it in a way so that in no stage would your emotion be easily slipped in.

. Save the review files somewhere (svn check-in the plain text .htm files if you are collaborating with me).

. Read the reviews once, grouped by reviewers instead of questions. This will give you a more coherent impression of each reviewer’s general stance. It also allows you a chance to vent (in a non-harmful way); if you feel angry now, go punch the wall or something. Do not proceed to the following stages until you are sufficiently relieved.

. Prepare a blank document for the rebuttal text. If you are collaborating with me via svn, check in the document first.

. Read the reviews again, and write down the questions in the rebuttal document that remotely seem to need answering. (Yes, we should keep the rebuttal document reasonably short and try not to answer every single question, but we need to identify the important ones first. This is why I prefer to be more conservative in listing questions in this early stage.) Focus on the questions now and do not worry about answers in this stage.

. It is likely that different reviewers might ask the same or similar questions. If so, consolidate the questions and mark the corresponding reviewer ids.

. Sort the questions in a roughly prioritized order, put in front these questions that are more important (e.g. factual misunderstanding/misinterpretation or specific questions asked by reviewers that they wish you to address in the rebuttal). Do not remove any question at this stage.

. Start adding answers to the questions. In the process, we might need to reorganize the questions and their orders. This is a natural process of writing rebuttal, just like writing the paper. We will iterate multiple rounds until every co-author is satisfied with the document.

. The official suggestion is to keep the rebuttal document short, but I would prefer to make it longer than necessary instead of risking omit important questions. (It is not always easy to correctly identify which questions are really important.) Of course the rebuttal text cannot run over the size limit (e.g. 2000 words), but to avoid confusing the reviewers, I usually separate the rebuttal into two parts: the main part for major and common questions, and the detailed part for individual reviewers. My personal experience as a reviewer (both primary/secondary and tertiary) is that I would not mind seeing a rebuttal running up to the length as long as the main part is clearly marked and shown up front. This allows me the flexibility to skip the more detailed parts if necessary. I have also observed that the reviewers tend to feel respected if the authors answer their questions, even for relatively minor ones.

. Before sending out the rebuttal text, add a short paragraph in the very beginning to thank all the reviewers for their effort in reading your paper and all the wonderful comments they have made.

. Submit the rebuttal. Do it early instead of waiting for the last minute, for obvious reasons. If the mechanism allows you to overwrite the previous uploads, definitely do so early. You can update the document later.

(Below is what I sent out to my collaborators for SIGGRAPH. I believe similar principles could be applied for other conferences as well.)

1. I recommend that only one person (the account creator) uploads materials to the sis account during the last day prior to the deadline, to avoid confusion and potential concurrent read/write hazards. I am usually a bad choice for material uploads as I will become a sequential bottleneck for uploading to multiple accounts. (Not to mention that I might get confused and mixed up the materials.) The same person should also be responsible to check all fields of the submission account to make sure everything is correct. I would usually take a pass to check everything, but the buck has to stop with the account creator.

2. The SIS server would usually be overloaded during the last few hours prior to the deadline, so upload all materials early and frequently. You can always overwrite the old materials with the new ones. If you wait until the last few hours or even minutes and find out that you cannot access the server, do not cry for help from me. I do not control the servers and there is nothing I can do.

3. On a similar vein, even though it is possible to upload only the checksums for the files prior to the deadline and upload materials with identical checksums later, I strongly recommend against doing so, unless you are very sure what you are doing. The main reason is that even with the same source files, different compilations could produce binary data with different checksums, e.g. pdflatex. So if you accidentally overwrite or lose the file, you are screwed. My overall recommendation is to avoid uploading anything in the last 2 to 3 hours prior to the deadline; this not only avoids potential server overloads but also help ensure that the files you uploaded are “sane”, for which humans tend not to be near the deadline according to my experience.

4. (For my collaborators in particular)
Do not assume I will be available during the last 8 hours prior to the deadline. Humans tend to procrastinate and end up with a lot of work for the last minutes. (I see this every single year, and I *never* understand why.) Unfortunately, given the number of projects I am usually involved with, it is mathematically impossible for me to spend the kind of time (every one of) you would prefer during these rush hours. So to lower people’s expectation and to encourage better time management, simply assume I will not be responding.