Confessions of a researchaholic

(This is mainly for SIGGRAPH, but can be adapted to other scenarios.)

Check out Aaron Hertzmann’s blog about rebuttal as well. In short, our goal of the rebuttal is to convince the reviewers to accept the paper, regardless of the official rules.

First of all, understand the rebuttal process by reading the relevant information, e.g. the email from paper chair as well as the SIGGRAPH website. These already provide useful hints. Below are my additional suggestions.

The basics
Rebuttal is a scripted process, and it is not about the scores. The content does depend on the reviews, but the process should remain the same if you are rational.

Rebuttal does make a difference, not always, but frequent enough to worth the efforts. I have had a paper accepted with average score $<$ 2.8, and a paper rejected with average score $>$ 4.0 (out of a 5.0 scale).

The psychology
It is human nature to rate their own works more highly than others would. So you will likely get disappointed, especially for venues with high quality bar like SIGGRAPH. Lowering your expectation can help. (I usually assume the worst case scenario.) As you become more experienced you might be able to stand in the reviewers shoes and judge your own work more objectively with less emotional attachment. But before that, I suggest following a pre-scripted procedure.

Procedure
Feel free to design your own procedure that works best for you, but here is mine for your reference. I designed it in a way so that in no stage would your emotion be easily slipped in.

. Save the review files somewhere (svn check-in the plain text .htm files if you are collaborating with me).

. Read the reviews once, grouped by reviewers instead of questions. This will give you a more coherent impression of each reviewer’s general stance. It also allows you a chance to vent (in a non-harmful way); if you feel angry now, go punch the wall or something. Do not proceed to the following stages until you are sufficiently relieved.

. Prepare a blank document for the rebuttal text. If you are collaborating with me via svn, check in the document first.

. Read the reviews again, and write down the questions in the rebuttal document that remotely seem to need answering. (Yes, we should keep the rebuttal document reasonably short and try not to answer every single question, but we need to identify the important ones first. This is why I prefer to be more conservative in listing questions in this early stage.) Focus on the questions now and do not worry about answers in this stage.

. It is likely that different reviewers might ask the same or similar questions. If so, consolidate the questions and mark the corresponding reviewer ids.

. Sort the questions in a roughly prioritized order, put in front these questions that are more important (e.g. factual misunderstanding/misinterpretation or specific questions asked by reviewers that they wish you to address in the rebuttal). Do not remove any question at this stage.

. Start adding answers to the questions. In the process, we might need to reorganize the questions and their orders. This is a natural process of writing rebuttal, just like writing the paper. We will iterate multiple rounds until every co-author is satisfied with the document.

. The official suggestion is to keep the rebuttal document short, but I would prefer to make it longer than necessary instead of risking omit important questions. (It is not always easy to correctly identify which questions are really important.) Of course the rebuttal text cannot run over the size limit (e.g. 2000 words), but to avoid confusing the reviewers, I usually separate the rebuttal into two parts: the main part for major and common questions, and the detailed part for individual reviewers. My personal experience as a reviewer (both primary/secondary and tertiary) is that I would not mind seeing a rebuttal running up to the length as long as the main part is clearly marked and shown up front. This allows me the flexibility to skip the more detailed parts if necessary. I have also observed that the reviewers tend to feel respected if the authors answer their questions, even for relatively minor ones.

. Before sending out the rebuttal text, add a short paragraph in the very beginning to thank all the reviewers for their effort in reading your paper and all the wonderful comments they have made.

. Submit the rebuttal. Do it early instead of waiting for the last minute, for obvious reasons. If the mechanism allows you to overwrite the previous uploads, definitely do so early. You can update the document later.

(Below is what I sent out to my collaborators for SIGGRAPH. I believe similar principles could be applied for other conferences as well.)

1. I recommend that only one person (the account creator) uploads materials to the sis account during the last day prior to the deadline, to avoid confusion and potential concurrent read/write hazards. I am usually a bad choice for material uploads as I will become a sequential bottleneck for uploading to multiple accounts. (Not to mention that I might get confused and mixed up the materials.) The same person should also be responsible to check all fields of the submission account to make sure everything is correct. I would usually take a pass to check everything, but the buck has to stop with the account creator.

2. The SIS server would usually be overloaded during the last few hours prior to the deadline, so upload all materials early and frequently. You can always overwrite the old materials with the new ones. If you wait until the last few hours or even minutes and find out that you cannot access the server, do not cry for help from me. I do not control the servers and there is nothing I can do.

3. On a similar vein, even though it is possible to upload only the checksums for the files prior to the deadline and upload materials with identical checksums later, I strongly recommend against doing so, unless you are very sure what you are doing. The main reason is that even with the same source files, different compilations could produce binary data with different checksums, e.g. pdflatex. So if you accidentally overwrite or lose the file, you are screwed. My overall recommendation is to avoid uploading anything in the last 2 to 3 hours prior to the deadline; this not only avoids potential server overloads but also help ensure that the files you uploaded are “sane”, for which humans tend not to be near the deadline according to my experience.

4. (For my collaborators in particular)
Do not assume I will be available during the last 8 hours prior to the deadline. Humans tend to procrastinate and end up with a lot of work for the last minutes. (I see this every single year, and I *never* understand why.) Unfortunately, given the number of projects I am usually involved with, it is mathematically impossible for me to spend the kind of time (every one of) you would prefer during these rush hours. So to lower people’s expectation and to encourage better time management, simply assume I will not be responding.

“If you give a person a fish, they’ll fish for a day. But if you train a person to fish, they’ll fish for a lifetime.” -Dan Quayle

I couldn’t agree more with this from my own advising experience. And I have one little add on: if the apprentice fails to catch any fish during a particular day of training, you have to let him come home empty-handed.

This is a vast subject, and probably has been covered by many articles (or even books). But here let me focus on the most crucial and fundamental issue.

Theorem
The nature of a research presentation is to convey ideas that you know to others who do not yet know.

Corollary: no need for stage fear
There was a study indicating that people fear public speaking more than death. Not everyone has this issue, but if you happen to be nervous about an upcoming presentation, just remember the basic theorem: you know much more than your audience about what you plan to talk about. So, even if you make mistakes, the audience would probably not find out anyway. Just stay cool, and do not let your tone, facial expression or body language disclose the fact that you are screwing up.

Corollary: never over-estimate your audience
Unless you could read mind, it is probably very difficult to devine what other people do not know. Since you already know what you want to talk about, it is very tempting for you to recite what is already in your mind instead of what your audiences really need. This is the single most important cause for a bad presentation in my personal observation. There is no magic solution, but a useful heuristic is to never over-estimate your audience. Always start your presentation with the basics; if the audiences do not know that, they will appreciate your effort, and if they do, they will feel they are smart. Convey high level information instead of details, and use easily absorbable medium, like images or videos, instead of texts, to convey your points.

Corollary: never over sell
The goal of the presentation is more to entice people to be interested in your research than to teach them the details. In a sense, it is more like an advertisement than a class room teaching. It is nearly impossible for average humans to learn a new subject within a 20 min presentation. So do not try to cram in every single detail of your algorithm into your talk; probably nobody is going to get that anyway. Instead, focus on getting the audiences’ attention for the first 10 minutes of your talk. And if you could achieve that, you are on your way for a great presentation.

Finally, like many other aspects of research, the best way to learn is by experience. The more presentations you give, the more likely you will learn how to give a great one. Do not worry about failures; I totally blew up my first research presentation, but eventually I figured out the deals, and now I am not only highly comfortable but also highly enjoy giving research talk, especially to huge audiences like SIGGRAPH.

There are usually two different parts of this question. Only one makes sense. Let me begin with the one that does not.

A common question asked by students (especially the new ones) is: what is the minimum amount of time that I should spend on research (e.g. how much time should I spend in the lab per day)? When one asks such a question, it usually implies that one is working for a certain superior, usually the mentor or advisor. For me this question does not make sense precisely because that is the wrong presumption. Notice that I am talking about how much time you should spend on “your” research, so it is about you, not your mentor, advisor, your parents, or anyone else. Why should you ask someone else how much time you should spend working for yourself? It is like asking what you should have for lunch.

If you are asking this question, it implies that you do not understand the nature of research. Unlike many other professions where one could toil from 9 to 5 and come home call it a day, your research is always with you, even during your sleep. It is a never ending effort. If you do not enjoy this or are unwilling to put in that amount of efforts, you are unlikely to be a good researcher, and I would recommend you choose another career. (I did not recall a single researcher that I know of who could afford to work only 9 to 5 for 5 days in a week and still manages to be good, except perhaps for these very senior ones who have established themselves long ago.)

If you have any doubt on this, let me share with you one simple fact: there are many very smart people in the world working very hard in your field, so ask yourself how you could compete with them. As far as I could see you have to do at least one of the following two things: outsmart them, or outwork them. The majority of you probably cannot do the former, so your best chance is the latter. Even if you are truly smart, it usually does not hurt to assume you are not. Humility takes you a long way. (Researchers and scientists who think themselves to be smarter than they really are will usually get punished very quickly by failures and rejections. This is one of the main reasons why I like research; it provides an objective feedback mechanism to keep ones ego in check. This is in sharp contrast to some other professions, e.g. politicians or corporate executives, whose egos could get boasted by a vicious cycle of self delusion.)

After you have realized that you probably need to work pretty hard to excel in research (either by taking someone’s advice or by your own experience in failures and rejections), you might ask if you should put a cap on the maximum amount of time on your research. Well, a quick answer is that, like I said above, your research will be with you all the time, so you cannot really shake it off (except maybe temporarily like after a major submission deadline). But on the other hand, it does not imply that you should stay in the lab staring at a computer all the time. It is counter-productive, making you fatigued, exhausted, uncreative, and prone to mistakes. I sometimes worked after dinner while in grad school, but after the end of one night when I accidentally typed “rm –r –f *.c” instead of “rm –r –f *.o” in the clean field of a make file (no, I did not properly back up), I have forced myself to stay away from my office and any computers after dinner.

So, in summary, my suggestion is to work on your research all the time while staying in the lab probably only from 9 to 6. This may sound self-contradictory but in fact it is not. Working on your research does not mandate that you have to be in the lab. You can keep your thinking going on as a background process in your brain while you are doing other stuff, like crashing a party, having dinner with your family and friends, attending a concert, exercising, or even sleeping. I am not joking; I actually got most of my ideas while doing all these random stuff while away from my lab or office. I am not a psychologist, but I believe creativity works at its best when you are simultaneously relaxed while having a thought going on in the back of your head.