Confessions of a researchaholic

Congratulations that your paper has been accepted to the most prestigious conference in your academic field. You can have a sigh of relief and take a vacation. You deserve it.

But your job is not over yet. You still have to present the paper.

It is important
The presentation is essentially a promo for your paper, and to some degree, for yourself. So it is to your advantage to give a good talk. And if you are relatively junior in the field, you should do it yourself (instead of your adviser or another co-author). This will give you the necessary exposure and recognition.

It is a sales pitch
This can be a personal preference, but for me, a conference talk is more of an advertisement than an education. A very common mistake is trying to cram in all the information into a presentation, hoping that the audience will understand everything. Unfortunately, the talk slots are usually too short and the audience too distracted for this to happen.

Instead, consider the conference as a social event, and your talk a sales pitch. Your goal is to attract people to get interested in your paper, so that they will later spend time reading it. And if you can give a good talk, it also helps sell you as a good researcher. This can translate into long term benefits, like job offers.

Content
The content of your talk should be mainly about the following: (1) why people should care about your paper, (2) why they should read it after your talk, and (3) what are the key points of your paper. Remember, it is more of an advertisement (1 and 2) than education (3).

Actually, there is one more: (4) what a cool and smart guy you are. But you really need to know what you are doing. And if you do, you are beyond this post already.

Practice
Finally, practice your talk, both with yourself and in front of an audience. If you are relatively inexperienced or have a certain degree of glossophobia (most people do), practice enough times so that you can give the talk while sleeping. I practiced at least 20 times for my first SIGGRAPH talk, even though nowadays I usually do just once or twice to avoid cruising through my talk with a too rigid or scripted performance.

Look for what is good in a paper. For the rest (that is not so good), try to make constructive comments so that the authors can improve the paper.

Looking for the good

I have seen many, many reviews, both as primary/secondary (for which I can see reviewer identities) as well as tertiary (for which I cannot). My overall impression is that on average, reviewers tend to have a negative bias. That is, I have seen more good papers rejected than bad ones accepted. This is sad, as the prosperity and health of the academic community (as well as the human civilization in general) depends crucially on the timely publication of good ideas. The risk/reward ratio is highly asymmetric; a bad idea, even though published, can do little harm as it tend to be quickly ignored and forgotten. (Notable exception: Das Kapital by Karl Marx.) A good idea, if not published, could take tremendous effort to recover (e.g. proving Fermat’s last theorem) or is forever lost (I cannot even give you examples on this).

Why you should be positive as a reviewer

Humans, by nature, tend to be more critical towards others than to themselves. But here are a few reasons why being positive as a reviewer can be good for you:

. This is a small world. What comes around goes around. If you are known as a paper killer, it can not only harm your reputation but also makes others more likely to be negative towards your papers. There are a few well known folks in the graphics community who are highly successful and yet unpopular enough so that they do not tend to be invited to serve as paper committee members. Their opinions also tend to get ignored during committee discussions.

. Your success depends highly on your followers. Do not discourage them. If you see people citing or extending your work, it is to your benefit to help them as they will in turn help you become more famous and more influential in the long run. (Trust me on this: I saw this happened to me before, and I want to see it happen again.) Some people (especially more junior guys) tend to treat this as a kind of threat as they want to claim all credits themselves. The very best researchers start a new field/subject (or at least help popularize it), and move on to a different one leaving others cite and expand their earlier works. People who have to stay on the same subject or field tend to lack sufficient capability or courage to move on.

. The health of a community depends on how the members treat each other. Nobody likes to be in a place where people ambush each other. So do not start to behave like one yourself.

All in all, my experience is that one’s reviews reflect one’s personality and intelligence. Good researchers tend to write good and positive reviews. Cynical and nonconstructive reviews tend to come from those who are sour or feel insecure.

Being fair and constructive

Note that I am not saying you should accept bad papers. Papers with incremental or boring ideas should not be published, as otherwise it is unfair and can drag down the quality of the venue you review for. Papers with good ideas but major flaws should also be fixed before publication, as otherwise they might not attract the attention they deserve. (A common example is a good idea expressed in a bad writing; if people have difficulty understanding an idea, it would probably not make as large an impact it should. In that case, it is usually much better for everyone involved if the writing can be improved before publication.)

Be specific and constructive in your reviews so that the authors could have concrete clues about how to improve their paper.

SIGGRAPH now publishes the technical papers committee list a few weeks prior to the submission deadline. Here is my suggestion on how to take this information to your advantage.

First, let me tell you what NOT to do. It is very common for people to try to guess who is likely to review your papers, but this is entirely useless and counterproductive. In general, there is indeed a positive correlation between the committee member expertise and your paper subject, but it is not a very strong correlation and there is a non-trivial chance that a paper will be assigned to an unexpected committee member. (This is mainly due to how the paper sorting and assignment process goes but I am not going to elaborate on this here because I do not think it is necessary for authoring a good paper.)

So my personal strategy is to make sure the paper will survive *any* committee member. This is the safest bet and you cannot go wrong with that. (But do check the committee member list constantly though as sometimes it may change prior to the deadline.) Specifically, you should write your paper (especially the title and abstract) in a way so that it is as clear as possible on what it is all about to minimize the chance of misunderstanding. (If the paper sorters misunderstand your paper, they are more likely to assign your paper to the wrong committee member. And if the senior reviewers misunderstand your paper, they are more likely to assign your paper to the wrong tertiaries. And if reviewers misunderstand your paper they are more likely to score it lowly.)

It also helps to read and cite papers from the committee members for two reasons. The first reason is that the background and expertise of a committee member will obviously have a huge influence on how she is going to review a paper, so if you understand her, you stand a better chance of appreciating how she might approach a paper. The second reason is that psychological studies indicate that it is almost never too much to flatter, and people are usually happy to see their own papers cited and attributed in a positive manner.

If you want to push this further, my personal philosophy is to assume that my paper will be reviewed by the worst possible reviewers and prepare for that. Like Andy Grove said, only the paranoid survive.

Anyone still questioning my emphasis and insistence on academic integrity (e.g. be honest in disclosing limitations and do not abuse authorship) please just take a look at this Economist article.

“In looking for people to hire, you look for three qualities: integrity, intelligence, and energy. And if they don’t have the first, the other two will kill you.” – Warren Buffet

(If you are doing graphics research, you probably know that this is about the time every year for people to feel frustrated about rejections. So allow me to share my personal thoughts on this in the hope to make this world better.)

Theorem: life is a stochastic sampling process

Corollary 1: keep on trying

The whole paper submission and reviewing process is also stochastic. Out of the whole world of about 6 to 7 billion people, only 5 review your paper, and they, along with about 50 other committee members, decide the fate of your paper. For these papers that are either very good or very bad, the outcome would probably be the same regardless of who are the reviewers. However, since most papers are in the borderline gray area, they can easily end up in very different outcomes depending on who review your papers.

So, in a sense, getting upset about paper rejections is about as useful as getting upset at slot machines in a casino.

Now, assuming the review process is reasonably independent and unbiased (issues mostly beyond your control anyway), then the best way for variance reduction is to generate more samples. So, if you are tough and persistent enough and keep on submitting, in the long run your aggregate acceptance rate should approach your “intrinsic” value. In my personal case, I have huge yearly variations on acceptance rates, but my life time acceptance rate (so far) for SIGGRAPH is about 1/3, which I heard is about right for a reasonably good graphics researcher.

Corollary 2: do what you love

I hope it is not too late, but it is very important that you do what you absolutely love (instead of for any other reasons). This will give you more buffers to live through rejections. For me, I will not regret doing research even assuming all my future submissions are rejected.

Corollary 3: look for good things at other peoples papers (and look for bad things at yours)

I have seen multiple occasions where someone with rejected papers looked at the accepted ones and said: “These papers are not better than mine! How come they got accepted while mine got rejected?”

First of all, according to Corollary 1, this might actually be true due to the stochastic nature of the review process.

However, this might be (and likely is) untrue as well. It is human nature to evaluate their own work more highly than that of the others, so such a comparison is inherently biased.

More importantly, I do not see how that will help. Very few papers are perfect, so if one is accepted, it might as well have some merits, despite its apparent flaws. If you keep on looking for bad things instead of good things of the accepted papers, you will never learn why they are accepted, and thus keep on getting yours rejected.

So, I always try my best to look for good things in others papers. This not only motivates me to learn new things but also keeps me positive.
And when I look at my own papers, I always try my best to look for flaws and defects. Obviously, it is much better for me to spot and fix these before the reviewers have a chance to do so.

Corollary 4: quit whining

According to http://www.100people.org/, among the world population, 50% live in poverty, 1% have a college education, and 1% own computers. So if you could manage to get papers rejected, you are in the lucky 1% minority. That is, you are 99% likely to have ended up in a situation where you will be worrying about other stuff rather than paper rejections. So quit whining.

(This is mainly for SIGGRAPH, but can be adapted to other scenarios.)

Check out Aaron Hertzmann’s blog about rebuttal as well. In short, our goal of the rebuttal is to convince the reviewers to accept the paper, regardless of the official rules.

First of all, understand the rebuttal process by reading the relevant information, e.g. the email from paper chair as well as the SIGGRAPH website. These already provide useful hints. Below are my additional suggestions.

The basics
Rebuttal is a scripted process, and it is not about the scores. The content does depend on the reviews, but the process should remain the same if you are rational.

Rebuttal does make a difference, not always, but frequent enough to worth the efforts. I have had a paper accepted with average score $<$ 2.8, and a paper rejected with average score $>$ 4.0 (out of a 5.0 scale).

The psychology
It is human nature to rate their own works more highly than others would. So you will likely get disappointed, especially for venues with high quality bar like SIGGRAPH. Lowering your expectation can help. (I usually assume the worst case scenario.) As you become more experienced you might be able to stand in the reviewers shoes and judge your own work more objectively with less emotional attachment. But before that, I suggest following a pre-scripted procedure.

Procedure
Feel free to design your own procedure that works best for you, but here is mine for your reference. I designed it in a way so that in no stage would your emotion be easily slipped in.

. Save the review files somewhere (svn check-in the plain text .htm files if you are collaborating with me).

. Read the reviews once, grouped by reviewers instead of questions. This will give you a more coherent impression of each reviewer’s general stance. It also allows you a chance to vent (in a non-harmful way); if you feel angry now, go punch the wall or something. Do not proceed to the following stages until you are sufficiently relieved.

. Prepare a blank document for the rebuttal text. If you are collaborating with me via svn, check in the document first.

. Read the reviews again, and write down the questions in the rebuttal document that remotely seem to need answering. (Yes, we should keep the rebuttal document reasonably short and try not to answer every single question, but we need to identify the important ones first. This is why I prefer to be more conservative in listing questions in this early stage.) Focus on the questions now and do not worry about answers in this stage.

. It is likely that different reviewers might ask the same or similar questions. If so, consolidate the questions and mark the corresponding reviewer ids.

. Sort the questions in a roughly prioritized order, put in front these questions that are more important (e.g. factual misunderstanding/misinterpretation or specific questions asked by reviewers that they wish you to address in the rebuttal). Do not remove any question at this stage.

. Start adding answers to the questions. In the process, we might need to reorganize the questions and their orders. This is a natural process of writing rebuttal, just like writing the paper. We will iterate multiple rounds until every co-author is satisfied with the document.

. The official suggestion is to keep the rebuttal document short, but I would prefer to make it longer than necessary instead of risking omit important questions. (It is not always easy to correctly identify which questions are really important.) Of course the rebuttal text cannot run over the size limit (e.g. 2000 words), but to avoid confusing the reviewers, I usually separate the rebuttal into two parts: the main part for major and common questions, and the detailed part for individual reviewers. My personal experience as a reviewer (both primary/secondary and tertiary) is that I would not mind seeing a rebuttal running up to the length as long as the main part is clearly marked and shown up front. This allows me the flexibility to skip the more detailed parts if necessary. I have also observed that the reviewers tend to feel respected if the authors answer their questions, even for relatively minor ones.

. Before sending out the rebuttal text, add a short paragraph in the very beginning to thank all the reviewers for their effort in reading your paper and all the wonderful comments they have made.

. Submit the rebuttal. Do it early instead of waiting for the last minute, for obvious reasons. If the mechanism allows you to overwrite the previous uploads, definitely do so early. You can update the document later.