Confessions of a researchaholic

Look for what is good in a paper. For the rest (that is not so good), try to make constructive comments so that the authors can improve the paper.

Looking for the good

I have seen many, many reviews, both as primary/secondary (for which I can see reviewer identities) as well as tertiary (for which I cannot). My overall impression is that on average, reviewers tend to have a negative bias. That is, I have seen more good papers rejected than bad ones accepted. This is sad, as the prosperity and health of the academic community (as well as the human civilization in general) depends crucially on the timely publication of good ideas. The risk/reward ratio is highly asymmetric; a bad idea, even though published, can do little harm as it tend to be quickly ignored and forgotten. (Notable exception: Das Kapital by Karl Marx.) A good idea, if not published, could take tremendous effort to recover (e.g. proving Fermat’s last theorem) or is forever lost (I cannot even give you examples on this).

Why you should be positive as a reviewer

Humans, by nature, tend to be more critical towards others than to themselves. But here are a few reasons why being positive as a reviewer can be good for you:

. This is a small world. What comes around goes around. If you are known as a paper killer, it can not only harm your reputation but also makes others more likely to be negative towards your papers. There are a few well known folks in the graphics community who are highly successful and yet unpopular enough so that they do not tend to be invited to serve as paper committee members. Their opinions also tend to get ignored during committee discussions.

. Your success depends highly on your followers. Do not discourage them. If you see people citing or extending your work, it is to your benefit to help them as they will in turn help you become more famous and more influential in the long run. (Trust me on this: I saw this happened to me before, and I want to see it happen again.) Some people (especially more junior guys) tend to treat this as a kind of threat as they want to claim all credits themselves. The very best researchers start a new field/subject (or at least help popularize it), and move on to a different one leaving others cite and expand their earlier works. People who have to stay on the same subject or field tend to lack sufficient capability or courage to move on.

. The health of a community depends on how the members treat each other. Nobody likes to be in a place where people ambush each other. So do not start to behave like one yourself.

All in all, my experience is that one’s reviews reflect one’s personality and intelligence. Good researchers tend to write good and positive reviews. Cynical and nonconstructive reviews tend to come from those who are sour or feel insecure.

Being fair and constructive

Note that I am not saying you should accept bad papers. Papers with incremental or boring ideas should not be published, as otherwise it is unfair and can drag down the quality of the venue you review for. Papers with good ideas but major flaws should also be fixed before publication, as otherwise they might not attract the attention they deserve. (A common example is a good idea expressed in a bad writing; if people have difficulty understanding an idea, it would probably not make as large an impact it should. In that case, it is usually much better for everyone involved if the writing can be improved before publication.)

Be specific and constructive in your reviews so that the authors could have concrete clues about how to improve their paper.

SIGGRAPH now publishes the technical papers committee list a few weeks prior to the submission deadline. Here is my suggestion on how to take this information to your advantage.

First, let me tell you what NOT to do. It is very common for people to try to guess who is likely to review your papers, but this is entirely useless and counterproductive. In general, there is indeed a positive correlation between the committee member expertise and your paper subject, but it is not a very strong correlation and there is a non-trivial chance that a paper will be assigned to an unexpected committee member. (This is mainly due to how the paper sorting and assignment process goes but I am not going to elaborate on this here because I do not think it is necessary for authoring a good paper.)

So my personal strategy is to make sure the paper will survive *any* committee member. This is the safest bet and you cannot go wrong with that. (But do check the committee member list constantly though as sometimes it may change prior to the deadline.) Specifically, you should write your paper (especially the title and abstract) in a way so that it is as clear as possible on what it is all about to minimize the chance of misunderstanding. (If the paper sorters misunderstand your paper, they are more likely to assign your paper to the wrong committee member. And if the senior reviewers misunderstand your paper, they are more likely to assign your paper to the wrong tertiaries. And if reviewers misunderstand your paper they are more likely to score it lowly.)

It also helps to read and cite papers from the committee members for two reasons. The first reason is that the background and expertise of a committee member will obviously have a huge influence on how she is going to review a paper, so if you understand her, you stand a better chance of appreciating how she might approach a paper. The second reason is that psychological studies indicate that it is almost never too much to flatter, and people are usually happy to see their own papers cited and attributed in a positive manner.

If you want to push this further, my personal philosophy is to assume that my paper will be reviewed by the worst possible reviewers and prepare for that. Like Andy Grove said, only the paranoid survive.