Confessions of a researchaholic

January 30, 2011

How to review papers

Filed under: Real — liyiwei @ 6:06 pm
Tags: ,

Look for what is good in a paper. For the rest (that is not so good), try to make constructive comments so that the authors can improve the paper.

Looking for the good

I have seen many, many reviews, both as primary/secondary (for which I can see reviewer identities) as well as tertiary (for which I cannot). My overall impression is that on average, reviewers tend to have a negative bias. That is, I have seen more good papers rejected than bad ones accepted. This is sad, as the prosperity and health of the academic community (as well as the human civilization in general) depends crucially on the timely publication of good ideas. The risk/reward ratio is highly asymmetric; a bad idea, even though published, can do little harm as it tend to be quickly ignored and forgotten. (Notable exception: Das Kapital by Karl Marx.) A good idea, if not published, could take tremendous effort to recover (e.g. proving Fermat’s last theorem) or is forever lost (I cannot even give you examples on this).

Why you should be positive as a reviewer

Humans, by nature, tend to be more critical towards others than to themselves. But here are a few reasons why being positive as a reviewer can be good for you:

. This is a small world. What comes around goes around. If you are known as a paper killer, it can not only harm your reputation but also makes others more likely to be negative towards your papers. There are a few well known folks in the graphics community who are highly successful and yet unpopular enough so that they do not tend to be invited to serve as paper committee members. Their opinions also tend to get ignored during committee discussions.

. Your success depends highly on your followers. Do not discourage them. If you see people citing or extending your work, it is to your benefit to help them as they will in turn help you become more famous and more influential in the long run. (Trust me on this: I saw this happened to me before, and I want to see it happen again.) Some people (especially more junior guys) tend to treat this as a kind of threat as they want to claim all credits themselves. The very best researchers start a new field/subject (or at least help popularize it), and move on to a different one leaving others cite and expand their earlier works. People who have to stay on the same subject or field tend to lack sufficient capability or courage to move on.

. The health of a community depends on how the members treat each other. Nobody likes to be in a place where people ambush each other. So do not start to behave like one yourself.

All in all, my experience is that one’s reviews reflect one’s personality and intelligence. Good researchers tend to write good and positive reviews. Cynical and nonconstructive reviews tend to come from those who are sour or feel insecure.

Being fair and constructive

Note that I am not saying you should accept bad papers. Papers with incremental or boring ideas should not be published, as otherwise it is unfair and can drag down the quality of the venue you review for. Papers with good ideas but major flaws should also be fixed before publication, as otherwise they might not attract the attention they deserve. (A common example is a good idea expressed in a bad writing; if people have difficulty understanding an idea, it would probably not make as large an impact it should. In that case, it is usually much better for everyone involved if the writing can be improved before publication.)

Be specific and constructive in your reviews so that the authors could have concrete clues about how to improve their paper.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Your email address will not be published. Required fields are marked *

 

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Theme: Rubric. Get a free blog at WordPress.com