Usual Criteria for Reviewing a Conference Paper

Clarity (1-5)
For the reasonably well-prepared reader, is it clear what was done and why? Is the 
paper well-written and well-structured? Does the English or the mathematics need 
cleaning up? Would the explanation benefit from more examples or pictures? Is there 
sufficient detail for an expert to validate the work, i.e., by replicating experiments 
or filling in theoretical steps? 

(Take into account whether any obscurity or minor English errors could be fixed with 
relatively little effort, or whether the paper requires more work than is likely to be 
carried out in the 3 weeks available from notifications being sent out to the camera-ready deadline.)

5 = Admirably clear. 
4 = Understandable by most readers. 
3 = Mostly understandable to me with some effort. 
2 = Important questions were hard to resolve even with effort. 
1 = Much of the paper is confusing. 


Originality / Innovativeness (1-5)
How original is the approach? Does this paper break new ground in topic, methodology, or 
content? How exciting and innovative is the research it describes? 

(Note that a paper could score high for originality even if the results did not show a convincing benefit.)

5 = Surprising: Noteworthy new problem, technique, methodology, or insight. 
4 = Creative: Relatively few people in our community would have put these ideas together. 
3 = Somewhat conventional: A number of people could have come up with this if they thought about it for a while. 
2 = Rather boring: Obvious, or a minor improvement on familiar techniques. 
1 = Significant portions have actually been done before or done better.


Soundness / Correctness (1-5)
First, is the technical approach sound and well-chosen? Second, can one trust the claims 
of the paper -- are they supported by proper experiments, proofs, or other argumentation?
 (Bear in mind that this is a 4- or 9-page conference paper, not a journal article).

5 = The approach is very apt, and the claims are convincingly supported. 
4 = Generally solid work, though I have a few suggestions about how to strengthen the technical approach or evaluation. 
3 = Fairly reasonable work. The approach is not bad, and at least the main claims are probably correct, but I am not entirely ready to accept them (based on the material in the paper). 
2 = Troublesome. There are some ideas worth salvaging here, but the work should really have been done or evaluated differently, or justified better. 
1 = Fatally flawed.


References / Meaningful Comparison (1-5)
Are the references adequate? Does the author make clear where the problems and methods 
sit with respect to existing literature? Are any experimental results meaningfully 
compared with the the best prior approaches? Note that if you feel references/comparators 
are missing, you should detail what they are in your review.

5 = Precise and complete comparison with related work. Good job given the space constraints. 
4 = Mostly solid bibliography and comparison, but I have some suggestions. 
3 = Bibliography and comparison are somewhat helpful, but it could be hard for a reader to determine exactly how this work relates to previous work. 
2 = Only partial awareness and understanding of related work, or a flawed empirical comparison. 
1 = Little awareness of related work, or lacks necessary empirical comparison.


Impact of Ideas or Results (1-5)
How significant is the work described? If the ideas are novel, will they also be useful or
 inspirational? If the results are sound, are they also important? 

5 = Will affect the field by altering other people's choice of research topics or basic approach. 
4 = Some of the ideas or results will substantially help other people's ongoing research. 
3 = Interesting but not too influential. The work will be cited, but mainly for comparison or as a source of minor contributions. 
2 = Marginally interesting. May or may not be cited. 
1 = Will have no impact on the field.

Impact of Resources (1-5)
In addition to its direct intellectual contributions, does the paper promise to release 
any new resources, such as an implementation, a toolkit, or new data? If so, is it clear 
what will be released and when? If so, will these resources be valuable to others in the 
form in which they are released? Do they fill an unmet need? Are they at least sufficient 
to replicate or better understand the research in the paper? 

5 = Enabling: The newly released resources should affect other people's choice of research or development projects to undertake. 
4 = Useful: I would recommend the new resources to other researchers or developers for their ongoing work. 
3 = Potentially useful: Someone might find the new resources useful for their work. 
2 = Documentary: The new resources are useful to study or replicate the reported research, although for other purposes they may have limited interest or limited usability. (this is a positive rating) 
1 = No usable resources released. (most submissions)

Overall Positives (1-4)
How strong are the (positive) contributions of the paper, putting aside, for a moment, 
any reservations you may have about things the paper does less well?

4 = The paper could really turn the field on its head/establish an entirely new research direction
3 = The paper breaks significant new ground/has a great idea which will change the direction of the field
2 = The paper builds incrementally on the shoulders of others
1 = The paper makes no positive contribution

Overall negatives (1-4)
How bad/unreconcilable are any flaws in the paper, ignoring for a moment, the things the 
paper does well?

1 = The paper is flawless, and executed perfectly
2 = I have minor quibbles about the execution of the paper/specifics of some ideas, but they are fixable within the confines of the current paper
3 = I have major concerns about the execution of the paper/specifics of some ideas, beyond a level which is reasonably fixable within the constraints of the current paper
4 = The paper is fatally flawed