Tags: * poor ARR meta/review quality * improve matching * measure quality of matching * against fully automatic matching
This survey is too focused on activities, and not enough on accomplishments. The problem is that people
do not have much confidence in the reviewing process. The reason is because too many of the reviewers
are unqualified and unsympathetic to the area. The solution is to focus on the matching process. The
process needs to have enough qualified people who know the area to make sure that the reviewers are
as qualified and as sympathetic as the target audience for the paper. This cannot be done by a computer,
or by keyword matching. It is more important than writing up the meta reviews, and needs to be given
as much time and attention by the program committee as any other step in the process. If the paper is
assigned to a reviewer that is unqualified, or unsympathetic, they are likely to abstain. That should
not kill a paper, but it does. (Reviewers should be encouraged to abstain when the assignment process
messed up. We should track how many times the assignment process messes up.)