Comment 184

next comment

Tags: * against review load proportional to submissions * return to pre-ARR system * for reviewer diversity * for discussion, author-reviewer * reviewer performance: rate, track, act upon

"As can be seen from my answers, I'd prefer returning to the classical ""submission for a conference"" 
or infrastructure system. Apart from this preference, I am strongly against assigning papers for reviewing 
to authors who submit a lot of papers, since quantity by no means warrants quality. Even if many of 
their papers are accepted, having funding for many post-docs by no means guarantees that the colleagues 
in question take the time to provide solid reviews. Having said that, I am strongly in favor of a diversity 
of reviewers. I highly appreciate the initiative of ACL to continuously improve the reviewing process. 
Nevertheless, one aspect that was in my mind missing from the questions is the handling of answers to 
reviews, given that answers can be provided. Even though I have not submitted many papers to ACL* type 
conferences, I have already met a reviewer of one of my papers by chance, who was not even aware of 
the fact that answers had been submitted and was waiting for a discussion on the reviews of the paper/authors' 
answers that had never happened. One more suggestion I have is that given the focus of the community, 
some methods for evaluating review quality would be very useful. There are reviewers who submit reviews 
that could be submitted for almost any paper, not even confirming that they have actually read the paper. 
Such reviewers should be closely monitored in future. And my final suggestion is that you may have wanted 
to limited the character length for this comment field - my apologies for the length of this feedback. 
"