Tags: * ARR is the only way to scale our conferences * for 12 week cycle * no reviewers from outside NLP * fix late reviews
"I want to thank all the ARR people for getting this big experiment started. I think it is ultimately
the only way to scale our conferences, and will be patient to work with in this experiment until the
kinks are ironed out. Additional feedback: 1. I like the suggestion to extend to 8-week review period.
I would actually prefer an even longer 12-week review period: 1 month for reviewers, 1 month for meta-reviewer,
discussion, and chasing late reviewers, plus 1 month for buffer. I think the cause with low review quality
is that everyone is overworked. We should slow down the pace. If we consider that in the past we strive
to submit to on average 3-4 major *ACL conferences per year, then it should be fine to have only 3-4
ARR deadlines per year. 2. I strongly support the idea of focusing ARR on major ACL conferences. This
will help reviewers and meta-reviewers know how to write their reviews and calibrate their scores. But
this needs to be communicated. I think currently reviewers and meta-reviewers do not have enough training
on what is expected. 3. I think we shouldn't require all authors of a submission to be considered for
reviewing assignments, because there may be collaborators who are outside our field and are not trained
in our ARR expectations. One remedy: we could perhaps require each submission to explicitly nominate
one author as reviewer. 4. As a meta-reviewer for ARR, I would say that the part I hate the most is
chasing late reviews. Or assigning emergency reviewers using a Google spreadsheet outside of the ARR
system. It is a significant time sink that doesn't utilize my technical expertise. I wish the process
can be streamlined somehow. Incremental improvements in the infrastructure for issues like these will
improve people's productivity, and for this reason I strongly support the suggestion of hiring a FTE."