Tags: * against tracks * against integrated system * poor ARR meta/review quality * against emphasis on career concerns * against findings
"Disclaimer: I haven't been involved in ARR at all. I have been declining review/AC requests, and not
submitting papers to *ACL/EMNLP (slightly) before ARR started (partially for personal circumstances,
but largely due to losing my belief in the effectiveness of the process). - My main criticism of ARR
is if we can have the same reviewing process for multiple conferences, maybe we should not have multiple
conferences. Regional ones are, of course, understandable but the benefits of the RR is questionable
(as those interested in joining a regional event are likely to target one of them). - In my previous
experience as an author, I have never seen as disparate (and some being simply clueless - not necessarily
negative) reviews as in ACL conferences than any other venue. I think this is the effect of recruiting
(almost) any author as a reviewer. This may be a solution to the shortage of the reviews, but I do not
think it results in high-quality reviewing. - Although it was also asked in the survey, a note on the
areas/tracks: I am strongly against it. An automatic matching system (e.g., as used by NeurIPS, ICML)
combined with competent chairs would work much better. - There is a strong emphasis on ""careers"" of
participants in the procedure in the discussion so far. Although this is understandable for an individual,
for the organization, the main concern should be advancing the field as a whole. Of course, being fair
is important, but if/when the advancement of science conflicts with the career concerns due to questionable
values/practices in academia, organizations like ACL should not adjust to these practices, but perhaps
act (outside the reviewing/publishing process) to reduce such questionable practices. For example, an
common problem (in my opinion, but I believe this is shared by many) is the extreme pressure to publish
in high quantities. Instead of making the conferences even larger, doubling the published papers in
the proceedings every few years, and creating publication ""opportunities"" like ""findings of ..."",
ACL should perhaps promote acceptability of high-quality, low-quantity output as a measure of success.
- On a related note, ""findings of"" proceedings are probably a good sign that the review procedure
is broken. Although some of the justifications make sense, we all know that this is a practical hack
to reduce the repeated submissions, by channeling some of the rejected papers to a ""less reputable""
venue. If a paper is not worth publishing, then it should not be published. If it is publishable, but
being rejected, we should look for the reasons for it to be rejected and fix them (as I believe is the
point of this survey SEMICOLON)."