Take-aways from the comments
Author:
Hinrich Schuetze
The focus of these take aways is feedback that is complementary
to Jonathan Kummerfeld's
summary of the
ACL
Reviewing Survey Results.
Abbreviations
AC = area chair = AE = action editor = metareviewer
SAC = senior area chair = SAE = senior action editor
Main pain points
Apart from OpenReview (see below), if we take number and
lengths of comments as an indicator of what the main
points are then I would come up with this list:
- Reviewer-paper
matching needs to be improved. This is a major cause
of problems because both reviewers and authors are
negatively affected if reviewers have to review papers
that are not a good match for them.
- Poor
(meta)review quality was the code discussed most
frequently in the comments. Clearly, something needs to
be done to improve it.
- Many survey participants felt that ARR
put
too much load on reviewers, partially because the ARR
model is to review year round, on a tight schedule, as
opposed to a few times a year, at a more leisurely schedule.
Maybe we could get used to more distributed reviewing if
load management is improved? Journals (including
TACL) also rely on "constant" reviewing.
- The lack of
author
response and
reviewer
discussion was also frequently criticized. My
understanding is that these have been added to ARR already.
OpenReview
Many authors and reviewers were new to OpenReview, so some
of the general unhappiness should probably be contributed
to the pain of having to learn a new system. But there are
many issues with OpenReview that would improve ARR a lot
if they could be fixed:
Actionable items
- Subreviewers
are frequently and successfully used in many reviewing
systems. As the comments point out, subreviewers are
a good tradeoff between making sure senior reviewers are
involved in many reviews on the one hand and being
realistic about how much work we can expect from them
on the other. I only see an upside in using them. The
challenge is that OpenReview does not currently support
subreviewers?
- Maybe we should take a serious look at strategies
that quickly weed out clearly bad papers in the
beginning, e.g., by
expanding
desk rejects or some other form of
paper triage.
Disagreements
On many issues, there is no consensus in the community, so
it seems unavoidable that not everybody will be happy
with whichever alternative ends up being selected.