Tags: * equal spacing of conferences * several rounds of acceptance * poor ARR meta/review quality * for tracks * for bidding * for opt-in
"We should do everything we can to improve the percentage of reviews that are submitted on time. If
this is easier in an 8-week cycle, then that would make sense to meSEMICOLON hunting down missing reviews
has always been one of the unfortunate parts of an area chair‘s job, and this gives them the time
to do it. By contrast, having four reviewers per paper sends a signal to reviewers that it is okay not
to submit a review that you committed to. Similarly, any method for spreading out the submissions over
the year to avoid peaks in the reviewing load seems like a good thing. I don‘t know if Kummerfeld‘s
system would achieve this, but it seems worth a shot. The trickiest problem seems to me how many papers
to accept in the first round, given constraints on the overall number of papers the conference can accept.
We don’t want to make it harder or easier to get a paper accepted in the different rounds. Perhaps
one needs to set aside a certain number of papers that can be accepted in the first round, with the
option that papers from the first round can be retroactively accepted in the second round if there is
leftover space. To me, the biggest problem with ARR is that the reviewing quality is abysmal. I have
observed this as an author, but as a reviewer I have also been assigned papers outside of my area of
expertise, and still had to review them although I brought this up to the action editor. This must be
fixed as soon as possible to maintain the credibility of our reviewing system. It seems obvious to me
that we need tracks back. In the olden days, we had area chairs for the tracks who personally recruited
their reviewers - this is a process that is now probably harder to scale, but I would rather provide
automated support in reviewer recruitment than in reviewer-to-paper assignment. This not only makes
it easier for the area chair to find reviewers who are competent to review a paper (they all come from
a somewhat coherent community), but it also establishes a personal relationship between the area chair
and the reviewer that is crucial when reviews are delayed. On the other side, authors put thought into
the areas to which they submit their paper, with an eye towards where it will be reviewed best. This
piece of information is lost when papers are automatically assigned to action editors instead of areas.
Personally I can‘t think of a single reason why one wouldn‘t just bring tracks back as they were.
I would also strongly vote to let reviewers bid on papers again. By bidding on papers, reviewers take
some ownership in the papers they need to review, which I bet will improve the percentage of reviews
that are submitted on time (and the amount of thought the reviewers will put into the paper). If there
are too many submissions (per track?) to make bidding on the whole list of submissions feasible, we
could do top-k predictions for the current paper-to-reviewer matchiing system to pre-filter the list
of submissions for each reviewer (to k=100 or so). Finally, for the same reason, I think opt-in is really
important. Reviewing for ARR right now feels like I am a remote-controlled reviewing droneSEMICOLON
an intransparent reviewing scheme requires me to review an unpredictable number of papers per month,
and I have no say at all in what I‘m supposed to review. I really dislike this feeling. If we want
to increase percentage of reviews on time and reviewing quality, we need to find ways to give the reviewers
a sense of agency back. Opt-out operates in exactly the opposite direction: I don‘t even have agency
in volunteering to be a reviewer. At the same time, if everyone who _submits_ to ARR becomes a reviewer
by default, ARR pollutes its reviewer pool with a large number of reviewers who have never even gotten
an ARR paper _accepted_ and thus have no idea of the standards of our community. It sucks, but opt-in
seems like the only option to me."