Tags: * for discussion, AC-SAC * maximize agency * fully test ARR system before launch * communicate plan for ARR * improve communication with reviewers * ARR is not a helpful resource for PCs
"Some of the questions are difficult to answer because some information is missing. In particular, the
lack of possibility for the conference SACs to go back to the AEs and encourage discussion between reviewers
is a big issue. How would that be factored in? In theory yes authors submitting lots of papers should
do their part in reviewing, but how will that be enforced? The three RR pillars were all good in theory,
but they were not implemented. In the panel, Hal Daumé made a very compelling point about agency. The
system that will be put in place needs to maximize it. It also needs to be fully tested prior to launch.
Even with longer cycles, a dedicated team of people needs to think through the implementation, in parallel
of running RR, (which is right now patching fundamental issues. As Pascale Fung pointed out at the business
meeting, they are building a plane while flying it.) It would be re-assuring for the community to know
what the plan is going to be to build a centralized system that actually provides what RR aims for.
On a side note, there is a big psychological effect at play too: even with opt-out, one keeps getting
RR emails every month. RR promised load balance but that was far from the reality. In the past we knew
when we would get around 5-6 papers to review, and we could plan our work/family plans around that.
This was not possible in the past months. This is something that could be fixed with clearer and better
communication. Note also that while it is clear that our previous system was broken, the current survey
obscures a lot of the work that is needed from the PCs for integrating the RR info with the OpenReview
conference site — special theme needs to be handled separately to ensure appropriate reviewers, and
the integration is far from easy. PCs nowadays have to juggle between ARR, OpenReview, Underline, and
the local site…and in the current state of affairs, RR is not a helpful resource for PCs."