Comment 087

next comment

Tags: * equal spacing of community review load * for kummerfeld proposal * consider frequent deadlines without revise-and-resubmit * fix late reviews * concrete proposal for how to proceed with ARR: review, analysis, proposal, community consultation, decision * against decoupling of meta/reviews * opinion about mentoring depends on implementation details * against opt-out for all authors

"I believe the most important objective should be to spread the review burden across the year. A system 
that achieved this but didn't achieve anything else would still add a lot of value. Assuming a fixed 
(and identifiable) pool of high-quality reviewers, spreading across the year increases the probability 
that a paper gets a high-quality review. In my experience as an action editor, ARR does not currently 
achieve this - some months I get 10+ papers, other months I get zero. For better or worse, this is probably 
not caused by infrastructure limitations but is more organisational - something like Jonathan Kummerfeld's 
suggestion could be a step in a good direction. I am not sure why ""frequent deadlines"" and ""revise-and-resubmit"" 
are always coupled together, we could have frequent non-conference-specific deadlines without any support 
for revise-and-resubmit. The most painful part of the process for me as an action editor is dealing 
with large numbers of missing reviews and the last-minute rush to find emergency reviewers. This also 
causes an uneven distribution of labour and stress over time. The problem seems to be much worse with 
ARR than it was with traditional conference reviewing (perhaps because reviewers traditionally opt in?). 
We need to find a way to know early whether some reviewers are not going to submit reviews, and have 
some accountability for reviewers that fail to submit reviews without informing AEs early. Extreme idea: 
if you fail to provide assigned reviews for N cycles, you are prevented from submitting a paper for 
the next M cycles. Overall it seems incorrect to conclude that because a particular implementation of 
rolling review could have been launched better, rolling review as an approach is flawed. I also do not 
believe it would be better to throw everything out and invent yet another system which would only get 
user feedback after a number of years (one version of Hal's proposal). I would suggest: - A formal review 
of everything we have learned from the ARR launch, conducted by a team that includes some community 
members that did not work on ARR - A strict timeline for the analysis (e.g. 2-3 months) - A proposal 
for where we need to get to (do we know what the desirable end state is? if not, how do we identify 
it?) and an analysis of whether the current ARR platform is the right starting point - A consultation 
period open to the full ACL membership - A final decision endorsed by the ACL Exec and other relevant 
stakeholders (e.g. conference PCs). ""Use a rolling review system for reviews, but not meta-reviews 
(which would then be handled by an Area Chair within the conference)"" -> it's critical that area chairs/action 
editors can interact with reviewers, e.g. to support reviewer discussion and to push reviewers to improve 
their reviews. This proposal seems likely to prevent this (e.g. the AC may be looking at a paper that 
was reviewed six months previously and the reviewers may have forgotten the details). ""Develop mentoring 
for both reviewers and action editors"" -> a noble objective, but who does the mentoring? Good mentoring 
can be a lot of work. I'm not sure whether I support or oppose without some detail about implementation. 
""By default, authors sign up to be considered for reviewing assignments. They must opt-out if they 
are unable to review."" -> This seems too extreme. Some coauthors may be very junior, or not from the 
field - putting them on the spot adds friction. Maybe there could be a requirement that for every paper, 
at least one author must confirm that they will review? "