raw data (comments) plus assigned codes
against ARR (20): 008 014 032 045 046 069 092 125 132 162 163 166 168 188 205 222 224 239 258 266
thank you to ARR team (12): 015 031 062 093 099 105 106 134 172 203 220 232
return to pre-ARR system (7): 053 076 092 166 184 188 237
for ARR (5): 007 088 106 131 262
for rolling review (3): 070 199 252
for kummerfeld proposal (3): 035 087 263
against rolling review (3): 041 191 253
adopt ICLR model (3): 163 199 208
proposal: 2 week review, two week author response, two week final decision (1) * increase acceptance rate for findings (1) * in favor of infrastructure, but what exactly is part of it? (1) * for integrated system (1) * for infrastructure (1) * for audit of ARR (1) * for ARR and for pre-ARR system (1) * Covid probable reason for reviewer behavior (1) * concrete proposal for how to proceed with ARR: review, analysis, proposal, community consultation, decision (1) * against integrated system (1) * against "permanent" workshops like sigmorphon (1) * adopt TACL model (1) * adopt ICWSM model (1) *
against revise-and-resubmit (2): 162 168
utility of revise-and-resubmit unclear (1) * shorten review period for resubmissions (1) * punish new submissions that are resubmissions (1) * Martin Haspelmath on revise-and-resubmit: href=https://dlc.hypotheses.org/2333 (1) * for single-conference rolling review (1) * for revise-and-resubmit (1) * for multi-conference rolling review (1) * consider frequent deadlines without revise-and-resubmit (1) * ban resubmission of bad papers (1) *
link all versions in openreview (8): 067 113 129 146 160 204 208 268
against openreview (4): 056 070 210 252
improve openreview interface (1) *
poor ARR meta/review quality (24): 001 004 009 027 032 054 060 063 068 096 115 136 143 159 166 179 199 219 235 251 253 265 267 268
against year-round reviewing (7): 002 023 031 043 113 156 166
need new reviewing criteria (6): 112 115 149 195 243 265
against junior reviewers (6): 095 149 154 190 209 239
against 4 reviews (5): 006 085 116 128 219
pay reviewers (4): 011 057 227 239
fix late reviews (4): 015 087 118 234
prohibit authors from reviewing for a conference they submit to (3): 066 095 178
incentivize reviewers (3): 019 109 128
for subreviewers (3): 052 147 149
senior reviewers are often bad reviewers (2): 056 145
release reviews earlier (2): 167 175
publish meta/reviews (2): 094 130
penalize bad reviewers (2): 128 219
make reviewer stats available (2): 007 048
improve interface for reviewers (2): 037 166
for reviewer diversity (2): 009 184
for authors as mandatory reviewers (2): 097 229
appeals process for bad reviews (2): 139 146
against structured review forms (2): 202 245
video conference for new (meta)reviewers (1) * use same scale for meta/reviews (1) * too much discrepancy between reviews and metareview (1) * the community does not have enough competent reviewers for all submissions (1) * short/long papers should have short/long review forms (1) * senior people do not have time for reviewing (1) * reviewing record should be factor in paper recommendation (1) * reviewer management too aggressive (1) * remove reviewers who guess author identity (1) * prohibit authors from reviewing competing papers (1) * previous reviews are often ignored (1) * predict need for emergency reviewers and recruit accordingly (1) * no reviewers from outside NLP (1) * no editing of reviews after release (1) * NLP reviewers do not have sufficient ML expertise (1) * max of 4 papers per reviewer per cycle (1) * many authors are not qualified reviewers (1) * make clear how to reach ARR (for authors, reviewers etc) (1) * let meta/reviewers rate each other (1) * increase reviewer diversity (1) * improve management of junior reviewers (1) * improve interface for reviewer assignment (1) * improve emergency reviewing (1) * force senior people to review (1) * force authors to review (1) * for paper updates during review process (1) * for more informative metareviews: potential for major conference acceptance, long or short etc (1) * for 4 reviews (1) * fix reviewer signup process (1) * fix random removal of reviewers (1) * first time authors should not review (1) * fire late (meta)reviewers (1) * financial rewards for good reviewers and authors (1) * favor/disfavor papers by good/bad reviewers for acceptance (1) * ensure each paper has senior reviewer (1) * enforce reviewer diversity (1) * emergency reviewing is too stressful (1) * do not force non-authors to review (1) * do not change score definitions (1) * disconnect between review scores and acceptance decision (1) * create reviewer certification program (1) * concerns about commitment of partially reviewed papers (1) * change "commitment" into "consideration-for", i.e., revision and reviewing in parallel to commitment (1) * ARR should sign up reviewers for a specific conference (1) * ARR bad fit for interdisciplinary reviewers (1) * against commitment of partially reviewed papers (1) * ACL reviewing is too complex (1) *
link meta/reviews to conferences (9): 047 050 082 093 111 122 197 235 242
link meta/reviews to acceptance (7): 025 080 089 144 146 186 236
for decoupling reviews and conferences (2): 103 259
separate reviewing for each conference (1) * for per-conference reviewing (1) * against decoupling of meta/reviews (1) *
for opt-out (4): 012 015 084 245
for easy opt-out (3): 019 039 077
sell opt-out better by contacting each author right after submission (1) * opt-out unfair to interdisciplinary and junior researchers (1) * improve interface for opt-in / opt-out (1) * for opt-in because of interdiscipinary authors (1) * for opt-in (1) * against opt-out for all authors (1) * against easy opt-out (1) *
for sticky meta/reviews (5): 010 050 119 173 231
soften sticky meta/reviews (4): 036 138 145 240
for stricter sticky meta/reviews (3): 058 170 223
against sticky meta/reviews (3): 180 235 259
for sticky meta/reviews (keep old reviews even when reviewers are changed) (1) *
improve communication with reviewers (15): 012 016 021 024 031 049 055 113 161 166 200 216 219 228 248
improve communication (5): 015 098 226 252 258
improve communication, reviewer-(S)AC (1) * improve communication with reviewers (scores) (1) * improve communication with authors (1) * improve communication between area chair and reviewers (1) * improve communication about ARR problems (1) * improve communication (use less email) (1) *
for author response (7): 001 096 160 175 196 213 254
for discussion, author-reviewer (6): 165 175 176 184 196 239
give reviewers time to revise based on author response (1) * for reviewer discussion (1) * for meta/reviewer discussion (1) * for discussion, reviewer-AC (1) * for discussion, author-reviewer and/or reviewer-meta/reviewer (1) * for discussion, author-reviewer and/or reviewer-AC (1) * for discussion, AC-SAC (1) * for discussion (1) * better AC/SAC-PC interaction (1) * against SAC interactions with ACs and reviewers to avoid inefficiency and confusion (1) *
improve matching (21): 009 020 027 049 063 077 095 128 151 152 155 158 174 175 176 179 194 245 248 255 258
against fully automatic matching (2): 111 179
measure quality of matching (1) * improve interface for reviewer matching (1) *
against 8 week cycle (4): 016 069 089 187
for longer cycles (3): 043 065 200
for 12 week cycle (2): 075 118
short cycle length means to much load on meta/reviewers (1) * make a findings decision in each cycle, separate from conference decision (1) * for short review cycle (1) * for 8 week cycle or longer (1) * for 8 week cycle (overlapping) (1) * for 8 week cycle (1) * for 6 week cycle (1) * cycle too short (1) *
review load proportional to submissions (7): 021 026 082 128 168 214 221
ARR review load too high (6): 092 097 108 156 177 249
improve interface for load management (4): 023 108 128 230
against review load proportional to submissions (3): 088 184 240
increase number of / load of senior reviewers (2): 149 219
ARR increases review load (2): 101 266
ARR does not decrease review load (2): 176 253
reduce review load for junior reviewers (1) * need statistics on (increased?) load due to ARR to make any decisions (1) * monthly reviewing increases review load? (1) * make ARR review load predictable (1) * load management important (1) * increase load of senior reviewers (1) * equal spacing of community review load (1) *
reviewer performance: rate, track, act upon (5): 000 026 127 137 184
reviewer performance: track how much authors review (1) * reviewer performance: flag bad reviewers (1) *
good reviewers do not get invited (4): 022 083 098 152
senior people do not get invited (1) *
opinion about mentoring depends on implementation details (1) * mentor area chairs (1) * for reviewer training (1) * for mentoring (1) * against reviewer training (1) * against ARR mentoring (1) *
blind reviewers to each other (2): 026 086
introduce new completely open, "no-blinding" conference (1) * for double blind (1) * blind (S)ACs to reviewers (1) * allow reviewers to unblind themselves (1) *
how to become AC? (2): 185 255
AC should read paper (2): 086 245
no ACs, just SACs (1) * increase AC seniority (1) * increase (S)AC diversity (1) * define roles of (S)ACs (1) * decrease arbitrariness / increase diversity of AC/SAC selection (1) *
distribute submissions over year (2): 001 199
charge submission fees (2): 042 227
time from submission to acceptance is too long (1) * shorten time between submission and acceptance (1) * several rounds of acceptance (1) * make ARR the sole submission system (1) * make all acceptance decisions for the next conference (1) * limit submissions per person (1) * incentivize early submission (1) * for workshop submissions to ARR (1) * enforce higher submission standards, e.g., punish authors who hide information (1) * EMNLP submission modalities were poorly communicated (1) * double submissions to softconf / ARR confusing (1) * allow double submissions both within ARR and ARR/softconf (1) *
ARR too complex (3): 008 104 239
publish ARR 'decisions' on papers (1) * pre-ARR system for EACL/AACL (1) * improve ARR tech support (1) * hire more staff, not only for ARR (1) * fully test ARR system before launch (1) * communicate plan for ARR (1) * ARR too stressful (1) * ARR not transparent (1) * ARR not timely (1) * ARR is not a helpful resource for PCs (1) * ARR harms diversity (1) * ARR good for authors (1) * ARR excludes interdisciplinary research (1) * ARR decisions improperly influenced (1) * ARR bad for diversity (1) * ARR alienates from ACL (1) * any decision on ARR should be democratic (1) * against new ARR roles (1) *
for tracks (12): 005 018 027 054 122 128 143 174 181 194 209 252
use keywords instead of tracks (1) * for tracks, but only for a subset (1) * for partial tracks (1) * do not use ARR for special themes/tracks (1) * against current set of tracks used at conferences (1) *
expand desk rejects (3): 079 081 151
soften desk reject (1) * against expanding desk rejects (1) *
allow authors to enter COIs (2): 038 238
make COIs private (1) * improve COI detection (acknowledgments) (1) * ensure that COIs are uptodate (1) *
publish papers before conference (2): 135 146
equal spacing of conferences (2): 143 146
reduce the number of conferences (1) * reduce in limbo times by increasing number of conferences (1) * make conferences more inclusive (LREC model) (1) * make conference PC honorary role (1) * increase number of ACL conferences (1) * if ARR continues, there should be only one conference, potentially with several instances (1) * fundamental flaw in ARR : papers are submitted for conferences (1) * for diversity of conferences (1) * differentiate tier 1 and tier 2 conferences (1) * ARR only for conferences (1) * ARR is the only way to scale our conferences (1) * ARR and legacy should have same deadline for a conference (1) *
abolish anonymity policy (4): 028 158 182 267
for anonymous open review (1) * ban arxiv (1) *
improve data collection interface (1) * for data collection (1) * data collection should be opt-in, not opt-out (1) * concerned about terms of use of data collection (1) *
track ethical breaches in the infrastructure (for both authors and reviewers) (1) * make handling ethics easier (1) * incorporate community feedback on ethics/reproducibility (1) *
put resources into TACL (3): 061 114 171
turn ACL anthology into a journal (1) * move ACL's focus from conferences to journals (1) * invest in journals not in ARR (1) *
survey confusing (3): 008 062 242
the survey should have explained what an action editor is (1) * survey unclear about partially reviewed papers (1) * survey question about extending ARR functionality poorly designed (1) * survey does not include masters students (1) * problems with announcement of survey (1) * future of reviewing document should have defined the roles of (senior) area chair / action editor (1) * decisions should be made based on evidence and research not based on surveys (1) *
march delay unacceptable (4): 004 093 164 234
use open research knowlege graph (1) * reduce online registration cost (1) * one FTE not enough (1) * maximize agency (1) * increase ACL paper length (1) * improve website (1) * horrible experience with special theme (1) * get Nihar Shah to give a tutorial (1) * focus ACL on "computation and language", encourage linguists to leave (1) * change citation style (1) * anywhere on earth time is confusing (1) * against emphasis on career concerns (1) * ACL is welcoming to new researchers (1) *