Tags: * thank you to ARR team * for paper triage
"(First of all, I would like to thank everyone that made ARR possible. I don't think we reached an ideal
system yet, but it's good to see drive for change.) Although my evidence is mostly anecdotal as an AC,
I believe many problems would be easier to solve if we didn't have the huge amount of submissions at
every cycle. This sounds obvious, but nonetheless, I sometimes miss a discussion on *why* we are experiencing
this increase. And I think one of the reasons is just the low standards we have with papers these days,
probably because big names like to spread the idea of ""dirty/messy science"" which runs rampant on
arXiv. Not sure if/how feasible this is, but a veto system for experimental/empirical papers could be
put in place? For instance, the guidelines could explicit state that an empirical paper submission should
have: - Detailed description of any models used (like a Model Card). - Same for datasets, if any are
proposed. - Statistical tests for any metric comparison. - Error analysis. - Code/container to replicate
all experiments and results with a single command. Yes, even for industry papers. Then, if any experimental
paper misses one of the above, it is desk-rejected. Alternatively, if people want to publish something
preliminary that doesn't follow all of the above (to get feedback for instance), they can submit it
to a workshop (much like ML workshops focus on work in progress). My hypothesis is that such a system
will drastically reduce the number of experimental/empirical paper submissions, in a way that pushes
the content towards quality over quantity. Of course, there are probably many caveats that I am missing
it so I am not claiming it is the best solution. But in general, I would like to hear more discussions
about moving the community and the field away from the ""dirty science"" mindset that is currently plaguing
us. "