I think this also applies at all levels. I'd argue that it is the big reason people feel very frustrated with reviewing, and especially in hyped areas (e.g. ML). There's plenty of incentives to reject papers (low acceptance rates mean "higher quality" publications, advantage yourself by rejecting or being overly critical of your competitors, you get no clout for reviewing and no one will get upset if you take all of 5 minutes to review), but very few incentives (I can't even name one) to accept papers. It is fairly easy to dismiss papers as not novel because we all build off the shoulders of giants and things are substantially more obvious post hoc. Metareviewers and Area Chairs will 99/100 times stand with reviewers even if they are in the wrong and can be proven so (I had a reviewer give me a strong reject claiming I should compare to another work, which was actually our main comparitor and we compared to in 5+ tables and 5+ graphs). I can't see these issues being resolved until we all agree that there needs to be some incentive to write high quality reviews. The worst of this is that the people it hurts the most is the grad students and junior researchers (prevents graduating and career advancement). I'm not saying we have to accept papers, but I am saying we need to ensure that we are providing high quality reviews. Rejections suck, but rejections that don't provide valuable feedback are worse.
If the publication system is a noisy signal then we need to fix it AND recognize it as such. There's been plenty of studies to show that this process is highly noisy but we're all acting like publications are all that matters.
This is all before we even talk about advantages linked to connections even in double blind reviews, collusion rings, or citation hacking. I feel we can't even get the first step right.
We all get papers rejected. Nobel prizes get also papers rejected. And rejection is one of the very few tools we have to stop the crap from flowing in. This is not to say that all rejected papers are crap. Sorry to hear about your bad time with rejections, this is a universal thing.
I don't think this response really is getting to what I'm complaining about. Rejects suck, but often they are deserving. I'm not saying that we need to throw out the peer review system. I even specifically said I don't think we need to accept more papers. But I do think we need a mechanism to ensure that reviews are good and high quality. Especially the reject ones. If your paper is being rejected without feedback that can convey what needs to be fixed to become a good paper, then this wasn't a peer review.
I've had plenty of good reviews and plenty of bad reviews. A reject doesn't sting nearly as much when I think "maybe they have a point." But if a reviewer tells me to compare to something I'm already comparing to and is in many graphs and tables, I'm entirely unconvinced that they even read the paper and I think we can all agree that that type of reviewer is not benefiting anyone. They are the system failing.
You may disagree with the number of bad reviewers there are (and that's okay) but I highly doubt you would actually defend them. I'm just saying we need a system where we either encourage these people to change the _quality_ of their reviews or stop reviewing (I am _not_ arguing that we need to change their scores).
"I do think we need a mechanism to ensure that reviews are good and high quality. Especially the reject ones." This is a very important point, IMHO more for proposals than for articles. The latter deserving less time because time is at a premium, although I am aware that this is a most painful truth!
I think this also applies at all levels. I'd argue that it is the big reason people feel very frustrated with reviewing, and especially in hyped areas (e.g. ML). There's plenty of incentives to reject papers (low acceptance rates mean "higher quality" publications, advantage yourself by rejecting or being overly critical of your competitors, you get no clout for reviewing and no one will get upset if you take all of 5 minutes to review), but very few incentives (I can't even name one) to accept papers. It is fairly easy to dismiss papers as not novel because we all build off the shoulders of giants and things are substantially more obvious post hoc. Metareviewers and Area Chairs will 99/100 times stand with reviewers even if they are in the wrong and can be proven so (I had a reviewer give me a strong reject claiming I should compare to another work, which was actually our main comparitor and we compared to in 5+ tables and 5+ graphs). I can't see these issues being resolved until we all agree that there needs to be some incentive to write high quality reviews. The worst of this is that the people it hurts the most is the grad students and junior researchers (prevents graduating and career advancement). I'm not saying we have to accept papers, but I am saying we need to ensure that we are providing high quality reviews. Rejections suck, but rejections that don't provide valuable feedback are worse.
If the publication system is a noisy signal then we need to fix it AND recognize it as such. There's been plenty of studies to show that this process is highly noisy but we're all acting like publications are all that matters.
This is all before we even talk about advantages linked to connections even in double blind reviews, collusion rings, or citation hacking. I feel we can't even get the first step right.