AI ruined reviewing and I am sad about it

I finished my batch of reviewing this week (I was a reviewer for AAAI, AISTATS, ICLR in parallel as they overlapped; NeurIPS was done already) and I am grieving something myself now: the slop destroyed reviewing for me.

You see, I feel a bit weird in this, but I really enjoyed reviewing ever since I got to review for ICLR and NeurIPS starting in 2018. It always felt like a sneak peek into the research frontier and I have learned a lot by reviewing (some of my favourite papers I found because I was a reviewer on them).

As such, I have been reviewing for TMLR, CVPR, AISTATS, IJCAI, NeurIPS, ICML, ICLR, AAAI, LoG, RLC and some other venues over the years, and kept doing so even after finishing my PhD and dialing down my research to focus on software at Adaptyv and now my new employer, and so didn’t have to since I wasn’t submitting papers.

Of course, I had noticed the trickle of slop over the last year or so, but in this batch (4 AISTATS papers, 4 ICLR, 4 round 1 AAAI and 2 round 2), I had to flag 3/4 AISTATS submissions, 2/4 for ICLR, 2/4 round 1 AAAI and 2/2 round 2 AAAI for LLM slop (either officially or as part of my critique).

I am not even opposed to writing papers with LLMs or vibe coding on a moral basis¹; if the papers were legible and good, if they still carried insight, I would tolerate some LLMisms².

But when I say slop, I really mean it: not even wrong drivel, the authors obviously not bothering to edit and make the reading experience or claims cohesive. One ICLR submission did use LLM for “polishing” which made it harder to read, but at least it was clear there was hand editing done and the authors cared.

Overall, out of 14 papers I had to review, 9+1 were slop or LLM influenced.

So yeah, thanks, I hate it :-(

There was something special to reviewing, which is why I always tried to actually engage with the papers and be constructive, even if they were bad or hard to read - one or more authors pouring large chunks of their life into 8 to 9 pages + appendix, polishing things to the best of their ability, trying to convince you of the cool result they had.

There was a connection to the authors, through this weird medium, and an exchange through the review and rebuttals - I learned about certain types of statistical rigour through reviewing, my own papers got better through rebuttals (although there were of course also absolute horseshit reviews at times), and I got acknowledged in some papers for having the authors do ablations that ended up strengthening the papers or suggesting rewrites (often doing part of said rewrites myself).

It felt like science at its collaborative best. And now it is gone, at least for a while, because of the slopocalypse hitting (computer/machine learning) science finally.

So yeah, I’m mourning the defacing and destruction of another piece of my little nerd corner that started between 2020 and 2022 when everything stopped being ML, optimization and statistics and became “genAI” and “AI” with the image and text generative models.

It was also very clear to me that the majority of reviewers are doing the rational thing and either using LLMs for their reviews or dialing down their efforts and simply mentioning the slop and impossibility to read things (I tried to steelman and understand each paper, critique it as if it wasn’t slop when possible, then call out the slop - something I won’t do again I think, I’m just tired).

I am sure that there will be a new equilibrium - the slop is indeed slop, the people who care will create new cultural norms shunning careless AI slop, the people who don’t care and just want the economic benefits at the end of the scientific process will realize the AI review calling a piece of slop “clear derivations” isn’t actually filtering the drivel from the pearls they want and will begrudgingly fund them to augment the industrial knowledgeworks with the special touch that only comes from obsessive nerdy engagement with the topic³, or be outcompeted economically by the ones that do so.

But for now, I am grieving.

Other commentary this week

Edited in as soon as I had time