Code Review Is the Hard Part

"Using a Magnifying Glass and Driving" by Mike "Dakinewavamon" Kline is licensed under CC BY 2.0.

There is currently some recognition of the fact that with code creation having become cheaper, the burden of code review has not necessarily become easier. This is especially the case for people who get LLM-written pull requests who don’t use LLM tools for review. I can only imagine how exhausting it must be. I have only seen one complete waste of time pull request on one of my projects, but it was pretty irritating. Having said this, for some of my projects, review has been the hard part for many years already. It’s often easy to send a drive-by pull request but it can be much harder to find a competent, careful reviewer who is familiar with a whole bunch of RFCs, some superseding the others, who can say “this is correct and probably won’t break a bunch of legacy code you don’t know about”.

Add to this the fact that you don’t get much credit on GitHub for “spent X hours in careful review of brittle pull request that had Y sharp edges that the contributor never considered”, but you do get kudos for every one of the low effort pull requests that you send (I’m looking at you, Hacktoberfest).

If you’re doing it for the likes, then code review is not the way to go, even if it’s equally as important as writing the code.

I don’t have an actual solution for this. I’m just saying that the gatekeepers in my universe seem to be in short supply.

Do I use LLMs for code review? I do, but while I might trust the LLMs to find a lot of the problems in a pull request, I don’t trust them to find all of the problems and I certainly don’t trust them to be the final arbiter of code correctness. For any non-trivial change to Open Source code that actually matters, I don’t see a good way to keep a human out of the loop. I write a lot of Open Source that doesn’t matter, but if it’s actually something that people use, I feel a responsibility to make a best effort not to explode whatever they’re working on.