Who Handles Your Security Reviews?

Door handle designed by Jasper Morrison by JT at JMLtd, licensed under CC BY-SA 4.0.

I’ve had a closer look at what’s happening in the world of OSS security these past few months, and I’ve seen firsthand where vulnerabilities are turning up. What I find particularly surprising is that it’s not just in legacy code. CVEs are also turning up in brand new code that’s being uploaded right now. Just because we now have more tools at our disposal doesn’t mean that security issues are a solved problem.

How do we solve this problem? Probably via review. But review by whom? Maybe you use an LLM. Or a friend who is interested in security. Possibly your ecosystem already has a program with engineers you can request reviews from. It could be all of the above.

#LLMs cut both ways

While LLMs are very good at finding exploits, they are also really good at creating them — just like humans. The cynical approach to this says that of course vibe-coded software will be riddled with errors, but to be fair, the vibe coders also have defensive tools at their disposal. They’re either not using them consistently or stuff is just getting through — in the same way that you can get a security review from your very annoying and brutally thorough colleague and they will likely still have missed something.

#Make security review a habit

If you use LLMs, one way to improve your odds is to get into the habit of invoking a specialized security reviewer on your new code. Here’s my standard security-review.md which might give you a starting point if you’re looking for inspiration. At the end of the day it’ll come down to what your LLM skills are directing the agent to look for but also which model you are using to implement the skill.

Depending on which model you are using, you may want to avoid doing things like asking for a proof of concept. Certain Claude models have been known to deny the request or even lock you out of your account. (I have that as first-hand anecdata). If you’re serious about doing security research and you use Claude, you may want to look at Anthropic’s Cyber Verification Program.

#Scan the code you already have

As an interesting exercise, try pointing an LLM scanner at your existing code. You may be surprised at what you find, and maybe you can even ship fixes before the security researchers find them. For example, just a few days ago I found security hardening worth doing in diff-lockfiles while making unrelated changes.

#Ask for help

Your ecosystem may already have a program you can turn to:

If you’re involved with one of these ecosystems or already know a security researcher, ask them to scan your code and send you the results. Or ask a human to review it the old-fashioned way — that has worked for decades and is still a fine choice.