What vulnerable code is and why it goes unnoticed

This is the third article in the series where we are going to teach a machine to find security flaws in code, all in .NET. In the first two, we stayed on the machine learning side: what it is, and how machines learned to read. In this article we switch sides and go to security. Because before teaching the machine to find these flaws, we need to understand what they are: what exactly is "vulnerable code"?

As always, in the simplest way I can manage. If you have never programmed, come along, you can understand everything.

A bug and a vulnerability are not the same thing

All software has defects, the famous bugs. The screen that freezes, the button that doesn't respond, the calculation that rounds wrong. Bugs are annoying, but in general they only bother whoever is using the system.

A vulnerability is a special kind of defect: one that someone can exploit on purpose. The difference is intention and consequence. Think of a house: a door that jams is a bug, it annoys you every day. Now a lock that opens with any hairpin is a vulnerability: it works perfectly day to day, you don't even notice it exists, until the day someone with bad intentions decides to try it.

And that is what makes vulnerabilities so treacherous: the system works. The tests pass, the clients use it, everything looks great. The flaw only shows up when someone, with bad intent, uses the system in a way nobody anticipated.

The celebrity: SQL injection

If vulnerabilities had a hall of fame, SQL injection would be the first name on the wall. It has existed for over 25 years, appears on every list of most exploited flaws, and keeps causing incidents in companies of every size to this day. And the best part for us: it is perfect for understanding how a vulnerability is born.

First, the scenario. Almost every system stores its data in a database, and talks to it in a language called SQL. When you type your name into a login form, the system builds a sentence in SQL more or less like this: "database, bring me the user whose name is JOHN". So far, so good.

The problem is how that sentence is built. The naive way, the system takes what you typed and glues it into the middle of the sentence, literally. And that is where the attacker comes in. Instead of typing "JOHN" in the name field, he types something like:

JOHN' OR 1=1

It looks like nonsense, but watch what happens: glued into the middle of the sentence, it becomes "database, bring me the user whose name is JOHN, or where 1=1". And since 1 always equals 1, that condition is true for everyone. The database, obedient, hands over all the users. Depending on the system, the attacker just logged in without a password, or downloaded the entire customer table.

Translating the essence: the field was supposed to receive data, but it received a command, and the system executed it. It's like a waiter who takes down orders at a restaurant and fulfills whatever is written on the paper: you write "one pizza" and he brings the pizza. Someone writes "one pizza and the key to the safe" and he brings the key to the safe. The waiter did his job just fine, it's the process that never instructed him to be suspicious of what comes on the paper.

The most impressive part: the fix has been known for decades (basically, teaching the system to never mix data with commands). But the naive pattern keeps showing up in new systems, every year. Deadline pressure exists in every project, and a line that works perfectly in the tests raises no suspicion at all. That is why it never dies.

Two other classics, in passing

Just so you can see the pattern repeating:

XSS (cross-site scripting): instead of injecting commands into the database, the attacker injects a script into the page. Think of a public message board: he leaves a "message" that, when read, executes an action in the browser of whoever opened it, like stealing that person's session. The field was supposed to receive a comment, it received a program.

Path traversal: the system lets you download your files, and the attacker asks for a file with a malicious name, full of "go up one folder, go up another", until he reaches the server's own files, like the passwords one. The field was supposed to receive a file name, it received a map to the safe.

Did you notice what repeats? User input treated with too much trust. That is the root of a gigantic family of vulnerabilities. Keep that pattern in mind, because it is exactly what we are going to teach a machine to recognize.

The official catalog: OWASP

The organization that keeps this subject in order is OWASP, a nonprofit foundation that became the world reference in application security. Its most famous product is the OWASP Top 10: the list of the ten most critical categories of web vulnerabilities, updated from time to time. Injection (the SQL injection family) has been on that list since the first edition, which shows the size of the challenge.

And how are these flaws found today?

Three defenses work on this today, each with its own blind spot:

1. Human review (code review). Another person reads your code before it enters the system. It is the richest defense, humans understand context like nobody else. But human attention is a finite resource, and the volume of code produced in a modern project is enormous. There is no team in the world capable of reading every line with the same depth.

2. Static analyzers (SAST). Tools that scan the code looking for dangerous patterns, without executing anything. They are essential and every serious project uses them. But here I hand you back to the previous article: how do you think these tools work inside? Handwritten rules. Thousands of them. And you already know how that story usually goes: the attack changes shape, the rule misses it. The innocent code looks suspicious, the tool cries wolf. And too many false alarms teach the team to ignore the real alarm.

3. Penetration tests (pentest). Specialists attack the system for real, the way an attacker would, but with authorization. They find what nobody else finds, but they are expensive, happen once in a while, and test the finished system, when fixing things is harder and more costly.

Do you see the empty space between the three? Human review understands context but doesn't scale. SAST scales but doesn't understand context, only fixed patterns. And the pentest arrives late.

The question that closes the article

Now put the pieces of the series together. In article 1: machines learn from examples, not rules. In article 2: transformers read text understanding context, and code is text. In this one: today's tools look for vulnerabilities using handwritten rules, the same house of cards the spam filter abandoned twenty years ago.

So the question asks itself: what if we showed a transformer thousands of vulnerable and safe code samples, and let it learn the difference? Not to replace the three defenses, but to add a fourth: one that reads like human review, scales like SAST and works all the time, not once a year.

That is exactly what exists, it has a name, and it is the subject of the next article: the transformers trained to read source code, with CodeBERT leading the family.

Conclusion

Recapping:

A vulnerability is the defect someone exploits on purpose: the lock that opens with a hairpin, not the door that jams
SQL injection, the classic of classics: the field expected data, received a command, and the system obeyed
The pattern that repeats in XSS, path traversal and so many others: too much trust in what comes from the user
OWASP Top 10 is the official catalog of these categories
Today's defenses: human review (doesn't scale), rule-based SAST (doesn't understand context) and pentest (arrives late)

In the end, the search for vulnerabilities today repeats the story of the 90s spam filter: handwritten rules, chasing a creative adversary. We have already seen how that race usually ends, and we already know which technology changed its ending.

Blog

Series