The previous article ended with two numbers: CodeBERT got 80.9% of the exam right, and the transformer built from scratch got 73.0%.
This article is the investigation of those numbers.
80.9%: the number that raises no suspicion at all
There is nothing wrong with 80.9% at first glance. It is not high enough to be a lie, not low enough to worry. It is exactly the kind of result you write down, celebrate quietly and move on from.
It was wrong. The real score was 77.3%.
I am keeping that three and a half point difference on purpose, because it is the point of the article. You have probably read that a score which is too high is a sign of leakage in the data. That is true, and it is the easy version of the problem. The hard version is this one: leakage that produces a modest, plausible number, exactly the size you expected. That one sets off no alarm.
That is how it got through.
If the number does not give it away, what does?
In this case it was a very simple habit, and it is what I would take from this article if I could only take one thing: before believing any result, open the exam and read a few questions. Do not look at the score, look at what was asked.
I took half a dozen examples that had landed in the exam and put them next to examples from the training set, to see with my own eyes what the model had faced. It took about two minutes.
That is where it became obvious.
Remembering how the exam was built
The examples come from moulds. Each mould is a snippet of code with holes, and each hole is filled eight different ways, generating eight examples.
After generating every example, I shuffled everything and set a portion aside for the exam. Like this:
var shuffled = examples.OrderBy(_ => random.Next()).ToList();
var evaluationSize = shuffled.Count * 20 / 100;
return new Dataset(
[.. shuffled.Skip(evaluationSize)],
[.. shuffled.Take(evaluationSize)]);It looks reasonable. Shuffle everything, set aside 20% for the exam, train with the other 80%. It is what almost every machine learning tutorial teaches, and in most situations it is correct.
Here it is wrong, and the reason is quite specific.
The defect: the same mould landed on both sides
Take any mould. It generated eight examples. The shuffle spread those eight across the whole pile.
Then, when I set aside 20% for the exam, what happens in practice is: six examples from that mould stay in training and two go to the exam. Not because anybody decided that, but because the draw distributed them.
Now look at what the model faces at exam time. The question is this:
var sql = "SELECT * FROM Orders WHERE Reference = '" + request.Email + "'";
using var command = new SqlCommand(sql, connection);And during training it had seen this one:
var sql = "SELECT * FROM Users WHERE Name = '" + userName + "'";
using var command = new SqlCommand(sql, connection);It is the same mould. Three names changed.
That is not an unseen question. It is the same exercise from the homework with the names swapped. And getting it right does not prove the model understood what SQL Injection is. It only proves it recognizes a shape it had already seen six times.
Leakage: when the answer arrives before the exam
This problem has a name: data leakage. It happens whenever some information from the exam set reaches the model during training, even if nobody copied anything on purpose.
Here what leaked was not the exact examples. It was the mould. And since the mould is the structure, and the structure is precisely what the model learns to recognize, leaking the mould is practically leaking the answer.
That explains why both numbers went up, and why the small model went up more. Memorizing shape is exactly what a small model does well, and the exam was only asking for that. CodeBERT gained less from the cheat sheet because it was not memorizing, it was generalizing, and generalizing sometimes misses a case that rote learning would get right.
Both scored higher than they deserved, because the exam was measuring the wrong skill.
The repair: split by mould, not by example
The fix is deciding the split before generating the examples. Instead of generating everything and drawing lots, I choose which moulds stay out of training, and all eight examples of that mould go to the exam together.
for (var index = 0; index < group.Count; index++)
{
var isHeldOut = index % EveryNthTemplateIsHeldOut == EveryNthTemplateIsHeldOut - 1;
Render(group[index], isHeldOut ? evaluation : training);
}Every fourth mould of each category goes to the exam in full. Render is what expands the mould into the eight examples, and it drops all eight on the same side.
Now the exam asks the right thing. The questions are code patterns the model has never seen, not even the shape of. If it gets them right, it is because it understood something about concatenating user data inside a query, and not because it memorized a mould.
The general rule, which applies outside this project
Worth keeping this in a form you can reuse elsewhere.
When your examples were generated by something, split the exam by that something, not by the row of the table.
If the examples came from moulds, split by mould. If they came from patients, and each patient has several visits, split by patient. If they came from users, split by user. If they are measurements over time, split by period, leaving the future in the exam.
The question that settles it on the spot is always the same: is there anything in common between a question in the exam and some example in the training set? If there is, that is where the score will inflate.
The real scores
With the exam rebuilt, both models were trained again, on exactly the same code, changing only the split.
CodeBERT dropped from 80.9% to 77.3%.
The from scratch transformer dropped from 73.0% to 65.6%.
Notice the damage is not the same size for the two. The small model lost more than twice what CodeBERT lost, and that makes sense: without having read code before, it depends more on recognizing what it has already seen, so the cheat sheet was worth more to it.
Before looking at the detail, worth defining the two numbers in the table, in plain terms.
Precision answers: when it flags a category, how often is it right? Low precision means it accuses a lot of code that was fine.
Recall answers: of the cases that actually existed, how many did it find? Low recall means it lets things through.
The detail of the from scratch transformer:
| Category | Precision | Recall |
|---|---|---|
| Safe | 60.3% | 74.3% |
| SQL Injection | 92.3% | 62.5% |
| Cross-Site Scripting | 62.7% | 100.0% |
| Command Injection | 95.8% | 71.9% |
| Path Traversal | 0.0% | 0.0% |
| Insecure Deserialization | 52.9% | 56.2% |
Zero in both Path Traversal columns does not mean it got a lot wrong there. It means it never flagged that category, not once. To it, Path Traversal simply does not exist.
And it partially learned the ones that remain. Not by accident, the ones it does best on are the ones with the strongest visual mark: the concatenation inside a quoted string, and the HTML tag being assembled.
Why NeuralGuard does not use the from scratch model
This is the result that settles the backend question.
The project has both models available, and the choice is a parameter. But the pipeline uses CodeBERT, and the reason is in the table above: a detector that does not recognize one of the five vulnerabilities it promises to recognize is not a complete detector, and Path Traversal is not a category you can do without.
Running both and adding up the findings does not pay off either. Adding up would only make sense if both had similar strength and made mistakes in different places, because then one would cover the other's gap. That is not the case. In the category where the from scratch model scores zero, it adds nothing, and in the others it is worse. What it would add is false positives, and a false positive here has a price: each one becomes a paid call to the next stage.
The small model stays in the project because it is the material of these two articles, and because the structure of the code allows swapping models with a parameter. If a better model shows up tomorrow, it goes in without touching anything else.
A second exam problem: the score changed between runs
There is a sequel to this story, and it appeared while I was writing this article.
I retrained CodeBERT to confirm nothing had broken, and the score came out more than a point away from the run before. Same code, same set of examples, same split, different number.
The cause is that training uses a technique called dropout, which switches off a random part of the network on each round so the model does not memorize. Without a fixed starting point, that randomness changes from one run to the next, and the final result changes with it.
For a series that publishes numbers and a repository anyone can clone, that is a real problem. If the reader's number does not match mine, either they think they did something wrong, or they conclude the numbers in the article were cherry picked.
The fix is one line, fixing the seed of the random number generator before training:
TorchSharp.torch.manual_seed(TrainingDefaults.Seed);After that I trained twice in a row and compared: identical results, down to the decimal. Every number in this article comes from that fixed seed, and running the repository reproduces them.
Different problems, the same lesson: a measurement you cannot trust is worse than no measurement, because it gives you confidence you did not earn.
The third problem: the exam did not look like real code
The two previous problems showed up by looking at the score. This one showed up when I stopped to analyze the detector already built, running against a real pull request.
The result was this:
## NeuralGuard
No suspicious code found in the 10 changed snippets analyzed.Nothing found. And the pull request was from the Playground, the project that exists precisely to have vulnerable code inside it, including a SQL Injection written in the most direct way there is.
I took the same vulnerable line and changed only what was around it:
| What was sent to the classifier | Answer |
|---|---|
| the query line, on its own | SQL Injection, 99.9% |
| it with two lines of context | SQL Injection, 100.0% |
| it inside the method | Safe, 71.3% |
it inside the method, with the [HttpGet] on top | Safe, 100.0% |
The line is the same in all four. What changes is the frame. And in the last one the model is not in doubt, it states that everything is fine with 100% confidence.
The cause is the usual one: the model learns what you show it. Every training example was a short loose fragment, so it learned the dangerous pattern together with the look of a short fragment, and came to need both to recognize either.
There is a practical aggravating factor. The tool groups the changed lines of the diff into blocks of twelve lines, a reasonable size for Claude to have context. Twelve lines is exactly where the model no longer sees anything:
| Block size | Blocks | Flagged |
|---|---|---|
| 2 lines | 22 | 6 |
| 3 lines | 15 | 4 |
| 4 lines | 11 | 2 |
| 6 lines | 8 | 1 |
| 8 lines | 6 | 0 |
| 12 lines | 4 | 0 |
It could be patched by shrinking the block to two or three lines, and the Playground would light up straight away. But that is adjusting the question until it fits what the model knows how to answer. It would keep failing on any code that is not a loose fragment, and it would give Claude too little context to judge whether the thing is exploitable.
The real repair is in the training, and it is what the previous article describes: every mould is now rendered in four shapes, from the loose fragment to the full controller. Repeating the same measurement with the new model:
| The same line | Trained on loose fragments only | Trained on the four shapes |
|---|---|---|
| on its own | SQL Injection 99.9% | SQL Injection 100.0% |
| with context | SQL Injection 100.0% | SQL Injection 100.0% |
| inside the method | Safe 71.3% | SQL Injection 99.7% |
| inside the endpoint | Safe 100.0% | SQL Injection 99.9% |
And the overall score? The old model, trained and examined only on loose fragments, got 81.9%. The new one, trained and examined on code wrapped like real code, gets 77.3%.
That looks like a regression and it is not. They are two different exams. The 81.9% one was made of loose fragments, the same shape as the training set, and that made it easy. The 77.3% one has code wrapped like real code. The model did not get worse, the ruler stopped being generous.
It is the third time in the same article. The first time, the exam was contaminated. The second, it was not reproducible. This time it was clean, it was reproducible, and it still measured the wrong thing: it measured whether the model recognizes fragments, and what we need to know is whether it recognizes code.
An exam is only worth what it can predict about real use. And the only way to find that out was to leave the table and run against real code.
The lesson that was already there
This result gives back, with a number attached, exactly what the last article of the previous series had concluded: the transformer's machinery is simple and within anyone's reach, and what has no shortcut is the experience of having read millions of lines beforehand.
What changed is that now we can state the size of the difference. And we can see it is not uniform: the model with no background gets close on the loudest patterns, and does not leave zero on the ones that require knowing what the called function does.
One last note about 77.3%
CodeBERT landed on 77.3%, and it is worth marking right now that this number is not good enough to block anybody's merge.
Nearly one in five verdicts is wrong. Some of those errors are letting something through, which is bad. Others are flagging correct code as vulnerable, and that is the one that stops the team's work.
Holding on to that number is what makes the rest of the series necessary. The next article starts the engineering, packaging the detector into a command line tool that reads a pull request diff. And right after it comes the article that solves the 77.3%.