The previous article explained why the classifier cannot have the final word. Now we build the one who does.
What calling Claude means
If you have never used a language model API, the mechanics are simpler than they look.
You make a request to an address on the internet, sending along the text you want the model to read. The server responds with the text the model wrote. It is an HTTP request like any other your application already makes.
What changes compared to an ordinary API is that the answer is not deterministic: the same question can produce slightly different text. That will matter shortly.
Installing and creating the client
Anthropic publishes an official package for .NET, so there is no need to assemble the request by hand.
dotnet add package AnthropicThe client is created with no parameters at all:
AnthropicClient client = new();It looks for the access key in an environment variable called ANTHROPIC_API_KEY. That is the right way to do it: the key never appears in the code, and whoever runs the tool sets it in the environment. In the pipeline, it comes from a repository secret.
The shape of a call
A call has four main pieces of information.
var response = await client.Messages.Create(new MessageCreateParams
{
Model = options.Model,
MaxTokens = options.MaxTokens,
System = SecondOpinionPrompt.System,
Messages = [new() { Role = Role.User, Content = "the case this time" }]
});Model is which model you want. Different models cost differently and get things right differently.
MaxTokens is a size ceiling for the answer. It exists so the model does not write an enormous text you then pay for. Since the answers here are short, 2048 is plenty.
System is the fixed instructions: who it is in this task, what the rules are, what it should consider. This does not change between one analysis and the next.
Messages is the conversation itself. Here there is only one message, with the snippet of code to analyze. It is the part that changes on every call.
The separation between System and Messages matters: System is the policy of the job, Messages is the concrete case.
The problem with loose text
Now the practical problem.
If you send that and ask for nothing special, the answer comes back as running text. Something like: "Yes, this code is vulnerable to SQL Injection, because the name parameter is concatenated directly into the query without sanitization."
For a human reading, perfect. For your program, a nightmare.
The program needs a yes or no decision, so it knows whether to take the check down. Extracting that from the text would mean searching for words like "yes", "vulnerable", "is not", and that approach breaks on the first answer that starts with "No, despite looking vulnerable...". You end up writing an interpreter for English full of special cases.
Structured output: asking for a fixed format
The solution is telling the API which format you want, not just which question.
This is called structured output. You describe the fields the answer must have, and the API guarantees that what comes back fits that description. It is not a suggestion in the prompt, it is a real constraint.
The description uses JSON Schema, which is a standardized way of saying "this object has these fields, of this type, and these are required".
In the project it is this one:
["type"] = JsonSerializer.SerializeToElement("object"),
["properties"] = JsonSerializer.SerializeToElement(new
{
exploitable = new
{
type = "boolean",
description = "True when an attacker controlling the input can change what the program does."
},
justification = new
{
type = "string",
description = "One or two sentences a developer can act on, naming the input and the sink."
}
}),
["required"] = JsonSerializer.SerializeToElement(new[] { "exploitable", "justification" }),
["additionalProperties"] = JsonSerializer.SerializeToElement(false)Translating: the answer is an object with two fields. A boolean called exploitable, which is the verdict. And a text called justification, which is the explanation that will show up in the pull request comment. Both are required and no extra field is accepted.
Notice each field has a description. That is not a comment, it travels with the request and helps the model understand what to fill in there.
And notice the choice of the boolean. I could have asked for a text with "Confirmed" or "FalsePositive", and then I would have to compare strings and handle the case where it comes back spelled differently. A boolean has none of that ambiguity.
With the schema defined, reading the answer becomes ordinary deserialization:
var answer = JsonSerializer.Deserialize<ReviewAnswer>(json, _jsonOptions);The instructions: what to write in the System
This is the part that changes the result the most, and it is where time is worth spending.
NeuralGuard's System has four blocks, each solving a specific problem. The repository keeps them in English, which is where the tool runs, but here is what each one says.
The first says who called and why:
You are a security engineer reviewing findings raised by a machine learning classifier during a pull request check. The classifier is small and often wrong: it recognizes shapes it saw during training and has no view of the surrounding codebase.
Saying the classifier is small and often wrong changes the behaviour. Without that sentence, the model tends to assume that if somebody flagged it, there is probably something there, and it starts looking for reasons to agree. With it, it understands that part of the job is disagreeing.
The second block says what counts as safe:
Parameterized queries, encoded output, argument lists instead of shell strings, validated paths and type-safe deserialization are safe. A finding is only confirmed when an attacker who controls the input can change what the program does.
Those are the fixes from the five fundamentals articles, listed. And the last sentence is the criterion: it is not "looks risky", it is "somebody who controls the input can change what the program does".
The third block explains the cost of being wrong, and it is the most important:
Be strict about false positives: an unconfirmed finding blocks a developer for no reason.
That is the entire previous article in one sentence. Without it, the model treats both kinds of error as equivalent, and when in doubt it confirms, because confirming looks like the cautious move. Explaining that confirming wrongly blocks a person, it starts demanding evidence.
The fourth block defines the limit of what it has:
Judge only the snippet you are given, and say plainly when the snippet alone is not enough to prove exploitability.
The model receives a small piece of code, without the rest of the project. Without that explicit permission to say "there is no way to tell", it would fill the gaps with assumptions.
The message for each case
The System is fixed. What changes on every call is the message with the case:
public static string Build(CodeSnippet snippet, VulnerabilityLabel suspectedLabel) =>
$"""
The classifier flagged this snippet as {Describe(suspectedLabel)}.
File: {snippet.FilePath}
Line: {snippet.StartLine}
{snippet.Content}
Is this a real vulnerability, or a false positive?
""";Four pieces of information: which category the classifier flagged, the file, the line and the code.
Saying which category was flagged helps focus the analysis. Without it, the model would do a general security review of the snippet, and might answer about something other than what we are judging.
Choosing the model
The model is configurable, and the choice is a balance.
Stronger models cost more per call and dismiss more false positives, because they reason better about what the code does. Simpler models cost less and tend to confirm more when in doubt, which gives back part of the false positive you were trying to remove.
The project's default is the most capable model, for the reason in the previous article: what is being protected here is the team's trust in the tool, and that is the most expensive thing to recover once lost. If cost is a constraint in your context, you can point at a cheaper model by changing a parameter, and it is worth measuring the effect on your own code before settling the decision.
And it is worth remembering the total cost is already low for another reason: the confidence threshold means most pull requests generate no call at all.
When the call fails
What is left is error handling, which has an important rule in a security tool.
catch (AnthropicApiException exception)
{
logger.LogError(exception, "Reviewing {FilePath} failed.", snippet.FilePath);
return Result<SecondOpinion>.Invalid("The reviewer is unavailable.");
}The full error goes to the log, where whoever maintains the pipeline can see it. What goes back to the caller is a generic message.
The reason is that error messages from an external service sometimes carry too much detail: part of a key, an internal address, quota information. That text could end up in a public pull request comment. The log is a controlled place, the comment is not.
Notice also that the failure does not become an exception. It becomes a failure result, following the same pattern as the rest of the project: an expected error is a return value, not a change of flow.
How the full flow looks
Putting it together, the service that orchestrates does this for each snippet of the diff:
var classificationResult = classifier.Classify(snippet.Content);
var classification = classificationResult.Value!;
if (!policy.DeservesSecondOpinion(classification))
return Result<Finding?>.Success(null);
var finding = Finding.Create(Guid.NewGuid(), snippet, classification).Value!;
var reviewResult = await secondOpinionReviewer.ReviewAsync(snippet, classification.Label, cancellationToken);
finding.Apply(reviewResult.Value!);Classify. If it did not clear the threshold, it ends there and cost nothing. If it did, ask Claude and store the verdict.
The next article takes this out of the command line and into GitHub: how to package it all into an Action, which parameters to expose, and how to make the model download happen once instead of on every run.