Blog

Notes on the .Net ecosystem, Angular, Software Architecture, Machine Learning and CyberSecurity

← All posts
Building a transformer from scratch with TorchSharp
  • TorchSharp
  • Transformers
  • Deep Learning

Building a transformer from scratch with TorchSharp

This is the last article in the series where we taught a machine to find security flaws in code. And it exists because of a debt. In the previous article, we assembled the detector by stacking attention layers as closed boxes: we plugged them in, used them, it worked. But I promised we would come back to open the boxes. The moment has come.

The plan: build a small transformer, from scratch, in C# with TorchSharp. No downloading a ready-made model, no borrowed knowledge. Every piece explained before it comes in. And at the end, an experiment that closes the whole series with the most important lesson of all.

Friendly warning: this is the most technical article of the series. But the house rule still applies: every concept explained in the simplest way I can. Come along slowly and we will get there together.

The assembly map

A transformer from the readers' family (the one of BERT and CodeBERT, remember article 2) is basically four floors stacked up:

  • Embeddings: the tokenizer's numbers become points on a map
  • Positions: each word gets its seat number
  • The attention blocks: the heart, repeated several times
  • The final layer: summarizes everything into an answer

And these are the measurements of our mini, declared in one place, which all the following snippets will draw from:

const int vocabularySize = 50_265;   // how many tokens exist in the vocabulary
const int maxSequenceLength = 256;   // the maximum size of the "sentence"
const int embeddingSize = 128;       // the coordinates of each token on the map
const int blockCount = 4;            // the attention floors
const int labelCount = 2;            // safe or vulnerable

Embeddings: the numbers become points on a map

From article 2 you already know the idea: words become points on a map, and similar words sit close together. What's new is seeing that, in practice, this is a table. Each token in the vocabulary has a row, and each row holds that token's coordinates on the map. Building this in TorchSharp is one line:

// The token table: each token in the vocabulary has its coordinates on the map.
// It starts all scrambled, and it is the training that tidies the map little by little.
var tokenEmbeddings = Embedding(vocabularySize, embeddingSize);

An important detail: this table starts filled with random values. The map begins fully scrambled, with "king" far from "queen", and it is the training that arranges the positions over time. In CodeBERT, this map arrived ready from college. In ours, it will be born raw.

Positions: the seat number

Here comes an elegant catch. Remember that the transformer looks at the whole sentence at once? Well, that creates a problem: looking at everything at the same time, it has no way of knowing the order. To it, "the client canceled the contract" and "the contract canceled the client" would be equal again, and we would be back to the bag of words problem from article 2!

The solution is simple and clever: give each position in the sentence its own point on the map as well. It's like numbering the seats in an auditorium: besides knowing who is present, the model knows where each one sat. In practice, it is a second embedding table, and each token's final entry is the sum of the two things: who it is, plus where it is.

// The snippet we are going to analyze, converted into numbers by the previous article's tokenizer
var suspiciousCode = """var sql = "SELECT * FROM Users WHERE Name = '" + name + "'";""";
var tokenIds = tokenizer.ToTensor(suspiciousCode);
var sequenceLength = tokenIds.shape[1];

// The position table: one coordinate for each possible seat in the auditorium
var positionEmbeddings = Embedding(maxSequenceLength, embeddingSize);
var seatNumbers = arange(sequenceLength);   // 0, 1, 2, ... one seat per token

// Each token's final entry: who it is + where it is sitting
var input = tokenEmbeddings.forward(tokenIds) + positionEmbeddings.forward(seatNumbers);

Attention: opening the box, at last

Now, yes. The box that stayed closed for the whole series.

In article 2 I described attention like this: each word decides which other words are worth paying attention to. Now you will see how it decides, and I promise it is simpler than it sounds. The mechanism works like a group dynamic, where each word prepares three things:

  • A question (the technical name is query): "what do I need to know to understand myself better?"
  • A badge (key): "here is the kind of information I offer"
  • A note (value): the actual content it has to share

Then comes the round: each word's question is compared with everyone else's badge. When question and badge match strongly, the score is high. In our old example, when "was hungry" asks "who is the subject around here?", the badge of "dog" matches very strongly, and the one of "yesterday" barely at all.

The scores then become percentages (through a function called softmax, which does only that: turns a list of scores into a list of percentages that add up to 100). And each word's final result is the blend of everyone else's notes, in those proportions: 80% of the note from "dog", 5% of the one from "yesterday", and so on.

// The three abilities, learned during training:
var questionProjection = Linear(embeddingSize, embeddingSize);   // formulating the question (query)
var badgeProjection = Linear(embeddingSize, embeddingSize);      // preparing the badge (key)
var noteProjection = Linear(embeddingSize, embeddingSize);       // writing the note (value)

// Each token prepares its question, its badge and its note
var questions = questionProjection.forward(input);
var badges = badgeProjection.forward(input);
var notes = noteProjection.forward(input);

// Compares each question with all the badges: the score table is born
var scoreScale = Math.Sqrt(embeddingSize);   // keeps the scores from growing too large
var matchScores = matmul(questions, badges.transpose(-2, -1)) / scoreScale;

// The scores become percentages that add up to 100%
var attentionWeights = softmax(matchScores, dim: -1);

// Each token's output: the blend of the notes, in the decided proportions
var attentionOutput = matmul(attentionWeights, notes);

Two operations in this snippet deserve a translation. matmul is matrix multiplication, and here it is just an efficiency shortcut: instead of comparing question with badge one pair at a time, it does all the comparisons at once, every question against every badge, in a single stroke. And transpose simply turns the badge table on its side, because that is how the multiplication can fit the two tables together. Nothing beyond organizing the math.

Done. You have just read the complete mechanism that sustains ChatGPT, CodeBERT and the whole modern AI revolution. About ten lines. The genius was never in the complexity of the mechanism, it was in the idea.

And one detail that answers a natural question: "who teaches each word to ask good questions?". The training. The three projections (question, badge, note) are learned, adjusted by the same guess, grade and correction ritual from the previous article. The model learns, on its own, what is worth asking.

Multi-head and digestion: the complete floor

In practice, transformers run several of these dynamics in parallel, each with its own questions and badges. The technical name is multi-head attention. The intuition: each "head" learns to specialize in one type of relation. In a sentence, one head may end up specialized in linking verbs to subjects, another in linking pronouns to the names they replace. In code, one may link variables to their uses, another link an if to the block it controls. Nobody programs those specialties: they emerge from training. In our mini, each floor uses a single head so the code stays clean to read. CodeBERT uses 12 per floor.

After the attention, each word still goes through a small individual network (the feed-forward), which I like to describe as the digestion moment: after listening to everyone, each word processes in silence what it collected. And that pair, attention plus digestion, forms a floor. In the real version, each step also goes through a stabilizer of numbers (the technical name is LayerNorm), an engineering detail that keeps the training healthy. I left it out of the snippet so as not to clutter it.

// The digestion: a small network each token goes through alone, after listening to the others
var digestion = Sequential(
    Linear(embeddingSize, embeddingSize * 4),
    GELU(),
    Linear(embeddingSize * 4, embeddingSize));

// A complete floor: listens to the others (without forgetting what it knew, hence the sum), then digests
var afterAttention = input + attentionOutput;
var blockOutput = afterAttention + digestion.forward(afterAttention);

The final assembly

With all the pieces on the table, the complete model is this:

// The 4 floors (each with its own attention and digestion) and the final summary
var blocks = CreateTransformerBlocks(blockCount, embeddingSize);
var classificationHead = Linear(embeddingSize, labelCount);

// The reading: the sentence comes in, and at each floor the understanding deepens
var understanding = input;
foreach (var block in blocks)
    understanding = block.forward(understanding);

// The summary of the whole sentence becomes the verdict
var summary = understanding.mean(dim: 1);
var verdict = classificationHead.forward(summary);   // safe or vulnerable?

The CreateTransformerBlocks method just stacks the floor you saw above, four times. Read the flow again calmly: the sentence comes in, who gets added to where, four rounds of conversation among the words, and a summary becomes the verdict. Six articles of history fit in these few commands.

The experiment that closes the series

And now, the million dollar question: if our mini transformer has the same gear as CodeBERT, does it detect vulnerabilities as well as the detector from the previous article?

I trained both on the same collection of examples, and the answer is a resounding no. The mini even learns the most blatant patterns, the text glued into the SQL query it catches. But any subtler variation escapes it, and on the brand-new questions of the final exam the difference to the article 5 detector is embarrassing.

And that result is the most valuable lesson of the whole series. The architecture of the two is the same. The difference is the baggage: CodeBERT arrived at our training after years of college reading millions of code samples, and the mini arrived raw, having seen the world through a few dozen examples. The gear is simple and within anyone's reach, as you just saw. What nobody can shortcut is the experience. That is why the whole world builds on top of pre-trained models, and that is why this series did the same.

Conclusion of the series

Look at the road we traveled together:

I hope that at some point along the way you looked at one of these concepts that seemed like magic and thought: "oh, so that's all it is?". Because that thought is the best sign that the magic became understanding. Until the next series.

Sources