Look, I hope you're excited.
You really should be.
You've been through 80% of the course and it's all been building up to this moment.
Today you will be training your own proprietary LLM for for fun and for profit.
It all starts here.
So what is actually involved in today?
We're going to start with some stuff that maybe isn't so thrilling.
We're going to talk hyperparameters one more time.
I've got some essential hyperparameters to go through with you.
And the reason this is so important is that you're going to be doing some hyperparameter optimization
yourself.
The fancy word for for trial and error.
And you need to understand the context of what it is that you're playing with.
And this is really the opportunity to build something that can beat other models.
It's about understanding what kind of levers you've got to experiment with.
That is at the heart of the R&D behind building leading models.
So we've got some hyperparameters to talk about.
And then we're going to set up a supervised fine tuning trainer, an SFT.
Trainer, which is the sort of core object behind running this training.
Um, looking at parts of the TRL library from Hugging Face and then we will kick off.
Our own proprietary LM training process.
It's going to be great.
So first of all, though, before we get to the great stuff, we do need to talk about some of the essential
hyperparameters that control this process, starting with Q Laura, and most of this is stuff that you're
very familiar with now.
So the first hyperparameter I'll mention is one more time to bring up the target modules.
And I think you now remember exactly what this is.
If you have an architecture of a transformer, a base model like Lama 3.1, it's way too big to try
and fine tune this enormous, great architecture.
So instead we pick a few layers in the architecture and we call those layers.
The target modules are the ones we're going to target.
We freeze everything.
We're not going to try and optimize these weights.
There's too many of them.
Even in these target modules.
We're not going to train these.
Rather, we're going to have onto one side a lower dimensional matrix that we will train this lower
dimensional matrix and we will apply it to this original target module.
We'll apply it in fact by multiplying them together, using that as a, as a as a delta on the weights
here.
Um, and so we train these little guys and we apply them to the target modules, selected layers in
the bigger architecture that's target modules.
And then ah, uh, with this greater 3D goggles as the, as the logo, as the icon.
Uh, R is how many dimensions do we have in this lower dimensional, uh, adapter matrix?
Uh, it's often common with learning with language learning tasks to start with eight.
Um, for this project you're going to see I've got 32 as the R, because we've got so much training
data that I figured we could use quite a few parameters to learn on.
But if that's running out of memory for you, you can you can have eight.
I actually I should say that the difference between 8 and 16 and 32 was quite marginal.
It did improve things, but not by a huge amount.
So if you have any memory problems, then stick with an R of eight.
If you're on a smaller box that will be just fine.
32 is splashing out a bit, but but but it was worth it given the amount of training data we have.
Alpha, you may remember, is the scaling factor.
It's used to multiply up the importance of this adapter when it's applied to the target module.
In fact, you may remember there are actually two Laura matrices.
One is called Laura A and one is called Laura B, and the formula is that the change in weights is actually,
uh alpha.
The scaling factor times A times B, as simple as that.
That is that's the most maths that we're going to get in this course.
And I think that's not not taking it too far.
So that's as simple as what alpha is.
It's the scaling factor.
And the rule of thumb is to have alpha to be double R.
That's what everyone does.
By all means you can experiment with other values of alpha.
But but the norm is is to do alpha is two r.
So we're going to start with an R of 32 and an alpha of 64.
Quantisation of course is just what we call it when we reduce the precision of the weights in the base
model.
The base model has 32 bit numbers.
In it.
We reduce it down to eight bits or even down to four bits, which sounds insane.
We did that with our base model and we saw that we were still getting results.
They weren't great results, but I think that would be true for the base model overall.
And we did see actually that the eight bit model did better than the four bit model, but they were
both pretty miserable at it.
And by all means you can try training with the eight bit model too.
But we're going to train with a four bit model because that's what will fit in in our in our memory.
And that's, uh, but I'd be interested if you try the eight bit to see whether you get significantly
different results.
And then the final hyperparameter is a new one that we've not talked about before, except to show you
it in the code dropout.
So dropout is a type.
It's a technique that's known as a regularization technique, of which there are a few, um, which
means that it's a technique designed to prevent the model from doing what's known as overfitting.
And overfitting is when a model gets so much training data, it goes through so much training that it
starts to just expect exactly the structure of the data in the training data set, and then give back
exactly that answer.
And it starts to to no longer understand the general trends of what's being suggested, but instead
it sort of hones in on precisely those words and the prediction that comes later.
And as a result of that, if you give it some new point that it hasn't seen in its training data set,
it performs really badly because it's not being learning the general themes, it's being learning to.
It's been too much learning the very specifics of this training data set.
I'm being a bit hand-wavy again, but hopefully you get the idea.
That's called overfitting when you are too precisely adhering to the training data set and the outcome.
And it's not learning the general flavor of what's of what it's trying to predict.
Um, and it's that flavor.
It's that nuance of what's going on.
That's what you're trying to teach the model.
Um, so that's that's the sort of that's the preamble, the explanation of what?
Of what overfitting is.
But now, to tell you exactly what dropout does, and it's really simple, what dropout actually does,
uh, is it quite simply removes a random subset of the neurons from the deep neural network.
From the transformer, it takes a random percentage.
We're going to start with 10%, takes 10% of the neurons, and it just wipes them out, sets the activations
to zero so that they are not involved in the forward pass or the backward pass.
They're not involved in predicting the next token and they're not involved in optimizing.
It's as if they're just not there.
And as a result, every time that you're going through training, the model is seeing a different subset,
a different 90% of the neural network, 10% of them have been removed randomly each time.
And so the, the, the weights are sort of discouraged from being too precise and to and looking too
precisely for one set of input tokens, but instead, because different neurons participate every time
in the training process, it starts to learn more.
The general theme than learning very specifically how to expect different tokens.
So it prevents any one neuron from becoming too specialized.
It supports this concept of more general understanding in the neural network, in this very simplistic
way of just removing 10% of the neurons from the process, a different 10% each time.
So that's dropout.
It's really very simple.
When you realize it, it's literally dropping out a bunch of the neurons and the the norm.
It's usually somewhere in the in the range of 5% through to 20%.
Um, I've picked 10% as the dropout that we're using.
You should absolutely experiment with 5% and 20% and see whether you get better results or not.
It is very much a hyperparameter to be experimented with.
Okay, so those are the five hyperparameters for Q.
Laura, next time we'll talk about five hyperparameters for the overall training process.