So at this point we're going to talk about hyperparameters.
And we're going to introduce three of them.
So a reminder of what is a hyperparameter.
We talked about it a bit last week.
A hyperparameter is one of these levers.
That is something which you as the experimenter just gets to choose what you want it to be.
There's no particular hard and fast rule about what it should be, and you're meant to use a process
known as hyperparameter optimization to try different values and see what works best for your task at
hand.
And in reality, what what we are actually doing is basically trial and error.
It's a bit of guesswork and then experimentation, because there aren't necessarily any theoretical
reasons why it should be set one way.
It's a matter of practical experiment.
And so often you find in these things when people have a something that's working well and there's a
few, a few controlling parameters, but they're not quite sure how what they should be set to.
We don't yet have the theory to say what they should be.
We just call it a hyperparameter.
That's what it's called.
And it means that you're in this world of trial and error and guesswork until you pick the right settings.
That works best for your model.
I'm oversimplifying a bit, of course, but hopefully you get the general idea.
So there's going to be three of them that are most critical.
In the case of Q Laura fine tuning, and I want to introduce them to you now and then.
We will be playing with them in our time on this.
The first of them is called R, which stands for rank.
And it means simply how many dimensions are we going to use for these lower rank matrices within within
the Lama architecture, the the inner layers have dimensionality of like 1004 thousand dimensions.
We're going to want much smaller number of dimensions in our lower rank matrices.
That's that's the whole idea of them.
Um, so typically, uh, as I said, there's no hard and fast rules different tasks look for different
values of R to start with when you're working in this kind of language generation models.
I think a good rule of thumb that I've, I've always used and that I see people use generally in the
community start with eight, which is a small number.
Um, and that means that it will use up very lower memory and it will run fairly fast.
Um, and then double it to 16, which will take up more memory and run more slowly and see whether or
not you get better results and then potentially double again until you reach a point where you're getting
diminishing returns.
It's slowing down and it's taking longer, but you're not seeing any improvement.
And then, you know, there's no point in having a higher R, you've already got the power you need
for the data that you've got.
So that's r uh, the next one that we'll talk about is alpha.
And Alpha is quite simply a scaling factor that is multiplied.
It's applied to these Laura A Laura B matrices.
And that is then used to change the weights in the model.
The formula for what it's worth is that the amount that you change the weights in the model by in your
in your target modules.
Is alpha times the A matrix times the B matrix.
They get all multiplied together.
So bigger alpha means more effect.
And in practice the rule of thumb that is used I think almost ubiquitously I've I've always used it
and I've always seen it this way in examples is to set alpha to be double R.
So if you start with an R of eight your alpha is 16.
Then when you go up to an R of 16 alpha is 32 and then 32 would be 64.
So that is the good rule of thumb.
But of course it's always worth experimenting with different alphas to see if that if that changes the
your your accuracy.
And then the third and final of our three essential hyperparameters is the actually saying, what will
be the target modules that you will focus on adapting in your architecture?
Which which of these layers are you going to select to focus on uh, and uh, generally the most common
choice and the one that we'll be using is that you focus on the attention layers.
That's very common.
You'll see that in the code that's going to make more sense when you see it.
Uh, there are situations when you want to to target other target modules.
Um, if for example, uh, you're generating something that, that where you want the output to be in
like a completely different language or something like that, then you might want to, uh, to target
some of those final layers.
So you'll see some, some, some that I'll give you more context in a moment about how that works.
But generally speaking, the most common by far is to target the attention head layers.
That's what we will do.
And you will see how that set up in a moment.
And with that, we are now going to head to Google Colab to look at this, to look at some models,
to talk about Laura, to talk about Q, Laura and to see these three hyperparameters in action.
Let's do it.