So in a future day, I'm going to be training, fine tuning a model and creating a fine tuned model.
And what I'm going to do now is load in one of the ones that that I've saved in the future, confusingly,
if you see what I mean.
Uh, and that will allow you to see the architecture of it, to get a sense of it.
And you do that by loading in not a base model.
Not not one of these, um, uh, auto model for causal LLM from pre-trained, but instead a Peft model,
a parameter efficient fine tuned model from pre-trained.
And I tell it the base model, which is the one that we've just got right here.
And fine tune model.
This is the name of the model that I saved after I did this.
Laura fine tuning.
And I'm doing this just to show you what it looks like.
Uh, and this is one that that will run very quickly because it's relatively small.
Uh, let's have a look at the size.
So the memory footprint of this is Gigabytes.
And that should be familiar to you because it's very close to this.
Uh, it is about 100MB difference between them.
Let's have a look at it.
So here is the architecture of this thing.
Uh, and now, yeah, I no longer play that trick.
This is definitely different to what we saw before.
And let me tell you what we're seeing.
So first of all, everything you're seeing up to this point here is the same up to the point where we
have the 32 Lama decoder layers.
And now we get to the this attention layer and we have the Q proj the k proj the V and the O uh, proj
uh layers.
And what you'll see is that each of these has a base layer.
And then it has lora A and lora B.
And these are the A and B matrices that I told you about before, uh which have come in here and you'll
see this number 32 here.
That is the R that I mentioned before, the Lora rank.
They are 32 rank matrices.
Um, and uh, yeah.
The, the uh, if, if you're in if you're familiar with the way that these the matrix math works out,
you can see this is designed so that the 32 dimensions will can be multiplied together in such a way
as it can be applied to this base layer and be used to make a small shift to that base layer.
I'm again being a bit hand-wavy because I want to get bogged down in the theory, but the idea is that
these will be multiplied together.
Lora A and Lora.
B together with alpha, the scaling factor, and that will then be used as a delta to apply.
On top of this base layer, there is also another hyperparameter called dropout.
And we'll be talking about that later.
That's not one of the big three that we talked about this week.
but you'll see that feature a few times here.
And so if we look at the other, uh, of the of the four target modules, you'll see that they all have
a Laura A and a Laura B here, where again, we have a Laura A and a Laura B, and finally a Laura A
and Laura B here, Laura A and Laura B, and so that is where our adapter matrices have been inserted
into the Lama architecture to adapt the bigger model, but with much fewer dimensions.
Uh, these, these 32 dimensions as specified by our R hyperparameter.
Uh, nothing else has been changed.
The the multi-layer perceptron layer is exactly the same.
Um, and everything else is the same.
And so just to mention again, we're trying not to get bogged down in this, but the you could look
back to convince yourself that this is the number of dimensions in those, those four, uh, Matrices
there is.
Each one has a Laura A and a Laura B, and I've just multiplied together the dimensions of that matrix
to tell you how many dimensions, how many weights in total we have across these adapters.
And then that means for each layer we sum up these four numbers.
I multiply that by 32 because there are 32 of these groups of modules.
And then each of these parameters is a four byte number.
It's 32 bits.
And so I calculate the size and divide that by a million to get it in megabytes.
I'm not sure if you're following all this, but hopefully you get a general idea just to give you a
sense of perspective, if you add up all of the weights in our Laura adapters, there's a total of 27
million parameters and the total size is about 109MB.
So 27 million parameters of size Hundred and nine megabytes.
That's how large our adapters are.
And of course, compare that to the fact that llama overall has 8 billion parameters and is 32GB in
size.
So it gives you a sense we're doing a lot here, a lot of parameters and a lot to be trained, but it's
tiny compared to the monstrosity that is llama 3.1, even the small variant.
So this whilst I realize there's been a fair bit of, uh, of, uh, stuff in here that, uh, you may
have to go back and check and see what I mean, but hopefully it gives you that intuition, that sense
that we're able to use these lower dimensional matrices to have an impact on the bigger architecture,
but with a smaller size, smaller number of weights that has to be adjusted.
Um, and just to give you just a sort of evidence that this number, this 109MB, is the size of the
parameters I can actually go into hugging face.
I'm now in hugging face and I'm looking at where I saved that particular Laura adapter, that fine tuned
model, and what we'll find.
When you look at these, you look for something called safe tensors.
That is the file which stores the parameters themselves.
Um, and if you look at this for llama 3.1, you'll see that it's 32GB large.
If I look at it for this, you'll see it's 109MB of parameters, 109MB, which matches this estimate
here, 109MB.
That is the size of the parameters that we are fine tuning using this Q Laura technique.
So I hope at least at the very least, it's given you a decent intuition for what's going on here and
how we're able to pull this trick of being able to fine tune a model without needing to have gigabytes
of data that we are optimizing over.
And so with that, back to the slides for a wrap up.