Well, here we are again in Google Colab.
It's been a minute since we were here, and welcome back to it.
Uh, this week we're going to spend our time here and it's going to be terrific.
It's actually it's going to be the best week yet.
I know I keep saying that, but this time it really, really is going to be peak peak.
The only thing that's going to be better than what we do here is what's coming next week in week eight,
which really, I mean, I can't wait to tell you about that.
But stay focused, keep on week seven.
There's a lot to go through here.
So first we do.
I've set up this week seven day one Colab.
We start with some installs.
Um, and one of the things we're installing is a new package you've not seen before, a hugging face
library called Peft, which stands for parameter efficient fine tuning, parameter Efficient Fine Tuning,
which is their name for the library that includes Laura.
Uh, it's within this library.
Pfft, pfft.
Just rolls off the tongue.
Um, so that is it.
I am on a T4 box, which is the lowest of the GPU boxes, which just has 15, gigabytes of GPU Ram.
You can see, most of which I'm already using up just a few cells into this.
So anyway, we do our Pip installs.
We do a bunch of imports here, uh, and set some constants.
We're going to be working with a base model which is llama 3.18 billion.
And I'm also setting a fine tuned model here.
I obviously we haven't done any fine tuning yet.
I'm.
This is from the future.
Uh, I'm bringing this in just so I can show you what a fine tuning model looks like in terms of the
the the Laura matrices applying to the target modules.
And then here are three hyper parameters which now you're an expert on are which I am setting to 32.
You probably remember I said start with eight and then go to 16 and then go to 32.
Well I got to 32.
Uh, and that was where I ended up.
Uh, and so I'm doing 32 here.
Uh, alpha is a rule of thumb, double R.
So there it is at 64.
And the target modules.
So these are the four names of the layers that we are targeting.
And you will see why shortly.
And this is by far the most common, uh, setup for llama models.
Other models may have different names of their layers, but yeah, you give the names of the layers
that you will be targeting in this list that you assign to target modules.
Okay.
So next, this is some standard stuff that you've done a few times now to log in to Hugging Face.
Um, and I've got the usual blurb that if you don't have hugging face account, but of course you have
a hugging face account by now.
Uh, but you log in there, it's free, you get a token, and then you go to this section in the Colab,
the, the key, and you use that to put in your, uh, your, your token, um, as you've done in the
past.
And then when you do that, you can run this cell and it will log in to hugging face.
Uh, the alternative is you could just type in your token there.
If you have any problems with accessing the notebook.
Okay, so without further ado, I am going to read in the base model without any quantization.
No funny business.
We're just going to read in the entire llama 3.18 billion base model, remembering that is the smallest
of the llama series.
Uh, device map equals auto means use a GPU if you've got one which which this box does.
And I'm not going to run this now because I just ran it and it took about five minutes.
Uh, and it came up with a warning that it couldn't fit it all in GPUs.
So some of it went on the CPU.
And that's why if you look at the resources over on the right, you'll see that my GPU is 11 out of
15 gigs are filled up, and almost 13 gigs of Ram are also filled up.
So it's really taken up both the reason for the spike here is because I did it once, and then I restarted
and did it again.
Uh, obviously all you will see is the the one rise up to the top when you run this.
Okay.
And so now I'm going to print how much memory is this base model using up.
And again if you if you wanted to train this it would it would take many many more times.
This this is just how much memory is the base model using up, and its memory footprint is just north
of 32GB, 32GB of memory being used.
And you may remember that's what we said earlier.
It's basically 32 bit floats for each of the 8 billion parameters.
Okay.
So that's, uh, it's big.
Uh, and let's just look at it.
You can just take a look by printing the base model itself.
And this now is a view on what it looks like, um, so briefly before, but we'll just pause for a moment.
And again, this isn't going to be a deeply theoretical class, so I'm not going to do too much in the
way of explaining this other than saying what you can.
What is made clear when you look at the architecture of this neural network is that there are it consists
of, first of all, an embedding layer, which is the thing that that takes text and turns it into it,
embeds it into vectors in the in the neural network.
So this is very it's like the, the encoder encoding LMS that we talked about before.
The first layer is embedding tokens, um, into a, um, a vector.
Uh, and in fact, that is the dimensionality of how many possible tokens we have.
And this is the dimensionality of the embedded vectors.
So there are then 32 layers called Lama decoder layers.
32 sets of them.
And each of those 32 looks like all of this.
Let's get that right to there.
Um, and so you can you can go through this, but you can see that it consists of the the set of attention
layers, which are called q, proj, k proj, v and o proj.
And these these are the layers that we have targeted in our target modules, which is typically what
you do.
You can try others too, but this is the most common approach.
And you'll see that, uh, some of these layers have 4000 odd dimensions in and out.
Uh, this one and this one and some of them are 4000 in and and 1000 out.
So they've got some different dimensionality there.
And that will be somewhat relevant when we look at the lora A and Lora B.
But I'm not going to get too deep into this, but this is yours to experiment with and read up on if
you want more more information about it.
There is then a multi-layer perceptron layer with something that, for example, the the up is something
that explodes out the number of dimensions, and then the down then reduces down the number of dimensions.
And that's followed by an activation function.
Again, for people that are more familiar with this stuff.
The activation function that's used for Lama is Selu, which you can see in the PyTorch documentation.
You can look at it and learn more about what that is and and why it is used.
Um, so that is then followed by by layer norm layers.
Um, and then at the very end there is a linear layer, the LM head.
And this is sometimes targeted.
This is sometimes added to target modules.
As I mentioned before, in the cases where you wanted to generate something, where part of what you
want it to learn is to generate results, that will take a different format of some sort.
Maybe you want a particular structure of JSON, or maybe something completely different, like you want
it to speak a different language, or you want it to to structure things in some, some very unique
way.
Then you might target this in your target modules as well.
Um, but that gives you a sense of the architecture.
And in a second, when we look at the Lora adapters, you'll see why I've taken a moment to dwell on
this.
All right.
So we used up 32GB.
The next thing we need to do is to restart this session by going to runtime and restart session to clear
out the memory so we can keep going.
There are some torch commands that will clear the cache, but in fact they're not aggressive enough.
It still holds on to too much because we've consumed so much.
The only way forwards now is to restart the session.
So that's what I'll do.
And I will see you in the next video once I have restarted and I'm back here again.