And here we are in Google Colab, ready for fun with models.
So first we do the usual Pip installs and some imports.
Now this will take a little bit longer for you because I have cheated and run this already right before
recording this so that it can be a bit faster.
The Pip installs will probably take 30s to a minute to to all go through for you.
So once we've done the Pip installs, we sign in to hugging face.
I think you're used to this now.
Hopefully you've got the token set up as your secret on the left.
Um, and I'm now going to set some constants for the names of the models we'll be playing with in the
Hugging Face hub.
As always, it's company slash, the name of the model or the model repo.
So we're going to be playing with llama with Phi three with Gemma two.
And then I'm leaving an exercise for you to repeat with quanto the mighty LLM from Alibaba Cloud.
And then I've also given Mistral here.
Now, I have to say, this is probably going to be a model that will be too big unless you've splashed
out on on some big GPUs.
Uh, in which case the the ask for you is to go to Huggingface hub and find a nice model that's 8 billion
parameters or fewer and, uh, use that instead.
Pick the one you like or one that's that's popular or is is doing well at the moment and see what you
make of that.
But do be sure to have completed at least five models.
So with that, let's set those constants, and then let's make a messages list in a format that we know
so well at this point, with a system message and a user message as two dicts in a list.
No more explanation needed for that.
So you remember from last time that you need to, uh, agree to the, um, llama 3.1 terms of service
by going to the model page and um, pressing the agree.
If you haven't already done that, um, then please do so so that you have access to llama 3.1 model.
Now this is something new.
I want to talk a bit about something called quantization, which I mentioned.
So quantization is a rather surprising thing.
Uh, the idea is that we can say, look, we, we want to load in this model into memory, but when
we do so we want to reduce the precision of the numbers of the weights that make up the model.
These weights are normally 32 bit floats.
Uh, 32 bit floating point numbers make up the weights in this deep neural network.
What if we brought them in with fewer bits?
Uh, so 32 bits, of course, is four bytes.
Um, and, uh, we might want to try and cram, uh, more of our, of our numbers into less memory.
Um, and that process of reducing the precision so that you have more coarse numbers in your model is
known as quantization.
And that's something that we're going to do.
And I remember being very surprised when I first heard about this, that the, uh, initially people
talked about taking your 32 bit numbers and replacing them with eight bit numbers, much, much lower
accuracy.
And the thinking was that, surprisingly, whilst of course the accuracy decreases a bit, it doesn't
decrease as much as you might think.
It doesn't decrease, it doesn't become four times worse.
Uh, it just becomes a bit worse, uh, and tolerably so and worth the trade off in terms of more memory.
I was surprised to hear that, and I was even more surprised to hear that you could do more than that.
You can actually reduce it, not down to eight bits, but all the way down to four bits.
That's half a byte if you're counting.
You can reduce from 32 bit number down to four bits going into the number.
And again accuracy sure is is is is hurt, but not by as much as you might expect.
Not I would have expected it would be profoundly different.
And it's not it's quite tolerable.
And it allows for much bigger models to fit into memory and load faster and run faster and so on.
So this is something that people do a lot quantization.
It's a powerful tool, uh, particularly when you get to training and you're having to do lots more
work with these models.
Uh, quantization is a lifesaver.
You may remember I mentioned that at some point in a few weeks time, we're going to get to a technique
called Q Laura, a way of fine tuning in an efficient way.
And in Q, Laura, the Q of Q, Laura stands for quantization.
So it is something we will be coming back to from time to time.
So in the meantime we are using a library called Bits and Bytes, which goes hand in hand with the hugging
face Transformers library.
It's a wonderful library and you can create a new bits and bytes config object to that will be using
shortly to describe what kind of quantization we want to do.
And we are going to say load in four bit equals true.
We're going to go all the way down to four bits.
You can also here say load in eight bit equals true.
Instead if you want to do eight bits and maybe you want to try doing both and see if you can tell the
difference in accuracy.
And now this again is very surprising.
But you also can do four bit use double quant equals true.
And this means that it quantizes all of the weights, not not once, but twice.
Uh, saving a little bit more memory.
And the results of doing this again doesn't massively impact the accuracy of the results.
So it's a good trade to make and people do it.
Um, this is this is saying that in doing the calculations use a this this data type A, B 16, which
makes, uh, makes a some improvement in performance.
So this is quite common as well.
Uh, and then this is about the so when you have reduced the numbers down to four bits, how how will
how to interpret how to treat that, that those four bit numbers, how to compress it down to four bits.
And this, uh, NF four is a four bit representation of numbers.
The N stands for normalized.
And I understand that it's to do with with considering these numbers to follow a normal distribution
allows for more more accuracy when you're compressing things down to just four bits.
So, um, these these two are probably less important.
They're not expected to make a massive difference.
They're meant to be.
It's meant to be.
Good settings to have, though.
And this one makes some difference.
And this one makes a huge amount of difference in terms of memory.
And none of it is too bad in terms of the output.
So with all of that chit chat, we've now created our quant config, our bits and bytes config.
This is something we're familiar with.
We are going to create a tokenizer for Lama.
This line is a new one that I haven't talked about before.
Uh, the uh, there is something called a pad token, which is which token is used to fill up the prompt
if there needs to be more added to the prompt when it's fed into the neural network.
Uh, and the, um, it's a sort of common practice to set that pad token to be the same as the special
token for the end of sentence, the end of the prompt token.
Uh, and if you don't do this, you get a warning.
It doesn't matter that you get a warning.
I don't think it makes any impact.
But if you don't want to get the warning, then you keep it in here and you see that people have this
in as very standard in, in a lot of code that you'll see.
Okay.
And so then we are going to use our tokenizer.
We're going to call the apply chat template function that you know.
Well that takes our messages as a list of dictionaries and converts it into tokens.
And there we are pushing that onto our GPU.
So let's run that and the tokenizer will get to work.
And what we're going to do next is load our model.
So what does this line do.
So first of all it's very analogous to this line.
Here we created a tokenizer by saying auto tokenizer dot frompretrained.
We create a model by saying auto model for causal LLM from pre-trained.
Now this is the general class for creating any LLM.
A causal LLM is the same as an autoregressive LLM.
And that means it's an LLM which takes some set of tokens in the past and predicts future tokens.
And basically all the llms we've talked about have been that kind of LLM.
Later in the course, we will look at one other kind of LLM, which has some some use from time to time.
But for everything that we're talking about for this sort of generative AI use case, we'll be working
with causal llms or autoregressive llms.
And this will be the way to create them from pre-trained we pass in.
Just as with the tokenizer, we pass in the name of the model.
We tell it that if we have a GPU, we want to use that GPU.
That's what Device map auto does.
And we pass in the quantization config, the quant config that we just set up and that is how we build
a model.
The model is the real code, which is actually our large language model as software, as Python code,
which we're going to be able to run.
And under the covers it is PyTorch.
It is a series of PyTorch layers, layers of a neural network that will be able to feed in inputs and
get out outputs.
So it's the real deal.
Now, it will probably take longer when you run this because I just ran it, and so it didn't have to
do as much work as if it was a completely fresh box.
What actually happens when when you run this is it downloads.
It connects to hugging face.
It downloads all of the model weights from the Hugging face hub, and it puts it locally on the disk
of this Google Colab instance in a cache in a special file, which is a temporary file on the desk of
this box, which will get deleted when we later disconnect from this box so that this model is now temporarily
stored on the box on disk, and it's also loaded into memory as well, ready for us to use.
We can we can ask the model how much memory it uses up by calling get memory footprint.
And so we will see what that says.
It says the memory footprint of this model is about 5.5GB.
And so if we look at the resources for this box, you can see that we are using about 5.5GB of space
on the box.
And it's bouncing around in the past because I've been running this already.
But you can imagine that when you look at it, you'll be starting from down here and it will bump up
to about five and a half.
And on the disk, we're using up plenty of space because it's been loaded into the cache of the disk.
Okay.
Almost ready for for for prime time here.
At first we're going to look at the model itself.
And we do that simply by printing the model.
What comes up when we print the model is a description of the actual deep neural network that is represented
by this model object.
This is what we're looking at here.
It's real layers of code representing the layers of the deep neural network.
And these are all this is showing PyTorch classes that have set up that are being that are that are
referenced by model.
Uh, and again, this is a practical class with only a touch of theory from time to time.
But it is worth looking at this, depending on your level of of knowledge of the innards of deep neural
networks and the layers.
Some of this may be super familiar to you.
You may be comfortable seeing that it begins with an embedding layer, which is how the tokens become
embedded into the neural network.
And you can imagine that this dimension, these are showing the dimensions and it's the dimensionality
of the vocab.
Uh, and you'll then see that there's a series of modules, each of the layers in the neural network.
There are attention layers that you'd be expecting to see, particularly as you know that attention
is all you need.
As the paper said, attention is all you need, and that is at the heart of what makes a transformer
a transformer, these attention layers.
And then we have multi-layer perceptron layers right here.
And there is an activation function.
Uh, again, as those those who are more familiar with the theory will be expecting to see this.
The activation function that is used by this llama 3.1 model is the ReLU activation function, which
is the sigmoid uh, linear unit, which is described in Pytorch's documentation right here.
Uh, and it is also known apparently as the swish function.
Uh, and it's, it's basically x times the logistic sigmoid of x.
And that's what the activation function looks like.
Again, if you're into the theory of deep neural networks, you know exactly what this is.
If you're not, then then don't worry.
Just get a general sense of what's happening here, and it's something that you can look more at as
you study this model and others afterwards.
At the end of that, there's a, uh, like a, some, some, uh, uh, layer norm layers and then the
linear layer at the end.
So this is worth looking at particularly, uh, depending on your level of knowledge of, uh, PyTorch
neural networks.
But also later when you look at other models, you could do the same thing.
Look at the model's output.
Look, look at the model, print the model, look at what it looks like and compare with lama3.
I'm going to break for the next video, but in the next video, we're then going to run this and then
run the other models too.
So don't go anywhere.
See you in a second.