And so now we talk about quantization the q and q Laura.
Q stands for quantized quantized.
Laura.
And I did mention quantization briefly, I believe in week three.
So you may remember some of this.
But now now we'll talk about it for reals.
So here's the problem.
The we're working with these smaller models, the 8 billion parameter version of llama 3.1.
And we've come up with this clever scheme that Laura for, for working with smaller dimensional matrices
so that we can get more memory.
But the the problem is that even that base model, even the small version of it, the 8 billion version
of it, will take up 32GB of Ram.
That's, uh, 8 billion floating point numbers, which are each 32 bits long.
Uh, and so it's going to fill up a GPU.
In fact, the cheap GPUs that will be using on T4 boxes only have 15GB of memory in them, so it won't
even fit the base model itself.
We're going to be out of memory right away, so that is a problem.
Lora is very useful in making things better for us, but it's not good enough because we can't even
fit the base model itself in memory.
Because 8 billion might be a small size of a model in some worlds, but it's still an enormous number
of parameters.
And so that that then gives us a challenge.
So this quite surprising discovery was made.
That is what we will be working with.
And at first it sounds almost too good to be true.
It sounds like this is a way to have your cake and eat it.
And it kind of is.
So the, the idea that that, that some people had was, okay, so we've got 8 billion parameters.
If we try and have fewer parameters like we have a 4 billion parameter model, we lose a lot of the
power of the model.
Those 8 billion parameters give us lots of knobs to turn.
It's in this very clever architecture.
It gives us a lot of power.
All right.
So let's not cut down the number of parameters.
Instead of doing that, let's just reduce the precision of each of these parameters.
It's like saying instead of being able to turn it through a very sort of finely grinded wheel, we're
going to make it go click, click, click, click through a few settings.
Uh, and so that that was a thinking let's just reduce the precision of each of these weights but keep
the same number of weights.
Now you might think logically all right, but you're just cutting the amount of information.
Surely it's going to be if you have half the amount of information, it's going to be quite similar
to having half the number of weights.
Uh, and it turns out that that's not the case.
For whatever reason, if you lower the precision, you do get some reduction in quality of the neural
network, but not as much as you might think.
It still retains a lot of its power.
And it turns out that this is just a great trick that allows you to fit bigger models in memory with
the same number of parameters.
Just lower precision means that it takes up less memory.
So it's surprising it works remarkably well.
And in fact, you could take the 32 bit floating point numbers that you normally have and reduce it
all the way down to eight bit numbers, and you still get good performance.
And then and now this is where it starts to sound really crazy.
You can reduce it all the way down to four bits.
So each number is just a four bit number.
If you think of that from from an integer point of view, that it's as if each number is going from
0 to 15 and that's it.
Just just in whole numbers.
That's how low the precision is.
Like a click that just has has 16 settings on it.
Um, and you still get pretty good performance.
Sure.
You do see a bit of a, of a drop in quality, but only a bit.
Um, and so this was the, the intuition.
And this, of course, dramatically reduces the memory requirement and allows one to fit bigger models
in memory.
There are a couple of of minor technical details that I'll mention.
One of them is that I just talked about the click switch.
You can think of it as being like a number from 0 to 15.
Typically, it's in fact not interpreted as an integer, but as a four bits are used to to be considered
as as floating point numbers just with lower granularity.
And you'll see that in reality in an example.
And the other thing to to point out, which is something I didn't understand early on when when color
first came out, is that this is the quantizing the base model, but it's not quantizing the, the,
the, the Lora adapters, they will still be 32 bit floats as you will see.
Um, so we're just talking about reducing the precision of the base model, this enormous great base
model so that we can fit it in memory.
So that is cu Lora.
That's quantization.
It's going to feel more real when we see it in the lab in just a second.
But first I want to talk to you about three important hyperparameters in the next video.