So we're now going to look at four bit quantization, the rather remarkable effect of reducing the precision
all the way down.
So very similar to before we create a quant config using bits and bytes.
But this time we asked to load in four bit equals.
True.
There are some other settings here too.
This time there's one called use double Quant, which again is slightly mysterious.
The idea here is that it does a pass through quantizing all of the weights, and then it does a second
pass through again.
And in doing so it's able to reduce memory by I think about 10 to 20% more.
It squeezes a bit more out of this.
And this is, uh, experimentally shown to make it very, very small difference to the power of the
neural network.
So it's almost a freebie.
It's again cake and eat it situation.
So recommended to use Double Quant as true.
This here compute dtype is about the data type that's used during computation.
And there generally you could work with 32 bit floats here.
But using the Bfloat16 data type binary float 16 is seen as something which improves the speed of training
and makes it faster, with only a tiny sacrifice to quality of the of the training.
Certainly when I've tried this, I've seen it run faster and I've not been able to detect any actual
change in the rate of of of optimization.
So this is recommended for sure.
But again, it's a hyper parameter that you can experiment with.
And then this here the the four bit quant type.
This is saying when we reduce the precision down to a four bit number, how should we interpret that
four bit number.
You might think okay, so if it's four bits 0000 through to 1111, then that represents an integer from
0 to 15.
That might be one way of doing it.
Um it's more common to interpret it, map it to a sort of a floating point number.
Um, and this nf4 approach maps it to something which has a normal distribution to it.
And so again, this is very common setting.
Uh, it's what I've used.
I tried something else and it wasn't as good.
So so this is the generally the recommended one.
But it is a hyper parameter which means it is yours for trial and error.
With that in mind, we create this quant config, which is very standard quant config for four bit quantization.
And we create a base model with that which I have done and will now print the memory footprint.
And remarkably, we are down to 5.6GB.
You may have already spotted that over here in my resources, but when you remember that the base model,
the real thing was 32GB in size, it's really come a long way down.
So this is now something which will comfortably fit in our GPU's memory for this cheap T4 box.
And if we look at the base model, we'll see the architecture.
I'm not going to try and make that stupid joke again.
It is, of course, identical to the architecture of the original, beefier 8 billion llama model.
Just that within this deep within this, the precision of the weights is lower.
It's four bit.
Okay.
In the next video, at this point, you should not restart your session.
We need to keep this session as it is, and in the next video we're going to go in and load in our example
of a fine tuned model and see how the Laura adaptations apply to this architecture.
I'll see you there.