Welcome back.
You may, like me, have just gone off and got a coffee while things loaded back up again.
And back here.
Uh, after the restart your session box, you needed to have come back and rerun the.
You don't need to rerun the installs, but you need to rerun the imports.
The constants log in to hugging face again.
Do not run this.
Otherwise we'll be back where we were with a big model in memory.
Uh, skip back to restart your session.
And here we are loading the base model using eight bit.
So again, you've seen this before.
We did briefly.
Uh, at least have this in the code if I didn't mention much about it.
Um, and it uses this wonderfully named there's a package called Bits and Bytes, uh, which is a great
name for a package.
Uh, and you create a class called Bits and Bytes config, which is where you specify, uh, what kind
of quantization config you want, how, how much you want to reduce the precision of your model.
And we are saying load in eight bit is true.
Uh, and then you when you load the base model, you pass that in as a parameter.
As I said, we did do this in the past, I think, but we didn't talk much about it.
Um, so this is how we load in the llama 3.1 base model, with the precision reduced from 32 bits down
to eight bits.
Um, so I've run that because it takes a couple of minutes.
And now let's look at the memory footprint that that's using up.
It's using up nine gigabytes just over nine gigs.
Uh, as you can see, if you look over here at the resources that we're using up nine gigs and we're
no longer having to spill into system memory.
Uh, and let's have a look at that base model's architecture.
So here then, is the model architecture after quantization.
Um, and now let me pause for a bit while, uh, to, for you to spot the difference.
Let's see what's changed in the architecture.
Uh, now that we've quantized, I'll give you just a second to identify that challenge for you.
And your time is up.
And, of course, you know that there's no difference in architecture.
Trick question.
Oh, don't hate me.
Uh, so, yeah, there's no difference in architecture.
This is identical because the whole idea of quantization is that whilst we've reduced the precision
of each of the numbers, each of the weights in the model, we haven't actually changed the architecture
of the model at all.
And somewhat surprisingly, it's not going to have a massive effect on the performance of the model.
So hopefully unless I've made some horrible gaffe, this architecture is identical to the one above.
You can just confirm and double check.
I haven't made any terrible mistake, but that's that is the whole idea of quantization.
We've got from 32GB down to nine gigabytes without, it seems, any, um, uh, well, we haven't yet
seen, but without without without significant consequences.
There will be some consequences.
Just not as big as you might expect.
Uh, so we will have to restart our sessions once more.
Uh, so for the final time, if I could ask you to restart your session as before, uh, runtime restart
session and then meet me back here again.
Uh, after this point, once you've run the imports, the constants, uh, and logged into huggingface
again, I will see you in one second.