From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
160 lines
4.4 KiB
160 lines
4.4 KiB
WEBVTT |
|
|
|
00:00.920 --> 00:02.120 |
|
Welcome back. |
|
|
|
00:02.150 --> 00:06.920 |
|
You may, like me, have just gone off and got a coffee while things loaded back up again. |
|
|
|
00:07.220 --> 00:09.140 |
|
And back here. |
|
|
|
00:09.350 --> 00:14.960 |
|
Uh, after the restart your session box, you needed to have come back and rerun the. |
|
|
|
00:14.990 --> 00:17.660 |
|
You don't need to rerun the installs, but you need to rerun the imports. |
|
|
|
00:17.660 --> 00:20.270 |
|
The constants log in to hugging face again. |
|
|
|
00:20.300 --> 00:21.740 |
|
Do not run this. |
|
|
|
00:21.740 --> 00:24.800 |
|
Otherwise we'll be back where we were with a big model in memory. |
|
|
|
00:24.830 --> 00:27.470 |
|
Uh, skip back to restart your session. |
|
|
|
00:27.470 --> 00:31.520 |
|
And here we are loading the base model using eight bit. |
|
|
|
00:31.520 --> 00:33.770 |
|
So again, you've seen this before. |
|
|
|
00:33.770 --> 00:35.060 |
|
We did briefly. |
|
|
|
00:35.060 --> 00:39.080 |
|
Uh, at least have this in the code if I didn't mention much about it. |
|
|
|
00:39.350 --> 00:45.890 |
|
Um, and it uses this wonderfully named there's a package called Bits and Bytes, uh, which is a great |
|
|
|
00:45.890 --> 00:47.090 |
|
name for a package. |
|
|
|
00:47.210 --> 00:52.850 |
|
Uh, and you create a class called Bits and Bytes config, which is where you specify, uh, what kind |
|
|
|
00:52.880 --> 00:58.310 |
|
of quantization config you want, how, how much you want to reduce the precision of your model. |
|
|
|
00:58.310 --> 01:01.940 |
|
And we are saying load in eight bit is true. |
|
|
|
01:02.060 --> 01:07.520 |
|
Uh, and then you when you load the base model, you pass that in as a parameter. |
|
|
|
01:07.520 --> 01:11.000 |
|
As I said, we did do this in the past, I think, but we didn't talk much about it. |
|
|
|
01:11.420 --> 01:20.660 |
|
Um, so this is how we load in the llama 3.1 base model, with the precision reduced from 32 bits down |
|
|
|
01:20.660 --> 01:21.980 |
|
to eight bits. |
|
|
|
01:22.250 --> 01:26.300 |
|
Um, so I've run that because it takes a couple of minutes. |
|
|
|
01:26.330 --> 01:30.230 |
|
And now let's look at the memory footprint that that's using up. |
|
|
|
01:30.260 --> 01:34.730 |
|
It's using up nine gigabytes just over nine gigs. |
|
|
|
01:34.790 --> 01:40.820 |
|
Uh, as you can see, if you look over here at the resources that we're using up nine gigs and we're |
|
|
|
01:40.820 --> 01:44.090 |
|
no longer having to spill into system memory. |
|
|
|
01:44.300 --> 01:48.320 |
|
Uh, and let's have a look at that base model's architecture. |
|
|
|
01:49.130 --> 01:54.350 |
|
So here then, is the model architecture after quantization. |
|
|
|
01:54.740 --> 01:59.030 |
|
Um, and now let me pause for a bit while, uh, to, for you to spot the difference. |
|
|
|
01:59.060 --> 02:01.910 |
|
Let's see what's changed in the architecture. |
|
|
|
02:01.940 --> 02:09.620 |
|
Uh, now that we've quantized, I'll give you just a second to identify that challenge for you. |
|
|
|
02:09.740 --> 02:11.330 |
|
And your time is up. |
|
|
|
02:11.330 --> 02:14.690 |
|
And, of course, you know that there's no difference in architecture. |
|
|
|
02:14.720 --> 02:15.770 |
|
Trick question. |
|
|
|
02:16.040 --> 02:17.120 |
|
Oh, don't hate me. |
|
|
|
02:17.300 --> 02:19.640 |
|
Uh, so, yeah, there's no difference in architecture. |
|
|
|
02:19.640 --> 02:24.520 |
|
This is identical because the whole idea of quantization is that whilst we've reduced the precision |
|
|
|
02:24.550 --> 02:29.020 |
|
of each of the numbers, each of the weights in the model, we haven't actually changed the architecture |
|
|
|
02:29.020 --> 02:30.160 |
|
of the model at all. |
|
|
|
02:30.340 --> 02:35.470 |
|
And somewhat surprisingly, it's not going to have a massive effect on the performance of the model. |
|
|
|
02:35.470 --> 02:42.010 |
|
So hopefully unless I've made some horrible gaffe, this architecture is identical to the one above. |
|
|
|
02:42.040 --> 02:44.290 |
|
You can just confirm and double check. |
|
|
|
02:44.320 --> 02:51.850 |
|
I haven't made any terrible mistake, but that's that is the whole idea of quantization. |
|
|
|
02:51.850 --> 03:04.900 |
|
We've got from 32GB down to nine gigabytes without, it seems, any, um, uh, well, we haven't yet |
|
|
|
03:04.900 --> 03:09.370 |
|
seen, but without without without significant consequences. |
|
|
|
03:09.370 --> 03:10.810 |
|
There will be some consequences. |
|
|
|
03:10.810 --> 03:13.930 |
|
Just not as big as you might expect. |
|
|
|
03:14.620 --> 03:17.830 |
|
Uh, so we will have to restart our sessions once more. |
|
|
|
03:17.830 --> 03:23.920 |
|
Uh, so for the final time, if I could ask you to restart your session as before, uh, runtime restart |
|
|
|
03:23.920 --> 03:26.410 |
|
session and then meet me back here again. |
|
|
|
03:26.440 --> 03:33.010 |
|
Uh, after this point, once you've run the imports, the constants, uh, and logged into huggingface |
|
|
|
03:33.010 --> 03:35.080 |
|
again, I will see you in one second.
|
|
|