From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
352 lines
10 KiB
352 lines
10 KiB
WEBVTT |
|
|
|
00:00.380 --> 00:07.730 |
|
So it's time to continue our journey into the world of open source and understand which models we should |
|
|
|
00:07.730 --> 00:13.460 |
|
pick to investigate the ability to generate open source code. |
|
|
|
00:13.460 --> 00:14.420 |
|
Where do we begin? |
|
|
|
00:14.420 --> 00:17.780 |
|
We begin, of course, with the big code models leaderboard. |
|
|
|
00:17.780 --> 00:23.300 |
|
This is the leaderboard running as a hugging face space that we looked at before, and it's the one |
|
|
|
00:23.300 --> 00:28.820 |
|
that compares the performance of open source models in code generation tasks. |
|
|
|
00:29.270 --> 00:34.370 |
|
You remember we started by just looking at base models, and we can see some of the base models that |
|
|
|
00:34.370 --> 00:39.530 |
|
are doing really well, including including Code Llama, Code Kwan and Deep Sikh Coder. |
|
|
|
00:39.770 --> 00:44.540 |
|
Um, but we're going to look at all models because we're interested in models that have been fine tuned |
|
|
|
00:44.690 --> 00:51.110 |
|
because they really do perform much, much better than some of the base models when they've been trained |
|
|
|
00:51.140 --> 00:55.370 |
|
specifically for, uh, for tasks such as C plus plus code. |
|
|
|
00:55.520 --> 01:01.820 |
|
Um, and we're going to be looking at Python and C plus plus test ability because that is, after all, |
|
|
|
01:01.820 --> 01:03.080 |
|
what we're going to be doing. |
|
|
|
01:03.440 --> 01:09.770 |
|
And when we look at these, one thing to note is that the ones that say exit here, their benchmark, |
|
|
|
01:09.890 --> 01:12.350 |
|
some of their scoring has happened externally. |
|
|
|
01:12.350 --> 01:17.720 |
|
So I'm personally going to tend to lean towards ones that don't have that symbol, that are the ones |
|
|
|
01:17.720 --> 01:25.790 |
|
that have been benchmarked as part of the hugging face, uh, spaces platform and the the one that is |
|
|
|
01:25.790 --> 01:33.050 |
|
top of the list here is code Kwan 1.57 billion, right up at the top. |
|
|
|
01:33.050 --> 01:35.090 |
|
The code variant of Kwan. |
|
|
|
01:35.090 --> 01:40.850 |
|
It's been trained for chat, which means that you can interact with it rather than just, for example, |
|
|
|
01:40.850 --> 01:43.670 |
|
giving it some Python code and expecting it to complete. |
|
|
|
01:43.670 --> 01:50.510 |
|
You can have a chat with it, such as saying, hey, could you convert this Python code to highly optimized |
|
|
|
01:50.510 --> 01:50.870 |
|
C plus? |
|
|
|
01:50.870 --> 01:51.500 |
|
Plus? |
|
|
|
01:51.500 --> 01:53.150 |
|
Let's go in and have a look. |
|
|
|
01:53.630 --> 02:00.710 |
|
We're now looking at the code Kwan 1.57 billion chat the top model. |
|
|
|
02:00.740 --> 02:04.250 |
|
And you can see here it describes some of what it can do. |
|
|
|
02:04.400 --> 02:07.310 |
|
Um and some of how you can interact with it. |
|
|
|
02:07.340 --> 02:11.270 |
|
Like you can say, write a quicksort algorithm in Python. |
|
|
|
02:11.450 --> 02:18.080 |
|
Um, and you can see that you can construct messages in much the way that we are used to, and then |
|
|
|
02:18.110 --> 02:25.070 |
|
use the tokenizer apply chat template to convert that in tokens and run that through the model. |
|
|
|
02:25.070 --> 02:31.580 |
|
So this is all very consistent with the way that we ran a number of models for other simple activities, |
|
|
|
02:31.580 --> 02:34.040 |
|
text generation activities in the past. |
|
|
|
02:34.310 --> 02:42.950 |
|
If we look at Codeclan we can see that it scores really very well indeed in Python coding and in C plus |
|
|
|
02:42.950 --> 02:43.760 |
|
plus coding. |
|
|
|
02:43.760 --> 02:51.380 |
|
It is only beaten by this one here, which is the externally run one which only fractionally does better. |
|
|
|
02:51.380 --> 02:56.780 |
|
And if you look, this model itself is in fact a fine tune of Quinn itself. |
|
|
|
02:56.780 --> 03:01.040 |
|
So I think frankly, we're doing pretty well by sticking with code. |
|
|
|
03:01.040 --> 03:01.610 |
|
Quinn. |
|
|
|
03:01.640 --> 03:04.250 |
|
There's a big drop off after that. |
|
|
|
03:04.250 --> 03:10.640 |
|
The other one that I was interested in showing you was I wanted to also show you Code Gemma down here, |
|
|
|
03:10.640 --> 03:13.550 |
|
which I thought would be great to try a different one and see Google. |
|
|
|
03:13.610 --> 03:15.590 |
|
But as it happens, when I try to use code. |
|
|
|
03:15.620 --> 03:22.400 |
|
Gemma uh, it was able to to answer very simple tasks, but it crashed whenever it was given harder |
|
|
|
03:22.400 --> 03:23.000 |
|
tasks. |
|
|
|
03:23.000 --> 03:25.730 |
|
You may want to try and see as an exercise whether you can get code. |
|
|
|
03:25.730 --> 03:31.010 |
|
Gemma to work, but I wasn't able to for this particular problem, at least not for any complicated, |
|
|
|
03:31.010 --> 03:35.000 |
|
uh, translation or reimplementation of Python in C plus plus. |
|
|
|
03:35.000 --> 03:40.040 |
|
But certainly an exercise for you will be to try out other models, because we have so many to choose |
|
|
|
03:40.040 --> 03:42.470 |
|
from and see what you find. |
|
|
|
03:42.470 --> 03:46.820 |
|
So we've picked, uh, Code Quan. |
|
|
|
03:46.850 --> 03:48.560 |
|
Let's go into it one more time. |
|
|
|
03:48.680 --> 03:52.970 |
|
Uh, the next thing to do is to figure out, okay, how are we going to run this various ways? |
|
|
|
03:52.970 --> 03:53.450 |
|
We could do it. |
|
|
|
03:53.450 --> 03:54.770 |
|
You already know of one of them. |
|
|
|
03:54.770 --> 04:01.460 |
|
We could go onto Google Colab and write the code to use the Hugging Face Transformers library to directly |
|
|
|
04:01.460 --> 04:07.280 |
|
access Codex code, load it in and run it in inference time. |
|
|
|
04:07.610 --> 04:08.810 |
|
And that would work great. |
|
|
|
04:08.810 --> 04:14.810 |
|
The thing is, I want to execute the compiled code on my box, so it would be a bit inconvenient. |
|
|
|
04:14.840 --> 04:20.810 |
|
We'd have to run that in Jupyter, in in Google Colab, and then copy and paste the code over to my |
|
|
|
04:20.810 --> 04:21.500 |
|
box. |
|
|
|
04:21.650 --> 04:25.250 |
|
And I was hoping to be able to do it all in one nice user interface. |
|
|
|
04:25.250 --> 04:30.230 |
|
And so basically I want to be able to run this model as it is, but from my box. |
|
|
|
04:30.260 --> 04:34.370 |
|
Now there are a few ways of doing that, and I'm not going to go into all of them, but there's one |
|
|
|
04:34.370 --> 04:40.550 |
|
that's super convenient, and it's actually going to show us a way that you can deploy models in production |
|
|
|
04:40.550 --> 04:41.990 |
|
for production purposes. |
|
|
|
04:41.990 --> 04:50.000 |
|
And that is to ask hugging face to run this model for you and to give you an endpoint which you can |
|
|
|
04:50.000 --> 04:53.540 |
|
use to call the model remotely from your code. |
|
|
|
04:53.540 --> 04:59.690 |
|
So it's a sort of hosting solution for running a model like this model, or any open source model you |
|
|
|
04:59.690 --> 05:02.810 |
|
want, and give you an endpoint to run it yourself. |
|
|
|
05:02.840 --> 05:04.790 |
|
And that sounds super advanced. |
|
|
|
05:04.880 --> 05:07.340 |
|
You may think that there's going to be a lot to it to do this. |
|
|
|
05:07.430 --> 05:09.680 |
|
It turns out it's not difficult at all. |
|
|
|
05:09.680 --> 05:12.740 |
|
All you have to do is go to this deploy button right here. |
|
|
|
05:12.740 --> 05:19.130 |
|
And the first option here is inference endpoints dedicated inference deployments for production. |
|
|
|
05:19.130 --> 05:24.380 |
|
And it's saying this is a way you can have your own deployment of this model. |
|
|
|
05:24.380 --> 05:30.350 |
|
And you'll be able to call it yourself from anywhere and use this model on your own. |
|
|
|
05:30.350 --> 05:32.900 |
|
Now of course you'll have to pay for that. |
|
|
|
05:33.230 --> 05:37.310 |
|
It doesn't come for free, but it's not massively expensive. |
|
|
|
05:37.310 --> 05:39.260 |
|
But this is one that is, it costs a little bit more. |
|
|
|
05:39.290 --> 05:41.480 |
|
It's something certainly to be aware of. |
|
|
|
05:41.630 --> 05:48.890 |
|
Um, so when you, when you come in to, to use one of these inference endpoints, you choose whether |
|
|
|
05:48.890 --> 05:53.450 |
|
you want to run on AWS or Azure or GCP, Google Cloud. |
|
|
|
05:53.660 --> 05:57.440 |
|
And you can also choose whether you want a CPU box or a GPU box. |
|
|
|
05:57.440 --> 06:05.690 |
|
In the case of this model, we would need to have a GPU box, and we would need to have at least an |
|
|
|
06:05.690 --> 06:07.100 |
|
Nvidia L4. |
|
|
|
06:07.130 --> 06:13.820 |
|
At least that is what it is recommending to us, which has 24GB of Ram on the GPU. |
|
|
|
06:14.210 --> 06:22.100 |
|
Uh, so we'd simply select this and then we would go ahead and, uh, fill in some of these things and |
|
|
|
06:22.100 --> 06:23.870 |
|
click Create Endpoint. |
|
|
|
06:23.870 --> 06:25.190 |
|
And it's as easy as that. |
|
|
|
06:25.190 --> 06:32.270 |
|
We would have our own dedicated endpoint for for your personal use running with hugging face paying |
|
|
|
06:32.300 --> 06:34.070 |
|
$0.80 an hour. |
|
|
|
06:34.070 --> 06:36.860 |
|
If you're in the US, you may see different pricing there. |
|
|
|
06:36.860 --> 06:41.360 |
|
So you know it's not super cheap, but you can turn it off and on whenever you want. |
|
|
|
06:41.390 --> 06:46.070 |
|
So for doing an experiment, if this is just going to take you an hour or two, uh, it's something |
|
|
|
06:46.070 --> 06:53.180 |
|
that's that's great fun to do, I recommend it, uh, if your budget allows for these things, but if |
|
|
|
06:53.180 --> 06:55.340 |
|
not, never fear, I've done it for you. |
|
|
|
06:55.370 --> 06:57.560 |
|
You can watch along and see what it's like. |
|
|
|
06:57.680 --> 07:01.700 |
|
I'm going to go to my dedicated endpoints using this menu right here. |
|
|
|
07:01.700 --> 07:05.780 |
|
And you'll see that I've got two dedicated endpoints that I've set up. |
|
|
|
07:05.780 --> 07:09.110 |
|
One of them is indeed Codeclan, the one we've just been looking at. |
|
|
|
07:09.110 --> 07:15.560 |
|
I basically followed through that screen and it's running on that box that was recommended, the L four. |
|
|
|
07:15.590 --> 07:21.410 |
|
So I am currently spending $0.80 an hour for the pleasure of running this endpoint. |
|
|
|
07:21.440 --> 07:23.390 |
|
Don't let it be said I don't do anything for you. |
|
|
|
07:23.930 --> 07:27.740 |
|
And I also, as I say, I tried code Gemma. |
|
|
|
07:28.070 --> 07:29.690 |
|
I tried it on an alpha box. |
|
|
|
07:29.690 --> 07:31.310 |
|
I was not able to get this to run. |
|
|
|
07:31.310 --> 07:35.960 |
|
It would do simple, simple stuff, but anything too complicated it would crash. |
|
|
|
07:36.200 --> 07:39.950 |
|
But I love this to be an exercise for you to come in and try this. |
|
|
|
07:39.950 --> 07:46.040 |
|
Try getting code Gemma to work, and then I'd be interested to see how it matches up with code. |
|
|
|
07:46.490 --> 07:52.100 |
|
But I think you'll find that just as we saw from the leaderboard, there will be very little doubt that |
|
|
|
07:52.100 --> 07:59.450 |
|
Codeclan is hugely superior and very, very capable of these kinds of tasks. |
|
|
|
07:59.540 --> 08:05.210 |
|
So at this point, you have hopefully followed this through. |
|
|
|
08:05.240 --> 08:10.550 |
|
You have to wait five minutes, go and get a coffee for for the codeclan endpoint to come up and running, |
|
|
|
08:10.550 --> 08:17.630 |
|
and then we'll be ready to get back to JupyterLab running on our box to call this endpoint and to try |
|
|
|
08:17.630 --> 08:19.190 |
|
out some code generation. |
|
|
|
08:19.190 --> 08:20.360 |
|
I will see you in a minute.
|
|
|