From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
319 lines
8.7 KiB
319 lines
8.7 KiB
WEBVTT |
|
|
|
00:00.980 --> 00:04.850 |
|
Welcome to the Jupyter Lab for day four. |
|
|
|
00:04.880 --> 00:10.820 |
|
It's going to look very familiar because it's actually I've started with a duplicate of day three. |
|
|
|
00:11.030 --> 00:14.720 |
|
And in this duplicate there's a few cells. |
|
|
|
00:14.720 --> 00:19.880 |
|
We're going to want to rerun the imports at the beginning connecting our environment. |
|
|
|
00:19.880 --> 00:22.340 |
|
We're going to want to connect to our frontier models. |
|
|
|
00:22.340 --> 00:23.630 |
|
Because guess what. |
|
|
|
00:23.630 --> 00:28.520 |
|
We're going to be able to get the frontier models to work side by side with open source models, which |
|
|
|
00:28.520 --> 00:29.900 |
|
is going to be terrific. |
|
|
|
00:30.260 --> 00:37.340 |
|
Uh, we'll get our prompts, our messages function, if you remember that something that can write to |
|
|
|
00:37.490 --> 00:39.530 |
|
output C plus plus file. |
|
|
|
00:39.920 --> 00:43.880 |
|
Um, this I don't think we need to run because it's going to be overwritten. |
|
|
|
00:43.880 --> 00:49.850 |
|
But then our code for our Pi program, we don't need to try it out because you'll remember that. |
|
|
|
00:49.850 --> 00:59.750 |
|
But we will get our code for the hard program that looks for subarrays of arrays with the largest cumulative |
|
|
|
00:59.750 --> 01:00.680 |
|
number. |
|
|
|
01:01.360 --> 01:07.870 |
|
Um, and we do need this stream, GPT function. |
|
|
|
01:07.870 --> 01:10.600 |
|
And we need this stream, Claude function. |
|
|
|
01:10.600 --> 01:14.710 |
|
And we need this optimize function, although we're about to overwrite it. |
|
|
|
01:14.710 --> 01:22.270 |
|
And you'll remember the gorgeous gradient functions will take execute Python and execute C plus plus, |
|
|
|
01:22.270 --> 01:28.180 |
|
which, uh, I hope you remember that this is something which is not to be shared with others. |
|
|
|
01:28.180 --> 01:33.280 |
|
You don't want to allow people to have the ability to execute arbitrary Python and C plus plus code |
|
|
|
01:33.310 --> 01:34.360 |
|
on your box. |
|
|
|
01:34.360 --> 01:35.650 |
|
That's not a good idea. |
|
|
|
01:35.650 --> 01:40.960 |
|
So keep the Gradio interface to yourself and don't say share equals true for that. |
|
|
|
01:40.960 --> 01:44.170 |
|
And then some CSS to make it look fancy. |
|
|
|
01:44.200 --> 01:45.430 |
|
All right. |
|
|
|
01:45.430 --> 01:47.770 |
|
And now we get to our new code. |
|
|
|
01:47.770 --> 01:50.710 |
|
And it is going to be extremely simple. |
|
|
|
01:50.710 --> 01:52.780 |
|
We start with some imports. |
|
|
|
01:52.780 --> 01:58.420 |
|
We're importing from Huggingface hub, the login function you've seen before, but also a class called |
|
|
|
01:58.420 --> 01:59.860 |
|
inference client. |
|
|
|
02:00.280 --> 02:03.210 |
|
And we're importing the the tokenizer. |
|
|
|
02:03.810 --> 02:05.790 |
|
We're going to log in to hugging face. |
|
|
|
02:05.790 --> 02:06.780 |
|
There we go. |
|
|
|
02:06.780 --> 02:09.030 |
|
And now just some constants. |
|
|
|
02:09.150 --> 02:15.000 |
|
Code is this is the name of the code model I've also got here code Gemma in case you want to give it |
|
|
|
02:15.000 --> 02:17.040 |
|
a try and see if you have more success than me. |
|
|
|
02:17.070 --> 02:22.200 |
|
This is the URL of my endpoint, so it's just taken from my endpoint page. |
|
|
|
02:22.200 --> 02:24.540 |
|
I can just take this this URL. |
|
|
|
02:24.540 --> 02:29.040 |
|
That is where my endpoint is running right now and costing me $0.80 an hour. |
|
|
|
02:29.040 --> 02:35.250 |
|
And this is code Gemma that is sadly paused as it was not particularly stable. |
|
|
|
02:35.640 --> 02:38.370 |
|
Okay, so let's give this a try. |
|
|
|
02:38.400 --> 02:45.090 |
|
So remember tokenizer you can create using the auto tokenizer class and say Frompretrained pass in the |
|
|
|
02:45.090 --> 02:49.020 |
|
name of your model and you will get the right tokenizer for that model. |
|
|
|
02:49.020 --> 02:51.390 |
|
And then I'm going to create this. |
|
|
|
02:51.390 --> 02:57.540 |
|
This is the function that we wrote before that's going to create the nice structured, um list of dictionaries |
|
|
|
02:57.540 --> 03:03.380 |
|
to ask for the simple function for pi to be rewritten in C plus plus. |
|
|
|
03:03.380 --> 03:09.980 |
|
And then we're going to use the Tokenizers apply chat template to turn that into text. |
|
|
|
03:10.010 --> 03:12.110 |
|
Let's just take a look at what I'm talking about here. |
|
|
|
03:12.110 --> 03:15.650 |
|
So if we remember to run that and then run this. |
|
|
|
03:16.250 --> 03:19.250 |
|
So first of all remember what messages is. |
|
|
|
03:19.250 --> 03:22.490 |
|
Messages is a list with two elements. |
|
|
|
03:22.490 --> 03:24.650 |
|
One of them has a system role. |
|
|
|
03:24.650 --> 03:25.760 |
|
And that's the content. |
|
|
|
03:25.790 --> 03:29.180 |
|
You were insistent that re-implements Python code blah blah blah. |
|
|
|
03:29.900 --> 03:34.850 |
|
And then user uh, and then the user method. |
|
|
|
03:34.880 --> 03:41.810 |
|
Now I should mention here just to in the spirit of being completely open, that I've added in a little |
|
|
|
03:41.810 --> 03:48.590 |
|
hint here, keep implementations of random number generators identical so that match, uh, results |
|
|
|
03:48.590 --> 03:49.940 |
|
match exactly. |
|
|
|
03:50.060 --> 03:57.410 |
|
Uh, so as you can imagine, uh, I've had to add hints here and there just to keep things on track. |
|
|
|
03:57.470 --> 04:00.830 |
|
And you will see the results of that in a moment. |
|
|
|
04:00.880 --> 04:04.240 |
|
So this is, of course, the messages that you're familiar with. |
|
|
|
04:04.240 --> 04:11.710 |
|
And now, if I show you what what text became after we applied the chat template, this if I print it, |
|
|
|
04:11.740 --> 04:13.240 |
|
it will be spaced out a bit better. |
|
|
|
04:13.270 --> 04:14.590 |
|
Let's see it printed. |
|
|
|
04:14.950 --> 04:16.360 |
|
Here we go. |
|
|
|
04:16.510 --> 04:17.440 |
|
Have a look at this. |
|
|
|
04:17.440 --> 04:24.490 |
|
So there is a special token I am start and then system and then the system message. |
|
|
|
04:24.490 --> 04:26.080 |
|
And then the same thing with user. |
|
|
|
04:26.080 --> 04:27.970 |
|
And then this is the code. |
|
|
|
04:27.970 --> 04:30.700 |
|
And then it ends with this little prompt. |
|
|
|
04:30.700 --> 04:37.330 |
|
That means that the LM is just going to keep going and give the assistance response. |
|
|
|
04:37.540 --> 04:46.090 |
|
So with that now you would imagine that making the call to a hugging face endpoint, making this kind |
|
|
|
04:46.090 --> 04:51.430 |
|
of message to an open source model and getting back the response might be really difficult. |
|
|
|
04:51.730 --> 04:52.840 |
|
And it's not. |
|
|
|
04:52.840 --> 04:55.390 |
|
It's one of those things that's just a couple of lines of code. |
|
|
|
04:55.390 --> 04:56.050 |
|
Here it is. |
|
|
|
04:56.050 --> 04:57.490 |
|
It's as simple as this. |
|
|
|
04:57.580 --> 05:04.020 |
|
Uh, you create an object, the inference client and you tell it the URL of your endpoint and your hugging |
|
|
|
05:04.050 --> 05:09.150 |
|
face token, and then you just say client dot text generation. |
|
|
|
05:09.150 --> 05:13.320 |
|
You pass in your text, you say, in this case I want to stream back results. |
|
|
|
05:13.320 --> 05:15.780 |
|
If you didn't have that, this would be even even shorter. |
|
|
|
05:15.780 --> 05:18.030 |
|
But it's nice to see the results streaming back. |
|
|
|
05:18.330 --> 05:24.030 |
|
Um, and then max new tokens, um, is where you tell it that it can keep going for a little while. |
|
|
|
05:24.270 --> 05:26.790 |
|
Uh, and let's see what happens. |
|
|
|
05:28.080 --> 05:36.270 |
|
We're getting streaming back from my endpoint, running in hugging face, the result of the, uh, of |
|
|
|
05:36.270 --> 05:38.610 |
|
of this conversion. |
|
|
|
05:40.650 --> 05:44.580 |
|
Now, one thing I will point out while it's doing this, hopefully you're recognizing this code and |
|
|
|
05:44.580 --> 05:50.940 |
|
seeing that it's that it looks very legit, very solid conversion of py, uh, code from Python into |
|
|
|
05:50.940 --> 05:51.300 |
|
C plus. |
|
|
|
05:51.300 --> 05:57.000 |
|
Plus, uh, it has given this, this stuff at the beginning about C plus plus code, you know, giving |
|
|
|
05:57.000 --> 05:57.990 |
|
a bit of an explanation. |
|
|
|
05:57.990 --> 06:01.850 |
|
And it's also ending with a bunch of text about why it did what it did. |
|
|
|
06:01.880 --> 06:07.070 |
|
And this is despite the fact that my system message insists that it shouldn't do that. |
|
|
|
06:07.190 --> 06:13.730 |
|
Um, it says, uh, do not provide any explanation, but I was not able, through the system message |
|
|
|
06:13.730 --> 06:15.980 |
|
to get it not to provide an explanation. |
|
|
|
06:16.010 --> 06:20.420 |
|
Although, as you remember, GPT four and Claude do obey the instructions. |
|
|
|
06:20.420 --> 06:25.610 |
|
So that's something I imagine you could spend more time working on that prompt to really try and make |
|
|
|
06:25.640 --> 06:26.900 |
|
sure that it doesn't happen. |
|
|
|
06:26.900 --> 06:32.360 |
|
But failing that, of course, it's very easy to do some post-processing and strip out the everything |
|
|
|
06:32.360 --> 06:39.800 |
|
that comes up till the, uh, the start of the C plus plus code and up until the, um, so you could |
|
|
|
06:39.800 --> 06:45.890 |
|
strip out everything up until this symbol here and take what comes between that and here. |
|
|
|
06:46.010 --> 06:49.850 |
|
So it'd be pretty easy to do that with some post-processing. |
|
|
|
06:50.270 --> 06:51.170 |
|
All right. |
|
|
|
06:51.170 --> 06:55.310 |
|
Anyway, we've seen how easy it is to run inference. |
|
|
|
06:55.460 --> 07:00.620 |
|
Uh, I will see you next time when we will put this together in a UI and give it a whirl. |
|
|
|
07:00.650 --> 07:01.520 |
|
See you then.
|
|
|