From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
319 lines
7.4 KiB
319 lines
7.4 KiB
WEBVTT |
|
|
|
00:00.830 --> 00:04.250 |
|
Welcome back to the radio day in the lab. |
|
|
|
00:04.250 --> 00:05.180 |
|
More to do. |
|
|
|
00:05.210 --> 00:06.620 |
|
Let's keep going. |
|
|
|
00:06.620 --> 00:14.150 |
|
Where we left off is we had just built a simple user interface that was calling an LLM and telling a |
|
|
|
00:14.150 --> 00:16.610 |
|
very, uh, good joke. |
|
|
|
00:16.850 --> 00:20.060 |
|
Uh, let's keep going with this. |
|
|
|
00:20.060 --> 00:28.250 |
|
What we're going to do next is ask for the, uh, assistant to respond in markdown, uh, as a way of, |
|
|
|
00:28.250 --> 00:32.540 |
|
uh, um, getting better looking user interfaces. |
|
|
|
00:32.960 --> 00:40.190 |
|
Um, and wouldn't it be nice if we wanted to show results in gradio with good formatting written in |
|
|
|
00:40.190 --> 00:40.940 |
|
markdown? |
|
|
|
00:40.940 --> 00:47.690 |
|
If we could just have that instead of text box, just replace it with the word markdown, and then the |
|
|
|
00:47.690 --> 00:51.560 |
|
output would be in perfectly formatted markdown. |
|
|
|
00:51.560 --> 00:53.180 |
|
That would be great wouldn't it? |
|
|
|
00:53.210 --> 00:54.380 |
|
Wouldn't it be nice? |
|
|
|
00:55.190 --> 00:56.900 |
|
You're probably getting the idea here. |
|
|
|
00:57.770 --> 00:59.030 |
|
Things just really are. |
|
|
|
00:59.030 --> 00:59.840 |
|
This good. |
|
|
|
01:00.020 --> 01:02.510 |
|
Uh, So let's say your message. |
|
|
|
01:02.510 --> 01:03.740 |
|
Let's say, um. |
|
|
|
01:03.890 --> 01:14.240 |
|
Um, how do I get from Times Square to Times Square like that to Grand Central? |
|
|
|
01:14.960 --> 01:17.120 |
|
I've got a question for New York navigation. |
|
|
|
01:17.120 --> 01:18.710 |
|
Let's see how it does. |
|
|
|
01:19.640 --> 01:21.440 |
|
It's thinking about that. |
|
|
|
01:22.400 --> 01:23.360 |
|
And there we go. |
|
|
|
01:23.360 --> 01:24.440 |
|
Here's a response. |
|
|
|
01:24.440 --> 01:28.370 |
|
And you can see to get from Times Square to Grand Central Terminal in New York City, it figured out |
|
|
|
01:28.370 --> 01:29.660 |
|
that's what I was talking about. |
|
|
|
01:29.870 --> 01:34.040 |
|
Follow these steps and you can see it's good headings. |
|
|
|
01:34.310 --> 01:41.810 |
|
And it's got a nice sub bullets and numbers and all the rest of it, as described in the markdown that |
|
|
|
01:41.810 --> 01:43.820 |
|
came back from GPT four. |
|
|
|
01:43.850 --> 01:47.000 |
|
Oh very easy, very nice. |
|
|
|
01:47.390 --> 01:50.360 |
|
Let's have a look at what else we can do. |
|
|
|
01:51.500 --> 01:53.420 |
|
Uh, streaming. |
|
|
|
01:53.420 --> 01:55.700 |
|
Streaming is something we got used to last time. |
|
|
|
01:55.700 --> 02:01.840 |
|
So can we stream results back to Gradio user interfaces, just as we did when it was coming back into |
|
|
|
02:01.840 --> 02:03.820 |
|
a Jupyter output cell. |
|
|
|
02:03.820 --> 02:04.960 |
|
So here we go. |
|
|
|
02:04.990 --> 02:07.300 |
|
We change our function. |
|
|
|
02:07.330 --> 02:10.840 |
|
It used to be the message GPT. |
|
|
|
02:10.870 --> 02:12.400 |
|
Now we're making it stream GPT. |
|
|
|
02:12.430 --> 02:13.420 |
|
Different function. |
|
|
|
02:13.420 --> 02:15.610 |
|
And the key thing is that this isn't actually a function. |
|
|
|
02:15.610 --> 02:20.320 |
|
It's a generator in that it's going to end by yielding a result. |
|
|
|
02:20.320 --> 02:26.170 |
|
And Gradio is going to detect that we're giving it a generator, not a function. |
|
|
|
02:26.170 --> 02:32.830 |
|
And because of that, Gradio is automatically going to be iterative and decide to fill in, um, piece |
|
|
|
02:32.830 --> 02:35.980 |
|
by piece as it comes back from this generator. |
|
|
|
02:36.100 --> 02:38.020 |
|
So usual story. |
|
|
|
02:38.020 --> 02:45.100 |
|
I create the messages and then you'll remember this time it's the same API call, but we pass in stream |
|
|
|
02:45.100 --> 02:46.120 |
|
equals true. |
|
|
|
02:46.150 --> 02:47.770 |
|
You remember how Claude does it? |
|
|
|
02:47.860 --> 02:48.790 |
|
Hopefully you do. |
|
|
|
02:48.820 --> 02:54.220 |
|
With Claude, you don't have an attribute, but instead you call dot stream instead of dot create. |
|
|
|
02:54.220 --> 02:56.290 |
|
But otherwise it's very similar. |
|
|
|
02:56.350 --> 02:58.810 |
|
So one thing that's worth noting here. |
|
|
|
02:58.840 --> 03:01.110 |
|
Just a tiny subtlety with Gradio. |
|
|
|
03:01.140 --> 03:08.820 |
|
When you are streaming back the results to Gradio, you don't stream back chunk by chunk of the results. |
|
|
|
03:08.820 --> 03:15.870 |
|
You have to stream back the entire cumulative result so far and stream back a longer and longer cumulative |
|
|
|
03:15.870 --> 03:16.410 |
|
result. |
|
|
|
03:16.410 --> 03:20.520 |
|
So you can see what I'm doing is I'm I'm sort of starting with an empty string. |
|
|
|
03:20.520 --> 03:28.110 |
|
And then for each chunk I'm adding that in and then yielding the total cumulative result so far. |
|
|
|
03:28.260 --> 03:34.830 |
|
Um, and if you don't do that, what you'll see is each individual chunk will appear in the output cell |
|
|
|
03:34.830 --> 03:37.140 |
|
and then disappear and will be replaced by something else. |
|
|
|
03:37.140 --> 03:38.910 |
|
So you have to do it this way. |
|
|
|
03:38.910 --> 03:42.900 |
|
If you don't see what I mean, try doing yield chunk instead of yield result. |
|
|
|
03:43.050 --> 03:43.290 |
|
Sorry. |
|
|
|
03:43.320 --> 03:46.200 |
|
Yield chunk .0. delta dot content. |
|
|
|
03:46.320 --> 03:46.590 |
|
Uh. |
|
|
|
03:46.590 --> 03:49.350 |
|
And you'll see exactly what I mean. |
|
|
|
03:49.350 --> 03:50.910 |
|
It's not not going to look good. |
|
|
|
03:51.150 --> 03:59.640 |
|
Uh, anyway, that is our stream GPT uh, and wouldn't it be nice if all we needed to do was replace |
|
|
|
03:59.760 --> 04:06.390 |
|
the the function that it used to be message GPT with stream, GPT and Gradio just figured out the rest. |
|
|
|
04:06.390 --> 04:09.300 |
|
It figured out that okay, this is a generator, not a function. |
|
|
|
04:09.300 --> 04:11.610 |
|
Therefore, they're going to want to stream back results. |
|
|
|
04:11.610 --> 04:15.360 |
|
Therefore, I need to have a sort of typewriter animation style effect. |
|
|
|
04:15.390 --> 04:18.660 |
|
Let's see if it really can be that simple. |
|
|
|
04:18.660 --> 04:20.190 |
|
Can it be that simple? |
|
|
|
04:20.520 --> 04:21.660 |
|
Here we go. |
|
|
|
04:21.690 --> 04:29.940 |
|
Uh, how do I get from Times Square to Grand Central? |
|
|
|
04:32.310 --> 04:33.360 |
|
And there we go. |
|
|
|
04:33.510 --> 04:34.470 |
|
Of course, it's that simple. |
|
|
|
04:34.500 --> 04:35.460 |
|
Of course it is. |
|
|
|
04:35.490 --> 04:36.930 |
|
Streams the results. |
|
|
|
04:36.930 --> 04:37.920 |
|
They look great. |
|
|
|
04:37.920 --> 04:40.380 |
|
Markdown looks fantastic. |
|
|
|
04:40.890 --> 04:46.710 |
|
Uh, so, uh, of course it wouldn't be doing my job if I didn't show you how easy it is with Claude |
|
|
|
04:46.710 --> 04:47.280 |
|
as well. |
|
|
|
04:47.280 --> 04:51.540 |
|
I mentioned it before, and now you can see there is Claude's API call. |
|
|
|
04:51.540 --> 04:52.800 |
|
It's very similar. |
|
|
|
04:52.800 --> 04:55.650 |
|
You call dot stream, you don't pass in the parameter. |
|
|
|
04:55.650 --> 05:00.470 |
|
You remember that you do have to specify max tokens and the system message goes in separately. |
|
|
|
05:00.500 --> 05:02.240 |
|
Otherwise very similar. |
|
|
|
05:02.270 --> 05:05.510 |
|
The streaming back is just for results as stream. |
|
|
|
05:05.690 --> 05:08.030 |
|
So with result as stream a context manager. |
|
|
|
05:08.030 --> 05:13.070 |
|
And then we yield the full response just as before. |
|
|
|
05:13.550 --> 05:16.670 |
|
And at this point it's going to be boringly simple. |
|
|
|
05:16.670 --> 05:17.780 |
|
You get the joke. |
|
|
|
05:17.810 --> 05:21.050 |
|
You simply pass in that function instead. |
|
|
|
05:21.050 --> 05:24.110 |
|
And now we are talking to Claude instead. |
|
|
|
05:24.110 --> 05:25.640 |
|
We'll ask it the same question though. |
|
|
|
05:25.670 --> 05:31.910 |
|
How do I get from Times Square to Grand Central? |
|
|
|
05:33.290 --> 05:40.250 |
|
And here comes Claude's response, a bit shorter, just two options, but nicely structured. |
|
|
|
05:40.250 --> 05:41.720 |
|
Nicely formatted. |
|
|
|
05:41.720 --> 05:43.820 |
|
Very good indeed. |
|
|
|
05:44.120 --> 05:53.750 |
|
Um, so we can take this one step forwards by having the ability to choose either GPT or Claude. |
|
|
|
05:53.750 --> 05:56.840 |
|
But I'm going to get to that in the very next session. |
|
|
|
05:56.840 --> 05:57.980 |
|
So hang on in there.
|
|
|