From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
304 lines
7.7 KiB
304 lines
7.7 KiB
WEBVTT |
|
|
|
00:00.740 --> 00:01.910 |
|
Welcome back. |
|
|
|
00:01.910 --> 00:04.220 |
|
It's time to make our full agent framework. |
|
|
|
00:04.220 --> 00:05.630 |
|
I'm super excited about this. |
|
|
|
00:05.660 --> 00:10.490 |
|
It's pulling everything together that we've been doing before, and I think you'll be very happy with |
|
|
|
00:10.490 --> 00:11.510 |
|
the outcome. |
|
|
|
00:11.720 --> 00:13.730 |
|
Uh, so just a quick recap. |
|
|
|
00:13.730 --> 00:14.870 |
|
An agent framework. |
|
|
|
00:14.960 --> 00:17.840 |
|
The term agent I as I said, it's an umbrella term. |
|
|
|
00:17.840 --> 00:20.120 |
|
It can refer to a bunch of different techniques. |
|
|
|
00:20.240 --> 00:23.060 |
|
Um, for example, it can be any of these five. |
|
|
|
00:23.090 --> 00:28.250 |
|
It can be about breaking a complex problem into smaller steps with multiple models carrying out different |
|
|
|
00:28.250 --> 00:29.420 |
|
specialized tasks. |
|
|
|
00:29.420 --> 00:33.890 |
|
It can be the ability for an LLM to have tools to give them extra capabilities. |
|
|
|
00:33.890 --> 00:41.000 |
|
It can be, uh, talking about the agent environment, which is the setup or the agent framework that |
|
|
|
00:41.000 --> 00:43.160 |
|
allows agents to collaborate. |
|
|
|
00:43.190 --> 00:50.270 |
|
Um, it can be the idea that one LLM can act as a planner, dividing tasks into smaller ones, that |
|
|
|
00:50.270 --> 00:55.100 |
|
specialists that can themselves be llms or bits of software can carry out. |
|
|
|
00:55.280 --> 01:00.210 |
|
Um, and then there is another point here, which is that people talk about agentic AI when you're thinking |
|
|
|
01:00.210 --> 01:07.740 |
|
about an agent having its own autonomy agency, uh, beyond necessarily just responding to a prompt, |
|
|
|
01:07.740 --> 01:13.050 |
|
such as having memory, being able to sort of, uh, I don't know, do something like, uh, scrape |
|
|
|
01:13.050 --> 01:18.690 |
|
the web for news information and using that to make decisions about buying or selling stocks, something |
|
|
|
01:18.690 --> 01:19.110 |
|
like that. |
|
|
|
01:19.110 --> 01:26.040 |
|
That is a kind of, uh, something that that exists outside the context of just say, a request response |
|
|
|
01:26.040 --> 01:26.670 |
|
chat. |
|
|
|
01:26.670 --> 01:32.100 |
|
So these are all the kinds of ways that that these are the kinds of things people are referring to when |
|
|
|
01:32.100 --> 01:34.890 |
|
they talk about agentic AI and the use of agents. |
|
|
|
01:34.890 --> 01:39.510 |
|
And what we're really doing here is we're talking about, uh, definitely number one and two there and |
|
|
|
01:39.510 --> 01:42.210 |
|
to a certain extent, numbers three and five. |
|
|
|
01:42.300 --> 01:45.090 |
|
But we're not we're not building an LLM that does the planning. |
|
|
|
01:45.090 --> 01:47.280 |
|
That's not something we'll be doing in this session. |
|
|
|
01:47.280 --> 01:55.710 |
|
So, uh, this should be somewhat familiar to you because this is the chat method that's quite close |
|
|
|
01:55.710 --> 01:57.000 |
|
to what we had before. |
|
|
|
01:57.000 --> 01:59.710 |
|
So you'll recognize a few things about this. |
|
|
|
01:59.710 --> 02:07.210 |
|
This section here is the usual chat radio function that we know really well. |
|
|
|
02:07.210 --> 02:16.180 |
|
It takes a message and a history, and it, uh, sort of unpacks that history into the format that OpenAI |
|
|
|
02:16.210 --> 02:20.200 |
|
will expect and then calls the response. |
|
|
|
02:20.440 --> 02:26.080 |
|
This part here will also look familiar to you because it's our use of tools. |
|
|
|
02:26.080 --> 02:32.860 |
|
It's where we find out if the model wants to call a tool, and if so, we handle that tool. |
|
|
|
02:33.010 --> 02:38.950 |
|
Uh, but there's one little extra line just inserted in there, and it's that line there that what we're |
|
|
|
02:38.950 --> 02:44.710 |
|
going to say is if the person does, if the model decides it needs to, to run the tool to find the |
|
|
|
02:44.710 --> 02:54.520 |
|
price of a ticket, then we will also have the, um, artist generate an image to represent that city |
|
|
|
02:54.520 --> 02:56.200 |
|
that's being looked up. |
|
|
|
02:56.200 --> 02:58.160 |
|
So there we have it. |
|
|
|
02:58.220 --> 03:00.050 |
|
Uh, that's, uh that's nice. |
|
|
|
03:00.050 --> 03:04.160 |
|
And also, now, if you remember before I told you there was a reason I passed back city that you're |
|
|
|
03:04.160 --> 03:04.940 |
|
going to find out. |
|
|
|
03:04.970 --> 03:05.690 |
|
Here it is. |
|
|
|
03:05.690 --> 03:09.800 |
|
That's why I needed the city to pass it to the artist. |
|
|
|
03:10.130 --> 03:13.880 |
|
Um, and then, uh, this is all exactly the same. |
|
|
|
03:13.880 --> 03:16.040 |
|
There's one more tiny change. |
|
|
|
03:16.040 --> 03:22.760 |
|
Which is this here, which is that, uh, once I've collected the response from the model, I then call |
|
|
|
03:22.790 --> 03:26.840 |
|
talker to make sure that we speak the response. |
|
|
|
03:26.840 --> 03:29.780 |
|
So that is our chat. |
|
|
|
03:29.960 --> 03:32.270 |
|
Uh, let's run that. |
|
|
|
03:33.920 --> 03:40.550 |
|
Now, this, I should say, since I've always showed off about how easy Gradio is, this code is a little |
|
|
|
03:40.580 --> 03:41.420 |
|
bit more involved. |
|
|
|
03:41.420 --> 03:46.970 |
|
You may notice the reason is because we're now because we want to do a little bit more and show images. |
|
|
|
03:46.970 --> 03:55.190 |
|
We're going outside the default, the sort of off the shelf, uh, chat user interface that Gradio provides |
|
|
|
03:55.190 --> 03:55.550 |
|
for us. |
|
|
|
03:55.550 --> 03:58.120 |
|
And we have to then build the interface ourselves. |
|
|
|
03:58.150 --> 04:04.960 |
|
And as a result, I've had to put together this interface that kind of puts together the various components |
|
|
|
04:04.960 --> 04:07.090 |
|
like the input and the buttons. |
|
|
|
04:07.300 --> 04:10.180 |
|
But what I'll say is this is still actually super straightforward. |
|
|
|
04:10.180 --> 04:11.470 |
|
It still reads like English. |
|
|
|
04:11.470 --> 04:13.030 |
|
It's very clear what's going on. |
|
|
|
04:13.030 --> 04:19.330 |
|
You'll see everything that's happening here, and hopefully this will be quite readable for you. |
|
|
|
04:19.330 --> 04:26.740 |
|
And you can use this to build more sophisticated chats, more sophisticated UIs yourself. |
|
|
|
04:26.740 --> 04:36.850 |
|
So with that background, we now are going to run this to it's running and we'll bring that up. |
|
|
|
04:37.240 --> 04:41.650 |
|
And here we have our chat with our new assistant. |
|
|
|
04:41.680 --> 04:43.300 |
|
Let's give it a try. |
|
|
|
04:47.740 --> 04:48.280 |
|
Hello. |
|
|
|
04:48.280 --> 04:49.630 |
|
How can I assist you today. |
|
|
|
04:51.040 --> 04:52.090 |
|
You like that? |
|
|
|
04:52.240 --> 04:53.650 |
|
It spoke to us. |
|
|
|
04:53.740 --> 04:54.520 |
|
There we go. |
|
|
|
04:54.520 --> 04:56.620 |
|
That's the first use of an agent. |
|
|
|
04:56.620 --> 05:04.420 |
|
We had a specialist model that's able to create, uh, audio, and we integrated that with our chatbot |
|
|
|
05:04.420 --> 05:07.510 |
|
so that it was able to speak back to us. |
|
|
|
05:15.760 --> 05:17.020 |
|
Great choice. |
|
|
|
05:17.080 --> 05:20.410 |
|
Would you like to know the to the ticket price for a return trip to London? |
|
|
|
05:22.930 --> 05:24.220 |
|
There we go. |
|
|
|
05:24.250 --> 05:27.220 |
|
That's entertaining, let's say. |
|
|
|
05:34.240 --> 05:35.710 |
|
We know there's a pause. |
|
|
|
05:43.090 --> 05:44.080 |
|
Here we go. |
|
|
|
05:48.040 --> 05:51.220 |
|
A return ticket to London is priced at 799. |
|
|
|
05:53.120 --> 05:54.920 |
|
And there we have it. |
|
|
|
05:54.950 --> 05:58.340 |
|
A return ticket to London is priced at $7.99. |
|
|
|
05:58.340 --> 06:00.290 |
|
And there is the image. |
|
|
|
06:00.290 --> 06:04.220 |
|
And that image looks spectacular. |
|
|
|
06:04.370 --> 06:06.290 |
|
A London bus in the middle. |
|
|
|
06:06.290 --> 06:07.700 |
|
It's got Big Ben. |
|
|
|
06:07.700 --> 06:10.400 |
|
It's got the bridge. |
|
|
|
06:10.400 --> 06:11.690 |
|
It's got, uh. |
|
|
|
06:11.720 --> 06:14.420 |
|
Yeah, I can see taxi there. |
|
|
|
06:14.450 --> 06:18.950 |
|
It's just a great montage of images. |
|
|
|
06:19.160 --> 06:26.570 |
|
Uh, and so I find this to be very compelling indeed, a wonderful example of what we're able to achieve |
|
|
|
06:26.570 --> 06:28.280 |
|
with just a little bit of code. |
|
|
|
06:28.520 --> 06:37.550 |
|
And so I present to you a multimodal app, complete with audio and some images running as part of what |
|
|
|
06:37.550 --> 06:48.050 |
|
is a in a, in a small way, a multimodal agentic framework for talking to an airline AI assistant. |
|
|
|
06:48.140 --> 06:49.280 |
|
Great work. |
|
|
|
06:49.310 --> 06:52.790 |
|
I'll see you for the challenge of the week and the wrap up.
|
|
|