From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
586 lines
15 KiB
586 lines
15 KiB
WEBVTT |
|
|
|
00:01.070 --> 00:04.640 |
|
It's always welcome back to JupyterLab, my favorite place to be. |
|
|
|
00:04.640 --> 00:07.790 |
|
And now we are, of course in the week five folder. |
|
|
|
00:07.790 --> 00:14.000 |
|
And I'm looking here at the day one Jupyter notebook in week five, which is going to be our home for |
|
|
|
00:14.000 --> 00:20.240 |
|
today, where we look at building our own DIY rag implementation in a very simplistic way, but it will |
|
|
|
00:20.240 --> 00:22.610 |
|
give you some real insight into how this works. |
|
|
|
00:22.700 --> 00:29.870 |
|
So first of all, let me tell you about our company in Shoreham, an insurance tech company. |
|
|
|
00:29.990 --> 00:37.400 |
|
We have got access to their company shared folder and it's called Knowledge Base and I have dropped |
|
|
|
00:37.400 --> 00:41.690 |
|
it right here in the same week five folder for your perusal. |
|
|
|
00:41.690 --> 00:48.260 |
|
If we open it up, you'll see that it contains four folders company contracts, employees and products |
|
|
|
00:48.260 --> 00:53.660 |
|
representing four different um divisions or four different areas of the company. |
|
|
|
00:54.020 --> 01:00.050 |
|
Um, now, one thing I will mention, as you may have guessed perhaps, is that you'll see a fair amount |
|
|
|
01:00.050 --> 01:03.450 |
|
of company data here, which is all completely fictitious. |
|
|
|
01:03.660 --> 01:07.440 |
|
This was, of course created by an LLM. |
|
|
|
01:07.470 --> 01:14.940 |
|
I took took my own exercise from a couple of weeks ago, wrote a data generator and used it to craft |
|
|
|
01:14.940 --> 01:19.020 |
|
some of this data and keep relationships between some of the documents and so on. |
|
|
|
01:19.050 --> 01:24.450 |
|
I did add in a little couple of tweaks here and there myself, but almost all of this is generated by |
|
|
|
01:24.450 --> 01:28.530 |
|
either GPT four or by Claude. |
|
|
|
01:29.370 --> 01:34.470 |
|
So just to have a quick peek at this in company, there's a few things here about the company. |
|
|
|
01:34.470 --> 01:39.180 |
|
I think these are quite short documents if these are in the form of markdown documents. |
|
|
|
01:39.180 --> 01:46.440 |
|
So if I can open it in this way, then we get to see some nice, um, write ups about the insurer Elm. |
|
|
|
01:46.440 --> 01:50.880 |
|
It was founded apparently by Avery Lancaster in 2015. |
|
|
|
01:50.880 --> 01:52.230 |
|
Remember, Avery's name will be. |
|
|
|
01:52.260 --> 01:53.910 |
|
We'll be meeting her a few times. |
|
|
|
01:54.240 --> 01:57.030 |
|
Um, and if you want to see the original documents, of course. |
|
|
|
01:57.030 --> 02:00.210 |
|
They look like the markdown that we are familiar with. |
|
|
|
02:00.210 --> 02:03.850 |
|
Um, then if we go back into knowledge base in contracts. |
|
|
|
02:03.850 --> 02:10.540 |
|
We've got some more substantive information about contracts, which includes the contract terms renewal, |
|
|
|
02:10.570 --> 02:15.130 |
|
the features that are included in that contract that's going to come back and be important later. |
|
|
|
02:15.130 --> 02:16.960 |
|
So so keep note of that. |
|
|
|
02:17.260 --> 02:19.150 |
|
Um, and some other things. |
|
|
|
02:19.510 --> 02:25.480 |
|
Uh, if we go back into knowledge base again and look at employees, these are the employee HR records |
|
|
|
02:25.480 --> 02:26.770 |
|
of the different employees. |
|
|
|
02:26.770 --> 02:32.440 |
|
And even the CEO, Avery Lancaster herself has an HR record that you'll see right here. |
|
|
|
02:32.770 --> 02:41.050 |
|
Uh, so all everyone has their their HR documents, um, and then products contains a write up of the |
|
|
|
02:41.050 --> 02:49.840 |
|
different offerings from insurer Elm, including car Elm, uh, for auto insurance, uh home Elm and |
|
|
|
02:49.870 --> 02:50.530 |
|
and so on. |
|
|
|
02:50.530 --> 02:55.600 |
|
You can look through these documents, but they are wonderful in how real they are. |
|
|
|
02:55.630 --> 02:59.800 |
|
All of course courtesy of Frontier Models. |
|
|
|
02:59.920 --> 03:03.480 |
|
A nice product summary right there with even with a roadmap. |
|
|
|
03:03.480 --> 03:04.500 |
|
Love it. |
|
|
|
03:04.980 --> 03:06.360 |
|
Uh, okay. |
|
|
|
03:06.390 --> 03:08.940 |
|
Back then to day one. |
|
|
|
03:08.940 --> 03:10.800 |
|
So what are we going to do? |
|
|
|
03:10.830 --> 03:16.380 |
|
We are going to write something now that is going to be asking questions about this company, and we |
|
|
|
03:16.410 --> 03:20.700 |
|
are going to artificially look up and insert context into the prompt. |
|
|
|
03:20.700 --> 03:22.110 |
|
So we do some imports. |
|
|
|
03:22.110 --> 03:25.470 |
|
We're going to use GPT four mini this time. |
|
|
|
03:25.530 --> 03:31.170 |
|
Um, and um uh, so now we get to the juice. |
|
|
|
03:31.170 --> 03:40.560 |
|
So what we do here this this is a Python function which will, uh, look in this folder and take all |
|
|
|
03:40.560 --> 03:43.350 |
|
of its contents and put it into a list. |
|
|
|
03:43.350 --> 03:44.100 |
|
Employees. |
|
|
|
03:44.100 --> 03:51.600 |
|
So employees is, in fact, now going to be a list of the file names in knowledge base slash employees. |
|
|
|
03:51.600 --> 03:59.400 |
|
And for each one, this is some janky code that that is just sort of hacked together, which basically |
|
|
|
03:59.640 --> 04:06.490 |
|
takes the name of the file, which is in fact an employee's Floyd's name and takes the their last name. |
|
|
|
04:06.520 --> 04:14.140 |
|
It just splits out their last name opens up that file and puts it into a dictionary where the key is |
|
|
|
04:14.140 --> 04:17.080 |
|
the employee name and the document is the document. |
|
|
|
04:17.080 --> 04:20.020 |
|
So let's just run this if we have a look at this. |
|
|
|
04:20.020 --> 04:22.090 |
|
So we're expecting to see a dictionary. |
|
|
|
04:23.080 --> 04:24.760 |
|
Uh sorry. |
|
|
|
04:25.210 --> 04:28.570 |
|
Um let's see it's called context or dictionary. |
|
|
|
04:28.570 --> 04:31.180 |
|
Let's just look at all of the keys in context. |
|
|
|
04:31.390 --> 04:33.130 |
|
So here are the keys. |
|
|
|
04:33.130 --> 04:38.410 |
|
The keys are indeed the last names of the employees, as you would expect. |
|
|
|
04:38.410 --> 04:42.040 |
|
And one of them is Lancaster, our CEO. |
|
|
|
04:42.040 --> 04:44.560 |
|
So let's just have a quick look at Lancaster. |
|
|
|
04:44.560 --> 04:48.520 |
|
So we should simply be able to look in our dictionary at Lancaster. |
|
|
|
04:48.520 --> 04:53.620 |
|
And we should just see bam the markdown document from her HR record. |
|
|
|
04:53.650 --> 04:58.990 |
|
Nothing magic at all going on here, reading in files, shoving them in a dictionary. |
|
|
|
04:58.990 --> 05:00.670 |
|
That's all there is to it. |
|
|
|
05:01.480 --> 05:09.200 |
|
So now let's just extend this And what we're going to do here is do exactly the same thing for the products. |
|
|
|
05:09.230 --> 05:16.520 |
|
Look in the products folder, iterate through each of the files, pluck out its name, and then shove |
|
|
|
05:16.520 --> 05:18.800 |
|
that in the same context dictionary. |
|
|
|
05:18.800 --> 05:22.520 |
|
So now if I look in the if I run that first, try not to forget that. |
|
|
|
05:22.520 --> 05:23.180 |
|
There we go. |
|
|
|
05:23.180 --> 05:24.650 |
|
And now look at the keys. |
|
|
|
05:24.650 --> 05:28.580 |
|
We should see a combination of employee last names and product names. |
|
|
|
05:28.940 --> 05:32.570 |
|
Here are the last names and their market. |
|
|
|
05:32.750 --> 05:33.620 |
|
L'm home. |
|
|
|
05:33.620 --> 05:34.910 |
|
L'm calm. |
|
|
|
05:35.030 --> 05:38.150 |
|
Are the product names okay? |
|
|
|
05:38.810 --> 05:40.130 |
|
Ready for action. |
|
|
|
05:40.160 --> 05:41.780 |
|
We'll have a system message. |
|
|
|
05:41.780 --> 05:47.180 |
|
You are an expert in answering accurate questions about ensure l'm the insurance tech company. |
|
|
|
05:47.210 --> 05:49.100 |
|
Give brief, accurate answers. |
|
|
|
05:49.100 --> 05:51.590 |
|
If you don't know the answer, say so. |
|
|
|
05:51.590 --> 05:52.820 |
|
Do not make anything up. |
|
|
|
05:52.820 --> 05:55.550 |
|
If you haven't been provided with relevant context. |
|
|
|
05:56.060 --> 06:02.570 |
|
It turns out that making these kinds of very authoritative instructions in the system prompt to tell |
|
|
|
06:02.570 --> 06:08.790 |
|
the model not to invent things is effective in stopping it from hallucinating and keeping accuracy high. |
|
|
|
06:08.970 --> 06:13.500 |
|
We're going to see that again later, too, but it's worth bearing in mind in keeping this one in your |
|
|
|
06:13.530 --> 06:19.500 |
|
in your notebook or in a list of useful things, because that kind of prompting is useful when accuracy |
|
|
|
06:19.500 --> 06:20.550 |
|
is essential. |
|
|
|
06:20.940 --> 06:32.610 |
|
So we will see that we will add that in to get we will make a function called get relevant context. |
|
|
|
06:32.610 --> 06:35.490 |
|
So what this function is going to do this is very important. |
|
|
|
06:35.490 --> 06:45.090 |
|
It's going to um uh it's going to take um a message, any kind of message. |
|
|
|
06:45.450 --> 06:53.160 |
|
Uh, and it's then going to iterate through the context, taking the title and the details out of each |
|
|
|
06:53.160 --> 06:55.020 |
|
item in the context. |
|
|
|
06:55.020 --> 07:02.370 |
|
And it's going to see whether or not the title, the, the, either the last name of the employee or |
|
|
|
07:02.370 --> 07:07.370 |
|
the product name, if it's a product, if that exists anywhere in the message. |
|
|
|
07:07.370 --> 07:14.030 |
|
If that text, like the word Lancaster, is anywhere in the message, then it's simply going to shove |
|
|
|
07:14.030 --> 07:18.920 |
|
that into this list of relevant context and return that list in the end. |
|
|
|
07:18.950 --> 07:21.110 |
|
So let me show you exactly what that means. |
|
|
|
07:21.140 --> 07:33.620 |
|
So if I say get relevant context and I pass in something completely irrelevant, like the cat, we'll |
|
|
|
07:33.620 --> 07:34.670 |
|
get nothing back. |
|
|
|
07:34.700 --> 07:38.870 |
|
The cat is not relevant to the questions about ensure. |
|
|
|
07:38.900 --> 07:46.160 |
|
And if I put who is Avery Lancaster then hooray! |
|
|
|
07:46.190 --> 07:53.690 |
|
We will in fact get a good robust, uh, answer in there, which is exactly what we would expect. |
|
|
|
07:53.840 --> 08:04.310 |
|
If I say, who is Avery Lancaster and what is calm, then we should hopefully let's have a look. |
|
|
|
08:04.370 --> 08:09.030 |
|
See that we'll get back a list of two things that will include. |
|
|
|
08:09.060 --> 08:10.800 |
|
Yes, you can see there's the second one beginning. |
|
|
|
08:10.800 --> 08:12.600 |
|
That's column in there. |
|
|
|
08:12.600 --> 08:17.940 |
|
We get back a list that contains Avery's document and the column document. |
|
|
|
08:18.450 --> 08:21.030 |
|
Um, but of course this is very brittle. |
|
|
|
08:21.030 --> 08:28.650 |
|
If I don't include Avery's last name, I just say Avery and I spell this without a capital C because |
|
|
|
08:28.650 --> 08:32.220 |
|
I haven't even done a case insensitive match there. |
|
|
|
08:32.220 --> 08:34.950 |
|
We're going to discover we get nothing back. |
|
|
|
08:34.950 --> 08:36.360 |
|
This is brittle. |
|
|
|
08:36.360 --> 08:39.780 |
|
It's only going to work in basic cases. |
|
|
|
08:40.620 --> 08:46.950 |
|
All right, so now I just build on this one step further by having a function add context that takes |
|
|
|
08:46.950 --> 08:47.970 |
|
a message. |
|
|
|
08:48.120 --> 08:53.190 |
|
Um, so the idea is that this will simply get any relevant context. |
|
|
|
08:53.190 --> 08:56.370 |
|
And if it finds some it's going to add that into the message. |
|
|
|
08:56.370 --> 09:01.290 |
|
It's going to say the following additional context might be relevant in answering this question. |
|
|
|
09:01.290 --> 09:04.320 |
|
So it's just sort of building on top of whatever it's passed. |
|
|
|
09:04.320 --> 09:07.900 |
|
So to give an example Let's just ask again the question. |
|
|
|
09:07.960 --> 09:12.160 |
|
Um, but yeah, let me start with who is Avery Lancaster, a positive example, and I'll give you a |
|
|
|
09:12.160 --> 09:12.880 |
|
bad example. |
|
|
|
09:12.880 --> 09:17.170 |
|
Who is Avery Lancaster and what, you have to run the cell. |
|
|
|
09:19.270 --> 09:23.410 |
|
Uh, so, uh, who is Avery Lancaster? |
|
|
|
09:23.410 --> 09:24.700 |
|
What do we see? |
|
|
|
09:24.940 --> 09:25.840 |
|
Uh, we see. |
|
|
|
09:25.840 --> 09:26.830 |
|
Here we go. |
|
|
|
09:26.830 --> 09:29.650 |
|
Uh, who is Avery Lancaster? |
|
|
|
09:29.680 --> 09:32.620 |
|
The following additional context might be relevant in answering this question. |
|
|
|
09:32.620 --> 09:34.540 |
|
And then there is the detail. |
|
|
|
09:34.750 --> 09:37.870 |
|
Uh, and then I can give a counter example. |
|
|
|
09:37.870 --> 09:39.340 |
|
Another kind of problem. |
|
|
|
09:39.340 --> 09:44.290 |
|
If I say who is Alex Lancaster, it will still see the word Lancaster in there. |
|
|
|
09:44.290 --> 09:47.530 |
|
And of course, it's still going to say here is relevant context. |
|
|
|
09:47.740 --> 09:50.380 |
|
Um, and it's of course missed the trick there. |
|
|
|
09:50.380 --> 09:53.320 |
|
So it is rough and ready. |
|
|
|
09:53.440 --> 09:56.470 |
|
Um, it really is just doing a string lookup. |
|
|
|
09:56.470 --> 10:01.570 |
|
But that's all we need, of course, to be writing our favorite chat function. |
|
|
|
10:01.570 --> 10:04.510 |
|
You remember, this is the function that Gradio expects. |
|
|
|
10:04.510 --> 10:09.230 |
|
If it's to build a quick user interface, it takes the current message and the history of messages, |
|
|
|
10:09.230 --> 10:12.200 |
|
and we hopefully this is now second nature to you. |
|
|
|
10:12.230 --> 10:20.510 |
|
We convert that history into the format that OpenAI expects, the list of dicts in that particular structure. |
|
|
|
10:20.510 --> 10:23.660 |
|
And then we make this lovely call here. |
|
|
|
10:23.660 --> 10:28.880 |
|
We added extra context to our message, and that's what we send to OpenAI. |
|
|
|
10:28.880 --> 10:36.590 |
|
And then this of course OpenAI chat completions create also second nature to you and we will stream |
|
|
|
10:36.590 --> 10:38.150 |
|
back the response. |
|
|
|
10:38.150 --> 10:40.190 |
|
So let's give this a run. |
|
|
|
10:40.280 --> 10:46.430 |
|
And now we know the last step is the one liner that brings this up in Gradio. |
|
|
|
10:46.760 --> 10:47.510 |
|
Here we go. |
|
|
|
10:47.540 --> 10:49.370 |
|
Let's see how this looks. |
|
|
|
10:54.230 --> 10:56.000 |
|
So let's say, uh. |
|
|
|
10:56.060 --> 10:56.780 |
|
Hi there. |
|
|
|
10:59.510 --> 11:00.050 |
|
Hello. |
|
|
|
11:00.050 --> 11:01.460 |
|
How can I assist you today? |
|
|
|
11:01.490 --> 11:06.410 |
|
So we'll say, who is Avery Lancaster? |
|
|
|
11:08.620 --> 11:09.190 |
|
That's right. |
|
|
|
11:09.190 --> 11:09.460 |
|
Yeah. |
|
|
|
11:09.490 --> 11:10.570 |
|
Lancaster. |
|
|
|
11:11.950 --> 11:21.670 |
|
And of course, we get back a nice good answer because we know full well that the Avery Lancaster information |
|
|
|
11:21.670 --> 11:24.940 |
|
was shoved into the prompt and sent to OpenAI. |
|
|
|
11:25.300 --> 11:30.040 |
|
Um, and now we can also say what is calm. |
|
|
|
11:32.350 --> 11:34.870 |
|
And innovative auto insurance product. |
|
|
|
11:34.900 --> 11:35.770 |
|
And there we go. |
|
|
|
11:35.770 --> 11:40.660 |
|
It's got a whole ton of information summarized from that big chunk of context that will have centered |
|
|
|
11:40.660 --> 11:41.830 |
|
about the product. |
|
|
|
11:41.950 --> 11:45.370 |
|
And it even mentions the roadmap for continuous improvement. |
|
|
|
11:45.400 --> 11:46.390 |
|
Lovely. |
|
|
|
11:46.990 --> 11:50.110 |
|
Uh, so you can see it's rag. |
|
|
|
11:50.110 --> 11:54.130 |
|
It's simplistic, but, you know, this is what it's all about. |
|
|
|
11:54.220 --> 11:56.440 |
|
Let's just break it for a second. |
|
|
|
11:56.440 --> 11:57.760 |
|
You can imagine what I'm going to do. |
|
|
|
11:57.790 --> 12:02.380 |
|
I'm just going to say, uh, who is Avery? |
|
|
|
12:02.410 --> 12:09.170 |
|
Well, it may it actually has the the conversation from the past, so we should probably start this |
|
|
|
12:09.170 --> 12:09.950 |
|
again. |
|
|
|
12:10.040 --> 12:11.570 |
|
Bring up a fresh chat. |
|
|
|
12:11.600 --> 12:15.470 |
|
Otherwise it will know who Avery it is because we just asked it before and it got the context. |
|
|
|
12:15.470 --> 12:22.250 |
|
So let's bring up a fresh new user interface and try it from scratch and not give it that benefit and |
|
|
|
12:22.250 --> 12:22.880 |
|
just ask it. |
|
|
|
12:22.910 --> 12:24.320 |
|
Who is Avery? |
|
|
|
12:25.760 --> 12:30.170 |
|
I'm sorry, but I don't have any information about Avery in relation to ensure. |
|
|
|
12:30.470 --> 12:32.540 |
|
So you can see that it's brittle. |
|
|
|
12:32.540 --> 12:34.430 |
|
It has to take the last name. |
|
|
|
12:34.430 --> 12:46.040 |
|
And even if I spell the last name wrong, who is Lancaster without, uh, a capital L, it's also going |
|
|
|
12:46.040 --> 12:48.200 |
|
to break, so you get the idea. |
|
|
|
12:48.200 --> 12:51.620 |
|
This is an effective way to add supplemental information into the prompt. |
|
|
|
12:51.650 --> 12:57.140 |
|
It gets back more accurate answers, but it is brittle and it requires exact text matching. |
|
|
|
12:57.140 --> 13:01.550 |
|
Not particularly flexible, not very scalable solution. |
|
|
|
13:01.670 --> 13:03.140 |
|
Uh, we can do better. |
|
|
|
13:03.140 --> 13:06.710 |
|
And that is what we are going to be doing in the in the next day. |
|
|
|
13:06.710 --> 13:09.350 |
|
But now back to the slides.
|
|
|