From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
325 lines
9.1 KiB
325 lines
9.1 KiB
WEBVTT |
|
|
|
00:00.800 --> 00:06.560 |
|
So last time we looked at a humble linear regression model with feature engineering, and now we say |
|
|
|
00:06.560 --> 00:12.080 |
|
goodbye to features and we start working with natural language processing. |
|
|
|
00:12.080 --> 00:19.490 |
|
And we're going to begin with a rather simplistic type of NLP, which is called Bag of words, which |
|
|
|
00:19.490 --> 00:21.110 |
|
I briefly mentioned before. |
|
|
|
00:21.140 --> 00:24.950 |
|
So we're going to start by putting making these two very useful things. |
|
|
|
00:24.950 --> 00:33.920 |
|
One is called prices and one is called documents and prices is just the list of all of the prices in |
|
|
|
00:33.920 --> 00:42.020 |
|
our training data set and documents is all of the test prompts from our training data set. |
|
|
|
00:42.230 --> 00:47.480 |
|
Just a bit of text, and what we're going to try and do now is build models which use the text in our |
|
|
|
00:47.480 --> 00:51.440 |
|
documents, rather than using features that we have engineered ourselves. |
|
|
|
00:51.440 --> 00:52.940 |
|
Just use the text. |
|
|
|
00:53.330 --> 01:00.660 |
|
Um, and yeah, one one little trick you might notice here is that I use the test prompt, not the training |
|
|
|
01:00.660 --> 01:01.440 |
|
prompt. |
|
|
|
01:01.590 --> 01:06.000 |
|
If I use the training prompt, it would include the price in the training prompt. |
|
|
|
01:06.120 --> 01:12.210 |
|
And that wouldn't work because then of course the model would just learn to spot the price itself is |
|
|
|
01:12.210 --> 01:13.470 |
|
actually in the prompt. |
|
|
|
01:13.470 --> 01:16.470 |
|
And then when it got to test time, it would fail rather miserably. |
|
|
|
01:16.470 --> 01:20.220 |
|
So that's a little trick to watch out for. |
|
|
|
01:20.460 --> 01:22.980 |
|
Um, a trap I'm more likely to watch out for. |
|
|
|
01:22.980 --> 01:25.710 |
|
So we prepare that. |
|
|
|
01:25.710 --> 01:31.770 |
|
We're now going to use something called a countvectorizer, which sounds super fancy. |
|
|
|
01:31.800 --> 01:36.330 |
|
At any time that you think you're going to be working today with a count of Vectorizer, sounds like |
|
|
|
01:36.330 --> 01:41.850 |
|
you're going to be building something that's highly sophisticated, but it is, alas, not sophisticated. |
|
|
|
01:41.940 --> 01:49.740 |
|
What it's going to do is simply look for the counting, the number of words, and then build a vector |
|
|
|
01:49.740 --> 01:57.740 |
|
where each location in the vector represents a particular word, and then how many times that word appears |
|
|
|
01:57.740 --> 01:59.420 |
|
in your document. |
|
|
|
01:59.540 --> 02:06.110 |
|
So each document will be a vector, and every row in that vector represents a word in your vocabulary. |
|
|
|
02:06.110 --> 02:09.170 |
|
And the counter counts how many times it's there. |
|
|
|
02:09.170 --> 02:15.050 |
|
And that whole way of thinking about things is known as a bag of words model, because it's like you |
|
|
|
02:15.050 --> 02:19.880 |
|
have a bag of words that you are counting up, and it doesn't matter what order the words appear in, |
|
|
|
02:19.880 --> 02:23.840 |
|
it's just the fact that they are there a certain number of times. |
|
|
|
02:24.170 --> 02:31.130 |
|
Uh, and so we're doing that we're going to count up to the 1000 most common or most important of the |
|
|
|
02:31.130 --> 02:31.700 |
|
words. |
|
|
|
02:31.700 --> 02:41.150 |
|
And we're going to use this parameter here to make sure that it only, uh, it, it removes, uh, common |
|
|
|
02:41.150 --> 02:47.270 |
|
stop words, which is what people call things like and and the and in and it that are going to be not |
|
|
|
02:47.270 --> 02:49.820 |
|
useful for the model and only a distraction. |
|
|
|
02:49.820 --> 02:54.320 |
|
And so they will get plucked out leaving us with juicy words behind. |
|
|
|
02:54.320 --> 03:00.000 |
|
And um, what we're then going to do is, uh, and now you can see we do things much quicker. |
|
|
|
03:00.000 --> 03:05.340 |
|
We're going to, uh, we're going to create our data set based on these documents. |
|
|
|
03:05.340 --> 03:09.240 |
|
We're going to create a linear regression model again just as before. |
|
|
|
03:09.270 --> 03:14.610 |
|
And we're going to fit our bag of words, um, to our prices. |
|
|
|
03:14.610 --> 03:21.090 |
|
In other words, we're instead of using features this time we're replacing our features with this bag |
|
|
|
03:21.090 --> 03:29.130 |
|
of words, this vector counting the number of words, um, for each word, each of our 1000 words in |
|
|
|
03:29.130 --> 03:32.850 |
|
our most common dictionary vocab. |
|
|
|
03:32.880 --> 03:34.650 |
|
So that's what we're going to do. |
|
|
|
03:34.680 --> 03:36.480 |
|
We're going to run that linear regression. |
|
|
|
03:36.480 --> 03:37.980 |
|
It's happening right now. |
|
|
|
03:37.980 --> 03:46.110 |
|
It's counting up the uh, the, the 1000, uh, vector points across all of our data set. |
|
|
|
03:46.440 --> 03:53.250 |
|
Um, and then once it's done that, we're going to put that into a bag of words, linear regression, |
|
|
|
03:53.250 --> 03:54.180 |
|
pricer. |
|
|
|
03:54.180 --> 03:58.130 |
|
and we are then going to test it, so we will test it again. |
|
|
|
03:58.160 --> 03:59.660 |
|
It's just completed running. |
|
|
|
03:59.690 --> 04:07.280 |
|
Um, so this is our function, our simple function that will try it out and we will test it with tester |
|
|
|
04:07.310 --> 04:08.570 |
|
dot test. |
|
|
|
04:09.320 --> 04:10.790 |
|
Bag of words. |
|
|
|
04:10.820 --> 04:13.520 |
|
Linear regression Pricer. |
|
|
|
04:16.010 --> 04:17.630 |
|
Let's see how it does. |
|
|
|
04:19.550 --> 04:21.050 |
|
Lots of greens there I see. |
|
|
|
04:21.050 --> 04:22.730 |
|
But also lots of reds. |
|
|
|
04:23.300 --> 04:23.960 |
|
You can see. |
|
|
|
04:23.990 --> 04:25.340 |
|
Let's just pluck one out. |
|
|
|
04:25.370 --> 04:27.140 |
|
It guest $74. |
|
|
|
04:27.140 --> 04:29.030 |
|
And it was in fact $46. |
|
|
|
04:29.030 --> 04:31.640 |
|
So you can see that it's got things that are right. |
|
|
|
04:31.640 --> 04:33.530 |
|
It's also got things that are way off. |
|
|
|
04:33.560 --> 04:35.540 |
|
How do you think the graph is going to look. |
|
|
|
04:35.570 --> 04:36.830 |
|
Let's see. |
|
|
|
04:37.430 --> 04:39.350 |
|
Okay okay. |
|
|
|
04:39.350 --> 04:42.680 |
|
Well it's looking more like there's some something good happening. |
|
|
|
04:42.680 --> 04:46.130 |
|
We're starting to see things converging around the line. |
|
|
|
04:46.130 --> 04:48.080 |
|
We're seeing a lot more green dots. |
|
|
|
04:48.080 --> 04:51.140 |
|
The average is $113. |
|
|
|
04:51.140 --> 04:59.310 |
|
So distinctly better than the linear regression with features and distinctly better than guessing. |
|
|
|
04:59.670 --> 05:01.980 |
|
Uh, so, uh, progress happening. |
|
|
|
05:01.980 --> 05:05.760 |
|
There's still some weird outliers here, some some problems. |
|
|
|
05:06.000 --> 05:10.860 |
|
Um, but, uh, that this was the true value and it should have been right up there somewhere. |
|
|
|
05:11.160 --> 05:15.810 |
|
Um, but there you can see that there is a progress being made. |
|
|
|
05:16.200 --> 05:24.540 |
|
So next, the the last of the of this set before we go into more advanced models is going to be using |
|
|
|
05:24.540 --> 05:31.560 |
|
the amazing Gensim library to, uh, introduce this word to vec model. |
|
|
|
05:31.560 --> 05:37.050 |
|
That was uh, it was one of the first times that I really encountered, uh, neural networks with NLP, |
|
|
|
05:37.410 --> 05:45.060 |
|
uh, using a vectorization model, um, with, with something that is a more sophisticated, uh, vector |
|
|
|
05:45.060 --> 05:46.440 |
|
embedding model. |
|
|
|
05:46.500 --> 05:54.950 |
|
Um, and we are going to use this word two vec function class from Gensim, and we are going to build |
|
|
|
05:54.950 --> 05:57.980 |
|
vectors with 400 dimensions. |
|
|
|
05:58.250 --> 06:02.960 |
|
And it's I've set it to use eight workers, which means it really hammers my box. |
|
|
|
06:02.960 --> 06:07.940 |
|
This still took several minutes to run, and I ran it in advance so we wouldn't have to wait for for |
|
|
|
06:07.940 --> 06:08.840 |
|
all of this. |
|
|
|
06:08.870 --> 06:19.610 |
|
Uh, and I also then ran this, and that should mean that hopefully we are ready just to run this and |
|
|
|
06:19.610 --> 06:24.410 |
|
to immediately see the results tester dot test. |
|
|
|
06:25.160 --> 06:33.200 |
|
We should be able to pass in the word two vec linear regression pricer and see how this fancy vectorization |
|
|
|
06:33.200 --> 06:36.140 |
|
model performs with linear regression. |
|
|
|
06:37.070 --> 06:39.950 |
|
Well, it looks good so far from the first two lines, but oh no. |
|
|
|
06:39.980 --> 06:46.130 |
|
Then there's some red, some green, some red, some greens, lots of reds that scoot down past the |
|
|
|
06:46.130 --> 06:49.290 |
|
250 test data points to the chart. |
|
|
|
06:50.100 --> 06:52.080 |
|
So here we have it. |
|
|
|
06:52.110 --> 06:54.150 |
|
It looks quite decent again. |
|
|
|
06:54.180 --> 07:01.980 |
|
Interestingly, the bad news is it's actually a hair worse than linear regression based on the simple |
|
|
|
07:02.010 --> 07:03.540 |
|
bag of words model. |
|
|
|
07:03.540 --> 07:11.850 |
|
So unveiling the lovely word two vec vector hasn't yet particularly helped us. |
|
|
|
07:11.850 --> 07:17.850 |
|
We're still much the same territory as before, and that may be because the linear regression model |
|
|
|
07:17.850 --> 07:23.730 |
|
isn't powerful enough to take advantage of all of the extra information that we have in these word two |
|
|
|
07:23.730 --> 07:25.140 |
|
vec vectors. |
|
|
|
07:25.410 --> 07:31.800 |
|
So in the next time we're going to explore some slightly more sophisticated models, and then we'll |
|
|
|
07:31.800 --> 07:34.890 |
|
be done with our with our traditional machine learning. |
|
|
|
07:34.890 --> 07:38.820 |
|
So just hang on in there for a bit longer, because I want to dig a little bit more juice out of this. |
|
|
|
07:38.850 --> 07:43.920 |
|
We want to get a bit better with our baseline models, because we don't want our LMS to have an easy |
|
|
|
07:43.920 --> 07:44.730 |
|
run at this at all. |
|
|
|
07:44.730 --> 07:46.350 |
|
We want to we want to put up a fight. |
|
|
|
07:46.350 --> 07:48.330 |
|
So see you next time.
|
|
|