From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
394 lines
10 KiB
394 lines
10 KiB
WEBVTT |
|
|
|
00:01.940 --> 00:03.290 |
|
Welcome back. |
|
|
|
00:03.320 --> 00:09.020 |
|
So hopefully you are still impressed by the GPT four mini results. |
|
|
|
00:09.260 --> 00:19.820 |
|
The frontier model coming in strong at just shy of $80 and with 52% hits just going above the 50% mark |
|
|
|
00:19.820 --> 00:22.820 |
|
in terms of the number of green dots and a nice chart there. |
|
|
|
00:22.880 --> 00:29.510 |
|
But the time has come for us to now run against its bigger cousin, big GPT four. |
|
|
|
00:29.540 --> 00:31.250 |
|
Oh, I just did. |
|
|
|
00:31.280 --> 00:37.520 |
|
I did want to mention that, uh, do be aware that at this point, uh, when we're talking about these |
|
|
|
00:37.520 --> 00:42.020 |
|
kinds of error numbers, the error is already, uh, reasonably low. |
|
|
|
00:42.050 --> 00:44.810 |
|
You think about the actual prices of products. |
|
|
|
00:44.810 --> 00:45.770 |
|
They do vary. |
|
|
|
00:45.770 --> 00:48.020 |
|
There is volatility baked into prices. |
|
|
|
00:48.020 --> 00:49.310 |
|
There are sales. |
|
|
|
00:49.340 --> 00:54.170 |
|
There's there's big swings when you come to things like the prices of a wheel or the price of a laptop. |
|
|
|
00:54.320 --> 01:02.320 |
|
No doubt they vary quite considerably from different sites and different, uh, Stores, so there is |
|
|
|
01:02.320 --> 01:05.380 |
|
some natural error built into this question anyway. |
|
|
|
01:05.380 --> 01:09.370 |
|
So we're certainly not expecting to to come down to small numbers. |
|
|
|
01:09.730 --> 01:16.090 |
|
So from that point of view, recognize that every dollar improvement we get at this point is a is a |
|
|
|
01:16.090 --> 01:17.350 |
|
victory in its own right. |
|
|
|
01:17.380 --> 01:21.670 |
|
We're going to be squeezing juice if we can, but don't expect a big change now. |
|
|
|
01:21.670 --> 01:29.110 |
|
We are now at a point where this is already super impressive and far ahead of any of the traditional |
|
|
|
01:29.110 --> 01:30.910 |
|
machine learning techniques. |
|
|
|
01:31.390 --> 01:41.290 |
|
Um, so, uh, the next thing to do, of course, is to unveil the bigger cousin, GPT four. |
|
|
|
01:41.290 --> 01:47.740 |
|
Oh, here is the same type of function again, the function that we'll be passing to our test program. |
|
|
|
01:47.890 --> 01:51.790 |
|
Uh, it's of course calling OpenAI ChatGPT completions dot create. |
|
|
|
01:51.790 --> 01:58.690 |
|
We're using the GPT four model as of August the 6th, which is very recent. |
|
|
|
01:58.960 --> 02:06.080 |
|
Um, passing in messages, passing in the seed and using the same way of getting back the results. |
|
|
|
02:06.110 --> 02:09.590 |
|
It's worth mentioning this is now going to cost a little bit more. |
|
|
|
02:09.620 --> 02:14.210 |
|
It costs somewhere between $0.10 and $0.20 for me to run this across 250 points. |
|
|
|
02:14.450 --> 02:19.160 |
|
Um, so, um, yeah, I mean, it's not going to break the bank, but you might want to think twice |
|
|
|
02:19.160 --> 02:21.200 |
|
before running this a hundred times. |
|
|
|
02:21.380 --> 02:24.650 |
|
Uh, so I do urge you to at least try it once. |
|
|
|
02:24.740 --> 02:26.330 |
|
It's it's great fun. |
|
|
|
02:26.510 --> 02:30.170 |
|
Um, but, yeah, it doesn't it doesn't necessarily need to be run a lot of times. |
|
|
|
02:30.200 --> 02:35.390 |
|
I also want to mention that you might wonder why I'm not trying to run this against oh one. |
|
|
|
02:35.450 --> 02:39.530 |
|
Uh, so the kind of problems are that oh one is designed for. |
|
|
|
02:39.560 --> 02:45.560 |
|
Ah, more the problems where there is deeper thinking required, a multi-step thought process and challenging. |
|
|
|
02:45.560 --> 02:49.100 |
|
And that doesn't really lend itself to the problem that we're tackling here. |
|
|
|
02:49.160 --> 02:53.090 |
|
Um, it would also be slower and more expensive and for no good reason. |
|
|
|
02:53.090 --> 02:56.270 |
|
We're not trying to solve, uh, math puzzles. |
|
|
|
02:56.300 --> 03:02.180 |
|
We're just trying to have the most likely next guess of a price based on worldly knowledge. |
|
|
|
03:02.330 --> 03:05.750 |
|
So GPT four frontier should be our best shot. |
|
|
|
03:05.870 --> 03:06.110 |
|
Um. |
|
|
|
03:06.110 --> 03:08.570 |
|
And I have, of course, already run it. |
|
|
|
03:08.660 --> 03:12.170 |
|
Uh, so it's a little bit slower than GPT four mini. |
|
|
|
03:12.170 --> 03:13.070 |
|
Just a little bit. |
|
|
|
03:13.100 --> 03:15.740 |
|
But I think you should run it yourself and you'll enjoy it. |
|
|
|
03:15.770 --> 03:18.470 |
|
And here are the results I get. |
|
|
|
03:18.740 --> 03:23.660 |
|
Um, so there's a lot of greens, but there's lots of reds as well. |
|
|
|
03:24.230 --> 03:27.830 |
|
So it's quite a variety here. |
|
|
|
03:28.700 --> 03:33.110 |
|
Um, no doubt you're betting. |
|
|
|
03:33.110 --> 03:35.000 |
|
And here are the results. |
|
|
|
03:35.000 --> 03:37.610 |
|
So the bottom line is it is better. |
|
|
|
03:37.610 --> 03:39.380 |
|
It is improved it for sure. |
|
|
|
03:39.380 --> 03:42.140 |
|
It's gone from about 80 to about 76. |
|
|
|
03:42.140 --> 03:44.840 |
|
And the hits from 52 to 58. |
|
|
|
03:44.840 --> 03:45.950 |
|
Uh, green dots. |
|
|
|
03:45.950 --> 03:47.990 |
|
It's not a massive change. |
|
|
|
03:47.990 --> 03:53.840 |
|
You may have expected more going from mini to GPT four, but remember GPT four mini is really good. |
|
|
|
03:53.840 --> 04:00.840 |
|
And despite the fact that it's a much cheaper model, it has much the very similar smarts to GPT four |
|
|
|
04:00.870 --> 04:01.380 |
|
zero. |
|
|
|
04:01.620 --> 04:08.310 |
|
Um, so I would say, first of all, it is to be celebrated that we've come down by by $6 by $4. |
|
|
|
04:08.340 --> 04:08.790 |
|
Sorry. |
|
|
|
04:08.910 --> 04:16.770 |
|
Uh, but but also that, um, we can already assume that GPT four mini has done very well indeed. |
|
|
|
04:17.040 --> 04:19.050 |
|
Um, so, um. |
|
|
|
04:19.200 --> 04:22.740 |
|
Yeah, this is this is, uh, this is the results. |
|
|
|
04:22.770 --> 04:25.170 |
|
It looks visually very impressive. |
|
|
|
04:25.200 --> 04:28.410 |
|
The the I mean, compare it to humanity. |
|
|
|
04:28.440 --> 04:33.180 |
|
Uh, at the end of this lecture, I think we'll end by going back and embarrassingly looking at the |
|
|
|
04:33.480 --> 04:37.980 |
|
chart one more time, uh, comparing it to what frontier models are able to achieve. |
|
|
|
04:38.040 --> 04:41.370 |
|
Uh, but this does give you a good sense. |
|
|
|
04:41.880 --> 04:42.990 |
|
Okay. |
|
|
|
04:42.990 --> 04:50.670 |
|
And then, of course, we do have one more frontier model that perhaps we should unveil, which is Claude |
|
|
|
04:50.700 --> 04:52.920 |
|
3.5 sonnet. |
|
|
|
04:53.220 --> 05:03.760 |
|
Um, which is the strongest, uh, model from anthropic And, um, you recognize, I hope the structure. |
|
|
|
05:04.000 --> 05:07.180 |
|
Uh, Claude messages create uh, again. |
|
|
|
05:07.180 --> 05:11.500 |
|
Now, with in the case of Claude, you can't pass in a seed. |
|
|
|
05:11.530 --> 05:12.430 |
|
It doesn't take that. |
|
|
|
05:12.430 --> 05:14.500 |
|
So the results will be different each time. |
|
|
|
05:14.500 --> 05:20.440 |
|
But I've just gone with the first run I've done in here, and that's what we'll have as, as the, uh, |
|
|
|
05:20.440 --> 05:24.850 |
|
the, uh, the results for, for our purposes. |
|
|
|
05:24.880 --> 05:28.990 |
|
And I guess I'll probably, uh, replicate and save this notebook. |
|
|
|
05:28.990 --> 05:30.970 |
|
So we have it for, for posterity. |
|
|
|
05:31.270 --> 05:37.780 |
|
Uh, so time for Claude starts with a couple of reds, then a sea of greens. |
|
|
|
05:40.510 --> 05:45.850 |
|
And I will get us straight to some outcomes. |
|
|
|
05:48.160 --> 05:50.260 |
|
So here's the results from Claude. |
|
|
|
05:50.290 --> 05:53.440 |
|
Interestingly, Claude actually does a little bit worse. |
|
|
|
05:53.560 --> 06:03.060 |
|
Uh, it's, uh, um, uh, worse, in fact, than GPT four mini in this task, and worse also than the |
|
|
|
06:03.060 --> 06:03.990 |
|
Random forest. |
|
|
|
06:04.020 --> 06:09.300 |
|
Of course, again, Random Forest had the benefit of all of its training data 400,000 training data. |
|
|
|
06:09.300 --> 06:12.750 |
|
So it's a very different kind of of of test. |
|
|
|
06:13.200 --> 06:20.520 |
|
And you can see one of the reasons for that is that you can tell from this axis that it has made some |
|
|
|
06:20.520 --> 06:21.180 |
|
prediction. |
|
|
|
06:21.180 --> 06:27.000 |
|
We can't quite see it because it's going to be a little tiny red dot, but it made some massively outsized |
|
|
|
06:27.000 --> 06:30.690 |
|
prediction, although that will of course been spread out across everything. |
|
|
|
06:30.690 --> 06:32.730 |
|
So it wouldn't have made a huge difference. |
|
|
|
06:32.730 --> 06:41.670 |
|
But still, uh, one guess would have been like way, way too high and will have certainly, uh, certainly |
|
|
|
06:41.670 --> 06:43.320 |
|
made its results be a bit distorted. |
|
|
|
06:43.350 --> 06:47.820 |
|
Let's see if we can spy that from scanning through here. |
|
|
|
06:48.000 --> 06:49.950 |
|
Where did it go so wrong? |
|
|
|
06:49.950 --> 06:51.060 |
|
Here it is. |
|
|
|
06:53.550 --> 06:56.940 |
|
G technology G speed ES pro high performance. |
|
|
|
06:56.940 --> 07:02.780 |
|
So this thing which actually cost $495.95. |
|
|
|
07:02.780 --> 07:07.250 |
|
It guest was $4,999 and 99. |
|
|
|
07:07.550 --> 07:14.510 |
|
So it was a very outsized guest there that introduced 4500 error. |
|
|
|
07:14.540 --> 07:18.590 |
|
Now, obviously, that that error in itself won't distort things too much because it gets averaged out |
|
|
|
07:18.590 --> 07:21.020 |
|
across the 250 data points. |
|
|
|
07:21.020 --> 07:26.570 |
|
But it's an example of one of the things that pushed it into the worst camp. |
|
|
|
07:27.110 --> 07:30.170 |
|
So those are the results from Claude. |
|
|
|
07:30.200 --> 07:38.060 |
|
Uh, still has a nice diagram here, but didn't quite measure up in this with this particular challenge, |
|
|
|
07:38.060 --> 07:43.790 |
|
uh, to GPT four or GPT four at the frontier was the winner of this challenge. |
|
|
|
07:43.820 --> 07:50.150 |
|
You'll remember Claude won when it came to the coding performance challenge, by a way, a big, big |
|
|
|
07:50.150 --> 07:50.900 |
|
margin. |
|
|
|
07:50.960 --> 07:56.150 |
|
Uh, but in this case, for this particular challenge, our winner is Claude. |
|
|
|
07:56.480 --> 07:59.730 |
|
Uh, is sorry, is GPT four at the frontier. |
|
|
|
07:59.730 --> 08:04.620 |
|
And as I promised, I will leave you by one more look at the. |
|
|
|
08:04.650 --> 08:07.200 |
|
At this human's feeble attempt. |
|
|
|
08:07.230 --> 08:14.820 |
|
Take a look at the nice, elegant diagram here with a number of green dots with GPT four guessing close |
|
|
|
08:14.820 --> 08:16.740 |
|
to the to a home run. |
|
|
|
08:16.740 --> 08:22.620 |
|
And now compare that with the performance of the resident human in the room. |
|
|
|
08:22.890 --> 08:26.580 |
|
Uh, and this was my chart. |
|
|
|
08:28.800 --> 08:30.600 |
|
That I don't want to look at that ever again. |
|
|
|
08:30.840 --> 08:31.860 |
|
Uh, all right. |
|
|
|
08:31.860 --> 08:36.570 |
|
Anyway, I hope that this was a, uh, educational exercise for you. |
|
|
|
08:36.570 --> 08:40.680 |
|
And you see how you would do, uh, experiments like this with your own business problems. |
|
|
|
08:40.680 --> 08:41.430 |
|
Please do. |
|
|
|
08:41.460 --> 08:43.200 |
|
Now, come replicate this. |
|
|
|
08:43.200 --> 08:46.890 |
|
Play around with this, uh, you'll find, particularly with Claude, of course, that you won't get |
|
|
|
08:46.890 --> 08:48.750 |
|
the same results because there's no seed. |
|
|
|
08:48.930 --> 08:53.430 |
|
Um, but you will get some something different and see how it performs. |
|
|
|
08:53.430 --> 08:55.050 |
|
Maybe you'll get better results than me. |
|
|
|
08:55.230 --> 09:00.060 |
|
Uh, but I will see you to wrap up with some more slides.
|
|
|