You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

394 lines
10 KiB

WEBVTT
00:01.940 --> 00:03.290
Welcome back.
00:03.320 --> 00:09.020
So hopefully you are still impressed by the GPT four mini results.
00:09.260 --> 00:19.820
The frontier model coming in strong at just shy of $80 and with 52% hits just going above the 50% mark
00:19.820 --> 00:22.820
in terms of the number of green dots and a nice chart there.
00:22.880 --> 00:29.510
But the time has come for us to now run against its bigger cousin, big GPT four.
00:29.540 --> 00:31.250
Oh, I just did.
00:31.280 --> 00:37.520
I did want to mention that, uh, do be aware that at this point, uh, when we're talking about these
00:37.520 --> 00:42.020
kinds of error numbers, the error is already, uh, reasonably low.
00:42.050 --> 00:44.810
You think about the actual prices of products.
00:44.810 --> 00:45.770
They do vary.
00:45.770 --> 00:48.020
There is volatility baked into prices.
00:48.020 --> 00:49.310
There are sales.
00:49.340 --> 00:54.170
There's there's big swings when you come to things like the prices of a wheel or the price of a laptop.
00:54.320 --> 01:02.320
No doubt they vary quite considerably from different sites and different, uh, Stores, so there is
01:02.320 --> 01:05.380
some natural error built into this question anyway.
01:05.380 --> 01:09.370
So we're certainly not expecting to to come down to small numbers.
01:09.730 --> 01:16.090
So from that point of view, recognize that every dollar improvement we get at this point is a is a
01:16.090 --> 01:17.350
victory in its own right.
01:17.380 --> 01:21.670
We're going to be squeezing juice if we can, but don't expect a big change now.
01:21.670 --> 01:29.110
We are now at a point where this is already super impressive and far ahead of any of the traditional
01:29.110 --> 01:30.910
machine learning techniques.
01:31.390 --> 01:41.290
Um, so, uh, the next thing to do, of course, is to unveil the bigger cousin, GPT four.
01:41.290 --> 01:47.740
Oh, here is the same type of function again, the function that we'll be passing to our test program.
01:47.890 --> 01:51.790
Uh, it's of course calling OpenAI ChatGPT completions dot create.
01:51.790 --> 01:58.690
We're using the GPT four model as of August the 6th, which is very recent.
01:58.960 --> 02:06.080
Um, passing in messages, passing in the seed and using the same way of getting back the results.
02:06.110 --> 02:09.590
It's worth mentioning this is now going to cost a little bit more.
02:09.620 --> 02:14.210
It costs somewhere between $0.10 and $0.20 for me to run this across 250 points.
02:14.450 --> 02:19.160
Um, so, um, yeah, I mean, it's not going to break the bank, but you might want to think twice
02:19.160 --> 02:21.200
before running this a hundred times.
02:21.380 --> 02:24.650
Uh, so I do urge you to at least try it once.
02:24.740 --> 02:26.330
It's it's great fun.
02:26.510 --> 02:30.170
Um, but, yeah, it doesn't it doesn't necessarily need to be run a lot of times.
02:30.200 --> 02:35.390
I also want to mention that you might wonder why I'm not trying to run this against oh one.
02:35.450 --> 02:39.530
Uh, so the kind of problems are that oh one is designed for.
02:39.560 --> 02:45.560
Ah, more the problems where there is deeper thinking required, a multi-step thought process and challenging.
02:45.560 --> 02:49.100
And that doesn't really lend itself to the problem that we're tackling here.
02:49.160 --> 02:53.090
Um, it would also be slower and more expensive and for no good reason.
02:53.090 --> 02:56.270
We're not trying to solve, uh, math puzzles.
02:56.300 --> 03:02.180
We're just trying to have the most likely next guess of a price based on worldly knowledge.
03:02.330 --> 03:05.750
So GPT four frontier should be our best shot.
03:05.870 --> 03:06.110
Um.
03:06.110 --> 03:08.570
And I have, of course, already run it.
03:08.660 --> 03:12.170
Uh, so it's a little bit slower than GPT four mini.
03:12.170 --> 03:13.070
Just a little bit.
03:13.100 --> 03:15.740
But I think you should run it yourself and you'll enjoy it.
03:15.770 --> 03:18.470
And here are the results I get.
03:18.740 --> 03:23.660
Um, so there's a lot of greens, but there's lots of reds as well.
03:24.230 --> 03:27.830
So it's quite a variety here.
03:28.700 --> 03:33.110
Um, no doubt you're betting.
03:33.110 --> 03:35.000
And here are the results.
03:35.000 --> 03:37.610
So the bottom line is it is better.
03:37.610 --> 03:39.380
It is improved it for sure.
03:39.380 --> 03:42.140
It's gone from about 80 to about 76.
03:42.140 --> 03:44.840
And the hits from 52 to 58.
03:44.840 --> 03:45.950
Uh, green dots.
03:45.950 --> 03:47.990
It's not a massive change.
03:47.990 --> 03:53.840
You may have expected more going from mini to GPT four, but remember GPT four mini is really good.
03:53.840 --> 04:00.840
And despite the fact that it's a much cheaper model, it has much the very similar smarts to GPT four
04:00.870 --> 04:01.380
zero.
04:01.620 --> 04:08.310
Um, so I would say, first of all, it is to be celebrated that we've come down by by $6 by $4.
04:08.340 --> 04:08.790
Sorry.
04:08.910 --> 04:16.770
Uh, but but also that, um, we can already assume that GPT four mini has done very well indeed.
04:17.040 --> 04:19.050
Um, so, um.
04:19.200 --> 04:22.740
Yeah, this is this is, uh, this is the results.
04:22.770 --> 04:25.170
It looks visually very impressive.
04:25.200 --> 04:28.410
The the I mean, compare it to humanity.
04:28.440 --> 04:33.180
Uh, at the end of this lecture, I think we'll end by going back and embarrassingly looking at the
04:33.480 --> 04:37.980
chart one more time, uh, comparing it to what frontier models are able to achieve.
04:38.040 --> 04:41.370
Uh, but this does give you a good sense.
04:41.880 --> 04:42.990
Okay.
04:42.990 --> 04:50.670
And then, of course, we do have one more frontier model that perhaps we should unveil, which is Claude
04:50.700 --> 04:52.920
3.5 sonnet.
04:53.220 --> 05:03.760
Um, which is the strongest, uh, model from anthropic And, um, you recognize, I hope the structure.
05:04.000 --> 05:07.180
Uh, Claude messages create uh, again.
05:07.180 --> 05:11.500
Now, with in the case of Claude, you can't pass in a seed.
05:11.530 --> 05:12.430
It doesn't take that.
05:12.430 --> 05:14.500
So the results will be different each time.
05:14.500 --> 05:20.440
But I've just gone with the first run I've done in here, and that's what we'll have as, as the, uh,
05:20.440 --> 05:24.850
the, uh, the results for, for our purposes.
05:24.880 --> 05:28.990
And I guess I'll probably, uh, replicate and save this notebook.
05:28.990 --> 05:30.970
So we have it for, for posterity.
05:31.270 --> 05:37.780
Uh, so time for Claude starts with a couple of reds, then a sea of greens.
05:40.510 --> 05:45.850
And I will get us straight to some outcomes.
05:48.160 --> 05:50.260
So here's the results from Claude.
05:50.290 --> 05:53.440
Interestingly, Claude actually does a little bit worse.
05:53.560 --> 06:03.060
Uh, it's, uh, um, uh, worse, in fact, than GPT four mini in this task, and worse also than the
06:03.060 --> 06:03.990
Random forest.
06:04.020 --> 06:09.300
Of course, again, Random Forest had the benefit of all of its training data 400,000 training data.
06:09.300 --> 06:12.750
So it's a very different kind of of of test.
06:13.200 --> 06:20.520
And you can see one of the reasons for that is that you can tell from this axis that it has made some
06:20.520 --> 06:21.180
prediction.
06:21.180 --> 06:27.000
We can't quite see it because it's going to be a little tiny red dot, but it made some massively outsized
06:27.000 --> 06:30.690
prediction, although that will of course been spread out across everything.
06:30.690 --> 06:32.730
So it wouldn't have made a huge difference.
06:32.730 --> 06:41.670
But still, uh, one guess would have been like way, way too high and will have certainly, uh, certainly
06:41.670 --> 06:43.320
made its results be a bit distorted.
06:43.350 --> 06:47.820
Let's see if we can spy that from scanning through here.
06:48.000 --> 06:49.950
Where did it go so wrong?
06:49.950 --> 06:51.060
Here it is.
06:53.550 --> 06:56.940
G technology G speed ES pro high performance.
06:56.940 --> 07:02.780
So this thing which actually cost $495.95.
07:02.780 --> 07:07.250
It guest was $4,999 and 99.
07:07.550 --> 07:14.510
So it was a very outsized guest there that introduced 4500 error.
07:14.540 --> 07:18.590
Now, obviously, that that error in itself won't distort things too much because it gets averaged out
07:18.590 --> 07:21.020
across the 250 data points.
07:21.020 --> 07:26.570
But it's an example of one of the things that pushed it into the worst camp.
07:27.110 --> 07:30.170
So those are the results from Claude.
07:30.200 --> 07:38.060
Uh, still has a nice diagram here, but didn't quite measure up in this with this particular challenge,
07:38.060 --> 07:43.790
uh, to GPT four or GPT four at the frontier was the winner of this challenge.
07:43.820 --> 07:50.150
You'll remember Claude won when it came to the coding performance challenge, by a way, a big, big
07:50.150 --> 07:50.900
margin.
07:50.960 --> 07:56.150
Uh, but in this case, for this particular challenge, our winner is Claude.
07:56.480 --> 07:59.730
Uh, is sorry, is GPT four at the frontier.
07:59.730 --> 08:04.620
And as I promised, I will leave you by one more look at the.
08:04.650 --> 08:07.200
At this human's feeble attempt.
08:07.230 --> 08:14.820
Take a look at the nice, elegant diagram here with a number of green dots with GPT four guessing close
08:14.820 --> 08:16.740
to the to a home run.
08:16.740 --> 08:22.620
And now compare that with the performance of the resident human in the room.
08:22.890 --> 08:26.580
Uh, and this was my chart.
08:28.800 --> 08:30.600
That I don't want to look at that ever again.
08:30.840 --> 08:31.860
Uh, all right.
08:31.860 --> 08:36.570
Anyway, I hope that this was a, uh, educational exercise for you.
08:36.570 --> 08:40.680
And you see how you would do, uh, experiments like this with your own business problems.
08:40.680 --> 08:41.430
Please do.
08:41.460 --> 08:43.200
Now, come replicate this.
08:43.200 --> 08:46.890
Play around with this, uh, you'll find, particularly with Claude, of course, that you won't get
08:46.890 --> 08:48.750
the same results because there's no seed.
08:48.930 --> 08:53.430
Um, but you will get some something different and see how it performs.
08:53.430 --> 08:55.050
Maybe you'll get better results than me.
08:55.230 --> 09:00.060
Uh, but I will see you to wrap up with some more slides.