Welcome back.
So hopefully you are still impressed by the GPT four mini results.
The frontier model coming in strong at just shy of $80 and with 52% hits just going above the 50% mark
in terms of the number of green dots and a nice chart there.
But the time has come for us to now run against its bigger cousin, big GPT four.
Oh, I just did.
I did want to mention that, uh, do be aware that at this point, uh, when we're talking about these
kinds of error numbers, the error is already, uh, reasonably low.
You think about the actual prices of products.
They do vary.
There is volatility baked into prices.
There are sales.
There's there's big swings when you come to things like the prices of a wheel or the price of a laptop.
No doubt they vary quite considerably from different sites and different, uh, Stores, so there is
some natural error built into this question anyway.
So we're certainly not expecting to to come down to small numbers.
So from that point of view, recognize that every dollar improvement we get at this point is a is a
victory in its own right.
We're going to be squeezing juice if we can, but don't expect a big change now.
We are now at a point where this is already super impressive and far ahead of any of the traditional
machine learning techniques.
Um, so, uh, the next thing to do, of course, is to unveil the bigger cousin, GPT four.
Oh, here is the same type of function again, the function that we'll be passing to our test program.
Uh, it's of course calling OpenAI ChatGPT completions dot create.
We're using the GPT four model as of August the 6th, which is very recent.
Um, passing in messages, passing in the seed and using the same way of getting back the results.
It's worth mentioning this is now going to cost a little bit more.
It costs somewhere between $0.10 and $0.20 for me to run this across 250 points.
Um, so, um, yeah, I mean, it's not going to break the bank, but you might want to think twice
before running this a hundred times.
Uh, so I do urge you to at least try it once.
It's it's great fun.
Um, but, yeah, it doesn't it doesn't necessarily need to be run a lot of times.
I also want to mention that you might wonder why I'm not trying to run this against oh one.
Uh, so the kind of problems are that oh one is designed for.
Ah, more the problems where there is deeper thinking required, a multi-step thought process and challenging.
And that doesn't really lend itself to the problem that we're tackling here.
Um, it would also be slower and more expensive and for no good reason.
We're not trying to solve, uh, math puzzles.
We're just trying to have the most likely next guess of a price based on worldly knowledge.
So GPT four frontier should be our best shot.
Um.
And I have, of course, already run it.
Uh, so it's a little bit slower than GPT four mini.
Just a little bit.
But I think you should run it yourself and you'll enjoy it.
And here are the results I get.
Um, so there's a lot of greens, but there's lots of reds as well.
So it's quite a variety here.
Um, no doubt you're betting.
And here are the results.
So the bottom line is it is better.
It is improved it for sure.
It's gone from about 80 to about 76.
And the hits from 52 to 58.
Uh, green dots.
It's not a massive change.
You may have expected more going from mini to GPT four, but remember GPT four mini is really good.
And despite the fact that it's a much cheaper model, it has much the very similar smarts to GPT four
zero.
Um, so I would say, first of all, it is to be celebrated that we've come down by by $6 by $4.
Sorry.
Uh, but but also that, um, we can already assume that GPT four mini has done very well indeed.
Um, so, um.
Yeah, this is this is, uh, this is the results.
It looks visually very impressive.
The the I mean, compare it to humanity.
Uh, at the end of this lecture, I think we'll end by going back and embarrassingly looking at the
chart one more time, uh, comparing it to what frontier models are able to achieve.
Uh, but this does give you a good sense.
Okay.
And then, of course, we do have one more frontier model that perhaps we should unveil, which is Claude
3.5 sonnet.
Um, which is the strongest, uh, model from anthropic And, um, you recognize, I hope the structure.
Uh, Claude messages create uh, again.
Now, with in the case of Claude, you can't pass in a seed.
It doesn't take that.
So the results will be different each time.
But I've just gone with the first run I've done in here, and that's what we'll have as, as the, uh,
the, uh, the results for, for our purposes.
And I guess I'll probably, uh, replicate and save this notebook.
So we have it for, for posterity.
Uh, so time for Claude starts with a couple of reds, then a sea of greens.
And I will get us straight to some outcomes.
So here's the results from Claude.
Interestingly, Claude actually does a little bit worse.
Uh, it's, uh, um, uh, worse, in fact, than GPT four mini in this task, and worse also than the
Random forest.
Of course, again, Random Forest had the benefit of all of its training data 400,000 training data.
So it's a very different kind of of of test.
And you can see one of the reasons for that is that you can tell from this axis that it has made some
prediction.
We can't quite see it because it's going to be a little tiny red dot, but it made some massively outsized
prediction, although that will of course been spread out across everything.
So it wouldn't have made a huge difference.
But still, uh, one guess would have been like way, way too high and will have certainly, uh, certainly
made its results be a bit distorted.
Let's see if we can spy that from scanning through here.
Where did it go so wrong?
Here it is.
G technology G speed ES pro high performance.
So this thing which actually cost $495.95.
It guest was $4,999 and 99.
So it was a very outsized guest there that introduced 4500 error.
Now, obviously, that that error in itself won't distort things too much because it gets averaged out
across the 250 data points.
But it's an example of one of the things that pushed it into the worst camp.
So those are the results from Claude.
Uh, still has a nice diagram here, but didn't quite measure up in this with this particular challenge,
uh, to GPT four or GPT four at the frontier was the winner of this challenge.
You'll remember Claude won when it came to the coding performance challenge, by a way, a big, big
margin.
Uh, but in this case, for this particular challenge, our winner is Claude.
Uh, is sorry, is GPT four at the frontier.
And as I promised, I will leave you by one more look at the.
At this human's feeble attempt.
Take a look at the nice, elegant diagram here with a number of green dots with GPT four guessing close
to the to a home run.
And now compare that with the performance of the resident human in the room.
Uh, and this was my chart.
That I don't want to look at that ever again.
Uh, all right.
Anyway, I hope that this was a, uh, educational exercise for you.
And you see how you would do, uh, experiments like this with your own business problems.
Please do.
Now, come replicate this.
Play around with this, uh, you'll find, particularly with Claude, of course, that you won't get
the same results because there's no seed.
Um, but you will get some something different and see how it performs.
Maybe you'll get better results than me.
Uh, but I will see you to wrap up with some more slides.