Well, I'm very relieved.
I've got that behind me.
No more human testing for me.
We'll have one final look at my abysmal results and move swiftly on.
The time has come for us to be working with the frontier.
And we're going to start with GPT four mini, the miniature version.
Um, but it's not so many.
It's still a mighty frontier.
LLM.
Um, and it's going to be exciting to see how it fares compared to the other models we've worked with.
So first of all, we need to work on our prompt for GPT four.
And um, this is I mean, this is an extremely important part of your learning because this is where
we see how we will solve real commercial problems like this using a frontier model.
But a lot of this should again be pretty much second nature to you, because it's so similar to many
of the tasks we've done before.
The first thing we're going to do is write a function messages for item.
Given our data point, we're going to want to turn that into a prompt for GPT four, which comes in
the form of one of these messages lists, which you remember is a list of dictionaries.
So this is how we build up that list of dictionaries.
It starts with a system message, which is going to be in our case you estimate prices of items, reply
only with the price, no explanation.
And then for the user prompt, we are going to take the test prompt from this item, which you've seen
many times.
But we are going to do a little bit of data munging to it.
So first of all, I'm going to strip out this text to the nearest dollar.
The reason I'm doing that well actually the reason I should say the reason I put that in in the first
place is that later when we work with open source models, um, we'll be we'll be wanting to give those
open source models, make the problem as easy as it possibly can be, because they've got a lot to learn
and they've only got the humble 8 billion parameters to do it in.
When it comes to these frontier models, they are obviously much more capable, much more powerful,
and they don't need the same kind of guardrails.
GPT four or mini is going to be very happy, uh, predicting prices, including cents, and it knows
exactly what it's working with.
So this was an unnecessary guardrail in this case.
So I took it out.
I also take out the very end of the prompt, which is where it says price is dollars.
The reason I take that out is you will see when it comes to building this list of messages, I first
put in the system prompt under the system role.
I then add user and I put in the user prompt, and then I put in one more message which comes from assistant.
In other words, it's like it's the reply from GPT four mini.
Uh.
And that reply is price is dollars and then nothing.
Uh, and that makes it unbearably desperate to fill in that next token with the price, because this
is apparently the response coming back from the assistant from it.
Uh, and so you've really we've, we've we've teed it up so that it feels, uh, like it absolutely
has to fill that in with the most plausible next token, which is the most likely price of this product.
So that that trick of sort of having it be the assistant's response and filling it in that way is,
uh, is a good one to experiment with.
Um, so that's what we do.
Let's just quickly, uh, try this out and have a look at this.
So, so if we say, like, messages for and put in the first test data point just to see what this looks
like, what we get is, uh, roll system and then this is the system prompt.
Hopefully that makes total sense.
Then roll user and the content is how much does this cost.
And then an item and, uh, yeah, I'm guessing, uh, some kind of, uh, horrible memory of having
to go through and guess the prices of all of these things.
Just looking at it fills me with dread.
Uh, but it won't be my problem this time.
It's going to be GPT four or Mini's problem.
So it gets given that as part of the user prompt, and then it's given the assistance response.
It's own response price is dollars.
And then it will try and continue this conversation.
Um, so this is a great, uh, prompt for us to be sending GPT four.
Oh, then next, just one more little quick function, a utility called Get Price that I've written
here, which takes a string s, uh, and, uh, extracts from that string a floating point number within
it.
Um, so that if, for example, we get a string back from GPT four, like, like, um, the price is
roughly dollar 99.99.
Um, because blah blah.
If it doesn't obey my instructions and it decides to just be talkative about it, we will still strip
out.
Oops.
Don't reveal.
We will still strip out the 99.99 there.
So it will.
It will work.
Uh, okay.
That brings us I will carefully scroll down.
That brings us to the big, uh, function.
This is.
This is the real deal.
Uh, we are going to call GPT four mini with an item, and this is how we're going to do it.
Of course, we say OpenAI ChatGPT completions dot create.
Uh, we pass in a model, uh, GPT four mini.
Uh, we, uh, pass in the messages for that item, which we use the function that we just talked about,
this seed is a way to tell GPT four that we want to try and have this be reproducible, so that it gives
the same answer with the same inputs.
Now, our GPT, OpenAI can't always honour this because they make changes to their model.
And so sometimes the model will change.
And so so this number will change.
But it does its best, all else being equal.
Then if you call this twice with the same seed you should get the same answer.
So that's good to know.
And then max tokens I keep that number small because you know we've given it such a great tee up, we
can expect it really will fill in the price as indeed it does.
And so we don't need to waste our money, uh, bringing back more tokens than that.
And I should say, because we do this, uh, this whole operation is very, very cheap across all 250
data points.
Um, it's still for me in the US anyway.
Costs way less than a than a, um.
Well, less that's that's an ascent less than so little that it won't show up in the user interface.
It's less than a fraction of a cent.
So it's very cheap indeed to use because it's a short input tokens and a and a five output tokens.
And just before I want to be careful not to reveal what goes on.
Uh, uh, just before let's just quickly try this out.
Let's run GPT four.
Oh, mini, and let's call it with Test of Zero to see how it performs.
Uh, we run that and it gets back the answer 260.0.
Uh, so you can see it went through it called GPT four mini.
It then got back, it sent in something.
It sent in exactly this uh, and then with what it got back, it put it through this function and stripped
out 260 as the price for the first item.
Uh, we might as well find out what is the price of the first item?
Some, uh.
So it gets to 60 and the number is three, seven, four.
So it was quite far out with that one.
Um.
All right, well, the time has come to call tester dot test passing in GPT for a mini and test.
And you've seen a few times now that I have already run this because it comes pretty fast, but it still
takes maybe 2 or 3 minutes to do the whole lot.
So I didn't want you to have to sit there while it was happening and so I can scroll through the outcome.
But before I do so, uh, have your guess about how GPT four mini will perform.
How will it fare against, uh, original?
Against the average number?
How will it fare against humanity as represented by by yours truly?
And how will it fare against the most advanced traditional models, which had the benefit of 400,000
examples to train on?
Let's see.
So I executed this and this is what I got.
So some yellows and reds we see indeed the reproducibility you can see was I it gave identical guess
of 260 um against the the truth there.
Um so you can see some greens and some reds.
Lots of greens, lots of greens but still yellows and reds as well.
Uh, I'm going to speed up.
Wow, a big bunch of greens there.
Some more reds, some more greens.
And here is the answer I give you the GPT four mini frontier model results.
It has crushed the others.
Crushed them.
You can see that, uh, it has come in at $79.58.
Uh, definitely doing better than humanity, doing better than the, um, the other, uh, um, all
of the other models, including, uh, random forest by far, and without having any training data at
all Based purely on getting the description and continuing that token sequence.
And as I say, whereas there is some risk of test of training data contamination, I think it's unlikely
because there are almost no cases of it getting it exactly right.
So it's not as if it was regurgitating something it already knew.
Um, it's just got such a significant, uh, worldly knowledge that it's been trained on that.
It has a great viewpoint as to how much a tire costs, how much a headlamp costs, how much a shower
faucet costs, and all the other things.
That gave me tremendous problem because I have no clue.
Uh, but it had a very good clue indeed.
Um, and so much of a clue that it beat all of the models that had worked on this.
So that's pretty cool.
I did enjoy seeing this a lot.
And, uh, in the next video, we will take it up a notch and see how it's it's bigger cousin fares.