Welcome to Jupyter Lab and welcome to our experiments at the frontier.
So we are going to put our frontier models to the test, trying out this challenge of predicting the
prices of products using a combination of GPT four and Claude.
Uh, and I do want to point out a couple of things about this.
First of all, it's worth pointing out that we're not doing any training here.
We're not going to give the frontier models any benefit of the training data.
We're simply going to be giving it the test data and asking it to predict the outcome.
So when we looked at traditional machine learning, we gave it the 400,000 training data points and
had it learn a model based on that.
In this case, we're simply giving it the test data and saying, given all of your phenomenal knowledge
of of everything that you know about the world, all of the world information stored in your trillions
of parameters, please predict the price of this product and do it by finishing this the sentence this
product is worth dollars, and then the model is convinced that the most likely next token is going
to be a plausible price for that product.
So we're taking advantage of its world knowledge.
So yeah, on the one hand it's not been trained for this task.
Uh, on the other hand though, something else that's worth mentioning that maybe some of you thought
already is that given the enormous, outrageously large training data set that has been put through
these models as part of training, it's entirely possible that they have, in fact, seen these products
before.
They may have been provided with scrapes of Amazon.
They may have been provided with hugging face data sets, for all we know.
Uh, so, um, it's possible now.
Now they don't.
You'll see that results aren't suspiciously spot on or something.
That would make one feel that it has the benefit of precise prices, but we do still have to bear in
mind that it might have an unfair advantage.
I haven't seen evidence of that, but it's certainly something that one wants to be worried about.
It's what people call testata contamination.
When there's a possibility that your test data set has been seen, or has aspects of it have been seen
during training time.
So we'll bear that in mind.
But that's just going to be a side note.
We're not going to dwell more on that.
I haven't seen significant evidence that that is at play.
So we're going to do some imports.
There they go.
We are now back to importing OpenAI and anthropic and we'll be making use of them.
Now you remember I wrote that lovely tester class that I do like, and I think it's going to be very
useful for you.
And I encourage you to be writing similar kinds of test harness frameworks for your own projects to
validate their results using the using as many business metrics as you can.
I've moved it out into a separate, uh, Python module of its own.
This is the same code that's out in a module like that, and that just means that we don't have to have
it in all of our Jupyter notebooks going forwards, because we will use it quite a lot.
We can just import it like so, and it will be there.
And the the signature has changed very slightly.
We'll have to say Tesla dot test, put in the function name and also pass in the test data set, of
which it will take the first 250 elements data points.
All right.
So with that we are now going to load in our environment variables.
We are going to log in to hugging face.
Uh again I don't think we're actually going to use that.
But um anyway might as well we get into the practice of it.
Uh, always nice to log into hugging face, isn't it?
We will initialize our two models.
We will, uh, tell matplotlib that we are going to be making charts, and we will load in our pickle
files for our training and test data set that we outputted.
Um, and they are loaded in.
Now, I did say we were just about to go straight to the frontier, but I am going to, uh, pause for
one more second, because I do have one other model to show you before we go to the frontier.
And you're thinking, oh, come on, you said it was frontier time, but I think you will be amused
by this.
And this came very much at my expense.
And this is why, at the start of today's videos, I said I was absolutely exhausted.
But it did occur to me that another, perhaps another thing that we should compare our models to would
be the efforts of humanity in trying to predict prices of products.
It seems like we should have that baseline as well, so that we can compare our model performance against
human performance.
And I couldn't find anybody that I could convince to go through the horror that is reading 250 product
descriptions and trying to figure out how much they cost.
And so I subjected myself to this torture and torture it was.
I can tell you it is way more difficult than I was expecting.
I said to you before that I thought it was it was quite hard, but it's way harder than I had realized
there are just things that I had no idea about.
I had no idea how much it costs to buy a wheel, and there are a couple of wheels in there.
Uh, then, even though I should know computers back to front, I found myself agonizing over the cost
of refurbished computers with 400GB of, uh, disk space.
And, yeah, it was just really, really hard.
And chandeliers.
I don't know how much a chandelier costs.
Anyway, I digress.
I wrote some code that outputs to a CSV file 250 test prompts, and after I ran that, uh, I'll run
it.
Now it creates this file human input dot csv.
And here is human input dot CSV.
And it contains the prompts every single one of the top 250 prompts and zero in this column to be filled
in by said human.
Uh, and I, you know, I'm not even sure if I'm going to check in this human output into into git.
If you see it there, then I then I've dared to because I after a while you become fatigued and I was
going through it probably too fast.
I probably made some real blunders in there.
If you look at it, you'll probably say, what were you thinking?
You should stick to teaching LM engineering.
Certainly don't.
You're not, not not someone of the world.
Uh, but, um, yeah, I gave it my best shot.
So anyways, we'll read back in the prices that I set.
Uh, and then let's just, uh, quickly get a sense for, for, for how this looks.
So we're going to write a function which is going to be the human, um, the predictor, the human processor.
So it needs to take an input.
And that input should be one of the items.
And its job is to return the cost of that item.
Um, so what I do at this point is I say, okay, so if I look in my training data set, I say my test
data set.
What is the index of that item?
So is it the the zeroth item in in test.
Is it the first the second the third.
And we will call that index.
So that is which number of the test items we are looking at here.
And then I have read in all of my hopeless estimates into human predictions.
And so we will simply return human predictions.
At uh at index.
All right.
And run that.
And now now we will see we will see tester dot test human pricer.
And pass in the test data set.
Here we go.
So there are the results.
Uh, well you can see that there's a fairly large number of reds in there.
There are some greens though.
I did respectably.
Uh, but still I was quite far out.
But look, this one.
What is this?
Richmond auto upholstery, I guess 260.
And it was 225.
And this one here, Gibson Performance exhaust.
I don't know how much a Gibson Performance exhaust costs, but I.
I guessed 499.
I thought I'd, you know, go go for it.
And the answer is 535.
But then some others in here.
What did I get wrong here?
A Street Series stainless performance something, uh, I guess $260 and it was $814, so I was just
way off there anyway to put me out of my misery.
If we scroll down, we will see.
Here is the chart then, uh, I uh, in particular, you'll see that I didn't do terribly.
I mean, I did.
All right, look, lots of green dots hit rate of 32%.
Um, I, I also I realized actually, as I was About two thirds of the way through that, all of my
prices, I'd never guessed anything much more than 4 or $500.
So I knew immediately that obviously I was.
I hadn't I hadn't spotted things that were expensive.
So that was obviously a failing.
Um, so my total error, as it happens, was $127.
And that means I come in better than than than the average.
It's not like I could have done better just by guessing the average number all the way through.
Uh, the, uh, I've written down to remind myself the the comparisons.
The average was 146.
Uh, was the error of the average price.
So I did better than that.
The straight up linear regression with feature engineering.
The basic one was 139.
So I beat that.
I beat a very, very basic feature engineering linear regression.
But you probably already put in more features and did better than that anyway.
Um, but then all of the other models are crushed me with the, the bag of words style models and the
word two vec models.
And then you remember that Random Forest came in at 97, significantly better than humanity.
So already as it would happen, good traditional machine learning models can do better than this human
anyway in predicting the prices of items, but you may be better informed than me if you put yourself
through this exercise, which I do not recommend.
Then you may find you may find that you do better.
Anyway, in all seriousness, I haven't just wasted your time.
This is the kind of exercise that's good to do because maybe for a few data points, but it gives you
a good sense of the type of problem you're solving, and where the bar is set in terms of human performance
is something which can be used to compare how well we're doing with models.
After all, if we can't do better than then, than the human performance, then we need to work harder.
So that gives you a sense.
And when we come back in the next video, we really will move on to Frontier Models.
It's happening.
See you then.