So it's quite an adventure that we had at the frontier of what's capable with Llms today, solving a
particular problem that required world knowledge and to take you through what we saw from the performance
of everything as a reminder from last time we started.
Well, actually, we started with the random model, but we'll forget about that because that was silly.
We're proper first starting point was a constant model that just predicted an average number.
We obviously were able to do better, but not that much better.
With a model that used feature engineering, you may have improved on that with better features.
Um, but our best one was a random forest model based on a not a bag of words.
A word to vec vectorized.
Uh, look at the prompts with 400 dimensional vectors.
And that brought our error.
The average difference between the prediction and the actual price of a product based on its description
down to $97 being after being trained on 400,000 example data points.
We then, uh, unveiled the human today.
That was our first model.
Uh, and the human got 127 in terms of error.
So you'll see, I was able to, uh, at least do better than the very primitive feature engineering.
And at least I did better than than constant.
I think I wouldn't have, uh, I might not have included the whole result if I hadn't done better than
a constant number.
Uh, but, uh, obviously, uh, the next, uh, Claude clearly, uh, did significantly better than
me.
And Claude was very, very similar to Random Forest, uh, so very much on par.
And again, one has to bear in mind, Claude is doing this without seeing any training data.
It's just purely based on its world knowledge.
And then being given this product.
And I can tell you from bitter personal experience that that is a challenging task.
But GPT four mini did better and got down to an $80 error, and GPT four did even better yet and brought
it down to $76 in terms of the difference.
So it shows you that out of the box, working with frontier models and APIs, you can build solutions
to problems, even problems which feel like they are regression problems.
They're not.
They're numerical problems.
They're not.
They don't necessarily naturally sound like they should be ones where just text completion will be able
to solve them.
But even given that kind of problem still out of the box, GPT four mini is able to outperform a random
forest model, a traditional machine learning model with 400,000 training data points.
So it just goes to show you how powerful these models are and how they can be applied to so many types
of commercial problem.
But with that, we can now finally move on to the world of training.
The next subject is going to be about how we take this further, by fine tuning a frontier model to
take what it's got and do better with training examples.
The thing that it hasn't had so far.
So that is a big and exciting topic.
It will then complete this week before next week.
We take it to a whole different world where we try and fine tune our own open source model to see if
we can compete, bearing in mind that we'll be dealing with something with massively fewer parameters.
So a very different world.
And to see whether or not we have a hope of beating traditional machine learning or frontier models.
Lots to be excited about.
But first, I will see you tomorrow for fine tuning.