Welcome to our favorite place to be to JupyterLab.
Here we are again now in day three.
In week six.
I'm really looking forward to this notebook again and I hope you enjoy it too.
I've got some good things cooked up for you.
So again, our plan today is to look at baseline models.
And so I'm going to start with a bunch of imports which are all imports that you've seen before.
Nothing very new here.
But then some new imports in this second cell that you'll see here.
Some imports for traditional machine learning pandas that you have probably encountered many times in
the journey, a wonderful way to organize your data into into things that are a bit like mini spreadsheets.
Um, numpy, of course, I'm sure is old hat for you.
And then sklearn.
Scikit learn is a machine learning library that is incredibly popular, incredibly popular with tons
and tons of common algorithms that we will be using plenty of today, but most most importantly, linear
regression, a standard part of any data scientist's toolkit for running linear regression models.
And then there's another one here, which is a little set of imports related to natural language processing
NLP, including Gensim, which is a very useful library for NLP related stuffs such as word two vec
that I mentioned a while ago.
I mentioned just then and is something that is a powerful model for turning words into vectors.
So make sure that I run that cell too.
Oh, and then there is one more, another set of imports.
Uh, more from uh, scikit learn again.
But just I kept these ones separate because it's, uh, two different imports we're doing for more advanced
machine learning.
One is for the support vector regression, part of the Support Vector Machines and package.
And then the other is the Random Forest Regressor.
Uh, we I mentioned random forests a moment ago, so we will bring that in as well.
Okay.
Now this these set of constants, they might they might surprise you.
They they look a bit unusual.
Uh, I'll tell you to hold that thought.
They will come in later.
You may recognize them if you've ever done anything like this before.
That, um, slightly strangely, for various reasons that are very historic.
Um, when you print a that particular symbol to the standard out, it changes the color to being in
the color green.
Um, and for all sorts of, of reasons that I won't go into.
Uh, and reset turns the color back to black or white or depending on what your foreground color is.
And so knowing these constants, having them to hand makes it easy to print things in color, which
we will be doing today.
Okay.
So run that constants run our usual environment setup that we know so well and log in to hugging face.
Um, I'm not sure.
I don't think we actually use hugging face today, so I don't think I needed to log into hugging face,
but we did it anyway just for kicks.
Um, and then make sure that matplotlib comes in the Jupyter notebook.
Uh, we will load in our data from the pickle files rather than having to recreate it.
And so in it comes, uh, let's just take another look at the training data.
So let's just take the first training data point.
And I'm just going to ask for its prompt to remind you again of what this was.
So this is I'm looking for the prompt attribute of one of these item objects that I really belabored
you with.
Uh, in, in the past, uh, in the two, two days ago.
But hopefully this is now something you're becoming more familiar with.
Let me print that so it prints out nicely.
Um, there we go.
Uh, here it is.
How much this cost to the nearest dollar?
And then there's the title, and then there's the detail and there is the price.
Now you might wonder why I'm spending so much time on things like this item class specifically for this
problem.
And it is really because this is the kind of stuff that you'll be doing when you come up with your,
your when you face your own commercial problems and look for ways to engineer it and to massage the
data.
So this is real world experience that will come in handy.
You won't use exactly this code, the item class, and you probably won't have a prompt exactly like
this.
But this kind of technique is something you'll be able to replicate, so it's important to understand
it and understand the decisions that I'm making as we come up with it, so that you'll be able to do
the same thing with confidence with your own projects.
So this then, is the training prompt that it came up with.
And now let's look at a test prompt.
So I'm going to take the first of our test items.
And I'm going to call the method test prompt.
And remember that basically takes its training prompt but strips out the actual price so that we don't
reveal the answer when we're trying to test our model.
And its job is to fill in that price.
And if I want to know what it's supposed to fill in, um, I'll take it for a training point.
You take a train and then you can just call price like that.
And that is the actual price, uh, associated with this, you'll see that that this has been rounded
to the nearest whole dollar.
But the real price is something slightly different.
So hopefully this reminds you refreshes your memory on what we're doing with these train and test,
uh, methods and these these lists of training and test items and how we call them.
So now I want to reveal something that I'm really quite pleased with, which is a chunk of code, which
again, whilst you may not use exactly this code in your projects, you'll do similar things.
So it's a nice kind of principle, a nice way of approaching the problem that you should, uh, take
on and be able to replicate for your own problems.
So I wanted to be able to test different models that we come up with in a really quick, simple way.
Um, and that involves taking a bunch of our test data points and running them through the model and
being able to visualize the results.
And this was something I used to have that in a function, and I ended up repeating that lots and having
to copy and paste my code a lot, because repeatedly I'd want to be doing the same thing.
And any time you do that, it sounds like it's a time for you to build some kind of a utility to do
it for you.
Um, and so what I came up with is this, this class tester, which is going to be able to test a model
and the way it will work is that you will be able to write any function you want, any function that
will be called like my, uh, sorry, my prediction function or anything like that.
And its only job will be to take an item and to respond, return the estimated price that is its job,
and you put whatever code you want in there to do a particular prediction.
And once you've written a function like that, you can just call tester.
This class I'm about to show you dot test and pass in the name of the function.
And it will take this function.
It will call it repeatedly, in fact, for 250 different test items and see how good it is at predicting
the results, and then summarize that information back visually.
That's the idea.
And it's going to simplify our workflow distinctly.
Um, and this is the class itself.
It's perfectly simple.
It's got some stuff to deal with with colors.
I told you we'd be printing some colors.
It runs a data point, and the run data point is the method that actually does the business for one
particular data point, it gets that data point.
This is where it calls the function that you provided.
It calls it with the data point to get your model's models.
Guess what your function says it should be worth.
And then it gets the truth by calling the price attribute that we just looked at just a moment ago.
And then the error is, of course, the absolute difference between the guess and the truth.
As simple as that.
It also calculates the something called the squared log error, and the formula for the squared log
error is exactly as it is here.
It's the log of the truth plus one minus the log of the guess plus one.
Um, and uh, yeah, you can in your, you can probably imagine why there's this plus one in the formula.
It's because if the truth were zero, you wouldn't want math.log to blow up.
So this formula works well for, for cases when, for example, the, the, the truth or the guess are
zero.
Um, and that gives us then the squared log error is the square of course, of this.
Uh, and uh, we're then going to, uh, do a little bit of processing.
We're going to have the ability to draw a little chart, which I will show you in a moment.
Uh, write a report.
And this ultimately is the, uh, function I mentioned a moment ago.
You can just call test to run this this test.
Okay.
Let me execute that cell.
So you don't need to particularly understand everything that I did in this test class.
It's the the the principle of creating a nice little test harness like this and having it be something
you invest a bit of time in to make sure you'll be able to get real insight into the results of running
your model.
That's the learning here.
So what's the simplest possible model that you could imagine?
What is the simplest possible model?
We're going to come up before we do real baseline models.
We're going to do two comedy models, silly models that are going to be the most basic thing we can
imagine.
And let me challenge you for a moment.
Have a think about what could be the simplest possible model, and it's probably going to be something
simpler than that.
Um, so the first is going to be two very simple models.
The first one, I reveal the answer already.
You probably saw that the first one will be we're just going to guess a random number.
That's all it's going to be.
So here is a function.
Here is a function that takes a takes a it doesn't take a prompt.
It takes an item that not that it matters because it's going to completely ignore the item and instead
it's going to not care what it's told.
It will return a random number between 1 and 1000.
Uh, sorry, that's between 1 and 999 inclusive.
Uh, we will set the random seed so that it's the same every every time that we run this test.
And now we run it.
So the way that we run this test again is I go with my, my tester class, I just showed you dot test.
And then I simply pass in the name of this function.
I don't call the function because if I, if I call the function, it will just call it once and that
will be it.
I pass in the function itself like so.
And now I'm going to execute this and you're going to see the results of my program.
So it happened very fast because this was a very quick model.
So I'm going to scroll back up and tell you what you're seeing here because it's a lot.
And it's something you're going to get very familiar with because we're going to do this a lot of times
in the next few classes.
So each row you are seeing here is representing a different one of our test data points.
And it's telling you what the item is over here on the right.
Like here is a Godox ML 60 by LED, LED light kit, handheld LED.
And then I cut it short after that.
And what you're seeing for this particular LED light kit is what did the model what did this function
guess for the LED light kit.
And it guessed $143 because it's a random number generator.
What is the truth?
Somewhat remarkably, the truth is $289, which is rather more than I would have expected for, uh,
but only based on that that truncated Description there.
Maybe.
Maybe it comes with a laptop on the side or something.
So that's the error.
That's how much we this this model gets it wrong by this.
Here is the squared log error that we'll probably talk about another day.
But it's something that is meant to better compensate, better reflect the difference between absolute
errors and relative percentage errors.
But we're really going to be focusing on this more than anything because it's so easy to understand
for for us, for, for for humans, just the difference between the guess and the truth.
Um, and it's colored in red because that's considered a really terrible guess.
So red is really terrible.
Yellow is, uh, and green is fair enough.
And the definitions for those, if we scroll back up, I've just come up with something that's a bit
bit of a, of a rule of thumb.
I call it green if it guesses within $40 or 20%.
If it's within $40 or 20%, then that's that's green.
You might think that's quite generous of me to say $40, but remember, there's a big range of prices
here and you're just given the description of something.
And really it's very hard to do this.
So I think if something guesses to within 40 bucks then it's doing a fine job.
So you could of course be stricter if you wish.
This this is all yours to tweak.
But that was my principle for this.
So here are all of the points.
And at the end there's a nice little visualization.
So what are we seeing here.
I love this diagram.
And we're going to see a lot of these diagrams.
So so get used to this one.
The x axis is showing you the ground truth the actual value of a product.
Uh also sometimes you will hear that described as y by data scientists, whereas this axis is showing
you y hat as data scientists would say, or what estimate did the model give for the value.
So we're seeing the model's estimate against the actual true value of the product.
So the true value is always spread from from 0 to 1000in our data set, the model's value is all over
the place.
A total random set of dots.
This blue line represents, of course, the line of of perfect guessing.
If the model ever happens to guess along this blue line, then it is guest exactly on the ground truth,
and you can see that it got lucky.
Of course it will get lucky a small amount of time, and these green dots that are close to the blue
line represent where it's done fairly well.
Yellow dots for where it's a and then red dots is when it has missed the trick and gone right out there.
Uh, so that was fun.
I hope you enjoyed it.
There's another very trivial model that we can do, and it may be the one that you were thinking of
before when I asked for it.
For a very basic model, uh, you may have thought one of the really basic one that occurred to me was,
let's just guess zero for everything, or guess one for everything.
We can do slightly better than that.
We can take the training data set and say, what is the average price of a product across all of the
training data set?
Because remember our model is provided with the training data set.
So we can consider that as our as a as a constant guess.
Let's just guess that everything is is the average price of anything in our training data set.
Um, so basically we'll calculate the, uh, the, the prices of our training data set and then we'll
find its average, the sum of all of the training prices divided by the count of them.
That will give us the, the mean, the mean, uh, price of a point in the training data set.
And here is our very sophisticated model, our very sophisticated model.
Again, it takes an item and it simply returns the, uh, the average.
So it ignores whatever it's passed and it just returns the average.
So let's have a look at what this is going to look like.
Test to see if you can picture in your mind what kind of diagram you're about to see.
Uh, hopefully you can imagine exactly what it's going to look like.
And if you're ready to see if you're right or not.
Bam!
This is, of course, the diagram.
It guessed at a fixed point, which if you thought it was going to be at 500, then remember that the
distribution is skewed more towards cheaper items.
Not not as badly as as it was originally.
We corrected for it, but only a bit.
Um, so it guessed this amount for absolutely everything.
And of course, at the point where that is the same as the value of the product, it got a green, otherwise
yellow or red.
So there is the spread.
Uh, and you can see the result that you expected.
If we scroll back through the actual results, you'll see that there's a sea of reds with just some
greens from time to time for things that cost close to the average.
Well, with that, I hope that you're enjoying it.
So far, we haven't actually looked at real machine learning models yet, but don't worry, we're just
about to do that.
Hang on in there.
See you next time.