Welcome back to Jupyter Lab.
Last time, we looked at some silly models for predicting the price of products to make our basic,
basic baselines.
Now we're going to look at some more interesting baseline models.
This of course, again is the diagram showing you that very simple model of predicting a flat average
price.
You may notice a tiny, tiny change here, which is that I've changed the color, the yellow color that
was here into a more pleasing orange color because I think the yellow is harder to see.
But otherwise this should be a familiar picture for you.
And you'll notice that on average, it's out by $145.
I should mention that when we looked at the the prior diagram, I'm not sure if I looked at this if
I showed this to you, but on average that was out by $340.
So considerably worse performance, if you guess randomly than if you take an average, for obvious
reasons.
Uh, because yeah, obviously, because the data set isn't, uh, evenly distributed.
It's uh and Nord is it's average 500.
Uh, okay.
So now let me at this point, uh, we're going to move to the topic of feature engineering, which is,
uh, one of the most fundamental of the traditional machine learning techniques.
And frankly, it's the way that data science used to work.
This is what we would do.
Uh, when this kind of problem came up, you would start trying to think about what are the aspects
of this problem, what are the aspects of each product from Amazon that would be most suitable to use
to try and predict the price?
And a lot of time was spent trying to to do what people call feature engineering, which is figure out
what properties of a particular item are most meaningful to predict its price.
And people used to spend lots of time working on that before we found that deep neural networks can
do all of that for you.
So anyways, uh, what we're going to do now is work on feature engineering and I should say one more
time that sometimes feature engineering and traditional machine learning will perform great.
Sometimes that is what your problem needs.
And you may think that in the case of an Amazon product, we're in that kind of territory.
Uh, but we'll we'll see how it performs.
So first of all, let me remind you of something.
If I look at one of my training data points, uh, you may remember this.
There is a field called details.
That was one of the fields that we sucked from our Amazon data set.
Um, and what this is, it looks a bit like a, like a Python dictionary.
At first blush.
You're seeing keys and values.
Uh, but then you'll notice that the whole thing is, in fact, a string.
Uh, it's all one big string.
It's a JSON blob representing a dictionary.
Uh, so it would be nice if we could read in this details field on every one of our data set points
in training and in test and convert it from being text into being a Python dictionary.
And luckily, the standard library gives us a way to do that.
Using the JSON package, we can do Json.loads or Loadstring and it will convert these strings into objects.
So we're going to run that and then we'll we'll run that.
It will just take a few seconds.
And then now what I can now do is say train zero dot features.
And we'll expect to see this same string but now converted into a Python dictionary.
Let's see.
Let's run that.
There we go.
You can see that.
Sorry.
As I zoom around a dictionary and you can see that it's the same as that text.
And in fact we can do dot keys and see its keys are right here.
Now there's a problem with our data, which is that turns out these dictionaries are populated differently
for different products.
Some products don't have any, uh, any features at all.
Some of them have, um, just, uh, sparse, uh, features.
So, so it's inconsistently populated.
Let's get a sense of that.
We can use another useful Python standard library, uh, tool, the counter, um, in the collections
package.
Uh, and what you can do with the counter is you can count things up, and then you can say things like,
uh, feature count, dot most common and asked to see the most common 40 of these.
So let's run that and you'll see what comes back.
So what we're seeing here is what are the most common, uh, 40 features that are populated against
all of our training data points.
Uh, and so date first available is populated a lot.
Uh, almost.
Uh, yeah.
90% of our, of our population has that populated, uh, it's, what, 360,000 of the 400,000 that we
have in the data set.
Uh, item weight is very well populated.
Manufacturer brand.
They're quite similar bestsellers.
Rank is also well populated and then it starts to tail off.
So what are good candidates for us to use for features?
Well, we're really looking for something that's well populated.
That's a good start.
We want it to be consistently populated, and we also want it to be something that feels like it's likely
to be meaningfully related to the price.
And so looking at these item weights, that feels like it's a pretty solid candidate.
Like you think that that that I mean, it's not clear, but probably there's some correlation some of
the time between weight and price.
Uh, you know, like a bigger, heavier thing, maybe more valuable on average, uh, brand seems like,
uh, obviously it's not going to, to exactly match with a feature, but maybe there's a way that we
can make it and maybe best sellers rank.
That could be something.
That's something that's a bestseller might do well.
So we'll start with those.
Those feel like they are reasonable features to begin with.
And we'll add on one more thing that just is a throwback to something we talked about a while ago.
Um, so I'm going to start with something that's a bit janky.
As I put here, this is a this is a little bit hokey.
Uh, so it turns out that the weight that's populated in this dictionary is just like, very, uh,
it's very dirty data.
In some cases, it's in pounds, in some cases it's in ounces, in some cases it's in hundredths of
pounds and a milligrams and kilograms and various other things.
So I've just got a big old if statement here that goes through, figures out what units is this weight
in, and converts it all to a number of pounds and returns that amount.
So that's what this is.
I'm not going to necessarily convince you that this does the job.
You could take my word for it.
Or if you distrust me, Then come on in and try it out for some of these.
Um, and, uh, yeah, I then I'm going to get all of the weights for all of my training items.
Um, and, uh, this line here, uh, if isn't obvious, filters out any, any nones from there so that
because I return none if there's something that I, that I don't recognize the units for, um, and
that allows me to calculate the average weight of all of our training data set.
The average weight is £13.6.
Uh, now you may say, why do I need to calculate the average weight?
Well, it's for a slightly technical reason that when we're dealing with this kind of linear regression,
you have to make some decisions about how are you going to handle the items which don't have a weight
populated the 10% of our items of our training set that doesn't have a weight.
And there are various techniques you can use, uh, people, data scientists amongst you probably know
that you can do some tricks where you you have a feature which represents whether or not there is a
weight.
And then you have to do some, some jiggery pokery with how you incorporate that in your model.
Um, and one approach that's a perfectly respectable approach is to say if something doesn't have a
weight, just pick the average and plonk that in there.
And so I have this function get weight with default, which takes an item, it tries to get its weight,
and it either returns the weight or if the weight is none or zero, because that's presumably a problem.
If something has no weight, then we swap it out for an average weight instead.
Uh, so that is the get weight with default.
I think this was a fair amount of, uh, grotty work as we do our feature engineering.
So I'm going to take a break, let you mull over the other features we've got to do.
And when we come back, we're going to go into best sellers rank before wrapping up feature engineering
and running our model.
And seeing how it predicts prices.
See you in a second.